Improving Web Search Ranking by Incorporating User Behavior Information

We consider two complementary approaches to ranking with implicit feedback: (1) treating implicit feedback as independent evidence for ranking results, and (2) integrating implicit feedback features directly into the ranking algorithm. We describe the two general ranking approaches next.

(1)Implicit Feedback as Independent Evidence
The general approach is to re-rank the results obtained by a web search engine according to observed clickthrough and other user interactions for the query in previous search sessions. Each result is assigned a score according to expected relevance/user satisfaction based on previous interactions, resulting in some preference ordering based on user interactions alone.


We experimented with a variety of merging functions on the
development set of queries (and using a set of interactions from a
different time period from final evaluation sets). We found that a
simple rank merging heuristic combination works well, and is
robust to variations in score values from original rankers. For a
given query q, the implicit score ISd is computed for each result d
from available user interaction features, resulting in the implicitrank Id for each result. We compute a merged score SM(d) for d by
combining the ranks obtained from implicit feedback, Id with the
original rank of d, Od:




where the weight wI is a heuristically tuned scaling factor
representing the relative “importance” of the implicit feedback. The
query results are ordered in by decreasing values of SM to produce
the final ranking. One special case of this model arises when setting
wI to a very large value, effectively forcing clicked results to be
ranked higher than un-clicked results – an intuitive and effective
heuristic that we will use as a baseline. Applying more
sophisticated classifier and ranker combination algorithms may
result in additional improvements, and is a promising direction for
future work.