Searching the Web: The Problem of Ranking

When you go to Google and type the word “Cornell,” the first result it shows you is www. cornelledu, the home page of Cornell University. It's certainly hard to argue with this as a first choice, but how did Google “know” that this was the best answer? Search engines determine how to rank pages using automated methods that look at the Web itself, not some external source of knowledge, so the conclusion is that there must be enough information intrinsic to the Web and its structure to figure this out.

Before discussing some of the ideas behind the ranking of pages, let's begin by considering a few basic reasons why it's a hard problem. First, search is a hard problem for computers to solve in any setting, not just on the Web. Indeed, the field of information retrieval [36, 360] dealt with this problem for decades before the creation of the Web: automated information retrieval systems starting in the 1960s were designed to search repositories of newspaper articles, scientific papers, patents, legal abstracts, and other document collections in reponse to keyword queries. Information retrieval systems have always had to deal with the problem that keywords are a very limited way to express a complex information need. In addition to the fact that a list of keywords is short and inexpressive, it suffers from the problems of synonymy (multiple ways to say the same thing, so that your search for recipes involving scallions fails because the recipe you wanted called them “green onions”) and polysemy (multiple meanings for the same term, so that your search for information about the animal called a jaguar instead produces results primarily about automobiles, football players, and an operating system for the Apple Macintosh.)