Anatomy lesson 1: Search industry structure

Search engines were once considered a failed business idea because they were onlya conduit to other pages. In other words, they lacked stickiness; no one stayed longenough to see the advertising. In response to this crisis, search engine portals triedto develop new services to attract and retain users. For example, AltaVista spentmillions to develop new portal content that it hoped would make it a comprehensiveweb portal for not only searches but other activities such as news, travel, andshopping. Google resisted such efforts, and insisted instead on focusing on beingthe best-syndicated search engine provider, with the most relevant search results.However, analysts mocked Google for its seeming lack of a means to make moneyfrom its singular mission of search excellence.

Then, search engine portals began experimenting with sponsored links—a list oftwo or three paying sites that appear above the actual search results. Because sponsored links are so highly targeted (they directly relate to the search terms thatusers type in), they became enormously profitable. A small company dealing withspecialized golf equipment, for example, could sponsor a link that accompanied auser’s search on golf, directly targeting the golfer. Oftentimes because users didn’tknow the difference between sponsored and actual searches, they were clickingsponsored links 12 to 17 percent of the time (Waters, 2003) far in excess of theless than one percent banner ad click-through rate today (Harvey, 2003b). Andevery time a user clicked on a sponsored link, the search engine earned money. Notsurprisingly, search engine services barely distinguished between the sponsoredand non-sponsored categories in order to generate as many click-throughs aspossible.

Let’s start with a little dissection. To understand the search engine industry and itsgradual and quiet commercialization, it is necessary to distinguish between the fourfacets of the search engine industry: directories, search engine providers, searchengine portals, and commercial search providers. The four areas of the searchengine industry have their own completely separate functions. Certain companies,like Google, may perform all the functions within a single company; other companies contract out to smaller ones, which each perform a singular functions,and are merged to form the basis of what most people consider to be a “searchengine.”

Directories Often mistaken for search engines, directories are nothing more than comparativelysmall databases that may or may not feed a search engine. Directories predatesearch engines. Librarians began cataloguing Web sites in the early 1990s, but itwas Yahoo! who developed the first commercial directory in 1994 by hiringnumerous editors to compile web pages and place them into logical categories. TheDirectory was steadily growing as individuals, organizations and companiessubmitted sites for Yahoo! employees to review and (hopefully) add to a category.

As the largest commercial directory on the Web, Yahoo! became instantlyrecognized as the place to look for information. Yahoo!’s initial income came frombanner advertising, but also from individuals, organizations, and (mostly)companies who paid Yahoo! for “express submissions” to the Yahoo! Directory (arecurring annual fee of $299 by 2004). Other online directories followed Yahoo!’s model, like the Australian-based directory Looksmart, which launched in 1996.Another significant directory (especially to the search engine industry) is the OpenDirectory Project, which was launched in 1998. This enormous non-commercialendeavor, constructed entirely by a global army of volunteers (or as they arereferred to, “net-citizens”) is by far the largest directory on the Web and continuesto grow in size every day.

Directories have become an important component to the search industry in thatthey provide massive lists of indexed pages to feed search engine providers (moreon these below). The Open Directory Project is the most significant directory to thisend for two reasons: it’s the largest directory on the Web, and it has licensed itscontent for open content distribution. That means that any search engine providercan draw upon the Open Directory Project as it conducts its searches. Search engineproviders all claim to have amassed their own proprietary webpage databasesthrough which they conduct their searches, but the Open Directory Project makesup a large portion of each of these databases (Fine Brand, 2003). Yahoo!’s directoryhas also been an important database that feeds the company’s larger search portal,and the MSN portal has long relied on Looksmart’s directory to distinguish itssearch results from other search services (although that relationship ended in2004).

Search engine providersSearch engine providers are the tools that actually do most (but not all) of thesearching. Sometimes referred to as “algorithmic search engines” or “crawler basedsearch engines,” search engine providers license highly complicated crawlingsoftware to search the web. Search engine provider companies have all got theirstart in a similar way: Various very smart teams of computer engineers first puttheir heads together and came up with a unique mathematical formula, or algorithm, for determining webpage relevance. They then got venture capital toamass a database containing billions of web pages, and, through extensive automated text indexing tools, built a “proprietary web index”—the bigger thebetter. Search engine providers then began to syndicate their service to othersearch portals (what we commonly think of as “search engines”). Because the job ofcreating and updating these web search algorithms, and then amassing andmanaging (i.e., updating, cleansing) these web indexes is so huge, there areactually only a handful of search engine providers in existence. At the time of thiswriting, the most prominent search engine providers can be counted on one hand:AltaVista, Inktomi, Google, AlltheWeb, and Teoma. The finite number of search engine providers is one reason why search results from different “search engines”are so similar—they are powered by the same provider.

AltaVista and Inktomi are the oldest existing search engine providers, both emerging in 1995.¹ The AltaVista algorithm was developed by scientists at theDigital Equipment Corporation in Palo Alto California, and the Inktomi algorithmwas created by a computer science professor and his graduate student at Berkeley.The reason that AltaVista is better known than Inktomi is that AltaVista developedits own branded search portal (while also syndicating its services elsewhere);Inktomi opted to just syndicate, with Yahoo! and MSN being its first majorcustomers. Google arrived in 1998, bringing a new kind of searching standard tothe industry. Created by Larry Page and Sergey Brin, two grad students at Stanford,Google’s formula became instantly recognized as better. While AltaVista and Inktomi had based webpage relevance on keywords (the greater number of keywords on a page, the more relevant), Google based relevance on links (thenumber of links that point to a given webpage, the more relevant). Google becamethe most sought-after search engine provider as a result of its superior algorithm(for example, Yahoo! dumped Inktomi for Google in 2000), and initially made halfits revenue from selling its search technology and access to its massive web index,to various web sites (Harvey, 2003a). The search engine provider became anovernight success as a branded search portal as well.

Two more search engine providers emerged after Google. AlltheWeb was created bya group of scientists at the Norwegian University of Science and Technology in1999, and became hugely popular throughout Europe. Finally, Teoma was born in aRutgers University computer lab in 2000. There are other, smaller search engineproviders, such as Wisenut, and numerous specialty search engines (focusing onmedieval or ancient topics, or the Chinese language, for example), but the reality ofthe small world of search engine providers is that it’s getting even smaller. By2004, three search engine providers, AltaVista, Inktomi, and AlltheWeb had allmerged under the Yahoo! umbrella (more details on this below), leaving just threemain search engines providers, Google, the Yahoo! group, and Teoma, to powernearly all the searches on the web. Therefore, the myth that there are many searchengines to choose from, each with varying algorithms, is simply untrue.

Search engine portalsYahoo!, Excite, MSN, AOL, Google, Ask Jeeves, Lycos—all these are search portals.They are web sites that offer a search toolbar, which is powered by a syndicatedsearch engine provider (most often one of the top five). For example, at the time ofthis writing, the portals Netscape, AOL, Go, Google, and thousands of smaller,specific-purpose portals (e.g., ESPN.com, Amazon.com, Walmart.com,Mamamedia.com) are all powered by Google. The search portal Lycos is powered byAlltheWeb; MSN, Amazon.com, and eBay are powered by Inktomi; AskJeeves ispowered by Teoma, and Hotbot is powered by four search engine providers—Google, AlltheWeb, Inktomi and Teoma. Again, these different relationships explain why some search engine services (e.g., Yahoo! and MSN) havemore similar results than others at a given point and time: they are powered by thesame search engine provider (Inktomi).

To complicate the matter of partnerships, a few search portals, such as Yahoo! andMSN, integrate directory listings alongside an algorithmic database. In other words,the Yahoo! search portal draws listings from its own Yahoo! Directory in addition tothe results provided by Inktomi; MSN, until recently, was drawing listings from theLooksmart Directory and the Inktomi search engine provider. Not surprisingly,whichever search engine provider or directory a search portal decides to partnerwith, it’s a big deal for that company. For example, Yahoo! vaulted Inktomi’s profilein 1998 from a relatively unknown search engine provider to a significant playerwhen it chose to outsource its searching technology to the company; then Yahoo!devastated Inktomi when it switched to Google in 2000; In a strategic move, Yahoo!then bought the diminished Inktomi outright in 2003, and now is in the business ofsyndicating Inktomi’s technology.

Which brings us to another complication: some branded portals, such as the newYahoo!, Google, and AlltheWeb, also syndicate their own search engine provider.Yahoo! and Google are now in intense competition, not only for portal prominence,but for search engine provider prominence. AlltheWeb remains an especiallypopular search engine portal throughout Europe and has a growing following in the

U.S. as a search engine provider (powering the searches, for example, for Lycos).AltaVista, which operated one of the first popular search engine portals (and searchengine providers), is now owned by Yahoo!, but it still operates its branded searchportal. Its technology has now merged with Yahoo!’s Inktomi, and is syndicated through Yahoo!’s growing partnership network.

Commercial search providersCommercial search providers build and manage indexes of web pages (like searchengine providers), provide an algorithm for searching these indexes (like searchengine providers), and then syndicate this service to search portals (also like searchengine providers). The main difference is that the indexes of commercial searchproviders are all advertisers who pay to be there. To date, every major search portalcombines both algorithmic and commercial searches, which are conducted side byside—key words are used to identify both relevant web pages and relevant advertisers. Typically, commercial search results appear in separate locations on thesearch result list (top, bottom, side) as “sponsored listings” or “featured sites,” butmany times with as little demarcation as possible between the sponsored and non-sponsored listings so as to inspire more click-throughs. For example, in 2004AllthyeWeb used a nearly imperceptible thin grey line to separate the first threesponsored links from “the rest”; AltaVista used a large font size, so the first foursponsored listings take up the entire page—a user must scroll down to get to thenon-sponsored results. And MSN listed sponsored sites ahead of the regular searchresults, but employed the same consecutive numbering system. In other words, ifthere are three sponsored matches, then the first non-paid match begins as number four.

The company GoTo.com (now called Overture) has been instrumental in activatingthe commercial search business, and now provides commercial results to themajority of search portals on the web. GoTo began as a search portal in its ownright, relying on Inktomi for its algorithmic searches and banner advertising for itsincome. However, in 1998 the company began a new income-generating scheme: itbrazenly auctioned off placement within the portal’s “impartial” search engineresult list itself. The higher an advertiser’s bid, the higher the web site appeared inGoTo’s search result list; GoTo’s search results were effectively stacked with payingcustomers. Although the move caused considerable controversy among consumeradvocates (and within the industry), advertisers were delighted.

High placement within a search result list is important for two reasons. First, userstrust this list because they believe it prioritizes relevant web sites according to thekey terms entered. Second, users typically don’t tend to look beyond the first twoor three pages in a search result list, believing that the first two pages are the mostrelevant (Lasica, 2001) Recalling the four myths I introduced at the beginning ofthis chapter, search engines are less impartial that people typically think. AnotherGoTo breakthrough was its “Pay-For-Performance™” strategy, whereby advertisers paid GoTo only if a user clicked on a sponsored link. By 2000, the company beganto syndicate its commercial search services to other search portals (AOL being itsfirst major customer). Because portals kept a portion of the “Pay-For-Performance”revenue every time a user clicked on one of GoTo’s paid placements, search portalswere as happy as advertisers. Banner advertising had become increasinglyineffective, and finally there was a way, through search itself, to make money. OneLycos executive justified the practice this way:

We thought long and hard and decided it doesn’t matter if we are paidfor a link, so long as the results are what the user wants … the industry has trained users to avoid anything that looks commercial. Bycalling them paid listings, it hurts the user. (Lasica, 2001, p. 2)

Indeed, the growing justification among internet industry folk was that peoplegenerally use the web for commercial purposes anyway. They use the web to findflower delivery services, or to purchase a barbecue grill.

Towards the end of 2000, GoTo renamed itself Overture, disbanded the GoTosearch portal, and concentrated solely on its new role as a commercial searchprovider. By 2002, Overture had signed up 80,000 advertisers (Overture, 2003a)and was distributing its for-profit search results to tens of thousands of webportals across the internet, including MSN, Yahoo!, Lycos, AltaVista, HotBot,Netscape, AOL, Infospace, Fast, and ESPN.com. These web sites retained theiralgorithmic search provider (e.g., Google), but cross-listed this database withOverture’s growing index of sponsored web links. Yahoo!, for example, mixed itsYahoo Directory results with Google’s algorithmic results and Overture’s Pay-For-Performance sponsorship results, although it’s not exactly clear whether the sponsored pages were always restricted to the “sponsored” section, or slippeddown into the regular web results. Because search portal companies are not obligedto disclose exactly where the commercial influence lies in a given search resultlist—in the sponsored listings above a search result list (Kopytoff, 2003), or withinthe result list itself, revealing their mode of user deception has not been commonpractice. MSN, which had mixed the Looksmart Directory results with Inktomi’salgorithmic results and Overture’s pay-per-click sponsorship results, has beenpurportedly following the GoTo.com model by stacking its result pages (Fine Brand,2003). Before AltaVista became a Yahoo! subsidiary, AltaVista experimented withselling its search results to the highest bidder independently of Overture (FineBrand, 2003; Hansell, 1999). Back to the four myths, search engines are notimpartial.

Regardless of visible above-the-line sponsorship or invisible within-the-list sponsorship, the advent of commercial search providers has meant that a searchwithin a search portal is more and more likely to be heavily commercialized. In asingle quarter of 2002, Overture facilitated 563 million “paid introductions” (clickthroughs) and made $126 million, compared to Google’s approximately $15 millionin revenue for its main business of running non-paid searches (Overture, 2003b).But Google was not about to stand by and watch. Google responded to Overture’ssuccess by amassing its own index of commercial sites and creating the “AdWords”program. AdWords essentially combines Overture’s auction system of selling keywords and placement to the highest bidder with an algorithm that factors relevance,or the ad’s click-through rate, into placement. In addition, Google also establisheda more expensive sponsored links option for links appearing above, rather than onthe side, of the search result list. Significantly, Google took the higher road bypromising to never place sponsored links within the company’s objective searchresults. Such company integrity, however, may be irrelevant, or, at worst, a convenient marketing tool. Already, plenty of other channels have evolved toundermine Google’s promise of integrity. For this discussion, we need another anatomy lesson.

Anatomy lesson 2: The crafty craft of search engine sponsorship

First, a slight review is in order. As we’ve seen, search results on any commercialsearch portal are already skewed towards commerce due to a number of basicpayment practices.

• Commercial directories charge a recurring annual fee for an expresssubmission (in Yahoo!’s case, $299), which ensures a continued listing oncea submission is approved. The Looksmart Directory, as of 2002, accepts onlypaid submissions from commercial sites to its directory. These enterprisingstrategies indirectly benefit well-endowed and for-profit sites that can affordelite treatment and positioning within a directory.
• Commercial search providers have instituted pay-for-placement deals designed to directly benefit commercial sites. The higher an organization

bids on a key word, the higher they are listed in the search portal.

Yet these developments are just the beginning of an evolving and commerciallyinnovative search industry.

Paid inclusion In 2001, Inktomi introduced a new variable that would serve commerce—and thesearch engine industry—extraordinarily well: paid inclusion. Paid inclusion means that customers who pay a flat fee are guaranteed to be included in every searchcompleted by the Inktomi search engine. Search engines do not search the entireweb, only parts (here, another myth is deflated). With paid inclusion, paying siteswould always be incorporated into the searchable index (unlike many other sites,which simply slip away as algorithms are updated). Although paid inclusion doesnot guarantee the web site’s rank within the search results, it does guaranteeinclusion, somewhere, each time a search is conducted. For niche topics especially,this bodes well for the advertiser. As reporter Chris Gaither (2003) explains,“Internet companies have realized that, if someone is hunting for information on atopic like mesothelioma, the person is ripe for specialized advertising” (p. F1).

Inktomi’s model was soon copied by every major search engine provider saveGoogle, meaning that by 2002, Inktomi, AlltheWeb, Teoma, and AltaVista—were alloffering paid inclusion as part of their overall syndicated package. As the practiceof paid inclusion exploded, advice about paid inclusion practices appeared asunproblematic common sense in countless business newsletters and magazines by2003:

The key to success is finding the words that will drive traffic to yoursite—and, more important, convert those potential customers intosales. [Martin Child, Overture’s managing director for Northern Europe]says generic terms such as travel may generate a large number ofleads but many of those will be wasted if your firm offers only a nicheservice. “’Travel’ may not convert as well as ‘Icelandic expedition,’” hesays—even though there would be far fewer searches that use thelatter, much less expensive term. Bunis recommends a mix of genericand specific descriptions of your business. (Durman, 2003, p. 16)

With such a profitable commercial system, search engine providers were finally seenas money-makers in their own right, and as such came to be regarded more aslucrative properties rather than services for mere syndication. By early 2003, Yahoo!had acquired Inktomi ($235 million) and Overture had scooped up AlltheWeb ($70million) and AltaVista ($140 million). “The paid-inclusion model is really icing onthe cake,” said Yahoo! Chief Financial Officer Sue Decker in 2003. “That alone reallyjustifies the price of the transaction” (Reuters, 2003). A few months later, Yahoo!then acquired Overture.

As surely as paid inclusion is lucrative to search engine providers, there is also anoteworthy fringe benefit to advertisers investing in paid inclusion: Part of the flat fee involves advice on how to write advertisers’ listings so as to further enhancetheir position. “Since [commercial search engines] alone understand how the algorithms inside their search engine ‘black boxes’ work,” Financial Times reporterRichard Waters observed, “they generally know how to game the system, though itis a power they claim to use responsibly” (Waters, 2003, p. 30). In other words,even if paying sites didn’t pay for prominent placement directly, at least they gotthe tools to figure out how to get there. As advisory material from an onlinemarketing firm called The Web Search Workshop related in 2004, these new“opportunities” could get their clients’ web sites “more (and faster) exposure in acrowded market”: There is some dissatisfaction that these paid services are now openingup a gap between those websites able or willing to pay and thereforechanging the balance of search results being offered. However, thistrend for providing a paid alternative in return for privileges is likely tostay and probably increase in the future. (The Web, 2004, p. 1)

For good reason, there was heady jubilation within the search industry and inbusiness circles over the Web’s commercial viability via search engine listings.Results had become so commercially skewed, however, that consumer advocacygroups were increasingly alarmed. A campaign, initiated by the media watchdoggroup Commercial Alert, led to a Federal Trade Commission (FTC) investigation intothe practice of undisclosed, paid search results within search portals. Completed inJune 2002, the study reported (not surprisingly) that the web’s largest searchengines did not reveal the preferred treatment they accorded to sponsors. The FTC’s response was typical in terms of the current political climate: a gentle rebukeand a call for self-regulation. The rebuke however, did seem to be effective: it led to more differentiation between sponsored sites and “nonpaid” sites. For example,Yahoo!, AskJeeves, Lycos, and numerous other portals began stressing the “objectivity” of their web results by using bold red headings to demarcate asponsored link from a non-sponsored link. These distinctions, though, more or lessveiled the incursions of paid inclusion, which had surreptitiously become theindustry norm.

Search engine marketers (SEMs)Only one search engine provider/portal has resisted both pay-for-placement andpaid inclusion. Google has taken an admirable stance on search engine integritysince its inception, and, although it has rigorously pushed its AdWords program,which matches sponsored sites to key words within a search, the company haswidely publicized its refusal to allow any direct commercial influence in its searchresult lists. This is not to say, however, that Google’s result lists are free frommarket influence. One of the most significant developments just outside of (butdirectly affecting) the search engine industry has been the rise of search enginemarketing. This mini-industry exists to influence placement within the databases ofsearch engine providers and maximize the overall visibility of their clients’ web site.The search engine optimization (SEO) market, which offers “positioning” and “advisory & marketing” services to its clients, is flourishing. These small SEO companies (or SEMs) try to guarantee prominent listings for their clients. Althoughmost business organizations know the four necessary steps towards Web sitevisibility (a subscription to Google’s AdWords, Overture, Yahoo!’s Directory, and asubmission to the Open Directory Project), SEMs aggressively act on their behalf.

Most SEMs are especially focused on Google, a sort of Holy Grail for SEMs. Google isthe most popular search portal/search engine provider [conducting 83 percent ofall searches in 2003 (Hindman, Tsioustsiouliklis & Johsnon, 2003)], yet a tough nutto crack for SEMs, because it doesn’t give anyone access to its algorithm. In fact,one of the most typical promotional statements appearing on these companies’ websites concerns the ability to decode the patterns behind Google’s objective searchresults. “We understand the ‘spidering’ schedule that Google employs,” says Morevisibility.com. “By submitting at the appropriate intervals, we are able to systematically deep-penetrate the Google database” (MoreVisibility, 2003).Meanwhile, as Fiona Harvey of the Financial Times has reported, “so many smallcompanies have sprung up in this field that Google engineers spend much of theirtime tweaking its search criteria in order not to fall prey to them” (2003b, p. 32).

Since Google’s PageRank algorithm strategy is partially based on the number oflinks pointing to a site (ostensibly making it more “popular,” and therefore more worthwhile to most web searchers), SEMs have become especially savvy to thelinking game, working with their clients to increase the number of links leading totheir clients’ web sites. We hear about this practice in popular culture: for example,pranksters and political activists have turned official websites for 2004 Democraticnominee John Kerry and President George W. Bush into the top listings for searchterms like “waffle” or “miserable failure.” This strategy is called “Google bombing”in the mainstream media, and considered a harmless novelty. Meanwhile,enterprising SEMs (who we don’t tend to hear about) use the term “horizontal marketing,” and do anything they can to increase linkage for paying clients. Thisincludes specializing in particular areas such as health and insurance-related sitesto better shape web rings of reciprocal links. Blogrolling is another common waySEMs have generated more links: by applying the popular software supplied byBlogrolling.com, a user can add links to a blogpage with one easy click, which inturn more easily leads to link-farming, the practice of creating web sites withnothing but links. As Jill Walker (2005) explains in her helpful analysis of the linkeconomy:

There is a black market for links. You can pay dollars or kroner or yento buy links to your site from link farms, circles of sites with nothingbut links. There is also a common law perception of link prostitutionor link slutting: shamelessly selling one’s own integrity for links. (p. 3)

Because Google’s market success is dependent upon its perceived credibility (thecompany’s motto is “Don’t Do Evil”), Google has heavily discouraged link farming,and has punished link-farming companies and their clients with lower searchresults. One such firm, the Oklahoma City-based SearchKing, which had practicedlink farming and was “punished” by Google, filed a federal lawsuit against Google in2002. However, Google won the case in 2003 on the grounds that it has FirstAmendment rights to present its search results in any order it sees fit (Kopytoff,2003). While the SearchKing vs. Google case pitted one commercial company’sinterests against another, the case has some interesting implications for any futureefforts to de-commercialize search engine result lists. Like the landmark Midwest Video case in 1979, which entitled cable companies to pick and choose whichchannels to carry and escape Federal Communication Commission (FCC) regulation,this case allows search engine providers to escape any rules that would force themto disclose why some content (namely commercial content) appears more heavilyconcentrated than others. Of course, these developments may bode well for internet consumers (as they are now universally referred to in the search industry),but not so well for internet users with noncommercial tasks.

Because of Google’s aggressive actions towards link farms and its win againstSearchKing, link-farms have become more risky than worthwhile. Meanwhile, asSEMs continue to be punished and are, as a consequence, losing a successfulmarketing tool, Google has emerged a winner on two counts. First, the companycan continue to boast about its commitment to search engine integrity. Second,with the demise of link farming, advertisers have become increasingly dependenton Google’s very successful AdWords program (Goodman, 2003). They have alsobecome increasingly dependent on reciprocal linking as a necessary marketing tool.In this regard, Google is doing a fine job to accelerate this trend, which in thebusiness world is referred to as contextual linking.

Contextual links As it stands, link farms have never been nearly as effective at influencing Google’sranking system as singular links from a highly prominent web site. A link, forexample, to the used pick-up truck company Bronco Graveyard(broncograveyard.com) that appears on the home page of the popular truckingmagazine Truckin’ (truckin.com) can do a world of good in terms of enhancing Bronco Graveyard’s visibility; a link on the popular Tennis.com website to the less-known raquetdepot.com also helps increase the small web site’s “popularity,” andthus its ranking on regular search results. Called “contextual links,” they are linksto other sites that match the context of the main Web site. Today, users can clickon contextual links at the bottom of nearly every online article in a commercialpublication. But, small as they are, they are effective far beyond the advertising spoton a given page; the act of linking is also an act of endorsement, and consequentlyincreases the company’s PageRank standing in Google search results. As Walker(2005) explains, “The economy of links is not product oriented. It is service oriented, and the service is the link. The link is an action rather than an item; anevent rather than a metaphor” (p. 2).

Contextual links are, not surprisingly, highly valued, with commercial onlinepublications quickly jumping into the contextual link business by giving advertisersthe option to buy links on their home page, as Tennis.com does through its “TennisMagazine MarketPlace Program.” Although SEMs have worked hard to establishreciprocal links between smaller sites, it turns out that it’s the more well-connectedand powerful search engine companies, Google and Yahoo!/Overture, that are themost busy brokering contextual link deals through their massive index of advertisers.

Google introduced its AdSense program in 2003, while Yahoo! introduced ContentMatch a month later (Acohido, 2003). Both programs broker contextual links oncontent Web sites. Yahoo!, for example, supplies sponsored links to CNN.com andWall Street Journal.com. Google supplies sponsored links to U.S. News & World Report, the Weather Channel, and ABC.com (Mangalindan, 2003). Its purchase ofSprinks in 2003 (a pay-per-click advertising network owned by media conglomerate Primedia), and a resulting relationship with Primedia (which, amongother media products, owns the largest number of niche magazines, all of whichhave an online presence), will allow Google to supply contextual links to all thesepublications. Google’s drive to plant more and more contextual links amongprominent pages across the web, a process that increases the prominence of allthese commercial pages within the Google PageRank system, actually underminesthe company’s line about search engine result integrity.

With Yahoo! increasingly mirroring Google’s PageRank system, and with suchprescribed contextual linking in place, the search results of both search engineprovider/portals are now nearly indistinguishable, especially in terms of theirpromotion of the most prominent sites (with which they have advertising relationships). Again, the myth about search engine variance is discredited. As Hindman et al (2003) have observed, “All modern search enginealgorithms—including those radically different from Google’s PageRank—tend toreturn these most connected sites first” (p. 27). These authors have also observedthat the Web, via search engines, now operates much like traditional media, withheavily concentrated oligopolies serving as gatekeepers for entry into the rest ofthe community.

With these developments, both Google and Yahoo! are also becoming more generalonline ad agencies than search engines, and like ad agencies, they increasinglymeasure “ad” performance and collect consumer data. Consequently, both measurethe results of ads (what the industry now euphemistically terms “customizes”) bytracking the clickstream data, cookies, pixel tags, and contact/personallyidentifying information of search engine users. While Yahoo!/Overture relies on thelarge, web-based company, Doubleclick for this purpose, Google relies on its newsubsidiary, Kaltix, a start-up company that has developed profile-tailoring softwareto better target individual users by tracking their choices on the web; Googlepurchased Kaltix in October 2003 (Mangalindan, 2003). As Wall Street Journal business reporter Mylene Mangalindan observed, “By gathering more data on each Google user, the reasoning goes, the search engine would know that a search for‘apple’ is one for fruit rather than computers” (p. B1). Both Yahoo! and Google arealso working towards providing successful geo-location functions to their marketing toolkits (Oct. 18). These ad services identify users’ specific locations, andthus enable local advertisers to use search engines as a marketing strategy.

Interestingly, Google defines these local business opportunities in terms of greaterdemocracy. In 2003, Google’s director of product management argued that hercompany enabled democracy because anyone, even small advertisers, couldadvertise via Google (Mangalindan, 2003). Accordingly, in the world of search (asspoken by representatives from the “search engine of integrity”), the notion ofonline democracy no longer has anything to do with regular users—the democracyof ideas—but applies only to the advertising world—the democracy of the marketplace. The word “relevancy” has also come to have new meaning in the worldof search business-speak. Rather than attempting to deliver the most relevantinformation to users, the task is now for search engines to deliver the most relevantconsumer information to consumers (Jan. 6). As business reporter David Crowe(2003) stated in the Australian Financial Review, “keeping advertisers happy with their paid searches is now the most important objective for the big search companies” (p. 29).

In five short years, an industry study revealed that the pay-per-click market hadgrown from under $100 million a year to a multi-billion industry in 2003, andincluded these startling statistics: 42 percent of those who bought from onlineretail sites arrived via search engines (Schacther, 2003). So successful was searchengine sponsorship that rates increased by 50 percent in 2003 alone, revealing that“the adoption of the internet as an advertising vehicle by traditional advertisers istruly taking place” (Hansell, 2003b). Industry reports also pointed to search enginesponsorship as an increasing, rather than diminishing trend: the business analystgroup Bancorp Piper Jaffray projected that the search industry would swell to $7billion by 2007 (Oct. 11).

Anatomy lesson 3: Google, Yahoo!, and the predatory goals of Microsoft

Three main companies, all American-based, dominate the search industry: Google,Yahoo!, and Microsoft. Google has been involved in search technology from itsinception (1998), syndicating its search engine provider services and becoming afavorite search portal among users. Yahoo!, which began as a directory and searchportal (outsourcing to Inktomi, and then Google, for its search technology), becamea dominant player with its purchase of Inktomi, and then Overture (which in turnhad just purchased AltaVista and AlltheWeb), in 2003. Since these acquisitions,Yahoo! has been in direct competition with Google for search portal prominence.Yet Microsoft, with its powerful MSN portal, is on the horizon as the most dominantplayer of all. As CEO Bill Gates has said, “we will catch them,” setting the date forthe summer of 2005 (Feb. 1, Markoff, 2003).

The Google/Yahoo! rivalryIn 2000, Yahoo! hired Google to conduct the searches on its search portal, givingthe company “its first big break” (Hansell, 2003a). By 2003, however, Google andYahoo! had become the fiercest competitors. To understand the extent of this rivalry, it’s helpful to look at the major acquisitions each company made prior,during, and just after 2003:

Oct. 2001 Overture achieves profitability, with “outstanding financial results”

Overture’s success establishes search as a key business strategy.May 2001 Inktomi introduces “paid inclusion”Oct. 2002 Google launches AdWords program, begins head-on competition The end result of this run of acquisitions, which mostly occurred during 2003,meant that two rival search engine companies provide the searching functions (bothcommercial and algorithmic) to nearly every search portal on the Web. The oneremaining search engine provider, Teoma (owned by Ask Jeeves) is considered aripe prospect for acquisition (Savitz, 2003). Google itself has been preyed upon bythe software giant, Microsoft. Indeed, apart from the Google/Yahoo! rivalry,Microsoft emerged as the third dominant player by the end of 2003.

with Overture
Dec. 2002 Mar. 2003 Apr. 2003Apr. 2003 May 2003June 2003	Google launches the Froogle shopping toolYahoo! enters the search engine provider business, acquiring Inktomi (and dumping Google)Overture acquires AlltheWeb and AltaVistaGoogle acquires Applied Semantics (an online advertising softwareco.)Google launches its AdSense programOverture introduces Content Match, directly competing with AdSense
Sept. 2003 Sept. 2003Oct. 2003 Oct. 2003 Oct. 2003 Mar 2004 Mar 2004 Mar 2004	Yahoo! expands Yahoo! Shopping (using Inktomi technology), tocompete with FroogleGoogle acquires Kaltix (online profiling and marketing company)Yahoo acquires Overture (which includes AltaVista and AlltheWeb)Yahoo! introduces SmartSort (which helps users quickly narrow asearch for certain products)Yahoo! acquires 3721 Network Software (Chinese letter searchengine)Yahoo! acquires Kelkoo, Europe's leading online comparisonshopping serviceYahoo!/Overture introduces Site Match (a controversial combination of paid inclusion and pay-per-click)Google introduces Google Local, allowing advertisers to connectwith local users

Microsoft makes a move Microsoft’s moves on Google were not surprising given the company’s nearly limitless resources and stated ambitions in content acquisition. As early as 1995, Gates was talking about going “well beyond simply providing a pipe for bits” (1995,

p. 241-242). Microsoft, in Dawson and Foster’s (1998) words, is “interested in moving up ‘the economic food chain’ from the delivery and distribution of bits atthe bottom to computer applications and services and content at the top. Suchcompanies want to own the bits, not simply deliver them” (p. 60). However, Googledenied the partnership or takeover opportunity (at least for the time being), andMicrosoft turned to Plan B. For most of 2003, MSN was still relying on Inktomi topower its algorithmic searches, Overture to power its commercial searches, andLooksmart as a fortifying directory. In other words, MSN was deeply dependent onsubsidiaries owned by Yahoo! But by October (after the Google talks unraveled(Teather, 2003), MSN had ditched Looksmart and had started work on its ownsearch engine platform, resolving to drop Inktomi and Overture sometime after2006, when it would roll out its “Google-killer” search engine algorithm (Bazeley,2004). In other words, Microsoft had begun to amass a proprietary index of sitesfrom which to conduct searches and was hiring hundreds of engineers to work onweb-searching algorithms to top Google. Moreover, the software company planned to integrate its search technology directly into its Windows operating system under a project code-named “Longhorn” (Mangalindan, 2003).

For anyone familiar with Microsoft’s history of annihilating the competition, thisstrategy seems hauntingly familiar to Microsoft’s triumph over Netscape in the web browser business. “Today we are number one in email, we are number one inmessenger. Our ambition is to be number one in search,” Sharon Babyle, thegeneral manager of MSN’s consumer Internet service, said at the end of November2003. (Conners, 2003). During the run-up to Google’s public offering, which finallyoccurred in August 2004, Microsoft was working hard to destabilize Google andsnag Google employees (Markoff, 2004). Given Google’s subsequent release of itsnew desktop computer search software (Google Desktop Search), which allowsusers to search their desktops far more efficiently than Microsoft (Bazeley, 2004), itis clear that, if anything, Google will put up a good fight. But, now that Google’sfuture requires the company to attend to the demands of shareholders, manyanalysts are forecasting damage to Google’s search integrity. As the opening sentence of a story in Wired plainly said, “The world’s biggest, best-loved searchengine owes its success to supreme technology and a simple rule: Don’t be evil.Now the geek icon is finding that moral compromise is just the cost of doing bigbusiness” (McHugh, 2003). For starters, the Google Desktop Search programincreases the company’s ability to target users with personalized advertising.Likewise, the company’s Gmail program, introduced in March 2004 examines thecontent of individual emails and sends users’ marketing information back tocompany headquarters. Moral compromise and commercial intrusions into Google’ssearch listings will certainly continue as Microsoft and Yahoo! continue the onslaught.

Conclusion

Despite the considerable implications of search engine commercialization forknowledge access, the topic has not gained much attention in academic and libraryspheres. One reason for this lack of attention is good public relations: the searchengine industry continues to highlight “integrity,” “relevance” and “objectivity” as amantra, meanwhile stealthily undermining internet users’ faith in search technology. Another reason is the general silence in the U.S. mass media when itcomes to any criticism of commercialism (since the mass media themselves aremajor participants in advertising-supported commercial media culture).Consequently, the four myths about search engine services resound: they areimpartial, they are all-inclusive, they vary greatly, and they are the most reliableplace to go for relevant online information.

We educators, for the most part, buy into these myths; we continue to have alargely optimistic outlook on search engines as helpful and trustworthy educationaland research tools. Despite its heavy commercialism, the web’s potential as a placefor online scholarship and diversity is still evident—when the right search terms areused—and positive experiences can certainly outweigh negative experiences, fornow, anyway. Moreover, educators and librarians have heavily promoted individualskills (advanced searching techniques, web page evaluation skills) as a way to copewith excessive commercialism. Although it may feel empowering to teach or possess these skills, a wholesale critique of the commercial web structure remainssidelined (Fabos, 2004).

Fortunately, researchers in economics, political science, communication studies,applied physics and computer science have begun to scrutinize the structuralcomponents of the web against the framework of democracy and knowledgeaccess. Economists Lucas D. Introna and Helen Nissenbaum (2000), for example,have examined search engines’ current trajectory as market-driven informationtools and have questioned the future of the web as a public good. They argue “ifsearch engines systematically highlight Web sites with popular appeal and mainstream commercial purpose, as well as Web sites backed by entrenchedeconomic powers, they amplify these presences on the Web at the expense ofothers” (p. 28).

Hindman et al (2003), also economists, are also concerned with the increasingconcentration (and visibility) of heavily linked sites. They have analyzed thehyperlinks surrounding political Web sites (measuring link structure on a massivescale), and have concluded that the number of highly visible sites is small, and thevisibility drop-off is rapid. They have called this organizational structure “googlearchy” (the rule of the most heavily linked), and suggest that the Web ismore like the current corporate mass media, with top-down control of limited content, rather than a diverse platform of ideas.

Another computer scientist, Susan L. Gerhart (2004), has researched a residualeffect of web site consolidation: the way search engines inherently suppresscontroversy. In her study, which tracked search engine results about five controversial topics, Gerhart found that three out of five were buried by commercialsites, and concluded that significant amounts of controversial material are unattainable via search engines. In her words, controversy is important because:

… controversies often express the richness and depth of a topic.Controversies dramatize change. Controversies may make a critical difference in life-altering decisions. Scientists, journalists, and intelligence analysts are professionally required to address multipleperspectives, facts, authorities, and opinions on topics. Search enginesmay significantly decrease their productivity or conceal incompetenceif controversies are overly difficult to investigate. (Gerhart, 2004, p. 4)

According to Gerhart, unless a controversy has already been recently highlighted inthe mainstream media, or unless a searcher is already aware that a controversyabout a particular topic already exists (and can type the appropriate keywords),controversial topics are largely invisible. Computer scientist Jill Walker (2005),another scholar researching the implications of a link-driven online economy,eloquently summarizes the mainstreaming of Web content this way:

We are participants in this power structure whether we like it or not.We can criticize it, reflect upon it, approve of it or try to subvert it. Wemust not ignore it. This standardization of links and their value willshape what the future finds. It defines what can be found. It definesknowledge. (p. 3)

Indeed, all of this research is important, because it goes beyond the net result ofsearch engine result lists (what librarians and most educators are typicallyconcerned with) and addresses the complex and economically-charged structure ofthe web that affects all search results regardless of how well one crafts an individual search.

As people habitually turn to commercial search engines to navigate a commercialenvironment, they are unaware of the increasing difficulties to locate content that isnot commercial. They are unaware of the misleading motives of the internetnavigation tools they use, and of the constant efforts among for-profit enterprise tobend the internet toward their ends. Communications scholar Robert McChesney(1999), who has been on the forefront of discussions about media concentration, isanother researcher who has touched upon the internet as the latest in a long line ofcommercial media that have been colonized by corporate power, turning a so-called democratic medium into one wholly controlled by big media oligopolies.“Despite its much ballyhooed ‘openness,’” he writes, “to the extent that it becomesa viable mass medium, it will likely be dominated by the usual corporate suspects”

(p. 183).

Today, the internet is an oligopoly, with Google, Yahoo! and Microsoft as the three companies controlling most of the internet’s information flow. These companies arecapable of bringing good things to the web search environment, and will do theirbest to appear as if they put the interests of users first, rolling out new plans tomake search experiences more worthwhile and exposing “the invisible web.” For example, Google has begun to digitize all the books in the Stanford Universitylibrary system that date before 1923 (those that are in the public domain); thecompany will include this wealth of information in its database. In March, 2004,Yahoo! initiated its “Content Acquisition Program,” part of which involves contentpartnerships with organizations like the Library of Congress, National Public Radio,The New York Public Library (owner of Project Gutenberg), the National ScienceDigital Library, and Michigan’s OAIster digital archive). These considerable archiveswill supply “premiere” (that is, educational) listings on the Yahoo! search result list.And as Microsoft’s search engine becomes a significant player, the company willmore than likely make well-publicized efforts to improve its database with rich,non-commercial archives. As with television, there has to be some decent contentto drive the ads—that’s the basis of how commercial mass media operates in theUnited States.

Regardless of these seemingly good intentions, however, the myths surroundingsearch engines must be debunked. Commercial search engines will always beeducationally compromised; they will always give advertisers premium treatment intheir services. The other part of Yahoo!’s “Content Acquisition Program,” for example, involves Site Match, a more vigorous paid inclusion program refreshedevery 48 hours, which allows web-heavy companies more control over which oftheir pages are frequently indexed. The creation of Yahoo!’s Site Match also marksa change in the paid inclusion payment model. Now, a finder’s fee is involvedbeyond the original flat fee: any time a user clicks on a commercial link, whetherit’s on page 1 or page 20 on the search results page, advertisers pay the searchengine a small listing fee—naturally giving search engines more incentive to getusers into the “click-through” mentality.

If we want to go beyond a mainstream, commercialized, sponsored onlineinformation repository we need to turn to a different structure that offers a moreinclusive, democratic information environment. We need to discredit the final myth:that search engines are the only place to go for relevant online information. Some places to start are the non-profit world of subject gateways (e.g., The NationalScience Digital Library, INFOMINE, OAIster, or the Resource Discovery Network),issue networking sites such as Govcom.org, or the collaborative environments ofwiki pages (e.g., wikipedia.org; disinfopedia.org), where people, not corporations,make the significant choices about the information—controversial and diverse—thatis accessible online.

NOTES

¹ Both Lycos and WebCrawler, which emerged in 1994, pre-date AltaVista and Inktomi,

but stopped developing their algorithms in favor of outsourcing search services from

AlltheWEb (Lycos) or multiple search engine providers (Webcrawler).

References

Acohido, B. (2003, November 17). Simplifying the info hunt. USA Today, 11E.

Bazeley, M. (2004, October 15). Google rolls our search software. San Jose Mercury News, p. B2.

Conners, E. (2003, November 25). Microsoft in search of top spot. Australian Financial Review, p. 32.

Crowe, D. (2003, October 18). In search of a profit engine. Australian Financial Review, 29.

Dawson, M. and Foster, J. B. (1998). Virtual capitalism. In R. McChesney, E. M. Wood& J.B. Foster (Eds.), Capitalism and the information age: The politicaleconomy of the global communication revolution (pp. 51-68). New York: Monthly Review Press.

Durman, P. (2003, Nov. 9). Customers clicking with online adverts. Sunday Times (London), 16.

Fabos, B. (2004). Wrong turn on the information superhighway: Education and thecommercialization of the internet. New York: Teachers College Press.

Fine Brand (2003). A brief history of search engine marketing on the Web. [Online.]Availailable: http://www.Fine Brand.com/ideacenter/search-engineoptimization/ abriefhistory-2.cfm [2004, March 7].

Gaither, Chris (2003, February 23). Searching for dollars as banner market flags,Net giants pin hopes on revenue from targeted ads related to specific queries. The Boston Globe, F1.

Gates, Bill (1995). The road ahead. New York: Viking.

Gerhart, Susan L. (2004, January). Do Web search engines suppress controversy?First Monday, 9(1). [Online].http://firstmonday.org/issues/issue9_1gerhart/index.html. (2004,January 23).

Goodman, A. (2003). Cracking Google’s new algorithm. WebProNews [Online].Available:http://www.webpronews.com/ebusiness/smallbusiness/wpn-220031203CrackingGooglesNewAlgorithm.html

Griffiths, J. T. and Brophy, P. (2005, Spring). Student Searching Behaviour and theWeb: Use of Academic Resources and Google. Library Trends, 53(4), pp. XX

XX. Hansell, S. (1999, April 15). Altavista invites advertisers to pay for top ranking. New York Times, C2.Hansell, S. (2003a, April 7). Yahoo plans improvements in effort to regain lostground. New York Times, 2.Hansell, S. (2003b, October 9). Yahoo's resurgent profit is led by strong advertising.New York Times, C7.

Harvey, Fiona (2003a, January 15). Online ads await click-through to profitability:Questions have been raised about whether the medium works, but analystsare optimistic. Financial Times, FT Report, 1.

Harvey, Fiona (2003b, March 4). Putting a price on results: Recent acquisitionssuggest the sponsored search sector is a bright spot in a gloomy market. The Financial Times, 32.

Hindman, M, Tsioutsiouliklis, K, and Johnson, J. (2003). “Googlearchy”: How a fewheavily-linked sites dominate politics on the Web. [Online]. Available:www.princeton.edu/~mhindman/googlearchy--hindman.pdf [2004, 10 April].

Introna, L. and Nissenbaum, H. (2000). Shaping the web: Why the politics of searchengines matters. Information Society, 16(3): 189-186. [Online]. Available at: http://www-us.ebsco.com/online/Reader.asp (if your library is subscribed).

Kopytoff, V. (2003, November 9). Google’s future looking good; Bulked-up searchfirm beats up rivals. San Francisco Chronicle, I1.

Lasica, J.D. (2001, July 23). Search engines and editorial integrity. Online Journalism R e v i e w [Online]. Available: http://www.ojr.org/ojr/technology/1017778969.php. [2001, December 12].

Mangalindan, M. (2003, October 16). Seeking growth, search engine Google actslike ad agency. Wall Street Journal, p. B1.

Markoff, J. (2003, October 31). Microsoft and Google: Partners or rivals? The New York Times, C1.

Markoff, J. (2004, February 1). The coming search wars. The New York Times,section 3, 1.

McChesney, R. (1999). Rich media, poor democracy: Communication politics in dubious times. Urbana: University of Illinois Press.

McHugh, Josh (2003, January). Google vs. Evil. Wired Magazine, 11(1) [Online]. Available: http://www.wired.com/wired/archive/11.01/googgle_pr.html. [2004, March 20].

MoreVisibility platform three: E-commerce initiative (2003). More visibility.com[Online]. Available: http://www.morevisibility.com/serices_three.html. [2003,April 18].

Overture (2003a, February 25). Overture to acquire web search unit of Fast Search &Transfer—FAST. Press release; Overture, Inc. [Online]. Available: http://www.corporateir.net/ireye/ir_site.zhtml?ticker=OVER&script=410&layout=0&item_id=385605 [2003, April 18].

Overture (2003b). Vision: Corporate overview. Overture, Inc. [Online]. Available: http://www.content.overture.com/d/Usm/about/company/vision.jhtml[2003, May 25].

Reuters (2003, December 24). Yahoo! to buy software maker. Los Angeles Times,Section 3, 2.

Savitz, E. (2003, October 30). Why are Jeeves shares so high? The Toronto Star, K07.

Schachter, K. (2003, December 5). The pay-per-click Internet market grows intomulti-billion dollar business. Dolan Media Newswires.

Teather, D. (2003, November 1). Microsoft runs search for a way to take overinternet giant Google. The Guardian, 1.

The Web Search WorkShop (2004). Materials: Paying for search engine and directorylistings. [Online]. Available: htttp://www.websearchworkshop.co.uk/paying.htm. [2004, March 21].

Walker, Jill (2005, Spring). Links and power: The political economy of linking on theweb. Library Trends, 53(4), pp. XX.

Waters, Richard (2003, March 12). Search industry scours scrap heap for bargains.The Financial Times, 30.