Internet Guide Logo

Yahoo! Slurp

Last Edit: 10/01/17

Slurp is a web crawler that was used by Inktomi and Yahoo! to power their search results. Inktomi was founded in 1996, and was one of the most successful search result providers before Google. Inktomi was one of the first search engines to integrates 'pay per click' advertisements into their search results, and in this regard, Inktomi's was a pioneering search technology. Inktomi's search results powered search engines like Hotbot, and the web crawler it used was named Slurp. Slurp was written in C/C++, and a number of versions were released: Slurp/2.0 replaced Slurp/1.0 in 1997; the UserAgent for Slurp indicated it's version number.

Inktomi's search technology was purchased by Yahoo! in 2002 for $234 million. The valuation of Inktomi dropped starkly after the dot-com bubble (1997-2000); Yahoo! purchased Inktomi for $200 less per share: when compared to Inktomi's peak share price. After it's purchase of Inktomi, Yahoo! launched it's 'Yahoo! Search' service - to compete with Google - that was powered by Inktomi's search index and web crawler. Therefore, when Yahoo! Search was launched in 2003, it's crawler was named Yahoo! Slurp; Yahoo! Slurp was an upgraded version of Inktomi Slurp. Yahoo! Slurp was one of the first web crawlers to crawl dynamic content: webpages with a .php, .asp, and .jsp extension.

When Yahoo! launched Yahoo! Slurp, they stated it performed two functions:

  1. It collected content - from content partners - to 'power' it's news, finance and sports services.
  2. It accessed, collected and stored information from websites across the World Wide Web: to power it's organic Yahoo! Search service.

Yahoo! stated that it's Slurp web crawler could be identified by it's user agent:

Yahoo! Slurp was discontinued in 2009 when Yahoo! signed a contract with the Bing search engine: Bing would power Yahoo!'s search results.