The World Wide Web Wanderer, or simple, 'the Wanderer', was one
of the earliest web crawlers. The Wanderer was created in 1993,
by a student of the Massachusetts Institute of Technology, named:
Matthew Gray. While studying at MIT, Gray was also a member of the
Student Information Processing Board (SIPB); which setup MIT's first
website: www.mit.edu. Gray left MIT in 1994, and setup a software
company that developed tools for web developers. Gray is currently
(2010) working for Google.
While crawlers are used by search engines to find content the World
Wide Web - and index it into a database that can be queried - that
was not the purpose behind creating the Wanderer. Gray did not create
the Wanderer as a tool for developing a search engine: he simple
wanted to research the size of the World Wide Web for academic purposes.
In 1993, when the Wanderer was created, the World Wide Web was
in it's infancy and it was only just beginning to become popular.
No search engines existed, and people found web content from lists
of hyperlinks that were created manually. Therefore, nobody had
an accurate picture of how many website existed: due to the time
it takes to create and edit a manual list of websites. The only
suitable option was to create a software tool that could automate
the process, and this tool was a crawler.
In 1993, using the data collected by the Wanderer, gray wrote a
document named: Measuring the Growth of the Web. His report highlighted
the phenomenal growth of the World Wide Web and the Internet. And
this document highlights the fact that he created the Wanderer to
find new websites, and evaluate the growth of the World Wide Web.