World Wide Web Wanderer

The World Wide Web Wanderer, also referred to as just the Wanderer, was a Perl-based web crawler that was first deployed in June 1993 to measure the size of the World Wide Web. The Wanderer was developed at the Massachusetts Institute of Technology by Matthew Gray, who, as of 2017, has spent a decade as a software engineer at Google. The crawler was used to generate an index called the Wandex later in 1993. While the Wanderer was probably the first web robot, and, with its index, clearly had the potential to become a general-purpose WWW search engine, the author does not make this claim[1] and elsewhere[2] it is stated that this was not its purpose. The Wanderer charted the growth of the web until late 1995.