Common Crawl

Common Crawl
Type of business 501(c)(3) non-profit
Available in English
Headquarters San Francisco, California, USA; Los Angeles, California, USA
Founder(s) Gil Elbaz
Key people Peter Norvig, Nova Spivack, Carl Malamud, Kurt Bollacker Joi Ito
Website commoncrawl.org

Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public.[1][2] Common Crawl's web archive consists of petabytes of data collected since 2011.[3] It completes crawls generally every month.[4]

Common Crawl was founded by Gil Elbaz.[5] Advisors to the non-profit include Peter Norvig and Joi Ito.[6] The organization's crawlers respect nofollow and robots.txt policies. Open source code for processing Common Crawl's data set is publicly available.

History

Amazon Web Services began hosting Common Crawl's archive through its Public Data Sets program in 2012.[7]

The organization began releasing metadata files and the text output of the crawlers alongside .arc files in July of that year.[8] Common Crawl's archives had only included .arc files previously.[8]

In December 2012, blekko donated to Common Crawl search engine metadata blekko gathered from crawls it conducted from February to October 2012.[9] The donated data helped Common Crawl "improve its crawl while avoiding spam, porn and the influence of excessive SEO."[9]

In 2013, Common Crawl began using Apache Software Foundation's Nutch webcrawler instead of a custom crawler.[10] Common Crawl switched from using .arc files to .warc files with its November 2013 crawl.[11]

History of Common Crawl data

The following data have been collected from the official Common Crawl Blog

Crawl DateAvailability dateSize in TBBillions of pagesComments
May 2018June 20182152.75
April 2018May 20182303.1
March 2018March 20182503.2
February 2018March 20182703.4
January 2018January 20182703.4
December 2017December 20172402.9
November 2017November 20172603.2
October 2017October 20173003.65
September 2017September 20172503.01
August 2017August 20172803.28
July 2017July 20172402.89
June 2017July 20172603.16
May 2017June 20172502.96
April 2017May 20172502.94
March 2017April 20172503.07
February 2017March 20172503.08
January 2017February 20172503.14
December 2016December 2016-2.85
October 2016November 2016-3.25
September 2016October 2016-1.72
August 2016September 2016-1.61
July 2016August 2016-1.73
June 2016July 2016-1.23
May 2016June 2016-1.46
April 2016May 2016-1.33
February 2016February 2016-1.73
November 2015December 20151511.82
September 2015November 20151061.32
August 2015October 20151491.84
July 2015August 20151451.81
June 2015July 20151311.67
May 2015July 20151592.05
April 2015May 20151682.11
March 2015May 20151241.64
February 2015March 20151451.9
January 2015March 20151391.82
December 2014January 20151602.08
November 2014December 20141351.95
October 2014November 20142543.7
September 2014November 20142202.8
August 2014September 20142002.8
July 2014August 20142663.6
April 2014July 20141832.6
March 2014March 20142232.8First Nutch crawl
January 2014January 20141482.3Crawls performed monthly
November 2013November 20131022Data in Warc file format
July 2012July 2012--Data in Arc file format
January 2012January 2012--Public Data Set of Amazon Web Services
November 2011November 2011405First availability on Amazon

Norvig Web Data Science Award

In corroboration with SURFsara, Common Crawl sponsors the Norvig Web Data Science Award, a competition open to students and researchers in Benelux.[12][13] The award is named for Peter Norvig who also chairs the judging committee for the award.[12]

References

  1. ↑ Rosanna Xia (February 5, 2012). "Tech entrepreneur Gil Elbaz made it big in L.A." Los Angeles Times. Retrieved July 31, 2014.
  2. ↑ "Gil Elbaz and Common Crawl". NBC News. April 4, 2013. Retrieved July 31, 2014.
  3. ↑ "So you're ready to get started". Retrieved 2018-06-02.
  4. ↑ Lisa Green (January 8, 2014). "Winter 2013 Crawl Data Now Available". Retrieved June 2, 2018.
  5. ↑ "Startups - Gil Elbaz and Nova Spivack of Common Crawl - TWiST #222". This Week In Startups. January 10, 2012.
  6. ↑ Tom Simonite (January 23, 2013). "A Free Database of the Entire Web May Spawn the Next Google". MIT Technology Review. Retrieved July 31, 2014.
  7. ↑ Jennifer Zaino (March 13, 2012). "Common Crawl To Add New Data In Amazon Web Services Bucket". Semantic Web. Retrieved July 31, 2014.
  8. 1 2 Jennifer Zaino (July 16, 2012). "Common Crawl Corpus Update Makes Web Crawl Data More Efficient, Approachable For Users To Explore". Semantic Web. Retrieved July 31, 2014.
  9. 1 2 Jennifer Zaino (December 18, 2012). "Blekko Data Donation Is A Big Benefit To Common Crawl". Semantic Web. Retrieved July 31, 2014.
  10. ↑ Jordan Mendelson (February 20, 2014). "Common Crawl's Move to Nutch". Common Crawl. Retrieved July 31, 2014.
  11. ↑ Jordan Mendelson (November 27, 2013). "New Crawl Data Available!". Common Crawl. Retrieved July 31, 2014.
  12. 1 2 Lisa Green (November 15, 2012). "The Norvig Web Data Science Award". Common Crawl. Retrieved July 31, 2014.
  13. ↑ "Norvig Web Data Science Award 2014". Dutch Techcentre for Life Sciences. Retrieved July 31, 2014.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.