Google Ngram Viewer

The Google Ngram Viewer or Google Books Ngram Viewer is an online search engine that charts the frequencies of any set of comma-delimited search strings using a yearly count of n-grams found in sources printed between 1500 and 2008[1][2][3][4][5] in Google's text corpora in English, Chinese (simplified), French, German, Hebrew, Italian, Russian, or Spanish.[2][6] There are also some specialized English corpora, such as American English, British English, English Fiction, and English One Million; and the 2009 version of most corpora is also available.[7]

The program can search for a single word or a phrase, including misspellings or gibberish.[6] The n-grams are matched with the text within the selected corpus, optionally using case-sensitive spelling (which compares the exact use of uppercase letters),[3] and, if found in 40 or more books, are then plotted on a graph.[8]

The Google Ngram Viewer, as of January 2016, supports searches for parts of speech and wildcards.[7]

History

The program was developed by Jon Orwant and Will Brockman and released in mid-December 2010.[2][4] It was inspired by a prototype (called "Bookworm") created by Jean-Baptiste Michel and Erez Aiden from Harvard's Cultural Observatory and Yuan Shen from MIT and Steven Pinker.[9]

The Ngram Viewer was initially based on the 2009 edition of the Google Books Ngram Corpus. As of January 2016, the program can search an individual language's corpus within the 2009 or the 2012 edition.

Operation and restrictions

Commas delimit user-entered search-terms, indicating each separate word or phrase to find.[8] The Ngram Viewer returns a plotted line chart within seconds of the user pressing the Enter key or the "Search" button on the screen.

As an adjustment for more books having been published during some years, the data is normalized, as a relative level, by the number of books published in each year.[8]

Google populated the database from over 5 million books published up to 2008. Accordingly, as of January  2016, no data will match beyond the year 2008, no matter if the corpus was generated in 2009 or 2012. Due to limitations on the size of the Ngram database, only matches found in at least 40 books are indexed in the database; otherwise the database could not have stored all possible combinations.[8]

Typically, search terms cannot end with punctuation, although a separate full stop (a period) can be searched.[8] Also, an ending question mark (as in "Why?") will cause a 2nd search for the question mark separately.[8]

Omitting the periods in abbreviations will allow a form of matching, such as using "R M S" to search for "R.M.S." versus "RMS".

Corpora

The corpora used for the search are composed of total_counts, 1-grams, 2-grams, 3-grams, 4-grams, and 5-grams files for each language. The file format of each of the files is tab-separated data. Each line has the following format:[10]

  • total_counts file
    year TAB match_count TAB page_count TAB volume_count NEWLINE
  • Version 1 ngram file (generated in July 2009)
    ngram TAB year TAB match_count TAB page_count TAB volume_count NEWLINE
  • Version 2 ngram file (generated in July 2012)
    ngram TAB year TAB match_count TAB volume_count NEWLINE

The Google Ngram Viewer uses match_count to plot the graph.

As an example, a word "Wikipedia" from the Version 2 file of the English 1-grams is stored as follows:[11]

ngramyearmatch_countvolume_count
Wikipedia190411
Wikipedia1912111
Wikipedia192411
Wikipedia1925111
Wikipedia1929111
Wikipedia1943111
Wikipedia1946111
Wikipedia1947111
Wikipedia1949111
Wikipedia1951111
Wikipedia1953222
Wikipedia1955111
Wikipedia195811
Wikipedia1961222
Wikipedia1964222
Wikipedia1965111
Wikipedia1966152
Wikipedia1969333
Wikipedia19701294
Wikipedia1971444
Wikipedia1972222
Wikipedia197311
Wikipedia197421
Wikipedia1975333
Wikipedia1976111
Wikipedia1977133
Wikipedia1978111
Wikipedia197911212
Wikipedia1980134
Wikipedia1982111
Wikipedia198332
Wikipedia1984483
Wikipedia1985373
Wikipedia198664
Wikipedia1987132
Wikipedia1988143
Wikipedia1990122
Wikipedia199185
Wikipedia199211
Wikipedia199311
Wikipedia1994233
Wikipedia199541
Wikipedia1996233
Wikipedia199761
Wikipedia19983210
Wikipedia19993911
Wikipedia20004312
Wikipedia20015914
Wikipedia200210519
Wikipedia200314953
Wikipedia2004803285
Wikipedia20052964911
Wikipedia200698182655
Wikipedia2007200175400
Wikipedia2008337226825

The graph plotted by the Google Ngram Viewer using the above data is here.

Criticism

The data set has been criticized for its reliance upon inaccurate OCR, an overabundance of scientific literature, and for including large numbers of incorrectly dated and categorized texts.[12][13] Because of these errors, and because it is uncontrolled for bias[14] (such as the increasing amount of scientific literature, which causes other terms to appear to decline in popularity), it is risky to use this corpus to study language or test theories.[15] Since the data set does not include metadata, it may not reflect general linguistic or cultural change[16] and can only hint at such an effect.

Another issue is that the corpus is in effect a library, containing one of each book. A single, prolific author is thereby able to noticeably insert new phrases into the Google Books lexicon, whether the author is widely read or not.[14]

OCR issues

Optical character recognition, or OCR, is not always reliable, and some characters may not be scanned correctly. In particular, systemic errors like the confusion of "s" and "f" in pre-19th century texts (due to the use of the long s which was similar in appearance to "f") can cause systemic bias. Although Google Ngram Viewer claims that the results are reliable from 1800 onwards, poor OCR and insufficient data mean that frequencies given for languages such as Chinese may only be accurate from 1970 onward, with earlier parts of the corpus showing no results at all for common terms, and data for some years containing more than 50% noise.[17][18]

See also

References

  1. ↑ "Quantitative analysis of culture using millions of digitized books" JB Michel et al, Science 2011, DOI: 10.1126/science.1199644
  2. 1 2 3 "Google Ngram Database Tracks Popularity Of 500 Billion Words" Huffington Post, 17 December 2010, webpage: HP8150.
  3. 1 2 "Google Ngram Viewer - Google Books", Books.Google.com, May 2012, webpage: G-Ngrams.
  4. 1 2 "Google's Ngram Viewer: A time machine for wordplay", Cnet.com, 17 December 2010, webpage: CN93.
  5. ↑ "A Picture is Worth 500 Billion Words – By Rusty S. Thompson", HarrisburgMagazine.com, 20 September 2011, webpage: HBMag20.
  6. 1 2 "Google Books Ngram Viewer - University at Buffalo Libraries", Lib.Buffalo.edu, 22 August 2011, webpage: Buf497.
  7. 1 2 Google Books Ngram Viewer info page: https://books.google.com/ngrams/info
  8. 1 2 3 4 5 6 "Google Ngram Viewer - Google Books" (Information), Books.Google.com, December 16, 2010, webpage: G-Ngrams-info: notes bigrams and use of quotes for words with apostrophes.
  9. ↑ The RSA (4 February 2010). "Steven Pinker - The Stuff of Thought: Language as a window into human nature" – via YouTube.
  10. ↑ "Google Books Ngram Viewer". Google.
  11. ↑ googlebooks-eng-all-1gram-20120701-w.gz at http://storage.googleapis.com/books/ngrams/books/datasetsv2.html
  12. ↑ Google Ngrams: OCR and Metadata. ResourceShelf, 19 December 2010
  13. ↑ Nunberg, Geoff (16 December 2010). "Humanities research with the Google Books corpus". Archived from the original on 10 March 2016.
  14. 1 2 Pechenick, Eitan Adam; Danforth, Christopher M.; Dodds, Peter Sheridan; Barrat, Alain (7 October 2015). "Characterizing the Google Books Corpus: Strong Limits to Inferences of Socio-Cultural and Linguistic Evolution". PLOS ONE. 10 (10): e0137041. doi:10.1371/journal.pone.0137041.
  15. ↑ Zhang, Sarah. "The Pitfalls of Using Google Ngram to Study Language". WIRED. Retrieved 2017-05-24.
  16. ↑ Koplenig, Alexander (2015-09-02). "The impact of lacking metadata for the measurement of cultural and linguistic change using the Google Ngram data sets—Reconstructing the composition of the German corpus in times of WWII". Digital Scholarship in the Humanities (published 2017-04-01). 32 (1): 169–188. doi:10.1093/llc/fqv037. ISSN 2055-7671.
  17. ↑ Google n-grams and pre-modern Chinese. digitalsinology.org.
  18. ↑ When n-grams go bad. digitalsinology.org.

Bibliography

  • Lin, Yuri; et al. (July 2012). "Syntactic Annotations for the Google Books Ngram Corpus" (pdf). Proceedings of the 50th Annual Meeting. Demo Papers. Jeju, Republic of Korea: Association for Computational Linguistics. 2: 169–174. 2390499. Whitepaper presenting the 2012 edition of the Google Books Ngram Corpus
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.