{
    "info": {
        "author": "",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [
            "Intended Audience :: Developers",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: MIT License",
            "Natural Language :: English",
            "Operating System :: MacOS",
            "Operating System :: Microsoft :: Windows",
            "Operating System :: POSIX",
            "Operating System :: Unix",
            "Programming Language :: Python :: 3",
            "Topic :: Scientific/Engineering",
            "Topic :: Software Development",
            "Topic :: Text Processing :: Linguistic"
        ],
        "description": "wordfreq is a Python library for looking up the frequencies of words in many\nlanguages, based on many sources of data.\n\nAuthor: Robyn Speer\n\n\n## Installation\n\nwordfreq requires Python 3 and depends on a few other Python modules\n(msgpack, langcodes, and regex). You can install it and its dependencies\nin the usual way, either by getting it from pip:\n\n    pip3 install wordfreq\n\nor by getting the repository and running its setup.py:\n\n    python3 setup.py install\n\nSee [Additional CJK installation](#additional-cjk-installation) for extra\nsteps that are necessary to get Chinese, Japanese, and Korean word frequencies.\n\n\n## Usage\n\nwordfreq provides access to estimates of the frequency with which a word is\nused, in 36 languages (see *Supported languages* below). It uses many different\ndata sources, not just one corpus.\n\nIt provides both 'small' and 'large' wordlists:\n\n- The 'small' lists take up very little memory and cover words that appear at\n  least once per million words.\n- The 'large' lists cover words that appear at least once per 100 million\n  words.\n\nThe default list is 'best', which uses 'large' if it's available for the\nlanguage, and 'small' otherwise.\n\nThe most straightforward function for looking up frequencies is:\n\n    word_frequency(word, lang, wordlist='best', minimum=0.0)\n\nThis function looks up a word's frequency in the given language, returning its\nfrequency as a decimal between 0 and 1.\n\n    >>> from wordfreq import word_frequency\n    >>> word_frequency('cafe', 'en')\n    1.07e-05\n\n    >>> word_frequency('caf\u00e9', 'en')\n    5.75e-06\n\n    >>> word_frequency('cafe', 'fr')\n    1.51e-06\n\n    >>> word_frequency('caf\u00e9', 'fr')\n    5.13e-05\n\n\n`zipf_frequency` is a variation on `word_frequency` that aims to return the\nword frequency on a human-friendly logarithmic scale. The Zipf scale was\nproposed by Marc Brysbaert, who created the SUBTLEX lists. The Zipf frequency\nof a word is the base-10 logarithm of the number of times it appears per\nbillion words. A word with Zipf value 6 appears once per thousand words, for\nexample, and a word with Zipf value 3 appears once per million words.\n\nReasonable Zipf values are between 0 and 8, but because of the cutoffs\ndescribed above, the minimum Zipf value appearing in these lists is 1.0 for the\n'large' wordlists and 3.0 for 'small'. We use 0 as the default Zipf value\nfor words that do not appear in the given wordlist, although it should mean\none occurrence per billion words.\n\n    >>> from wordfreq import zipf_frequency\n    >>> zipf_frequency('the', 'en')\n    7.77\n\n    >>> zipf_frequency('word', 'en')\n    5.29\n\n    >>> zipf_frequency('frequency', 'en')\n    4.43\n\n    >>> zipf_frequency('zipf', 'en')\n    1.57\n\n    >>> zipf_frequency('zipf', 'en', wordlist='small')\n    0.0\n\n\nThe parameters to `word_frequency` and `zipf_frequency` are:\n\n- `word`: a Unicode string containing the word to look up. Ideally the word\n  is a single token according to our tokenizer, but if not, there is still\n  hope -- see *Tokenization* below.\n\n- `lang`: the BCP 47 or ISO 639 code of the language to use, such as 'en'.\n\n- `wordlist`: which set of word frequencies to use. Current options are\n  'small', 'large', and 'best'.\n\n- `minimum`: If the word is not in the list or has a frequency lower than\n  `minimum`, return `minimum` instead. You may want to set this to the minimum\n  value contained in the wordlist, to avoid a discontinuity where the wordlist\n  ends.\n\n\n## Frequency bins\n\nwordfreq's wordlists are designed to load quickly and take up little space in\nthe repository.  We accomplish this by avoiding meaningless precision and\npacking the words into frequency bins.\n\nIn wordfreq, all words that have the same Zipf frequency rounded to the nearest\nhundredth have the same frequency. We don't store any more precision than that.\nSo instead of having to store that the frequency of a word is\n.000011748975549395302, where most of those digits are meaningless, we just store\nthe frequency bins and the words they contain.\n\nBecause the Zipf scale is a logarithmic scale, this preserves the same relative\nprecision no matter how far down you are in the word list. The frequency of any\nword is precise to within 1%.\n\n(This is not a claim about _accuracy_, but about _precision_. We believe that\nthe way we use multiple data sources and discard outliers makes wordfreq a\nmore accurate measurement of the way these words are really used in written\nlanguage, but it's unclear how one would measure this accuracy.)\n\n\n## The figure-skating metric\n\nWe combine word frequencies from different sources in a way that's designed\nto minimize the impact of outliers. The method reminds me of the scoring system\nin Olympic figure skating:\n\n- Find the frequency of each word according to each data source.\n- For each word, drop the sources that give it the highest and lowest frequency.\n- Average the remaining frequencies.\n- Rescale the resulting frequency list to add up to 1.\n\n\n## Sources and supported languages\n\nThis data comes from a Luminoso project called [Exquisite Corpus][xc], whose\ngoal is to download good, varied, multilingual corpus data, process it\nappropriately, and combine it into unified resources such as wordfreq.\n\n[xc]: https://github.com/LuminosoInsight/exquisite-corpus\n\nExquisite Corpus compiles 8 different domains of text, some of which themselves\ncome from multiple sources:\n\n- **Wikipedia**, representing encyclopedic text\n- **Subtitles**, from OPUS OpenSubtitles 2018 and SUBTLEX\n- **News**, from NewsCrawl 2014 and GlobalVoices\n- **Books**, from Google Books Ngrams 2012\n- **Web** text, from ParaCrawl, the Leeds Internet Corpus, and the MOKK\n  Hungarian Webcorpus\n- **Twitter**, representing short-form social media\n- **Reddit**, representing potentially longer Internet comments\n- **Miscellaneous** word frequencies: in Chinese, we import a free wordlist\n  that comes with the Jieba word segmenter, whose provenance we don't really\n  know\n\nThe following languages are supported, with reasonable tokenization and at\nleast 3 different sources of word frequencies:\n\n    Language    Code    #  Large?   WP    Subs  News  Books Web   Twit. Redd. Misc.\n    \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n    Arabic      ar      5  Yes    \u2502 Yes   Yes   Yes   -     Yes   Yes   -     -\n    Bengali     bn      3  -      \u2502 Yes   -     Yes   -     -     Yes   -     -\n    Bosnian     bs [1]  3  -      \u2502 Yes   Yes   -     -     -     Yes   -     -\n    Bulgarian   bg      3  -      \u2502 Yes   Yes   -     -     -     Yes   -     -\n    Catalan     ca      4  -      \u2502 Yes   Yes   Yes   -     -     Yes   -     -\n    Chinese     zh [3]  7  Yes    \u2502 Yes   Yes   Yes   Yes   Yes   Yes   -     Jieba\n    Croatian    hr [1]  3         \u2502 Yes   Yes   -     -     -     Yes   -     -\n    Czech       cs      5  Yes    \u2502 Yes   Yes   Yes   -     Yes   Yes   -     -\n    Danish      da      3  -      \u2502 Yes   Yes   -     -     -     Yes   -     -\n    Dutch       nl      5  Yes    \u2502 Yes   Yes   Yes   -     Yes   Yes   -     -\n    English     en      7  Yes    \u2502 Yes   Yes   Yes   Yes   Yes   Yes   Yes   -\n    Finnish     fi      6  Yes    \u2502 Yes   Yes   Yes   -     Yes   Yes   Yes   -\n    French      fr      7  Yes    \u2502 Yes   Yes   Yes   Yes   Yes   Yes   Yes   -\n    German      de      7  Yes    \u2502 Yes   Yes   Yes   Yes   Yes   Yes   Yes   -\n    Greek       el      3  -      \u2502 Yes   Yes   -     -     Yes   -     -     -\n    Hebrew      he      4  -      \u2502 Yes   Yes   -     Yes   -     Yes   -     -\n    Hindi       hi      3  -      \u2502 Yes   -     -     -     -     Yes   Yes   -\n    Hungarian   hu      3  -      \u2502 Yes   Yes   -     -     Yes   -     -     -\n    Indonesian  id      3  -      \u2502 Yes   Yes   -     -     -     Yes   -     -\n    Italian     it      7  Yes    \u2502 Yes   Yes   Yes   Yes   Yes   Yes   Yes   -\n    Japanese    ja      5  Yes    \u2502 Yes   Yes   -     -     Yes   Yes   Yes   -\n    Korean      ko      4  -      \u2502 Yes   Yes   -     -     -     Yes   Yes   -\n    Latvian     lv      4  -      \u2502 Yes   Yes   -     -     Yes   Yes   -     -\n    Macedonian  mk      3  -      \u2502 Yes   Yes   Yes   -     -     -     -     -\n    Malay       ms      3  -      \u2502 Yes   Yes   -     -     -     Yes   -     -\n    Norwegian   nb [2]  4  -      \u2502 Yes   Yes   -     -     -     Yes   Yes   -\n    Persian     fa      3  -      \u2502 Yes   Yes   -     -     -     Yes   -     -\n    Polish      pl      6  Yes    \u2502 Yes   Yes   Yes   -     Yes   Yes   Yes   -\n    Portuguese  pt      5  Yes    \u2502 Yes   Yes   Yes   -     Yes   Yes   -     -\n    Romanian    ro      4  -      \u2502 Yes   Yes   -     -     Yes   Yes   -     -\n    Russian     ru      6  Yes    \u2502 Yes   Yes   Yes   Yes   Yes   Yes   -     -\n    Serbian     sr [1]  3  -      \u2502 Yes   Yes   -     -     -     Yes   -     -\n    Spanish     es      7  Yes    \u2502 Yes   Yes   Yes   Yes   Yes   Yes   Yes   -\n    Swedish     sv      4  -      \u2502 Yes   Yes   -     -     -     Yes   Yes   -\n    Turkish     tr      3  -      \u2502 Yes   Yes   -     -     -     Yes   -     -\n    Ukrainian   uk      4  -      \u2502 Yes   Yes   -     -     -     Yes   Yes   -\n\n[1] Bosnian, Croatian, and Serbian use the same underlying word list, because\nthey share most of their vocabulary and grammar, they were once considered the\nsame language, and language detection cannot distinguish them. This word list\ncan also be accessed with the language code `sh`.\n\n[2] The Norwegian text we have is specifically written in Norwegian Bokm\u00e5l, so\nwe give it the language code 'nb' instead of the vaguer code 'no'. We would use\n'nn' for Nynorsk, but there isn't enough data to include it in wordfreq.\n\n[3] This data represents text written in both Simplified and Traditional\nChinese, with primarily Mandarin Chinese vocabulary. See \"Multi-script\nlanguages\" below.\n\nSome languages provide 'large' wordlists, including words with a Zipf frequency\nbetween 1.0 and 3.0. These are available in 14 languages that are covered by\nenough data sources.\n\n\n## Other functions\n\n`tokenize(text, lang)` splits text in the given language into words, in the same\nway that the words in wordfreq's data were counted in the first place. See\n*Tokenization*.\n\n`top_n_list(lang, n, wordlist='best')` returns the most common *n* words in\nthe list, in descending frequency order.\n\n    >>> from wordfreq import top_n_list\n    >>> top_n_list('en', 10)\n    ['the', 'of', 'to', 'and', 'a', 'in', 'i', 'is', 'that', 'for']\n\n    >>> top_n_list('es', 10)\n    ['de', 'la', 'que', 'el', 'en', 'y', 'a', 'los', 'no', 'se']\n\n`iter_wordlist(lang, wordlist='best')` iterates through all the words in a\nwordlist, in descending frequency order.\n\n`get_frequency_dict(lang, wordlist='best')` returns all the frequencies in\na wordlist as a dictionary, for cases where you'll want to look up a lot of\nwords and don't need the wrapper that `word_frequency` provides.\n\n`available_languages(wordlist='best')` returns a dictionary whose keys are\nlanguage codes, and whose values are the data file that will be loaded to\nprovide the requested wordlist in each language.\n\n`get_language_info(lang)` returns a dictionary of information about how we\npreprocess text in this language, such as what script we expect it to be\nwritten in, which characters we normalize together, and how we tokenize it.\nSee its docstring for more information.\n\n`random_words(lang='en', wordlist='best', nwords=5, bits_per_word=12)`\nreturns a selection of random words, separated by spaces. `bits_per_word=n`\nwill select each random word from 2^n words.\n\nIf you happen to want an easy way to get [a memorable, xkcd-style\npassword][xkcd936] with 60 bits of entropy, this function will almost do the\njob. In this case, you should actually run the similar function\n`random_ascii_words`, limiting the selection to words that can be typed in\nASCII. But maybe you should just use [xkpa][].\n\n[xkcd936]: https://xkcd.com/936/\n[xkpa]: https://github.com/beala/xkcd-password\n\n\n## Tokenization\n\nwordfreq uses the Python package `regex`, which is a more advanced\nimplementation of regular expressions than the standard library, to\nseparate text into tokens that can be counted consistently. `regex`\nproduces tokens that follow the recommendations in [Unicode\nAnnex #29, Text Segmentation][uax29], including the optional rule that\nsplits words between apostrophes and vowels.\n\nThere are exceptions where we change the tokenization to work better\nwith certain languages:\n\n- In Arabic and Hebrew, it additionally normalizes ligatures and removes\n  combining marks.\n\n- In Japanese and Korean, instead of using the regex library, it uses the\n  external library `mecab-python3`. This is an optional dependency of wordfreq,\n  and compiling it requires the `libmecab-dev` system package to be installed.\n\n- In Chinese, it uses the external Python library `jieba`, another optional\n  dependency.\n\n- While the @ sign is usually considered a symbol and not part of a word,\n  wordfreq will allow a word to end with \"@\" or \"@s\". This is one way of\n  writing gender-neutral words in Spanish and Portuguese.\n\n[uax29]: http://unicode.org/reports/tr29/\n\nWhen wordfreq's frequency lists are built in the first place, the words are\ntokenized according to this function.\n\n    >>> from wordfreq import tokenize\n    >>> tokenize('l@s ni\u00f1@s', 'es')\n    ['l@s', 'ni\u00f1@s']\n    >>> zipf_frequency('l@s', 'es')\n    2.8\n\nBecause tokenization in the real world is far from consistent, wordfreq will\nalso try to deal gracefully when you query it with texts that actually break\ninto multiple tokens:\n\n    >>> zipf_frequency('New York', 'en')\n    5.28\n    >>> zipf_frequency('\u5317\u4eac\u5730\u94c1', 'zh')  # \"Beijing Subway\"\n    3.61\n\nThe word frequencies are combined with the half-harmonic-mean function in order\nto provide an estimate of what their combined frequency would be. In Chinese,\nwhere the word breaks must be inferred from the frequency of the resulting\nwords, there is also a penalty to the word frequency for each word break that\nmust be inferred.\n\nThis method of combining word frequencies implicitly assumes that you're asking\nabout words that frequently appear together. It's not multiplying the\nfrequencies, because that would assume they are statistically unrelated. So if\nyou give it an uncommon combination of tokens, it will hugely over-estimate\ntheir frequency:\n\n    >>> zipf_frequency('owl-flavored', 'en')\n    3.2\n\n\n## Multi-script languages\n\nTwo of the languages we support, Serbian and Chinese, are written in multiple\nscripts. To avoid spurious differences in word frequencies, we automatically\ntransliterate the characters in these languages when looking up their words.\n\nSerbian text written in Cyrillic letters is automatically converted to Latin\nletters, using standard Serbian transliteration, when the requested language is\n`sr` or `sh`. If you request the word list as `hr` (Croatian) or `bs`\n(Bosnian), no transliteration will occur.\n\nChinese text is converted internally to a representation we call\n\"Oversimplified Chinese\", where all Traditional Chinese characters are replaced\nwith their Simplified Chinese equivalent, *even if* they would not be written\nthat way in context. This representation lets us use a straightforward mapping\nthat matches both Traditional and Simplified words, unifying their frequencies\nwhen appropriate, and does not appear to create clashes between unrelated words.\n\nEnumerating the Chinese wordlist will produce some unfamiliar words, because\npeople don't actually write in Oversimplified Chinese, and because in\npractice Traditional and Simplified Chinese also have different word usage.\n\n\n## Similar, overlapping, and varying languages\n\nAs much as we would like to give each language its own distinct code and its\nown distinct word list with distinct source data, there aren't actually sharp\nboundaries between languages.\n\nSometimes, it's convenient to pretend that the boundaries between\nlanguages coincide with national borders, following the maxim that \"a language\nis a dialect with an army and a navy\" (Max Weinreich). This gets complicated\nwhen the linguistic situation and the political situation diverge.\nMoreover, some of our data sources rely on language detection, which of course\nhas no idea which country the writer of the text belongs to.\n\nSo we've had to make some arbitrary decisions about how to represent the\nfuzzier language boundaries, such as those within Chinese, Malay, and\nCroatian/Bosnian/Serbian.  See [Language Log][] for some firsthand reports of\nthe mutual intelligibility or unintelligibility of languages.\n\n[Language Log]: http://languagelog.ldc.upenn.edu/nll/?p=12633\n\nSmoothing over our arbitrary decisions is the fact that we use the `langcodes`\nmodule to find the best match for a language code. If you ask for word\nfrequencies in `cmn-Hans` (the fully specific language code for Mandarin in\nSimplified Chinese), you will get the `zh` wordlist, for example.\n\n\n## Additional CJK installation\n\nChinese, Japanese, and Korean have additional external dependencies so that\nthey can be tokenized correctly. Here we'll explain how to set them up,\nin increasing order of difficulty.\n\n\n### Chinese\n\nTo be able to look up word frequencies in Chinese, you need Jieba, a\npure-Python Chinese tokenizer:\n\n    pip3 install jieba\n\n\n### Japanese\n\nWe use MeCab, by Taku Kudo, to tokenize Japanese. To use this in wordfreq, three\nthings need to be installed:\n\n  * The MeCab development library (called `libmecab-dev` on Ubuntu)\n  * The UTF-8 version of the `ipadic` Japanese dictionary\n    (called `mecab-ipadic-utf8` on Ubuntu)\n  * The `mecab-python3` Python interface\n\nTo install these three things on Ubuntu, you can run:\n\n```sh\nsudo apt-get install python3-dev libmecab-dev mecab-ipadic-utf8\npip3 install mecab-python3\n```\n\nIf you choose to install `ipadic` from somewhere else or from its source code,\nbe sure it's configured to use UTF-8. By default it will use EUC-JP, which will\ngive you nonsense results.\n\n\n### Korean\n\nKorean also uses MeCab, with a Korean dictionary package by Yongwoon Lee and\nYungho Yu. This dictionary is not available as an Ubuntu package.\n\nHere's a process you can use to install the Korean dictionary and the other\nMeCab dependencies:\n\n```sh\nsudo apt-get install libmecab-dev mecab-utils\npip3 install mecab-python3\nwget https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/mecab-ko-dic-2.0.1-20150920.tar.gz\ntar xvf mecab-ko-dic-2.0.1-20150920.tar.gz\ncd mecab-ko-dic-2.0.1-20150920\n./autogen.sh\n./configure\nmake\nsudo make install\n```\n\nIf wordfreq cannot find the Japanese or Korean data for MeCab when asked to\ntokenize those languages, it will raise an error and show you the list of\npaths it searched.\n\nSorry that this is difficult. We tried to just package the data files we need\nwith wordfreq, like we do for Chinese, but PyPI would reject the package for\nbeing too large.\n\n\n## License\n\n`wordfreq` is freely redistributable under the MIT license (see\n`MIT-LICENSE.txt`), and it includes data files that may be\nredistributed under a Creative Commons Attribution-ShareAlike 4.0\nlicense (https://creativecommons.org/licenses/by-sa/4.0/).\n\n`wordfreq` contains data extracted from Google Books Ngrams\n(http://books.google.com/ngrams) and Google Books Syntactic Ngrams\n(http://commondatastorage.googleapis.com/books/syntactic-ngrams/index.html).\nThe terms of use of this data are:\n\n    Ngram Viewer graphs and data may be freely used for any purpose, although\n    acknowledgement of Google Books Ngram Viewer as the source, and inclusion\n    of a link to http://books.google.com/ngrams, would be appreciated.\n\n`wordfreq` also contains data derived from the following Creative Commons-licensed\nsources:\n\n- The Leeds Internet Corpus, from the University of Leeds Centre for Translation\n  Studies (http://corpus.leeds.ac.uk/list.html)\n\n- Wikipedia, the free encyclopedia (http://www.wikipedia.org)\n\n- ParaCrawl, a multilingual Web crawl (https://paracrawl.eu)\n\nIt contains data from OPUS OpenSubtitles 2018\n(http://opus.nlpl.eu/OpenSubtitles.php), whose data originates from the\nOpenSubtitles project (http://www.opensubtitles.org/) and may be used with\nattribution to OpenSubtitles.\n\nIt contains data from various SUBTLEX word lists: SUBTLEX-US, SUBTLEX-UK,\nSUBTLEX-CH, SUBTLEX-DE, and SUBTLEX-NL, created by Marc Brysbaert et al.\n(see citations below) and available at\nhttp://crr.ugent.be/programs-data/subtitle-frequencies.\n\nI (Robyn Speer) have obtained permission by e-mail from Marc Brysbaert to\ndistribute these wordlists in wordfreq, to be used for any purpose, not just\nfor academic use, under these conditions:\n\n- Wordfreq and code derived from it must credit the SUBTLEX authors.\n- It must remain clear that SUBTLEX is freely available data.\n\nThese terms are similar to the Creative Commons Attribution-ShareAlike license.\n\nSome additional data was collected by a custom application that watches the\nstreaming Twitter API, in accordance with Twitter's Developer Agreement &\nPolicy. This software gives statistics about words that are commonly used on\nTwitter; it does not display or republish any Twitter content.\n\n\n## Citing wordfreq\n\nIf you use wordfreq in your research, please cite it! We publish the code\nthrough Zenodo so that it can be reliably cited using a DOI. The current\ncitation is:\n\n> Robyn Speer, Joshua Chin, Andrew Lin, Sara Jewett, & Lance Nathan.\n> (2018, October 3). LuminosoInsight/wordfreq: v2.2. Zenodo.\n> https://doi.org/10.5281/zenodo.1443582\n\n\nThe same citation in BibTex format:\n\n```\n@misc{robyn_speer_2018_1443582,\n  author       = {Robyn Speer and\n                  Joshua Chin and\n                  Andrew Lin and\n                  Sara Jewett and\n                  Lance Nathan},\n  title        = {LuminosoInsight/wordfreq: v2.2},\n  month        = oct,\n  year         = 2018,\n  doi          = {10.5281/zenodo.1443582},\n  url          = {https://doi.org/10.5281/zenodo.1443582}\n}\n```\n\n\n## Citations to work that wordfreq is built on\n\n- Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Huck, M., Hokamp, C.,\n  Koehn, P., Logacheva, V., Monz, C., Negri, M., Post, M., Scarton, C.,\n  Specia, L., & Turchi, M. (2015). Findings of the 2015 Workshop on Statistical\n  Machine Translation.\n  http://www.statmt.org/wmt15/results.html\n\n- Brysbaert, M. & New, B. (2009). Moving beyond Kucera and Francis: A Critical\n  Evaluation of Current Word Frequency Norms and the Introduction of a New and\n  Improved Word Frequency Measure for American English. Behavior Research\n  Methods, 41 (4), 977-990.\n  http://sites.google.com/site/borisnew/pub/BrysbaertNew2009.pdf\n\n- Brysbaert, M., Buchmeier, M., Conrad, M., Jacobs, A.M., B\u00f6lte, J., & B\u00f6hl, A.\n  (2011). The word frequency effect: A review of recent developments and\n  implications for the choice of frequency estimates in German. Experimental\n  Psychology, 58, 412-424.\n\n- Cai, Q., & Brysbaert, M. (2010). SUBTLEX-CH: Chinese word and character\n  frequencies based on film subtitles. PLoS One, 5(6), e10729.\n  http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0010729\n\n- Davis, M. (2012). Unicode text segmentation. Unicode Standard Annex, 29.\n  http://unicode.org/reports/tr29/\n\n- Hal\u00e1csy, P., Kornai, A., N\u00e9meth, L., Rung, A., Szakad\u00e1t, I., & Tr\u00f3n, V.\n  (2004). Creating open language resources for Hungarian. In Proceedings of the\n  4th international conference on Language Resources and Evaluation (LREC2004).\n  http://mokk.bme.hu/resources/webcorpus/\n\n- Keuleers, E., Brysbaert, M. & New, B. (2010). SUBTLEX-NL: A new frequency\n  measure for Dutch words based on film subtitles. Behavior Research Methods,\n  42(3), 643-650.\n  http://crr.ugent.be/papers/SUBTLEX-NL_BRM.pdf\n\n- Kudo, T. (2005). Mecab: Yet another part-of-speech and morphological\n  analyzer.\n  http://mecab.sourceforge.net/\n\n- Lin, Y., Michel, J.-B., Aiden, E. L., Orwant, J., Brockman, W., and Petrov,\n  S. (2012). Syntactic annotations for the Google Books Ngram Corpus.\n  Proceedings of the ACL 2012 system demonstrations, 169-174.\n  http://aclweb.org/anthology/P12-3029\n\n- Lison, P. and Tiedemann, J. (2016). OpenSubtitles2016: Extracting Large\n  Parallel Corpora from Movie and TV Subtitles. In Proceedings of the 10th\n  International Conference on Language Resources and Evaluation (LREC 2016).\n  http://stp.lingfil.uu.se/~joerg/paper/opensubs2016.pdf\n\n- ParaCrawl (2018). Provision of Web-Scale Parallel Corpora for Official\n  European Languages. https://paracrawl.eu/\n\n- van Heuven, W. J., Mandera, P., Keuleers, E., & Brysbaert, M. (2014).\n  SUBTLEX-UK: A new and improved word frequency database for British English.\n  The Quarterly Journal of Experimental Psychology, 67(6), 1176-1190.\n  http://www.tandfonline.com/doi/pdf/10.1080/17470218.2013.850521\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://github.com/LuminosoInsight/wordfreq/",
        "keywords": "",
        "license": "MIT",
        "maintainer": "Robyn Speer",
        "maintainer_email": "rspeer@luminoso.com",
        "name": "wordfreq",
        "package_url": "https://pypi.org/project/wordfreq/",
        "platform": "any",
        "project_url": "https://pypi.org/project/wordfreq/",
        "project_urls": {
            "Homepage": "http://github.com/LuminosoInsight/wordfreq/"
        },
        "release_url": "https://pypi.org/project/wordfreq/2.2.1/",
        "requires_dist": [
            "msgpack",
            "langcodes (>=1.4.1)",
            "regex (<=2018.02.21,>=2017.07.11)",
            "jieba; extra == 'jieba'",
            "mecab-python3; extra == 'mecab'"
        ],
        "requires_python": ">=3.3",
        "summary": "wordfreq is a Python library for looking up the frequencies of words in many",
        "version": "2.2.1"
    },
    "last_serial": 4783827,
    "releases": {
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "1de07b2b1d5f15a74f7062a4c9210985",
                    "sha256": "18b0ba05fa2ee379e97e0060edaf75cbebaeb59c4d393626baa09dd2b06acb2a"
                },
                "downloads": -1,
                "filename": "wordfreq-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "1de07b2b1d5f15a74f7062a4c9210985",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7834,
                "upload_time": "2014-01-29T16:32:28",
                "url": "https://files.pythonhosted.org/packages/11/b3/5f0a2b732a350d007fc2613031ea6e337a1ef48b9b436398245774fb8c39/wordfreq-0.3.0.tar.gz"
            }
        ],
        "0.4.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "85bb3245fbf886a3d01a44524d23c958",
                    "sha256": "5865b1552560471b42c44a18416e0a6bc787c7e6a28539520b709fe21762e92d"
                },
                "downloads": -1,
                "filename": "wordfreq-0.4.1.tar.gz",
                "has_sig": false,
                "md5_digest": "85bb3245fbf886a3d01a44524d23c958",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8185,
                "upload_time": "2014-09-08T15:00:18",
                "url": "https://files.pythonhosted.org/packages/a4/4b/f96b68787d9bf8f2d3606b2aaf69eca07ecb73ada7a5494dce64b12f2c70/wordfreq-0.4.1.tar.gz"
            }
        ],
        "0.5.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f7cd87ed36545accebc95b8f5a5b4836",
                    "sha256": "c8de63388c389b86f919011d2fa4634465f6bea104463fe4e563f4874409d5a1"
                },
                "downloads": -1,
                "filename": "wordfreq-0.5.0.tar.gz",
                "has_sig": false,
                "md5_digest": "f7cd87ed36545accebc95b8f5a5b4836",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8324,
                "upload_time": "2015-03-13T16:14:19",
                "url": "https://files.pythonhosted.org/packages/da/79/5303ceec5c58bedf8377c3a38f242b2a41712d5debac186dd714fc0bbd4a/wordfreq-0.5.0.tar.gz"
            }
        ],
        "1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f2f2c7e4581d6156acdd0fb3552cfe44",
                    "sha256": "2f4c4bfcdcd404a5c55fecf38fc674e21243c109e5207cec4bab15a748068400"
                },
                "downloads": -1,
                "filename": "wordfreq-1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "f2f2c7e4581d6156acdd0fb3552cfe44",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5638142,
                "upload_time": "2015-07-29T15:53:40",
                "url": "https://files.pythonhosted.org/packages/41/23/3bf94cc5a2378f4c4608d66f4cc70cc58655bde823e4ccc010d7e54fc343/wordfreq-1.0.tar.gz"
            }
        ],
        "1.0b4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bf08a3f383e6346e1e6b41b8ecacf6b6",
                    "sha256": "e9709ee4fcb8bcdfdb158e0d8a14ce415bac64c105164b821201984d94a402e6"
                },
                "downloads": -1,
                "filename": "wordfreq-1.0b4.tar.gz",
                "has_sig": false,
                "md5_digest": "bf08a3f383e6346e1e6b41b8ecacf6b6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5702665,
                "upload_time": "2015-07-28T16:30:07",
                "url": "https://files.pythonhosted.org/packages/f1/44/7e4a9447917988e4955045b0275ab053acff2b74b722fe2f42a68aef273f/wordfreq-1.0b4.tar.gz"
            }
        ],
        "1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "61ec443bda50acedc9f99a8fa194a553",
                    "sha256": "65fcb7d2f567e18219a8f08146fcb8271d500aba92a001520be2bfef43f2f81f"
                },
                "downloads": -1,
                "filename": "wordfreq-1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "61ec443bda50acedc9f99a8fa194a553",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5556388,
                "upload_time": "2015-08-27T15:58:25",
                "url": "https://files.pythonhosted.org/packages/71/d9/78d6a43af2f65626ab1b7c0b8c69399e9b9150d159da539c655f20ae94c4/wordfreq-1.1.tar.gz"
            }
        ],
        "1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d7d2b75885fc5c3471f8f29b65fdaee6",
                    "sha256": "4671b592f666ae885509991b4fa239fd0990d6b2c8eb5487d1a0dff00a56a2d6"
                },
                "downloads": -1,
                "filename": "wordfreq-1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "d7d2b75885fc5c3471f8f29b65fdaee6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10054191,
                "upload_time": "2015-10-27T17:24:27",
                "url": "https://files.pythonhosted.org/packages/df/d1/24204e30b35f55e71980512d1a8597d0f25123a0474dab2bb1e4e9baa205/wordfreq-1.2.tar.gz"
            }
        ],
        "1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5ca4e4dfc425ecbf1b4b7c57fe900da7",
                    "sha256": "44d098e3c8cd2f3a980bd5f005566129c5ef089224a9451db642511579458822"
                },
                "downloads": -1,
                "filename": "wordfreq-1.3.tar.gz",
                "has_sig": false,
                "md5_digest": "5ca4e4dfc425ecbf1b4b7c57fe900da7",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10052311,
                "upload_time": "2016-01-21T18:41:05",
                "url": "https://files.pythonhosted.org/packages/14/fa/6d7d22a733bc02c8c6db16b5d3805a4970a68aa73e1a881920f52e640aec/wordfreq-1.3.tar.gz"
            }
        ],
        "1.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "328c23ec08947f09816ce762a7cda903",
                    "sha256": "5790b6d4ba86732863dc4486a39f971a0fbfc1451084c167cef3d09721367109"
                },
                "downloads": -1,
                "filename": "wordfreq-1.4.tar.gz",
                "has_sig": false,
                "md5_digest": "328c23ec08947f09816ce762a7cda903",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 23272162,
                "upload_time": "2016-05-19T21:15:54",
                "url": "https://files.pythonhosted.org/packages/ec/09/7faab37e193958a0318c67519fd6ae21bff9b4b97af33cbd5bd0a79ccc25/wordfreq-1.4.tar.gz"
            }
        ],
        "1.4.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a023bf4b873cdeb6c5502715a408e0ca",
                    "sha256": "2f067f40f2515af86cfb51589e3b9b790a5b97cd53d0b74815a270cbd724047c"
                },
                "downloads": -1,
                "filename": "wordfreq-1.4.1.tar.gz",
                "has_sig": false,
                "md5_digest": "a023bf4b873cdeb6c5502715a408e0ca",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 23272507,
                "upload_time": "2016-07-01T22:03:12",
                "url": "https://files.pythonhosted.org/packages/10/c1/2079f8409a4afdb2928b049c9ddfbc5164867bcb4bb310da8442242c8e48/wordfreq-1.4.1.tar.gz"
            }
        ],
        "1.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b4f12da451de62a82fbd65f1f66126a5",
                    "sha256": "13a9257be7ad21e84505d381d46e3c4b159b0c81d8913d55f2f7c21d75dad41e"
                },
                "downloads": -1,
                "filename": "wordfreq-1.5.tar.gz",
                "has_sig": false,
                "md5_digest": "b4f12da451de62a82fbd65f1f66126a5",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 32424007,
                "upload_time": "2016-08-08T22:00:06",
                "url": "https://files.pythonhosted.org/packages/07/5e/d1c1014067dcd54896b4d89d8a3bc6e6e152e6d24462d4d56af5bafb8309/wordfreq-1.5.tar.gz"
            }
        ],
        "1.5.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2141c681ef814a824f0609241444c34a",
                    "sha256": "cee6cfc02f69741b3bac1e3b0cf720e04eaf3d5eea981501c80ce82035190c3f"
                },
                "downloads": -1,
                "filename": "wordfreq-1.5.1.tar.gz",
                "has_sig": false,
                "md5_digest": "2141c681ef814a824f0609241444c34a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 32422888,
                "upload_time": "2016-08-19T18:33:41",
                "url": "https://files.pythonhosted.org/packages/e4/8a/b60b0c145f86d311706253b811903b67de72a1895793e02e76e4270c0f95/wordfreq-1.5.1.tar.gz"
            }
        ],
        "1.6.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d2730bdefb4e5e6144b58f97b15ed6d2",
                    "sha256": "671ed2731025cce00ef3aa651d6f36b9dc2c8afe2d8082448c6f378d0181a167"
                },
                "downloads": -1,
                "filename": "wordfreq-1.6.1.tar.gz",
                "has_sig": false,
                "md5_digest": "d2730bdefb4e5e6144b58f97b15ed6d2",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 30244192,
                "upload_time": "2017-05-16T21:11:30",
                "url": "https://files.pythonhosted.org/packages/0d/f7/ca97a45dc0342b4e2b7fbe05dbf7ce9021ae104d3e3a3ad3880ef1fc0c52/wordfreq-1.6.1.tar.gz"
            }
        ],
        "1.7.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f63f965d4c9fe0929fae2d895d0f6dc5",
                    "sha256": "f317d31aafa2e6e2b64de9841fbea4e7d54d8a31509210e0d78002765e58fb74"
                },
                "downloads": -1,
                "filename": "wordfreq-1.7.0.tar.gz",
                "has_sig": false,
                "md5_digest": "f63f965d4c9fe0929fae2d895d0f6dc5",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 31327991,
                "upload_time": "2018-03-07T20:13:00",
                "url": "https://files.pythonhosted.org/packages/23/51/66c169d5bb725c977024f918e5fbfcaceb24ec9703780c785a3cea1407a7/wordfreq-1.7.0.tar.gz"
            }
        ],
        "2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bd6506af4f33c56fba913bc9a8aea10e",
                    "sha256": "6f29fd58ed949c728ca4694331898fe17cbd9a3beb164206a6fd7cb3e193777d"
                },
                "downloads": -1,
                "filename": "wordfreq-2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "bd6506af4f33c56fba913bc9a8aea10e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 27530739,
                "upload_time": "2018-03-15T18:27:34",
                "url": "https://files.pythonhosted.org/packages/41/93/3cc57bd18d95cf547f8713f9a93a1150124f03acb0fd84d4db0588433b6c/wordfreq-2.0.tar.gz"
            }
        ],
        "2.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6c99826e830a68af64035506c9309329",
                    "sha256": "58b783972d0a0321ee7204cf7b5c1c4c50e28cc8cf580deafaa5909e5082cec5"
                },
                "downloads": -1,
                "filename": "wordfreq-2.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "6c99826e830a68af64035506c9309329",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.3",
                "size": 27531342,
                "upload_time": "2018-05-01T21:12:09",
                "url": "https://files.pythonhosted.org/packages/e5/b0/7c43e521bde6569dfc458447dca8b1f179576c9014b8252b0b4b4ab8a549/wordfreq-2.0.1.tar.gz"
            }
        ],
        "2.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "340d1cedabcc7e46ed9f3a39fb96fdd0",
                    "sha256": "47d51f3b7197eb396009ae3fad910e2661f75ee0e0667710849367dbebc5f042"
                },
                "downloads": -1,
                "filename": "wordfreq-2.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "340d1cedabcc7e46ed9f3a39fb96fdd0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.3",
                "size": 33201599,
                "upload_time": "2018-06-26T15:34:10",
                "url": "https://files.pythonhosted.org/packages/49/e3/4d810f175f9ad65a05fde662071123bbe422d729f7d59a8ec67f2809a868/wordfreq-2.1.0.tar.gz"
            }
        ],
        "2.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c2296d37a6c66ec8c31a80121fc21a6d",
                    "sha256": "29fc02ee145f243ea7facc52c737603bf67c18f3c8c8e0e0d7066037e12539a1"
                },
                "downloads": -1,
                "filename": "wordfreq-2.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "c2296d37a6c66ec8c31a80121fc21a6d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.3",
                "size": 32827924,
                "upload_time": "2018-07-24T23:03:09",
                "url": "https://files.pythonhosted.org/packages/00/2c/fe6a10280dedf5b8b6707bf76822e53e06365bf38ce8d1622e49cf8f4a9d/wordfreq-2.2.0.tar.gz"
            }
        ],
        "2.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "65d8b94c0d1bddb87d3bdf00728f323b",
                    "sha256": "7cc7058859b8ef195110f8dae0764706cf33e4d0e72a7ed6a9f6adf89cb109d5"
                },
                "downloads": -1,
                "filename": "wordfreq-2.2.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "65d8b94c0d1bddb87d3bdf00728f323b",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.3",
                "size": 32816663,
                "upload_time": "2019-02-05T21:01:17",
                "url": "https://files.pythonhosted.org/packages/d0/82/233c39f350ac66c740266dafb348f9e67ba3a5e5dad6a949a2c3715c34f5/wordfreq-2.2.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "912500159b14b7c70ad2d9da7b8ace4f",
                    "sha256": "15e85da56abd458909a514860219b91b16512cadc81aefe76e87141669dd72ca"
                },
                "downloads": -1,
                "filename": "wordfreq-2.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "912500159b14b7c70ad2d9da7b8ace4f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.3",
                "size": 32827827,
                "upload_time": "2019-02-05T21:01:26",
                "url": "https://files.pythonhosted.org/packages/33/e2/4da80daa4084f7748b262b466f85a5d1bea034c1476ea16867e50476f646/wordfreq-2.2.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "65d8b94c0d1bddb87d3bdf00728f323b",
                "sha256": "7cc7058859b8ef195110f8dae0764706cf33e4d0e72a7ed6a9f6adf89cb109d5"
            },
            "downloads": -1,
            "filename": "wordfreq-2.2.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "65d8b94c0d1bddb87d3bdf00728f323b",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.3",
            "size": 32816663,
            "upload_time": "2019-02-05T21:01:17",
            "url": "https://files.pythonhosted.org/packages/d0/82/233c39f350ac66c740266dafb348f9e67ba3a5e5dad6a949a2c3715c34f5/wordfreq-2.2.1-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "912500159b14b7c70ad2d9da7b8ace4f",
                "sha256": "15e85da56abd458909a514860219b91b16512cadc81aefe76e87141669dd72ca"
            },
            "downloads": -1,
            "filename": "wordfreq-2.2.1.tar.gz",
            "has_sig": false,
            "md5_digest": "912500159b14b7c70ad2d9da7b8ace4f",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.3",
            "size": 32827827,
            "upload_time": "2019-02-05T21:01:26",
            "url": "https://files.pythonhosted.org/packages/33/e2/4da80daa4084f7748b262b466f85a5d1bea034c1476ea16867e50476f646/wordfreq-2.2.1.tar.gz"
        }
    ]
}