{
    "info": {
        "author": "Filip Stefanak",
        "author_email": "f.stefanak@rare-technologies.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Environment :: Console",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3.3",
            "Programming Language :: Python :: 3.4",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6",
            "Topic :: Scientific/Engineering :: Artificial Intelligence"
        ],
        "description": "Bounter -- Counter for large datasets\n=====================================\n\n|Build Status|\\ |GitHub release|\\ |Mailing List|\\ |Gitter|\\ |Follow|\n\nBounter is a Python library, written in C, for extremely fast\nprobabilistic counting of item frequencies in massive datasets, using\nonly a small fixed memory footprint.\n\nWhy Bounter?\n------------\n\nBounter lets you count how many times an item appears, similar to\nPython's built-in ``dict`` or ``Counter``:\n\n.. code:: python\n\n    from bounter import bounter\n\n    counts = bounter(size_mb=1024)  # use at most 1 GB of RAM\n    counts.update([u'a', 'few', u'words', u'a', u'few', u'times'])  # count item frequencies\n\n    print(counts[u'few'])  # query the counts\n    2\n\nHowever, unlike ``dict`` or ``Counter``, Bounter can process huge\ncollections where the items would not even fit in RAM. This commonly\nhappens in Machine Learning and NLP, with tasks like **dictionary\nbuilding** or **collocation detection** that need to estimate counts of\nbillions of items (token ngrams) for their statistical scoring and\nsubsequent filtering.\n\nBounter implements approximative algorithms using optimized low-level C\nstructures, to avoid the overhead of Python objects. It lets you specify\nthe maximum amount of RAM you want to use. In the Wikipedia example\nbelow, Bounter uses 31x less memory compared to ``Counter``.\n\nBounter is also marginally faster than the built-in ``dict`` and\n``Counter``, so wherever you can represent your **items as strings**\n(both byte-strings and unicode are fine, and Bounter works in both\nPython2 and Python3), there's no reason not to use Bounter instead\nexcept:\n\nWhen not to use Bounter?\n------------------------\n\nBeware, Bounter is only a probabilistic frequency counter and cannot be\nrelied on for exact counting. (You can't expect a data structure with\nfinite size to hold infinite data.) Example of Bounter failing:\n\n.. code:: python\n\n    from bounter import bounter\n    bounts = bounter(size_mb=1)\n    bounts.update(str(i) for i in range(10000000))\n    bounts['100']\n    0\n\nPlease use ``Counter`` or ``dict`` when such exact counts matter. When they don't matter, like in most NLP and ML applications with huge datasets, Bounter is a very good alternative.\n\nInstallation\n------------\n\nBounter has no dependencies beyond Python >= 2.7 or Python >= 3.3 and a\nC compiler:\n\n.. code:: bash\n\n    pip install bounter  # install from PyPI\n\nOr, if you prefer to install from the `source\ntar.gz <https://pypi.python.org/pypi/bounter>`__:\n\n.. code:: bash\n\n    python setup.py test  # run unit tests\n    python setup.py install\n\nHow does it work?\n-----------------\n\nNo magic, just some clever use of approximative algorithms and solid\nengineering.\n\nIn particular, Bounter implements three different algorithms under the\nhood, depending on what type of \"counting\" you need:\n\n1. **`Cardinality\n   estimation <https://en.wikipedia.org/wiki/Count-distinct_problem>`__:\n   \"How many unique items are there?\"**\n\n   .. code:: python\n\n       from bounter import bounter\n\n       counts = bounter(need_counts=False)\n       counts.update(['a', 'b', 'c', 'a', 'b'])\n\n       print(counts.cardinality())  # cardinality estimation\n       3\n       print(counts.total())  # efficiently accumulates counts across all items\n       5\n\n   This is the simplest use case and needs the least amount of memory, by\n   using the `HyperLogLog\n   algorithm <http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf>`__\n   (built on top of Joshua Andersen's\n   `HLL <https://github.com/ascv/HyperLogLog>`__ code).\n\n2. **Item frequencies: \"How many times did this item appear?\"**\n\n   .. code:: python\n\n       from bounter import bounter\n\n       counts = bounter(need_iteration=False, size_mb=200)\n       counts.update(['a', 'b', 'c', 'a', 'b'])\n       print(counts.total(), counts.cardinality())  # total and cardinality still work\n       (5L, 3L)\n\n       print(counts['a'])  # supports asking for counts of individual items\n       2\n\n   This uses the `Count-min Sketch\n   algorithm <https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch>`__ to\n   estimate item counts efficiently, in a **fixed amount of memory**. See\n   the `API\n   docs <https://github.com/RaRe-Technologies/bounter/blob/master/bounter/bounter.py>`__\n   for full details and parameters.\n\n   As a further optimization, Count-min Sketch optionally support a\n   `logarithmic probabilistic\n   counter <https://en.wikipedia.org/wiki/Approximate_counting_algorithm>`__:\n\n   -  ``bounter(need_iteration=False)``: default option. Exact counter, no\n      probabilistic counting. Occupies 4 bytes (max value 2^32) per bucket.\n   -  ``bounter(need_iteration=False, log_counting=1024)``: an integer\n      counter that occupies 2 bytes. Values up to 2048 are exact; larger\n      values are off by +/- 2%. The maximum representable value is around\n      2^71.\n   -  ``bounter(need_iteration=False, log_counting=8)``: a more aggressive\n      probabilistic counter that fits into just 1 byte. Values up to 8 are\n      exact and larger values can be off by +/- 30%. The maximum\n      representable value is about 2^33.\n\n   Such memory vs. accuracy tradeoffs are sometimes desirable in NLP, where\n   being able to handle very large collections is more important than\n   whether an event occurs exactly 55,482x or 55,519x.\n\n3. **Full item iteration: \"What are the items and their frequencies?\"**\n\n   .. code:: python\n\n       from bounter import bounter\n\n       counts = bounter(size_mb=200)  # default version, unless you specify need_items or need_counts\n       counts.update(['a', 'b', 'c', 'a', 'b'])\n       print(counts.total(), counts.cardinality())  # total and cardinality still work\n       (5L, 3)\n       print(counts['a'])  # individual item frequency still works\n       2\n\n       print(list(counts))  # iterator returns keys, just like Counter\n       [u'b', u'a', u'c']\n       print(list(counts.iteritems()))  # supports iterating over key-count pairs, etc.\n       [(u'b', 2L), (u'a', 2L), (u'c', 1L)]\n\n   Stores the keys (strings) themselves in addition to the total\n   cardinality and individual item frequency (8 bytes). Uses the most\n   memory, but supports the widest range of functionality.\n\n   This option uses a custom C hash table underneath, with optimized string\n   storage. It will remove its low-count objects when nearing the maximum\n   alotted memory, instead of expanding the table.\n\n--------------\n\nFor more details, see the `API\ndocstrings <https://github.com/RaRe-Technologies/bounter/blob/master/bounter/bounter.py>`__.\n\nExample on the English Wikipedia\n--------------------------------\n\nLet's count the frequencies of all bigrams in the English Wikipedia\ncorpus:\n\n.. code:: python\n\n    with smart_open('wikipedia_tokens.txt.gz') as wiki:\n        for line in wiki:\n            words = line.decode().split()\n            bigrams = zip(words, words[1:])\n            counter.update(u' '.join(pair) for pair in bigrams)\n\n    print(counter[u'czech republic'])\n    42099\n\nThe Wikipedia dataset contained 7,661,318 distinct words across\n1,860,927,726 total words, and 179,413,989 distinct bigrams across\n1,857,420,106 total bigrams. Storing them in a naive built-in ``dict``\nwould consume over 31 GB RAM.\n\nTo test the accuracy of Bounter, we automatically extracted\n`collocations <https://en.wikipedia.org/wiki/Collocation>`__ (common\nmulti-word expressions, such as \"New York\", \"network license\", \"Supreme\nCourt\" or \"elementary school\") from these bigram counts.\n\nWe compared the set of collocations extracted from Counter (exact\ncounts, needs lots of memory) vs Bounter (approximate counts, bounded\nmemory) and present the precision and recall here:\n\n+----------------------------------------------+----------+---------+-----------+----------+----------+\n| Algorithm                                    | Time to  | Memory  | Precision |   Recall | F1 score |\n|                                              | build    |         |           |          |          |\n+==============================================+==========+=========+===========+==========+==========+\n| ``Counter`` (built-in)                       | 32m 26s  | 31 GB   | 100%      |   100%   |   100%   |\n+----------------------------------------------+----------+---------+-----------+----------+----------+\n| ``bounter(size_mb=128, need_iteration=False, | 19m 53s  | **128   | 95.02%    |   97.10% |   96.04% |\n| log_counting=8)``                            |          | MB**    |           |          |          |\n+----------------------------------------------+----------+---------+-----------+----------+----------+\n| ``bounter(size_mb=1024)``                    | 17m 54s  | 1 GB    | 100%      | 99.27%   |   99.64% |\n+----------------------------------------------+----------+---------+-----------+----------+----------+\n| ``bounter(size_mb=1024,                      | 19m 58s  | 1 GB    |    99.64% |   100%   |   99.82% |\n| need_iteration=False)``                      |          |         |           |          |          |\n+----------------------------------------------+----------+---------+-----------+----------+----------+\n| ``bounter(size_mb=1024,                      | 20m 05s  | 1 GB    |  **100%** | **100%** | **100%** |\n| need_iteration=False, log_counting=1024)``   |          |         |           |          |          |\n+----------------------------------------------+----------+---------+-----------+----------+----------+\n| ``bounter(size_mb=1024,                      | 19m 59s  | 1 GB    |    97.45% |   97.45% |   97.45% |\n| need_iteration=False, log_counting=8)``      |          |         |           |          |          |\n+----------------------------------------------+----------+---------+-----------+----------+----------+\n| ``bounter(size_mb=4096)``                    | **16m    | 4 GB    | 100%      |   100%   |   100%   |\n|                                              | 21s**    |         |           |          |          |\n+----------------------------------------------+----------+---------+-----------+----------+----------+\n| ``bounter(size_mb=4096,                      | 20m 14s  | 4 GB    | 100%      |   100%   |   100%   |\n| need_iteration=False)``                      |          |         |           |          |          |\n+----------------------------------------------+----------+---------+-----------+----------+----------+\n| ``bounter(size_mb=4096,                      | 20m 14s  | 4 GB    | 100%      |   99.64% |   99.82% |\n| need_iteration=False, log_counting=1024)``   |          |         |           |          |          |\n+----------------------------------------------+----------+---------+-----------+----------+----------+\n\nBounter achieves a perfect F1 score of 100% at 31x less memory (1GB vs\n31GB), compared to a built-in ``Counter`` or ``dict``. It is also 61%\nfaster.\n\nEven with just 128 MB (250x less memory), its F1 score is still 96.04%.\n\nSupport\n=======\n\nUse `Github\nissues <https://github.com/RaRe-Technologies/bounter/issues>`__ to\nreport bugs, and our `mailing\nlist <https://groups.google.com/forum/#!forum/gensim>`__ for general\ndiscussion and feature ideas.\n\n--------------\n\n``Bounter`` is open source software released under the `MIT\nlicense <https://github.com/rare-technologies/bounter/blob/master/LICENSE>`__.\n\nCopyright (c) 2017 `RaRe\nTechnologies <https://rare-technologies.com/>`__\n\n.. |Build Status| image:: https://travis-ci.org/RaRe-Technologies/bounter.svg?branch=master\n   :target: https://travis-ci.org/RaRe-Technologies/bounter\n.. |GitHub release| image:: https://img.shields.io/github/release/rare-technologies/bounter.svg?maxAge=3600\n   :target: https://github.com/RaRe-Technologies/bounter/releases\n.. |Mailing List| image:: https://img.shields.io/badge/-Mailing%20List-lightgrey.svg\n   :target: https://groups.google.com/forum/#!forum/gensim\n.. |Gitter| image:: https://img.shields.io/badge/gitter-join%20chat%20%E2%86%92-09a3d5.svg\n   :target: https://gitter.im/RaRe-Technologies/gensim\n.. |Follow| image:: https://img.shields.io/twitter/follow/gensim_py.svg?style=social&label=Follow\n   :target: https://twitter.com/gensim_py",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "http://pypi.python.org/pypi/bounter",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/RaRe-Technologies/bounter",
        "keywords": "counter,count-min sketch,bounded memory,hyperloglog,approximative counting,cardinality estimation",
        "license": "MIT",
        "maintainer": "RARE Technologies",
        "maintainer_email": "opensource@rare-technologies.com",
        "name": "bounter",
        "package_url": "https://pypi.org/project/bounter/",
        "platform": "any",
        "project_url": "https://pypi.org/project/bounter/",
        "project_urls": {
            "Download": "http://pypi.python.org/pypi/bounter",
            "Homepage": "https://github.com/RaRe-Technologies/bounter"
        },
        "release_url": "https://pypi.org/project/bounter/1.1.0/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Counter for large datasets",
        "version": "1.1.0"
    },
    "last_serial": 5118173,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7decbe9db613f5032df3a294a84e757e",
                    "sha256": "84c4ddd94a7ff5ec43cdeb9976b3b0e391ba0e19c912fe381c8c62216b38739e"
                },
                "downloads": -1,
                "filename": "bounter-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "7decbe9db613f5032df3a294a84e757e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6376,
                "upload_time": "2017-09-01T14:00:49",
                "url": "https://files.pythonhosted.org/packages/d3/20/e817bfeb93b0f0de8e4fcdf67e1c4fb4616b074bb3b7434ff2fbf1c2a447/bounter-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "01a4da1e97d41faf5e6f8ce36ab83b91",
                    "sha256": "672e0c45404a9ffe8b21d63c951f85cb9501325e7c9fa2713672a6843a29493c"
                },
                "downloads": -1,
                "filename": "bounter-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "01a4da1e97d41faf5e6f8ce36ab83b91",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6422,
                "upload_time": "2017-09-01T14:15:26",
                "url": "https://files.pythonhosted.org/packages/d6/60/6eae3d85f2b941d25c805b66b2020a6a9aec3626b58a8138a4fb8492831b/bounter-0.1.1.tar.gz"
            }
        ],
        "1.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "127c257a9a2b3cd4c46f91f641394014",
                    "sha256": "4bdaadf7ee06420d6f434b64dbb318ff7bdb690a7aea4b56c393d6eee868f8e4"
                },
                "downloads": -1,
                "filename": "bounter-1.0.1-cp27-cp27m-win_amd64.whl",
                "has_sig": false,
                "md5_digest": "127c257a9a2b3cd4c46f91f641394014",
                "packagetype": "bdist_wheel",
                "python_version": "cp27",
                "requires_python": null,
                "size": 62756,
                "upload_time": "2017-10-18T10:53:06",
                "url": "https://files.pythonhosted.org/packages/8d/ff/f5ba383bde1bf2f78e537db6224f8bd56aa88e4a251cdf167667e05a9001/bounter-1.0.1-cp27-cp27m-win_amd64.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "bc66121af2657c2dfc1cc4257b860a43",
                    "sha256": "8898eddf14d19a8617c32b11ba33d07382b0cc2174303730496958114ad980fd"
                },
                "downloads": -1,
                "filename": "bounter-1.0.1-cp36-cp36m-win_amd64.whl",
                "has_sig": false,
                "md5_digest": "bc66121af2657c2dfc1cc4257b860a43",
                "packagetype": "bdist_wheel",
                "python_version": "cp36",
                "requires_python": null,
                "size": 63270,
                "upload_time": "2017-10-18T10:53:13",
                "url": "https://files.pythonhosted.org/packages/7f/e8/a03d415b8486cf62d3a2cc0c57f38a34418350d2a63bd9c59bc150581a2f/bounter-1.0.1-cp36-cp36m-win_amd64.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e70e6e1f64ea6635b7a399e83f31103e",
                    "sha256": "07f4051795ab2583f5b6c1cb637510198e58c74026bf030aa42525632dfd0f01"
                },
                "downloads": -1,
                "filename": "bounter-1.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "e70e6e1f64ea6635b7a399e83f31103e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 39327,
                "upload_time": "2017-10-18T09:47:24",
                "url": "https://files.pythonhosted.org/packages/ec/91/e3967852de45667a9ac2271f69b62a94a972d27b0f0935cb58268093f2f3/bounter-1.0.1.tar.gz"
            }
        ],
        "1.0.1rc1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e40c62f47f0f7f5db83a04ca3682e99f",
                    "sha256": "ce76d1538d3c86e8fd55071012565dd1f83f398febb4e593ff075932131cabf8"
                },
                "downloads": -1,
                "filename": "bounter-1.0.1rc1-cp27-cp27m-win_amd64.whl",
                "has_sig": false,
                "md5_digest": "e40c62f47f0f7f5db83a04ca3682e99f",
                "packagetype": "bdist_wheel",
                "python_version": "cp27",
                "requires_python": null,
                "size": 55378,
                "upload_time": "2017-10-17T16:27:13",
                "url": "https://files.pythonhosted.org/packages/e3/4c/7f312329ebe9df0d40532586b6cf91428252ae754ed057fe111fa7bd322a/bounter-1.0.1rc1-cp27-cp27m-win_amd64.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ef8732104a22e553d05bb99779e22993",
                    "sha256": "c2e3513d6a664342dd7001e917e0827a3d80c736e93a60649804d8356e2a228a"
                },
                "downloads": -1,
                "filename": "bounter-1.0.1rc1-cp36-cp36m-win_amd64.whl",
                "has_sig": false,
                "md5_digest": "ef8732104a22e553d05bb99779e22993",
                "packagetype": "bdist_wheel",
                "python_version": "cp36",
                "requires_python": null,
                "size": 55942,
                "upload_time": "2017-10-17T16:26:56",
                "url": "https://files.pythonhosted.org/packages/0f/65/a4f9568205ab6c941046d411b11692a9f4b07a942162c5c67d7d9775b48e/bounter-1.0.1rc1-cp36-cp36m-win_amd64.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "bb829c5a48d08124be7ddb61f2fb616a",
                    "sha256": "cf7d522311222c48fc433b47b01700d526819fb46597462b87eb9f8dbdea8dd9"
                },
                "downloads": -1,
                "filename": "bounter-1.0.1rc1.tar.gz",
                "has_sig": false,
                "md5_digest": "bb829c5a48d08124be7ddb61f2fb616a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 39315,
                "upload_time": "2017-10-17T15:39:09",
                "url": "https://files.pythonhosted.org/packages/00/99/48b40e41cd4d14447baf78437bec9f0fdd015ce3a9b4d6c6526e4794ceb2/bounter-1.0.1rc1.tar.gz"
            }
        ],
        "1.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d59f8aed2f0acee885521d173591deaa",
                    "sha256": "0559d74c9bfa9f9a164eadd527afa21f61090810c8d1636f0f091dbc78f7345e"
                },
                "downloads": -1,
                "filename": "bounter-1.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "d59f8aed2f0acee885521d173591deaa",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 42937,
                "upload_time": "2019-01-17T09:39:52",
                "url": "https://files.pythonhosted.org/packages/c5/17/b10b661f6395629b74c33f4b12be7c0d9206bc877402b4f3ccb5208aa16e/bounter-1.1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "d59f8aed2f0acee885521d173591deaa",
                "sha256": "0559d74c9bfa9f9a164eadd527afa21f61090810c8d1636f0f091dbc78f7345e"
            },
            "downloads": -1,
            "filename": "bounter-1.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "d59f8aed2f0acee885521d173591deaa",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 42937,
            "upload_time": "2019-01-17T09:39:52",
            "url": "https://files.pythonhosted.org/packages/c5/17/b10b661f6395629b74c33f4b12be7c0d9206bc877402b4f3ccb5208aa16e/bounter-1.1.0.tar.gz"
        }
    ]
}