{ "info": { "author": "John Bailey", "author_email": "dev@brightsilence.com", "bugtrack_url": null, "classifiers": [], "description": "Variable Length Integer Encoding\n================================\n\n|Build Status|\n\nThis is a Python module which is intended to assist with variable-length\nencoding integers (and lists of integers) into more compact\nrepresentations which use less memory.\n\nNotes on Memory Usage\n---------------------\n\nGenerally in Python, integers are stored as\n`long `__\nmeaning that they will use at least 32 bits. When storing many numbers\nwhich do not require 32 bits, this would seem to be significantly\nwasteful; variable length representation should be able to assist in\nsuch cases.\n\nUnfortunately, Python 2 gives us the following\n\n.. code:: python\n\n >>> import sys\n >>> i = 1\n >>> sys.getsizeof(i)\n 12\n >>> b = bytearray([0] * 4)\n >>> sys.getsizeof(b)\n 29\n\nand the situation is no better in Python 3\n\n.. code:: python\n\n >>> import sys\n >>> i = 1\n >>> sys.getsizeof(i)\n 14\n >>> b = bytearray([0] * 4)\n >>> sys.getsizeof(b)\n 33\n\nWhat we can see however is that the Python overhead for bytearray is\nfixed. Increasing the size of the bytearray only increases the memory\nusage by the amount of bytes we've used:\n\n.. code:: python\n\n >>> import sys\n >>> b1 = bytearray([0])\n >>> sys.getsizeof(b1)\n 30\n >>> b10 = bytearray([0] * 10)\n >>> sys.getsizeof(b10)\n 40\n\nSo this means that currently:\n\n- Memory overhead for bytearray objects is higher in Python 3 than\n Python 2\n- Using varint encoding will actually *cost* us memory rather than\n saving us memory\n\nIf we consider arrays of numbers the situation is somewhat better. If we\ntake the example where we want to store ten zeros. A varint encoding\nshould mean that each zero can be stored in a single byte, meaning that\nwe'd end up with a bytearray with 10 elements. So ...\n\n.. code:: python\n\n >>> import sys\n >>> i1 = [0] * 10\n >>> i1\n [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n >>> sys.getsizeof(i1)\n 72\n >>> b1 = bytearray(i1)\n >>> sys.getsizeof(b1)\n 36\n\nMeaning that:\n\n- There is potential for saving memory by utilising varint encoding if\n we are storing arrays of integers\n- The amount of memory we save will depend on the numbers that are\n being stored. The larger the varint representation, the smaller the\n saving, hence the more numbers will need to be stored in order to\n compensate for the bytearray overhead\n- We will incur a processing overhead in order to save this memory.\n e.g.\n- Random access to varints stored in a bytearray would be O(n) rather\n than O(1)\n- We will incur an overhead each time we want to convert to and from\n varint representation\n\nSo why use varint in Python? In the case that we need a compact method\nto store a list of (frequently small) numbers, and we do not generally\nneed random access to the numbers contained.\n\nOne application is during\n`tree-search `__. Typically\nwe will end up with a number of nodes held in memory and not being\naccessed while other nodes in the tree are being processed. If we want\nto store a state associated with each node (e.g. pieces on a chess\nboard), then we can represent these as as list of integers and minimise\nthe memory usage by using varint representations.\n\n.. |Build Status| image:: https://travis-ci.org/bright-tools/varints.svg?branch=master\n :target: https://travis-ci.org/bright-tools/varints", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://github.com/bright-tools/varints", "keywords": "varint,integer,variable,length", "license": "Apache", "maintainer": null, "maintainer_email": null, "name": "varints", "package_url": "https://pypi.org/project/varints/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/varints/", "project_urls": { "Download": "UNKNOWN", "Homepage": "http://github.com/bright-tools/varints" }, "release_url": "https://pypi.org/project/varints/0.1.3/", "requires_dist": null, "requires_python": null, "summary": "Variable-length encoding of integers", "version": "0.1.3" }, "last_serial": 2636693, "releases": { "0.1.3": [] }, "urls": [] }