{
    "info": {
        "author": "Jim Kalafut",
        "author_email": "jim@kalafut.net",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python :: 2",
            "Programming Language :: Python :: 2.6",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.3",
            "Programming Language :: Python :: 3.4",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6",
            "Topic :: Software Development",
            "Topic :: Software Development :: Libraries :: Python Modules",
            "Topic :: Utilities"
        ],
        "description": "imohash\n=======\n\nimohash is a fast, constant-time hashing library. It uses file\nsize and sampling to calculate hashes quickly, regardless of file size.\nIt was originally released as a `Go library <https://github.com/kalafut/imohash>`__.\n\n``imosum`` is a sample application to hash files from the command line, similar to\nmd5sum.\n\nInstallation\n------------\n\n``pip install imohash``\n\nUsage\n-----\n\nAs a library:\n\n.. code-block:: python\n\n    from imohash import hashfile\n\n    hashfile('foo.txt')\n    'O\\x9b\\xbd\\xd3[\\x86\\x9dE\\x0e3LI\\x83\\r~\\xa3'\n\n    hashfile('foo.txt', hexdigest=True)\n    'a608658926d8aa86b3db8208ad279bfe'\n\n    # just hash the whole file if smaller then 200000 bytes. Default is 128K\n    hashfile('foo.txt', sample_threshhold=200000)\n    'x86\\x9dE\\x0e3LI\\x83\\r~\\xa3O\\x9b\\xbd\\xd3[E'\n\n    # use samples of 1000 bytes. Default is 16K\n    hashfile('foo.txt', sample_size=1000)\n    'E\\x0e3LI\\x83\\r~\\xa3O\\x9b\\xbd\\xd3[E\\x23\\x25'\n\n    # hash an already opened file\n    f = open('foo.txt')\n    hashfileobject(f)\n    'O\\x9b\\xbd\\xd3[\\x86\\x9dE\\x0e3LI\\x83\\r~\\xa3'\n\n    # hash a file on a remote server\n    import paramiko\n    ssh = paramiko.SSHClient()\n    ssh.connect('host', username='username', password='verysecurepassword')\n    ftp = ssh.open_sftp()\n    hashfileobject(ftp.file('/path/to/remote/file/foo.txt'))\n    'O\\x9b\\xbd\\xd3[\\x86\\x9dE\\x0e3LI\\x83\\r~\\xa3'\n\nOr from the command line:\n\n``imosum *.jpg``\n\nUses\n----\n\nBecause imohash only reads a small portion of a file's data, it is very\nfast and well suited to file synchronization and deduplication,\nespecially over a fairly slow network. A need to manage media (photos\nand video) over Wi-Fi between a NAS and multiple family computers is how\nthe library was born.\n\nIf you just need to check whether two files are the same, and understand\nthe limitations that sampling imposes (see below), imohash may be a good\nfit.\n\nMisuses\n-------\n\nBecause imohash only reads a small portion of a file's data, it is not\nsuitable for:\n\n-  file verification or integrity monitoring\n-  cases where fixed-size files are manipulated\n-  anything cryptographic\n\nDesign\n------\n\n(Note: a more precise description is provided in the `algorithm\ndescription <https://github.com/kalafut/imohash/blob/master/algorithm.md>`__.)\n\nimohash works by hashing small chunks of data from the beginning,\nmiddle and end of a file. It also incorporates the file size into the\nfinal 128-bit hash. This approach is based on a few assumptions which\nwill vary by application. First, file size alone *tends* (1) to be a\npretty good differentiator, especially as file size increases. And when\npeople do things to files (such as editing photos), size tends to\nchange. So size is used directly in the hash, and **any files that have\ndifferent sizes will have different hashes**.\n\nSize is an effective differentiator but isn't sufficient. It can show\nthat two files aren't the same, but to increase confidence that\nlike-size files are the same, a few segments are hashed using\n`murmur3 <https://en.wikipedia.org/wiki/MurmurHash>`__, a fast and\neffective hashing algorithm. By default, 16K chunks from the beginning,\nmiddle and end of the file are used. The ends of files often contain\nmetadata which is more prone to changing without affecting file size.\nThe middle is for good measure. The sample size can be changed for your\napplication.\n\n1 Try ``du -a . | sort -nr | less`` on a sample of your files to check\nthis assertion.\n\nSmall file exemption\n~~~~~~~~~~~~~~~~~~~~\n\nSmall files are more likely to collide on size than large ones. They're\nalso probably more likely to change in subtle ways that sampling will\nmiss (e.g. editing a large text file). For this reason, imohash will\nsimply hash the entire file if it is less than 128K. This parameter is\nalso configurable.\n\nPerformance\n-----------\n\nThe standard hash performance metrics make no sense for imohash since\nit's only reading a limited set of the data. That said, the real-world\nperformance is very good. If you are working with large files and/or a\nslow network, expect huge speedups. (**spoiler**: reading 48K is quicker\nthan reading 500MB.)\n\nName\n----\n\nInspired by `ILS marker\nbeacons <https://en.wikipedia.org/wiki/Marker_beacon>`__.\n\nCredits\n-------\n\n-  The \"sparseFingerprints\" used in\n   `TMSU <https://github.com/oniony/TMSU>`__ gave me some confidence in\n   this approach to hashing.\n-  S\u00e9bastien Paolacci's\n   `murmur3 <https://github.com/spaolacci/murmur3>`__ library does all\n   of the heavy lifting in the Go version.\n-  As does Hajime Senuma's\n   `mmh3 <https://github.com/hajimes/mmh3>`__ library for the Python version.\n\n\n",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/kalafut/py-imohash",
        "keywords": "hash hashing imohash",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "imohash",
        "package_url": "https://pypi.org/project/imohash/",
        "platform": "",
        "project_url": "https://pypi.org/project/imohash/",
        "project_urls": {
            "Homepage": "https://github.com/kalafut/py-imohash"
        },
        "release_url": "https://pypi.org/project/imohash/1.0.4/",
        "requires_dist": [
            "mmh3 (>=2.5.1)",
            "varint (>=1.0.2)"
        ],
        "requires_python": "",
        "summary": "Fast hashing for large files",
        "version": "1.0.4"
    },
    "last_serial": 4114620,
    "releases": {
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "00e1b7646fe4d6b2603468cafb245fbb",
                    "sha256": "4465290c69ae7318723ff1f6600f8cd8b777112c21fa532f78793cf8101fa214"
                },
                "downloads": -1,
                "filename": "imohash-1.0.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "00e1b7646fe4d6b2603468cafb245fbb",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 7939,
                "upload_time": "2017-01-16T23:32:09",
                "url": "https://files.pythonhosted.org/packages/78/31/45d6c0d394750f5c0630217278da672dd77401b5f6407a93c58381b00b6c/imohash-1.0.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d14227d3f58a342e93390e5585280782",
                    "sha256": "944054b4e5078dc15c48c91c762967196e5485a31ccab9c79601f3e94c23722f"
                },
                "downloads": -1,
                "filename": "imohash-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "d14227d3f58a342e93390e5585280782",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5108,
                "upload_time": "2017-01-16T23:32:11",
                "url": "https://files.pythonhosted.org/packages/44/bc/b50b42707f2784c58989e5cc37a56b3b6169815c287096f0b490ae147644/imohash-1.0.0.tar.gz"
            }
        ],
        "1.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "854045d281afcf393fbc48ea2e93e383",
                    "sha256": "1c1f04c21efd4b85039f94f2e2235f9a2c07a7104c5696e7f7f6d858aaa4a866"
                },
                "downloads": -1,
                "filename": "imohash-1.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "854045d281afcf393fbc48ea2e93e383",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5125,
                "upload_time": "2017-09-29T04:10:11",
                "url": "https://files.pythonhosted.org/packages/21/ef/099098c3d2e9b3216342ed7934f39820cdd2115d7211531c4d879a7451ce/imohash-1.0.1.tar.gz"
            }
        ],
        "1.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f7ff185f0267c8e9acbd0b0727e1a976",
                    "sha256": "43475cc5d7e0fd7d8d5c94e30a28a59b2276f20b2825e084103d26c62d47a6a7"
                },
                "downloads": -1,
                "filename": "imohash-1.0.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "f7ff185f0267c8e9acbd0b0727e1a976",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 7954,
                "upload_time": "2017-09-29T04:21:37",
                "url": "https://files.pythonhosted.org/packages/49/0d/5ffc55864437c1b06acba10874e16785d9bc34eb477615c48816ab215826/imohash-1.0.2-py2.py3-none-any.whl"
            }
        ],
        "1.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6455c4fe9d772e03168e4d6567c256da",
                    "sha256": "26add43c891b1d688b6111a5c4ea2ad05ff99ed15716179d923ef39ee9e9cebd"
                },
                "downloads": -1,
                "filename": "imohash-1.0.3-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "6455c4fe9d772e03168e4d6567c256da",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 8376,
                "upload_time": "2018-07-29T22:54:44",
                "url": "https://files.pythonhosted.org/packages/c7/db/6475ff3590178584c3177611e50db6c1b104932ac01a689aa0638f3c9aad/imohash-1.0.3-py2.py3-none-any.whl"
            }
        ],
        "1.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "079f4f71c2d7e08ed62c7b6624d6f4e5",
                    "sha256": "0bd1a1220a63697b81025e7dd343d1a414fee65ff97617bf4c7c22fc93fe6470"
                },
                "downloads": -1,
                "filename": "imohash-1.0.4-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "079f4f71c2d7e08ed62c7b6624d6f4e5",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 8386,
                "upload_time": "2018-07-29T23:04:44",
                "url": "https://files.pythonhosted.org/packages/72/5d/27347eaeedb44f5a5d965a096dd6177109550cc0e5918c3164e4b2bf754c/imohash-1.0.4-py2.py3-none-any.whl"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "079f4f71c2d7e08ed62c7b6624d6f4e5",
                "sha256": "0bd1a1220a63697b81025e7dd343d1a414fee65ff97617bf4c7c22fc93fe6470"
            },
            "downloads": -1,
            "filename": "imohash-1.0.4-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "079f4f71c2d7e08ed62c7b6624d6f4e5",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 8386,
            "upload_time": "2018-07-29T23:04:44",
            "url": "https://files.pythonhosted.org/packages/72/5d/27347eaeedb44f5a5d965a096dd6177109550cc0e5918c3164e4b2bf754c/imohash-1.0.4-py2.py3-none-any.whl"
        }
    ]
}