{
    "info": {
        "author": "Philippe Muller",
        "author_email": "philippe.muller@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Intended Audience :: Developers",
            "Operating System :: POSIX :: Linux",
            "Programming Language :: Python :: 3.5"
        ],
        "description": "S3 consistency checker\n======================\n\n``aws s3 sync`` is great!\n\nBut if you are truly paranoid about your precious files safety,\nit's always better to double check what was uploaded to S3 before deleting\nthem from your local file system.\n\nThis tool does exactly that:\n\n#. List recursively your local files\n#. Ask S3 for their sizes and ETags\n#. Check this against locally computed ETags\n\n\nUsage\n-----\n\n.. code-block:: console\n\n    $ time s3-consistency-checker /data/foo s3://bucketname/foo\n    2017/10/14 16:30:58.729 INFO Comparing 2093 files from /data/foo with s3://bucketname/foo\n    2017/10/14 17:43:18.222 INFO success=2093 errors=0 files=2093 bytes=1.3TiB\n\n    real    72m20.097s\n    user    55m6.975s\n    sys     778m45.112s\n\n    $ time s3-consistency-checker /data/bar s3://bucketname/baz\n    2017/10/14 18:47:08.620 INFO Comparing 26531 files from /data/bar with s3://bucketname/baz\n    2017/10/14 19:21:48.425 INFO success=26531 errors=0 files=26531 bytes=220.1GiB\n\n    real    34m42.023s\n    user    40m22.292s\n    sys     33m57.729s\n\n    $ time s3-consistency-checker /data/foobar s3://bucketname/foobar\n    2017/10/15 02:11:00.904 INFO Comparing 11224 files from /data/foobar with s3://bucketname/foobar\n    2017/10/15 02:25:18.397 INFO success=11224 errors=0 files=11224 bytes=84.8GiB\n\n    real    14m18.873s\n    user    17m3.899s\n    sys     10m33.841s\n\n\nInternals\n---------\n\nThis tool is designed to process a lot of big files as quickly as possible.\nIt uses the ``split`` command to split big files in chunks,\nstores them in ``/dev/shm``,\nthen computes their checksums using the ``md5sum`` command.\nIt does this in parallel using separate processes and threads to drive them.\n\n\nInstallation\n------------\n\n.. code-block:: console\n\n    $ pip install s3-consistency-checker\n\n\nRequirements\n------------\n\n* Python 3.x\n* boto3\n* coreutils: md5sum, split\n",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/pmuller/s3-consistency-checker",
        "keywords": "s3",
        "license": "Apache License 2.0",
        "maintainer": "",
        "maintainer_email": "",
        "name": "s3-consistency-checker",
        "package_url": "https://pypi.org/project/s3-consistency-checker/",
        "platform": "",
        "project_url": "https://pypi.org/project/s3-consistency-checker/",
        "project_urls": {
            "Homepage": "https://github.com/pmuller/s3-consistency-checker"
        },
        "release_url": "https://pypi.org/project/s3-consistency-checker/1.1.1/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Check consistency of files stored on S3 against local files",
        "version": "1.1.1"
    },
    "last_serial": 3251212,
    "releases": {
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "62bcc0af0bb47e0d084ffe370a963858",
                    "sha256": "a2297d86f6c4257e0b231c90bc8a482e1b41b6b93b0fa8b24e2ae440fd52358e"
                },
                "downloads": -1,
                "filename": "s3-consistency-checker-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "62bcc0af0bb47e0d084ffe370a963858",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6244,
                "upload_time": "2017-10-15T02:46:29",
                "url": "https://files.pythonhosted.org/packages/dc/c6/7e4f086b3c1a8e34e68cb31cd71efdbb8f4404a8a19f083b47405546b41b/s3-consistency-checker-1.0.0.tar.gz"
            }
        ],
        "1.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "cd2a773435000204448550cc38a49780",
                    "sha256": "9b69206192d42edc3cbb8e7aace1c7bc789e87ea9abf68bb11520cc4a2783077"
                },
                "downloads": -1,
                "filename": "s3-consistency-checker-1.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "cd2a773435000204448550cc38a49780",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6755,
                "upload_time": "2017-10-15T08:51:49",
                "url": "https://files.pythonhosted.org/packages/e5/1c/77feaa7cad7be42cc7932e080a1d8fc8558e4dd0281086ce588303036124/s3-consistency-checker-1.1.0.tar.gz"
            }
        ],
        "1.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3b56115dc76f406512aebfc3828fffba",
                    "sha256": "ff5830381aa7d32b730e8e93079d70e299315beeca8b2cd574239539048ecbd0"
                },
                "downloads": -1,
                "filename": "s3-consistency-checker-1.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "3b56115dc76f406512aebfc3828fffba",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6749,
                "upload_time": "2017-10-15T09:10:43",
                "url": "https://files.pythonhosted.org/packages/6c/fc/750f3f8a6d669586ea586fc3b974d4fab4d542a54ad42ecd1ccc9f555b4f/s3-consistency-checker-1.1.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "3b56115dc76f406512aebfc3828fffba",
                "sha256": "ff5830381aa7d32b730e8e93079d70e299315beeca8b2cd574239539048ecbd0"
            },
            "downloads": -1,
            "filename": "s3-consistency-checker-1.1.1.tar.gz",
            "has_sig": false,
            "md5_digest": "3b56115dc76f406512aebfc3828fffba",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 6749,
            "upload_time": "2017-10-15T09:10:43",
            "url": "https://files.pythonhosted.org/packages/6c/fc/750f3f8a6d669586ea586fc3b974d4fab4d542a54ad42ecd1ccc9f555b4f/s3-consistency-checker-1.1.1.tar.gz"
        }
    ]
}