{
    "info": {
        "author": "jmatt",
        "author_email": "jmatt@lsst.org",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "Intended Audience :: Information Technology",
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3.4",
            "Programming Language :: Python :: 3.5",
            "Topic :: Software Development :: Libraries",
            "Topic :: System",
            "Topic :: System :: Archiving :: Backup",
            "Topic :: System :: Systems Administration"
        ],
        "description": "s3s3 0.1.10\nA microservice to move files from S3 APIs (Swift or Ceph) to other S3 APIs.\n\nTo install use `pip install s3s3`\n\n----\n\n####\ns3s3\n####\n\ns3s3 is a microservice to move files from one S3 compliant service to another (Swift, Ceph, AWS).\n\ns3s3 has two services. ``s3s3.scripts.listen`` subscribes and listens to a redis pubsub channel waiting for s3 object names. When notified it calls ``s3s3.api.upload`` and uploads that S3 object from the source to destination(s). ``s3s3.scripts.bucket`` uploads all keys in one s3 bucket to another.\n\n``s3s3.scripts.listen`` is meant to be a real-time daemon.\n\n``s3s3.scripts.bucket`` is meant to run periodically as a cron job. Anything missed by ``s3s3.scripts.listen`` will be handled by this service.\n\nSummary\n=======\n\ns3s3 is required because Ceph and Swift are not feature complete with AWS S3. Many available libraries that work well with AWS S3 do not work with Ceph and Swift.\n\nSome examples: ``key.size`` and ``key.md5`` do not work with Ceph S3 without fetching the contents of the key (or s3 object). Multipart uploads are not reliable with Ceph S3. V4 signatures are not supported by Ceph S3.\n\nConfiguration\n=============\n\ns3s3 must be configured. Configuration can be found in the ``s3s3.config`` module. A source, destination and pubsub section are required. Multiple destination connections are supported but there was minimal testing. To signify a section is a destination connection it must start with ``'dest'``.\n\nExample template: https://github.com/jmatt/s3s3/blob/master/s3s3/s3s3.ini.dist\n\n.. code-block:: ini\n\n   [source]\n        aws_access_key_id = {YOUR_AWS_ACCESS_KEY_ID}\n        aws_secret_access_key = {YOUR_AWS_SECRET_ACCESS_KEY}\n        bucket_name = {YOUR_S3_BUCKET}\n        host = {YOUR_CEPH_S3_ENDPOINT}\n        verify_md5 = False # Verify md5 during s3 operations. \n        is_secure = True # Optional\n        calling_format = OrdinaryCallingFormat # Optional\n   [destination]\n        aws_access_key_id = {YOUR_AWS_ACCESS_KEY_ID}\n        aws_secret_access_key = {YOUR_AWS_SECRET_ACCESS_KEY}\n        bucket_name = {YOUR_S3_BUCKET}\n        verify_md5 = False # Verify md5 during s3 operations. \n        is_secure = True # Optional\n\nInstall\n=======\n\ns3s3 requires python3 and redis. It was tested with python3.4 and python3.5. And redis 3.x.\n\n.. code-block:: bash\n\n   pip install s3s3\n\nClient\n======\n\nThere are two clients. One for each service. ``s3s3.client.ListenClient`` is a client to listen to the ``'backup'`` redis pubsub channel and call ``s3s3.client.on_notify``. ``s3s3.client.BucketClient`` is a client that uses the configuration to provide access to the duplicate_bucket API function.\n\nCommand Line\n============\n\nBoth clients are available as command line scripts.\n\n.. code-block:: bash\n\n   s3s3listen --config /path/to/s3s3.ini\n\nThis will use the configuration to build source and destination boto connections, connect to redis and start listening on the ``backup`` channel. Any messages pushed to that channel will be considered source s3 key names and will attempt to be uploaded to the destination connection(s).\n\n.. code-block:: bash\n\n   s3s3bucket --config /path/pto/s3s3.ini\n\nThis will use the configuration to build source and destination boto connections and duplicate the source bucket in the destination bucket.\n\nAPI\n===\n\nThe API can be found in ``s3s3.api`` module.\n\n.. code-block:: python\n\n   def create_connection(connection_args):\n\nCreates a boto connection from the ``connection_args`` dictionary.\n\n.. code-block:: python\n\n   def upload(source_key, dest_key, verify_md5=False):\n\nUpload the source key (S3 object) to the destination key. If ``verify_md5`` is true then verify md5s match.\n\n.. code-block:: python\n\n   def duplicate_bucket(source_bucket, dest_bucket, verify_md5=False):\n\nDuplicate the source bucket to the destination bucket. If ``verify_md5`` is true then verify md5s match. If the md5 is not available compute it and verify it matches.\n\nDeploy\n======\n\ns3s3 requires redis, python3 and supervisord.\n\n.. code-block:: bash\n\n   mkdir -p /opt/env\n   cd /opt/env\n   virtualenv -p python3 s3s3\n   . /opt/env/s3s3/bin/activate\n   pip install s3s3\n   echo_s3s3_supervisord_conf > /etc/supervisor/conf.d/s3s3.conf\n   echo_s3s3_ini_template > /usr/local/etc/s3s3.ini\n   # Update ini file with your source and destination s3 information.\n   service supervisor restart # or... start if it's not running.\n\nLICENSE\n=======\n\nSee the `LICENSE file </LICENSE>`_.\n\n\n----\n\nFor more information, please see: https://github.com/lsst-sqre/s3s3",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/lsst-sqre/s3s3",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "s3s3",
        "package_url": "https://pypi.org/project/s3s3/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/s3s3/",
        "project_urls": {
            "Homepage": "https://github.com/lsst-sqre/s3s3"
        },
        "release_url": "https://pypi.org/project/s3s3/0.1.10/",
        "requires_dist": [
            "boto (==2.38.0)",
            "redis (==2.10.3)"
        ],
        "requires_python": "",
        "summary": "A microservice to move files from S3 APIs (Swift or Ceph) to other S3 APIs.",
        "version": "0.1.10"
    },
    "last_serial": 1813895,
    "releases": {
        "0.1.10": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f07baab46c51b726d7822d53697ad4e1",
                    "sha256": "703b93e5af151bcfb5c730b9f375daa9729bc743bd91ca3f3882bb207928778d"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.10-py3-none-any.whl",
                "has_sig": true,
                "md5_digest": "f07baab46c51b726d7822d53697ad4e1",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 16022,
                "upload_time": "2015-11-12T22:45:53",
                "url": "https://files.pythonhosted.org/packages/34/62/b3635cab7fbb69efeddff5b9cf0ea1c7004c02ef003b3b60f1926c1093bb/s3s3-0.1.10-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "492f4c36a85bc45ead57b97cfed34467",
                    "sha256": "75eabd703be61379667dc9284e8cf1945d1255f242825e993156807cb18226ac"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.10.tar.gz",
                "has_sig": true,
                "md5_digest": "492f4c36a85bc45ead57b97cfed34467",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11483,
                "upload_time": "2015-11-12T22:45:59",
                "url": "https://files.pythonhosted.org/packages/4b/23/a5b82a4d42474818df77d3e56d27e01a595f77f8df33c581def86c536f14/s3s3-0.1.10.tar.gz"
            }
        ],
        "0.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a993f8dddb1f3b46287eab756e1cee42",
                    "sha256": "685ef64810ecc390d725103966b2d4ccd5434726726cad70bc2a855886065617"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.3-py3-none-any.whl",
                "has_sig": true,
                "md5_digest": "a993f8dddb1f3b46287eab756e1cee42",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10209,
                "upload_time": "2015-10-28T05:51:58",
                "url": "https://files.pythonhosted.org/packages/3f/96/461495594328b40a7d00ca5e1056975171503f9bdb82ac2e468307a9d383/s3s3-0.1.3-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "0a272b52c741cab141afc55d0356704e",
                    "sha256": "bbacf84b0ee559186101a786bc122e1b21b8364aa6719a2a0483f8f3e0883009"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.3.tar.gz",
                "has_sig": true,
                "md5_digest": "0a272b52c741cab141afc55d0356704e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6935,
                "upload_time": "2015-10-28T05:52:18",
                "url": "https://files.pythonhosted.org/packages/fe/cd/9e9303e0a138ea5d8d59ce3956baac0999b7b9e2b953a7ffdc8f32d30836/s3s3-0.1.3.tar.gz"
            }
        ],
        "0.1.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "528088b0ce799a25d474cafe9bf63954",
                    "sha256": "a8295b85d2aadfdda0930ff8497eca8b13e6c9aec9a27bf70cfaeee3b8439232"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.4-py3-none-any.whl",
                "has_sig": true,
                "md5_digest": "528088b0ce799a25d474cafe9bf63954",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10187,
                "upload_time": "2015-10-28T05:57:47",
                "url": "https://files.pythonhosted.org/packages/76/52/bbff11185b129ccecdd0c413438f83da3d9d4f91b5fb2cfd6795ee771533/s3s3-0.1.4-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "595d1c0a60e7cc81a4c626f7670f3059",
                    "sha256": "5fc05c78e3e3d0de9fee6aa61bdb25086c127a5396e20801f6e1b24006dd8638"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.4.tar.gz",
                "has_sig": true,
                "md5_digest": "595d1c0a60e7cc81a4c626f7670f3059",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6918,
                "upload_time": "2015-10-28T05:57:52",
                "url": "https://files.pythonhosted.org/packages/71/e4/291c07196df2bba835e0caf179d549386dad7dc7666a6e1e9527acf93e5b/s3s3-0.1.4.tar.gz"
            }
        ],
        "0.1.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "cf39b2233dcc78683444f079efa9a3aa",
                    "sha256": "b954099f8b524bdc08e693e513a77dfaab88286aa0db13c68a154940dded3889"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.5-py3-none-any.whl",
                "has_sig": true,
                "md5_digest": "cf39b2233dcc78683444f079efa9a3aa",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 13510,
                "upload_time": "2015-10-28T19:11:46",
                "url": "https://files.pythonhosted.org/packages/b7/7e/41fce0f5cb0d15584a76aa6df5eb783658ff37c9ce8a739e0b7c690d6be7/s3s3-0.1.5-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c4e4c14112e87428a9e1dacb9067290f",
                    "sha256": "8acdd83901b7d2ee0da3c5c396197e75ef258b6d1b1bea8d59175db730313fe9"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.5.tar.gz",
                "has_sig": true,
                "md5_digest": "c4e4c14112e87428a9e1dacb9067290f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9752,
                "upload_time": "2015-10-28T19:11:51",
                "url": "https://files.pythonhosted.org/packages/f9/39/f83b7f4770912688e86d8a6ac54f107decb8fd894b12682baee2c319585c/s3s3-0.1.5.tar.gz"
            }
        ],
        "0.1.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3d94cff2b7991b999a4def0a2313bc0d",
                    "sha256": "76f44baf433cde0f3a01150dbf0528d51554433cbc1c94c36aae2bf71bb398b3"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.6-py3-none-any.whl",
                "has_sig": true,
                "md5_digest": "3d94cff2b7991b999a4def0a2313bc0d",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 15352,
                "upload_time": "2015-10-28T21:53:42",
                "url": "https://files.pythonhosted.org/packages/ac/bf/cd3bbc7e0300529e99da59e0295a1604eafca2948c7bd3f24ce8733d94fe/s3s3-0.1.6-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "1db9a6c2369eb29efc63cf0b40382be8",
                    "sha256": "2ceab0df862ef67391742bb5119521ec86250692a2baabfaed3049e6f5aa292d"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.6.tar.gz",
                "has_sig": true,
                "md5_digest": "1db9a6c2369eb29efc63cf0b40382be8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10536,
                "upload_time": "2015-10-28T21:53:48",
                "url": "https://files.pythonhosted.org/packages/47/41/f46b59e9d2e46a49dbdec94a51234fd81bafc783ab6f8620018314b50e99/s3s3-0.1.6.tar.gz"
            }
        ],
        "0.1.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c8d6da9889c51ac8529b94d1958a78d2",
                    "sha256": "040b1cc91e5dead38f55788592eed08428b1b813847fbf2f15db95c3f90b5620"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.7-py3-none-any.whl",
                "has_sig": true,
                "md5_digest": "c8d6da9889c51ac8529b94d1958a78d2",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 15378,
                "upload_time": "2015-10-29T00:04:57",
                "url": "https://files.pythonhosted.org/packages/41/6d/22e5c9a5b17c627a4bc37c11ccda9fe65d125e1ae4f3f5fa0d712f12ea59/s3s3-0.1.7-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "3a18c9c05868c43f8e8d9b1979d7236b",
                    "sha256": "bebafc9da19eec985cf4395b1e752ca715f70b4611dfdf11f68afb51f7b4e4ee"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.7.tar.gz",
                "has_sig": true,
                "md5_digest": "3a18c9c05868c43f8e8d9b1979d7236b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10986,
                "upload_time": "2015-10-29T00:05:11",
                "url": "https://files.pythonhosted.org/packages/13/87/249350b9945d7507137678a847d23aa79c395f75ee7bd5e91709a77a2db3/s3s3-0.1.7.tar.gz"
            }
        ],
        "0.1.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9bf6d1cdd2efd18979a8229929808e12",
                    "sha256": "e49f128a8f0b0dec81bfe2f78b0891873c6e3b0b8dedf695e191012ed3a8fd75"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.8-py3-none-any.whl",
                "has_sig": true,
                "md5_digest": "9bf6d1cdd2efd18979a8229929808e12",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 15658,
                "upload_time": "2015-10-29T01:17:35",
                "url": "https://files.pythonhosted.org/packages/70/9a/81d296b618c647ef7e32fdec30cdee2c81773c99f9c4c74e46c41dfc5064/s3s3-0.1.8-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c99dd0b06cd6a42972f273b5ad245dd9",
                    "sha256": "af3a74b31904b690f7cee2aef194028e8ad00f23fc87b987e795ee34c3ded6ca"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.8.tar.gz",
                "has_sig": true,
                "md5_digest": "c99dd0b06cd6a42972f273b5ad245dd9",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11117,
                "upload_time": "2015-10-29T01:17:41",
                "url": "https://files.pythonhosted.org/packages/5b/d5/0b1d0512cf1cf93aaba59cfa773a14eb35f989a39b00af7128ad3bebde5e/s3s3-0.1.8.tar.gz"
            }
        ],
        "0.1.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b4ccf54a141ac627887ff2905a8599d4",
                    "sha256": "f8c250f4644f83c87afea258b8938d97d8f6a08739984a20954eafbbde742fc1"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.9-py3-none-any.whl",
                "has_sig": true,
                "md5_digest": "b4ccf54a141ac627887ff2905a8599d4",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 15809,
                "upload_time": "2015-10-29T09:59:37",
                "url": "https://files.pythonhosted.org/packages/57/b9/11b0987bd2c1d38f31486cfd90061bf88d2c86ff984fea32d7c35474d867/s3s3-0.1.9-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e9a589aa4a2768ccfb8390257c45fb30",
                    "sha256": "956219e2124100928e4155442b4d4586eb9bf64f79bc849d491ecc310eff5cc8"
                },
                "downloads": -1,
                "filename": "s3s3-0.1.9.tar.gz",
                "has_sig": true,
                "md5_digest": "e9a589aa4a2768ccfb8390257c45fb30",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11294,
                "upload_time": "2015-10-29T09:59:44",
                "url": "https://files.pythonhosted.org/packages/44/05/8ee1f635f0a8ca198453234139cb1ad7bf98cb6fb47349bcbb90354fde3a/s3s3-0.1.9.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "f07baab46c51b726d7822d53697ad4e1",
                "sha256": "703b93e5af151bcfb5c730b9f375daa9729bc743bd91ca3f3882bb207928778d"
            },
            "downloads": -1,
            "filename": "s3s3-0.1.10-py3-none-any.whl",
            "has_sig": true,
            "md5_digest": "f07baab46c51b726d7822d53697ad4e1",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 16022,
            "upload_time": "2015-11-12T22:45:53",
            "url": "https://files.pythonhosted.org/packages/34/62/b3635cab7fbb69efeddff5b9cf0ea1c7004c02ef003b3b60f1926c1093bb/s3s3-0.1.10-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "492f4c36a85bc45ead57b97cfed34467",
                "sha256": "75eabd703be61379667dc9284e8cf1945d1255f242825e993156807cb18226ac"
            },
            "downloads": -1,
            "filename": "s3s3-0.1.10.tar.gz",
            "has_sig": true,
            "md5_digest": "492f4c36a85bc45ead57b97cfed34467",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 11483,
            "upload_time": "2015-11-12T22:45:59",
            "url": "https://files.pythonhosted.org/packages/4b/23/a5b82a4d42474818df77d3e56d27e01a595f77f8df33c581def86c536f14/s3s3-0.1.10.tar.gz"
        }
    ]
}