{ "info": { "author": "Rob Golding", "author_email": "rob@robgolding.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "License :: OSI Approved :: BSD License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.6", "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: System :: Distributed Computing" ], "description": "# Celery-S3\n\n[![Build Status](https://travis-ci.org/robgolding/celery-s3.svg?branch=master)](https://travis-ci.org/robgolding/celery-s3)\n[![Coverage Status](https://coveralls.io/repos/github/robgolding/celery-s3/badge.svg?branch=master)](https://coveralls.io/github/robgolding/celery-s3?branch=master)\n\nCelery-S3 is a simple S3 result backend for Celery.\n\nIf used in conjunction with the SQS broker, it allows for Celery deployments\nthat use only distributed AWS services -- with no dependency on individual\nmachines within your infrastructure.\n\nThis backend probably isn't suitable for particularly high-traffic Celery\ndeployments, but it works just fine in general -- and imposes no limits on the\nnumber of workers in the pool.\n\n## Installation\n\nInstall via pip:\n\n`pip install celery-s3`\n\nThen configure Celery to use the `S3Backend`:\n\n CELERY_RESULT_BACKEND = 'celery_s3.backends.S3Backend'\n\n CELERY_S3_BACKEND_SETTINGS = {\n 'aws_access_key_id': '',\n 'aws_secret_access_key': '',\n 'bucket': '',\n }\n\n## Configuration\n\nTo use a folder within the specified bucket, set the `base_path` in your\n`CELERY_S3_BACKEND_SETTINGS`:\n\n\n CELERY_S3_BACKEND_SETTINGS = {\n ...\n 'base_path': '/celery/',\n ...\n }\n\nTo use a region other than the default (`us-east-1`), set the `aws_region`\nparameter:\n\n CELERY_S3_BACKEND_SETTINGS = {\n ...\n 'aws_region': 'us-east-1',\n ...\n }\n\nTo use [reduced redundancy storage](https://aws.amazon.com/s3/reduced-redundancy/),\nset the `reduced_redundancy` parameter:\n\n CELERY_S3_BACKEND_SETTINGS = {\n ...\n 'reduced_redundancy': True,\n ...\n }\n\nTo use [server-side encryption](http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html),\nset the `encrypt_key` parameter:\n\n CELERY_S3_BACKEND_SETTINGS = {\n ...\n 'encrypt_key': True,\n ...\n }\n\n## Notes\n\nStoring Celery results with this backend will obviously result in API calls\nbeing made to Amazon S3. For each result, at least one `PUT` request will be\nmade (priced at $0.01 per 1,000 requests at the time of writing). Also, the\ndata contained within the result object will be stored indefinitely, unless\notherwise specified.\n\nTo fetch a result for a task that has already finished, at least two requests\nwill be made (one `HEAD` and one `GET`). If you use Celery's `result.get()` to\nwait for a task to finish, S3 will be polled continuously until the task has\nfinished.\n\nBy default, the poll interval is set to 0.5 seconds, which could result in\na lot of requests (two `HEAD` requests per second until the task has finished,\nthen one `GET` request to fetch the result). If you need to use\n`result.get()`, consider increasing the interval and using a timeout to prevent\npolling forever: `result.get(interval=5, timeout=600)`.\n\nAlso, for tasks whose result you don't need, be sure to use `ignore_result`:\n\n @celery.task(ignore_result=True)\n def process_data(obj):\n obj.do_processing()\n\nOnce task results have been used and are no longer needed, be sure to call\n`result.forget()` to delete the corresponding S3 key. Otherwise, old results\nwill remain forever and contribute to storage costs (storage is priced at\n$0.095 per GB per month at the time of writing).\n\nAlso, the S3 lifecycle can be used to archive or delete old keys after\na certain period of time.", "description_content_type": null, "docs_url": null, "download_url": "https://github.com/robgolding63/celery-s3/downloads", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/robgolding63/celery-s3", "keywords": "", "license": "BSD", "maintainer": "", "maintainer_email": "", "name": "celery-s3", "package_url": "https://pypi.org/project/celery-s3/", "platform": "", "project_url": "https://pypi.org/project/celery-s3/", "project_urls": { "Download": "https://github.com/robgolding63/celery-s3/downloads", "Homepage": "https://github.com/robgolding63/celery-s3" }, "release_url": "https://pypi.org/project/celery-s3/1.0.1/", "requires_dist": null, "requires_python": "", "summary": "An S3 result store backend for Celery", "version": "1.0.1" }, "last_serial": 3356134, "releases": { "0.1": [ { "comment_text": "", "digests": { "md5": "c65e5680cc3208cb1d6f3ca8e90b4d8d", "sha256": "9baf6981585868a6382f98884311a2142761eea4bd205de7f86becaec0eceaaf" }, "downloads": -1, "filename": "celery-s3-0.1.tar.gz", "has_sig": false, "md5_digest": "c65e5680cc3208cb1d6f3ca8e90b4d8d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3234, "upload_time": "2013-02-20T10:11:59", "url": "https://files.pythonhosted.org/packages/4a/f8/ec9d6add9aaae066c317c4f8ae3fe14eadc7c561c661a417b1adaeea17a7/celery-s3-0.1.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "95b81e2e82215f4f775bb7539c84a7d0", "sha256": "37d6843f58c61c18c8e1c511c4028a8182b0e492912b1ecd7c32433dd51a093c" }, "downloads": -1, "filename": "celery-s3-0.1.1.tar.gz", "has_sig": false, "md5_digest": "95b81e2e82215f4f775bb7539c84a7d0", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4133, "upload_time": "2013-03-03T19:22:02", "url": "https://files.pythonhosted.org/packages/f3/64/eb7cfb9880a74b2db110b9ea903f562da41944c81549b3b15def9cd2f864/celery-s3-0.1.1.tar.gz" } ], "1.0.0": [ { "comment_text": "", "digests": { "md5": "03d6a66772b6c5f695a64ad249f54d98", "sha256": "786123e68b2279673632bbf262df39c58e9ce879984cfdf9e011c1ddfea8b674" }, "downloads": -1, "filename": "celery-s3-1.0.0.tar.gz", "has_sig": false, "md5_digest": "03d6a66772b6c5f695a64ad249f54d98", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 5393, "upload_time": "2017-10-04T19:05:04", "url": "https://files.pythonhosted.org/packages/8a/f1/e6465a3e4349513d231e854ef3812e992d19568004617c87a80a8870e3b1/celery-s3-1.0.0.tar.gz" } ], "1.0.1": [ { "comment_text": "", "digests": { "md5": "c18904b81cb422ae9c376ee8473d0a15", "sha256": "3fa225cb2d013bbf2f4066546679c268144c6c640d4cfa028a9c9ce37fca0259" }, "downloads": -1, "filename": "celery-s3-1.0.1.tar.gz", "has_sig": false, "md5_digest": "c18904b81cb422ae9c376ee8473d0a15", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 5534, "upload_time": "2017-11-22T19:19:41", "url": "https://files.pythonhosted.org/packages/30/b8/83c83c8e2d867c38e30e8025cff5a43814979e457a283b928d5a9c06ef7f/celery-s3-1.0.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "c18904b81cb422ae9c376ee8473d0a15", "sha256": "3fa225cb2d013bbf2f4066546679c268144c6c640d4cfa028a9c9ce37fca0259" }, "downloads": -1, "filename": "celery-s3-1.0.1.tar.gz", "has_sig": false, "md5_digest": "c18904b81cb422ae9c376ee8473d0a15", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 5534, "upload_time": "2017-11-22T19:19:41", "url": "https://files.pythonhosted.org/packages/30/b8/83c83c8e2d867c38e30e8025cff5a43814979e457a283b928d5a9c06ef7f/celery-s3-1.0.1.tar.gz" } ] }