{ "info": { "author": "James Arthur", "author_email": "username: thruflo, domain: gmail.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Environment :: Web Environment", "Intended Audience :: Developers", "License :: Public Domain", "Programming Language :: Python :: 2.7" ], "description": "\n# nTorque - web hook task queue\n\n[nTorque][] is a [task](http://www.celeryproject.org)\n[queue](https://github.com/resque/resque) service that uses [web hooks][].\nIt is free, open source software [released into the public domain][] that\nyou can use from any programming language (that speaks HTTP) to queue\nup and reliably execute idempotent tasks. For example, in Python:\n\n```python\nimport os\nimport requests\n\nparams = {'url': 'http://example.com/myhooks/send_email'}\ndata = {'user_id': 1234}\n\nendpoint = os.environ.get('NTORQUE_URL')\nresponse = requests.post(endpoint, data=data, params=params)\n```\n\n[nTorque]: http://ntorque.com\n[web hooks]: http://timothyfitz.com/2009/02/09/what-webhooks-are-and-why-you-should-care/\n[released into the public domain]: http://unlicense.org/UNLICENSE\n\n\n## Rationale\n\nnTorque is designed to be a good solution when you need more reliability than\nfire-and-forget but you don't need an [AMPQ][] / [ESB][] sledgehammer to crack\nyour \"do this later\" nut.\n\nBecause it uses web hooks, you can:\n\n1. use it from (and to integrate) applications written in any language\n1. use DNS / web server load balancing to distribute tasks\n1. bootstrap your task execution environment the way you bootstrap a web\n application -- i.e.: once at startup, re-using your web app's configuration\n and middleware\n\n[AMPQ]: http://www.rabbitmq.com\n[ESB]: http://en.wikipedia.org/wiki/Enterprise_service_bus\n\n\n## Functionality\n\nnTorque provides the following endpoints:\n\n* `POST /` to enqueue a task\n* `GET /tasks/:id` to view task status\n\nAnd the following features:\n\n* persistent task storage\n* non-blocking, concurrent task execution\n* HTTPS and redirect support\n* configurable (linear or exponential) backoff\n\n\n## Implementation\n\nnTorque is a Python application comprising of a web application and one or more\nworker processes. These use a [PostgreSQL][] database to persist tasks and a\n[Redis][] database as a notification channel.\n\n
+------+  |  +--------+    +--------+    +--------+  |\n|POST /|     |Frontend|    |Web app |    |Postgres|\n|------|  |  |--------|    |--------|    |--------|  |\n|- url |+- ->|- auth  |+-->|- store |+-->|- tasks |\n|- data|  |  |- rate  |    |- notify|    |        |  |\n|      |     |  limits|    |        |    |        |\n+------+  |  +--------+    +--------+    +--------+\n                               +           ^    +    |\n          |                    |           |   url\n                             rpush        get  data  |\n          |                    |           |    |\n                               v           +    v    |\n          |                 +--------+   +--------+     +---------+\n                            |Redis   |   |Worker  |  |  |Web hook |\n          |                 +--------+   |--------|     |---------|\n                               |         |- POST  |+-|->|- perform|\n          | nTorque            +-blpop-> |  data  |     |  task   |\n                                         +--------+  |  +---------+\n
\n\nIn the event of a response with status code:\n\n* 200 or 201: the task is marked as successfully completed\n* 202 - 499: the task is marked as failed and is not retried\n* 500 (or network error): the task is retried\n\n[Hack here][] if you'd like a different strategy.\n\n[Hack here]: https://github.com/thruflo/ntorque/blob/master/src/ntorque/work/perform.py#L133\n\n## Algorithm\n\nThe real crux of nTorque is a trade-off between request timeout and retry delay.\nIt's worth understanding this before deploying -- and how to simply mitigate\nit by a) specifying an appropriate default timeout and b) overriding this as\nnecessary on a task by task basis.\n\nLike [RQ][] and [Resque][], nTorque uses Redis as a push messaging channel. A\nrequest comes in, a notification is `rpush`d onto a channel and `blpop`d off.\nThis means that tasks are executed immediately, with a nice evented / push\nnotification pattern.\n\nUnlike [RQ][] and [Resque][], nTorque doesn't trust Redis as a persistence layer.\nInstead, it relies on good-old-fashioned PostgreSQL: the first thing nTorque does\nwhen a new task arrives is write it to disk. It then notifies a consumer process\nusing Redis [BLPOP][]. The consumer then reads the data from disk and performs\nthe task by making an HTTP request to its url.\n\nIn most cases, this request will succeed, the task will be marked as completed\nand no more needs to be done. However, this won't happen *every time*, e.g.: when\nthere's a network error or the webhook server is temporarily down. Because there\nare edge case failure scenarios where the web hook response is unreliable, nTorque\nrefuses to rely on it as the source of truth™ about a task's status. Instead,\nthe single source of truth is the PostgreSQL database.\n\nThis is achieved by automatically setting a task to retry every time it's read\n(\"acquired\") from the database. Specifically, the query that reads the task data\nis performed within a transaction that also updates the task's due date and retry\ncount. This means that in any failure scenario, nTorque can always just be restarted\n(potentially on a new server as long as it connects to the same database) and you\ncan be sure that tasks will be performed at least once no matter where they were\nin the pipeline when whatever it was fell over.\n\nIncidentally, tasks due to be retried are picked up by a background process that\npolls the database every `NTORQUE_REQUEUE_INTERVAL` seconds.\n\nMore importantly, and where this description has been heading, is the relation\nbetween the due date of the task as it lies, gloriously in repose, and the\ntimeout of the web hook call. For there is one thing we don't want to do, and\nthat is keep retrying tasks before they've had a chance to complete.\n\nIn order to prevent this behaviour, we impose a simple constraint:\n\n> **The due date set when the task is transactionally read and incremented must\n be longer than the web hook timeout.**\n\nThis means that, in the worst case (when a web hook request does timeout or\nfail to respond), you must wait for the full timeout duration before your task\nis retried. So whilst you may naturally want to set a relatively high timeout\nfor long running tasks, you may want to keep it shorter for simper tasks like\nsending your new user's welcome or reset password email: so that they're\nretried faster.\n\nThe good news is that, in addition to the global `NTORQUE_DEFAULT_TIMEOUT`\nconfiguration variable, you can set an appropriate timeout for different tasks\nusing the [`timeout` query parameter](#usage-api/post).\n\nSimple -- once you know how the system works.\n\n[BLPOP]: http://redis.io/commands/blpop\n[Gevent]: http://www.gevent.org\n[PostgreSQL]: http://www.postgresql.org\n[Redis]: http://redis.io\n[Resque]: https://github.com/resque/resque\n[RQ]: http://python-rq.org/\n\n\n## Installation\n\nClone the repo, install the Python app using:\n\n bash pip_install.sh\n\nYou need Redis and Postgres running. If necessary, create the database:\n\n createdb -T template0 -E UTF8 ntorque\n\nIf you like, install Foreman, to run the multiple processes, using:\n\n bundle install\n\nRun the migrations:\n\n foreman run alembic upgrade head\n\nBootstrap an app (if you'd like to authenticate access with an API key):\n\n foreman run python alembic/scripts/create_application.py --name YOURAPP\n\nYou should then be able to:\n\n foreman start\n\nAlternatively, skip the Foreman stuff and run the commands listed in `Processes`\nmanually / using a Docker / Chef / init.d wrapper. Or push to Heroku, run the\nmigrations and it should just work.\n\n\n## Configuration\n\nAlgorithm / Behaviour:\n\n* `NTORQUE_BACKOFF`: `exponential` (default) or `linear`\n* `NTORQUE_CLEANUP_AFTER_DAYS`: how many days to leave tasks in the db for, defaults\n to `7`\n* `NTORQUE_DEFAULT_TIMEOUT`: how long, in seconds, to wait before treating a web\n hook request as having failed -- defaults to `60` see the algorithm section\n above for details\n* `NTORQUE_MIN_DUE_DELAY`: minimum delay before retrying -- don't set any lower\n than `2`\n* `NTORQUE_MAX_DUE_DELAY`: maximum retry delay -- defaults to `7200` but you\n should make sure its longer than `NTORQUE_DEFAULT_TIMEOUT`\n* `NTORQUE_MAX_RETRIES`: how many attempts before giving up on a task -- defaults\n to `36`\n* `NTORQUE_REQUEUE_INTERVAL`: how often, in seconds, to poll the database for\n tasks to requeue -- defaults to 5\n* `NTORQUE_TRANSIENT_REQUEST_ERRORS`: 4xx errors which ntorque should retry -- defaults to '408,423,429,449'\n\nDeployment:\n\n* `NTORQUE_AUTHENTICATE`: whether to require authentication; defaults to `True`\n -- see authentication section in Usage below\n* `NTORQUE_ENABLE_HSTS`: set this to `True` if you're using [HSTS][]\n* `HSTS_PROTOCOL_HEADER`: set this to, e.g.: `X-Forwarded-Proto` if you're running\n behind an https proxy frontend (see [pyramid_hsts][] for more details)\n* `MODE`: if set to `development` this will run [Gunicorn][] in watch mode (so the app\n server restarts when a Python file changes) and will raise HTTP exceptions in the\n API views (rather than returning them). If set to `production` it will run Gunicorn\n behind a [newrelic][] client. If this isn't quite what you want then either don't\n set it or set it to any other string (or hack the `run.sh` and / or `gunicorn.py`\n scripts)\n\nRedis:\n\n* `NTORQUE_REDIS_CHANNEL`: name of your Redis list used as a notification channel;\n defaults to `ntorque`\n* `REDIS_URL`, etc.: see [pyramid_redis][] for details on how to configure your\n Redis connection\n\nDatabase:\n\n* `DATABASE_URL`, defaults to `postgresql:///ntorque`\n* `SQLALCHEMY_MAX_OVERFLOW`, `SQLALCHEMY_POOL_CLASS`, `SQLALCHEMY_POOL_SIZE` and\n `SQLALCHEMY_POOL_RECYCLE` -- see the SQLAlchemy docs on [engine configuration][]\n and [pyramid_basemodel][] for more information; if you don't provide these\n then SQLAlchemy will use sensible defaults, also note that if you're using\n [pgbouncer][] you should set `SQLALCHEMY_POOL_CLASS=sqlalchemy.pool.NullPool`\n\n[engine configuration]: http://docs.sqlalchemy.org/en/rel_0_9/core/engines.html\n[gunicorn]: http://gunicorn.org\n[hsts]: http://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security\n[newrelic]: https://addons.heroku.com/newrelic\n[pgbouncer]: https://wiki.postgresql.org/wiki/PgBouncer\n[pyramid_basemodel]: https://github.com/thruflo/pyramid_basemodel\n[pyramid_hsts]: https://github.com/thruflo/pyramid_hsts\n[pyramid_redis]: https://github.com/thruflo/pyramid_redis\n\n## Usage / API\n\n### Authentication\n\nIf you set `NTORQUE_AUTHENTICATE` to `True` then you need to create at least one\napplication (e.g.: using the `alembic/scripts/create_application.py` script) and\nprovide its api key in the `NTORQUE_API_KEY` header when enqueuing a task.\n\n### `POST /`\n\nTo enqueue a task, make a POST request to the root path of your nTorque\ninstallation.\n\n**Required**:\n\n* a `url` query parameter; this is the url to your web hook that you want nTorque\n to call to perform your task\n\n**Optional**:\n\n* a `method` query parameter; which http method to use when calling the webhook --\n the default is POST, but you can alternatively specify DELETE, PUT or PATCH.\n* a `timeout` query parameter; how long, in seconds, to wait before treating the\n web hook call as having timed out -- see the Algorithm section above for context\n\n**Data**:\n\nThis aside, you can pass through any POST data, encoded as any content type you\nlike. The data, content type and character encoding will be passed on in the POST\n(or DELETE, PUT or PATCH) request to your web hook.\n\n**Headers**:\n\nAside from the content type, length and charset headers, derived from your\nrequest, you can specify headers to pass through to your web hook, by prefixing\nthe header name with `NTORQUE-PASSTHROUGH-`. So, for example, to pass through\na `FOO: Bar` header, you would provide `NTORQUE-PASSTHROUGH-FOO: Bar` in your\nrequest headers.\n\n**Response**:\n\nYou should receive a 201 response with the url to the task in the `Location`\nheader.\n\n### `GET /task/:id`\n\nReturns a JSON data dict with status information about a task.\n\n#### `POST /task/:id/push`\n\nPushes a task onto the redis notification channel to be consumed, aquired and\nperformed. You should *not* normally need to use this. It's exposed as an\noptimisation for [hybrid][] integrations.\n\n[hybrid]: https://github.com/thruflo/ntorque/blob/master/src/ntorque/client.py#L141\n\n\n## Pro-Tips\n\nnTorque is a system for reliably calling web hook task handlers: not for\nimplementing them. You are responsible for implementing and exposing your own\nweb hooks. In most languages and frameworks this is simple, e.g.: in Ruby\nusing [Sinatra][]:\n\n```ruby\npost '/hooks/foo' do\n # your code here\nend\n```\n\nOr in Python using [Flask][]:\n\n```python\n@app.route('/hooks/foo', methods=['POST'])\ndef foo():\n # your code here\n```\n\nKey things to bear in mind are:\n\n[Sinatra]: http://www.sinatrarb.com\n[Flask]: http://flask.pocoo.org\n\n#### Status Code\n\nAfter successfully performing their task, your web hooks are expected to return\nan HTTP response with a `200` or `201` status code. If not, nTorque will keep\nretrying the task.\n\n#### Avoid Timeouts\n\nYour web server must be configured with a high enough timeout to allow tasks\nenough time to complete. If not, you may be responding with an error when tasks\nare actually being performed successfully.\n\nFor example, for a 30 minute timeout with [Apache][] as a proxy:\n\n```text\nTimeout 1800\nProxyTimeout 1800\n```\n\nOr with [Nginx][]:\n\n```text\nsend_timeout 1800;\nproxy_send_timeout 1800;\n```\n\n[Apache]: http://httpd.apache.org\n[Nginx]: http://nginx.org\n\n#### Secure Public Hooks\n\nIf your web hooks are exposed on a public IP, you are likely to want to secure\nthem, e.g.: using HTTPS and an authentication credential like an API key.\n\nIt's also worth noting that you may need to turn off [CSRF validation][].\n\n[CSRF validation]: http://en.wikipedia.org/wiki/Cross-site_request_forgery#Prevention\n\n\n## Support\n\nRaise [bugs / issues on GitHub](https://github.com/thruflo/ntorque/issues).\n", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://ntorque.com", "keywords": null, "license": "http://unlicense.org", "maintainer": null, "maintainer_email": null, "name": "ntorque", "package_url": "https://pypi.org/project/ntorque/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/ntorque/", "project_urls": { "Download": "UNKNOWN", "Homepage": "http://ntorque.com" }, "release_url": "https://pypi.org/project/ntorque/0.4.2/", "requires_dist": null, "requires_python": null, "summary": "Web hook task queue service.", "version": "0.4.2" }, "last_serial": 2384634, "releases": { "0.3": [ { "comment_text": "", "digests": { "md5": "79fd6fd0b9268c70d086d3449bf87670", "sha256": "cfe8515528caf5f2ca85d0bbca137f0a5b7fb3fca1c8b93724adae34214b4d03" }, "downloads": -1, "filename": "ntorque-0.3.tar.gz", "has_sig": false, "md5_digest": "79fd6fd0b9268c70d086d3449bf87670", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 43051, "upload_time": "2015-06-22T16:45:00", "url": "https://files.pythonhosted.org/packages/49/7a/959174cef97a8df7a4185e4f779225e5225437d813aaff374f4ec9b6b5e0/ntorque-0.3.tar.gz" } ], "0.3.2": [ { "comment_text": "", "digests": { "md5": "5136ca2cce9d5272d77d7e7c7078150e", "sha256": "5e596504fb23cb89cab2b59f2c93c90f1c3cc89e6fef7011af88271c04ceaafb" }, "downloads": -1, "filename": "ntorque-0.3.2.tar.gz", "has_sig": false, "md5_digest": "5136ca2cce9d5272d77d7e7c7078150e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 34607, "upload_time": "2015-09-17T17:57:38", "url": "https://files.pythonhosted.org/packages/46/24/4b3cd54cbc0b29063bfa91aceafdf2bb3f4437b3de0208ca772ec3a39a1d/ntorque-0.3.2.tar.gz" } ], "0.3.3": [ { "comment_text": "", "digests": { "md5": "4bf07cdcb6e6d86016860b1d41fd73be", "sha256": "df64ccb077b7695578a9ffac444fa3e2007eceb75cfadd40d118305d73a9f123" }, "downloads": -1, "filename": "ntorque-0.3.3.tar.gz", "has_sig": false, "md5_digest": "4bf07cdcb6e6d86016860b1d41fd73be", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 34650, "upload_time": "2016-05-13T15:29:20", "url": "https://files.pythonhosted.org/packages/28/9b/b3bdc82ef90a3ef0ef343a4cbf8f0c3abe6f4237f9cb772b4592a71f103e/ntorque-0.3.3.tar.gz" } ], "0.3.4": [ { "comment_text": "", "digests": { "md5": "3487849ddd0b45cbd1bcb369ca2cafe2", "sha256": "ee616ff0f96f6b01a66c33953a2ef0caa5ec32e3cf005dda3b08a7c569c42fb3" }, "downloads": -1, "filename": "ntorque-0.3.4.tar.gz", "has_sig": false, "md5_digest": "3487849ddd0b45cbd1bcb369ca2cafe2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 23545, "upload_time": "2016-06-07T17:32:26", "url": "https://files.pythonhosted.org/packages/cd/05/4d3cb0b062c6eafa235724a7475fba738a16c8baece0056b065bdcf3e5bb/ntorque-0.3.4.tar.gz" } ], "0.4.0": [ { "comment_text": "", "digests": { "md5": "7bfba8fdf8d59cf21437f21de968392c", "sha256": "f97ba096181ebd81c13cb024a21867d7fc72b54af71b4613c67b897fea36eadc" }, "downloads": -1, "filename": "ntorque-0.4.0.tar.gz", "has_sig": false, "md5_digest": "7bfba8fdf8d59cf21437f21de968392c", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 35188, "upload_time": "2016-06-09T17:09:21", "url": "https://files.pythonhosted.org/packages/f6/13/1495be3e9c9a6ac88bbf8ea34836e3e9fd57ca0c9ac2477a26086fa69d1f/ntorque-0.4.0.tar.gz" } ], "0.4.1": [ { "comment_text": "", "digests": { "md5": "33c1e574f704bb51ce8cf1bc0efb773f", "sha256": "a56e81bb13de1a408973b99ed0917e44f6054bcb189975629b0ee40e7b57692d" }, "downloads": -1, "filename": "ntorque-0.4.1.tar.gz", "has_sig": false, "md5_digest": "33c1e574f704bb51ce8cf1bc0efb773f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 35218, "upload_time": "2016-09-09T15:28:24", "url": "https://files.pythonhosted.org/packages/99/0c/e7cd5c705510e242e585777d05cf88fcd370a808d86e6da8273af5d98235/ntorque-0.4.1.tar.gz" } ], "0.4.2": [ { "comment_text": "", "digests": { "md5": "aa57cd050ec48340d882941d23d9dd43", "sha256": "09ddad5ec2d41579f699a85843db2cc1598435bb76f84120dc37ec897284647d" }, "downloads": -1, "filename": "ntorque-0.4.2.tar.gz", "has_sig": false, "md5_digest": "aa57cd050ec48340d882941d23d9dd43", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 35251, "upload_time": "2016-10-06T16:03:31", "url": "https://files.pythonhosted.org/packages/fb/18/0920f212e301fa0dc97f32cbe35b3fb08b76830f336ea02d048239b5dbb3/ntorque-0.4.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "aa57cd050ec48340d882941d23d9dd43", "sha256": "09ddad5ec2d41579f699a85843db2cc1598435bb76f84120dc37ec897284647d" }, "downloads": -1, "filename": "ntorque-0.4.2.tar.gz", "has_sig": false, "md5_digest": "aa57cd050ec48340d882941d23d9dd43", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 35251, "upload_time": "2016-10-06T16:03:31", "url": "https://files.pythonhosted.org/packages/fb/18/0920f212e301fa0dc97f32cbe35b3fb08b76830f336ea02d048239b5dbb3/ntorque-0.4.2.tar.gz" } ] }