{ "info": { "author": "Michael Dorman and Stephen Tonkin", "author_email": "mjdorma@gmail.com sptonkin@outlook.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Intended Audience :: Other Audience", "License :: OSI Approved :: Apache Software License", "Operating System :: Microsoft :: Windows", "Operating System :: POSIX :: Linux", "Programming Language :: Python", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.2", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: Implementation :: CPython", "Topic :: Security", "Topic :: System :: Monitoring" ], "description": "Introduction to restq \n*********************\n\nWhy restq?\n\nWe wanted to have a simple platform independent solution for managing the\ncoordination and distribution of batched execution across our analysis\nplatforms. restq solved our wants into a system that could:\n\n * segregate execution based on a category or type (realm),\n * manage priorities of job execution (ordered queues),\n * enqueue, check-out, and expiry time based (almost FIFO) dequeuing of jobs\n from a realm.\n * status of jobs remaining against arbitrary tag indices.\n * zero configuration for the concepts talked about above. \n\n\nWhat's in restq:\n\n * An implementation of the execution management system described above. \n * A RESTful web API that exposes complete control over the execution\n management system.\n * A Python client that seamlessly interfaces the RESTful web API.\n * Default configuration configurable through environment variables or\n /etc/restq.conf, ~/.restq.conf\n * A command line interface accessible in the shell through the entry point\n 'restq'. The CLI makes it trivial to kick off a restq server. It also\n implements a set of commands which allow users to enqueue and dequeue commands into\n a realm. This makes it super trivial to deploy scheduled execution jobs\n across a pool of servers. \n\n\nFor additional tips / tricks with this restq feel free to post a question at \nthe github `restq/issues`_ page. \n\n\nProject hosting provided by `github.com`_.\n\n\n|pypi_version| |build_status| |coverage|\n\n\n[mjdorma+restq@gmail.com] \n\n\nInstall and run\n===============\n\nSimply run the following::\n\n > python setup.py install\n\nor `PyPi`_:: \n\n > pip install restq\n\n\nCoding with restq\n=================\n\nA simple example on how ::\n\n > restq web &\n > ipython\n\n In [1]: from restq import Realms\n\n In [2]: realms = Realms()\n\n In [3]: realms.test.\n realms.test.add\n realms.test.bulk_add\n realms.test.bulk_flush\n realms.test.get_job\n realms.test.get_tag_status\n realms.test.get_tagged_jobs\n realms.test.name\n realms.test.pull\n realms.test.remove_job\n realms.test.remove_tagged_jobs\n realms.test.request\n realms.test.requester\n realms.test.set_default_lease_time\n realms.test.set_queue_lease_time\n realms.test.status\n\n In [3]: realms.test.add('job 1', 0, 'do the dishes', tags=['house work'])\n\n In [4]: realms.test.add('job 2', 0, 'cut the grass', tags=['house work'])\n\n In [5]: realms.test.add('job 3', 1, 'fix bugs in restq', tags=['devel'])\n\n In [6]: realms.test.add('job 4', 3, 'document restq', tags=['devel'])\n\n In [7]: realms.test.add('job 5', 0, 'go for walk', tags=['sport'])\n\n In [8]: realms.test.status\n Out[8]: \n {u'queues': {u'0': 4, u'1': 1, u'2': 1, u'3': 1},\n u'total_jobs': 7,\n u'total_tags': 3}\n\n In [9]: jobs = realms.test.pull(count=7)\n\n In [10]: jobs\n Out[10]: \n {u'job 1': [0, u'do the dishes'],\n u'job 2': [0, u'cut the grass'],\n u'job 3': [1, u'fix bugs in restq'],\n u'job 4': [3, u'document restq'],\n u'job 5': [0, u'go for walk'],\n u'job 6': [0, u'go for walk with dog'],\n u'job 7': [2, u'go for bike ride']}\n\n In [11]: realms.test.get_tag_status('house work')\n Out[11]: {u'count': 2}\n\n In [12]: realms.test.get_tagged_jobs('devel')\n Out[12]: \n {u'job 3': {u'data': u'fix bugs in restq',\n u'queues': [[1, 82.17003393173218]],\n u'tags': [u'devel']},\n u'job 4': {u'data': u'document restq',\n u'queues': [[3, 82.16989994049072]],\n u'tags': [u'devel']}}\n\n\nUsing restq's CLI\n=================\n\nAdding arguments into the default realm\n---------------------------------------\n\nAdd the argument \"ls -lah\" into the default realm. ::\n\n > restq add \"ls -lah\"\n\nIf we want to refer to a group of commands we can tag a command (even if it\nalready exists). \n\nTag the argument \"ls -lah\" with a label of 'work'. ::\n\n > restq add --tags=work \"ls -lah\"\n\nAdd another argument to the realm, but this time we'll tag it with work and\nfun. ::\n\n > restq add --tags=work,fun pwd\n\nCheckout the status of the realm. ::\n\n > restq status\n Status of realm default:\n Contains 2 tags with 2 jobs\n Defined queues: 0\n\nTime to add pwd to another queue. ::\n \n > restq add --queue=1 pwd\n >\n > restq status\n Status of realm default:\n Contains 2 tags with 2 jobs\n Defined queues: 1, 0\n\n\nPulling (or doing a checkout) of arguments for execution\n--------------------------------------------------------\n\nContinuation from the previous example.\n\nPull and execute a maximum of two arguments from the default realm. After the\ndefault time out, these arguments will be available for checkout once again. ::\n\n > while read i; do eval \"$i\"; done < <(restq pull --count=2)\n drwxr-xr-x 9 mick mick 4.0K Jul 18 08:01 .\n drwxrwxr-x 9 mick mick 4.0K Jul 14 03:07 ..\n drwxrwxr-x 3 mick mick 4.0K Jul 12 00:04 docs\n -rw-rw-r-- 1 mick mick 72 Jul 12 00:04 MANIFEST.in\n -rw-rw-r-- 1 mick mick 3.7K Jul 12 00:04 README.rst\n drwxrwxr-x 2 mick mick 4.0K Jul 17 23:13 restq\n -rw-rw-r-- 1 mick mick 2.1K Jul 17 19:57 setup.py\n drwxrwxr-x 2 mick mick 4.0K Jul 12 00:04 tests\n -rw-rw-r-- 1 mick mick 321 Jul 12 00:04 .travis.yml\n /home/mick/work/restq\n\nThe argument pwd was placed into two queues. The next pull will see pwd being\ndequeued from queue 1. ::\n\n > restq pull\n pwd\n\nLets check the status of the pwd argument since checkout. This shows what\nqueues a specific argument is in, what tags it has, and how long it has been\nsince it was checked out (pulled). ::\n\n > restq status arg pwd\n Status of argument pwd:\n Tagged with: work\n queue id | (s) since dequeue\n 1 | 35.22\n 0 | 454.49\n\nTime to remove pwd from our realm... We're done with this argument and we no\nlonger require it for execution. You will notice that the fun tag no longer\nexists in the realm as it was only attached to pwd. ::\n\n > restq remove arg pwd\n >\n\nThe default lease time for a dequeue of an argument is 600s. After this\nexpiry time, 'ls -lah' will once again be available for dequeue. :: \n\n > restq pull\n ls -lah\n\n\nHow to distribute a shell script for execution \n----------------------------------------------\n\nAdd 'work.sh' script into the default realm. :: \n\n > restq add --file=work.sh \"chmod +x work.sh; ./work.sh\"\n\nNow when this job is dequeued using the restq cli, the path './work.sh' will\nbe written to using the data read from the original 'work.sh' and the\narguments will be written out to stdout. :: \n\n > eval \"`restq pull`\"\n\nThe following is an example of a script that could be deployed across multiple\nmachines to continuously pull and execute jobs that have been added into the\ndefault realm. ::\n\n > while [ 1 ]; do \n > while read i; do eval \"$i\"; done < <(restq pull);\n > sleep 1;\n > done\n\n\nIssues\n======\n\nSource code for *restq* is hosted on `GitHub `_. \nPlease file `bug reports `_\nwith GitHub's issues system.\n\n\nChange log\n==========\n\nversion 0.1.2 (26/08/2013)\n \n * bulk add and removal\n \nversion 0.1.0 (18/07/2013)\n\n * implemented cli controls. \n * realms now using yaml -> breaks compatibility with previous version.\n\nversion 0.0.4 (09/06/2013)\n\n * config and cli shell implementation\n\nversion 0.0.3 (06/06/2013)\n \n * bulk post & stable error handling\n\nversion 0.0.1 (10/04/2013)\n\n * pre life\n\n\nContributions\n=============\n\nContributions to restq:\n\n * [sptonkin@outlook.com]\n\n \n.. _github.com: https://github.com/provoke-vagueness/restq\n.. _PyPi: http://pypi.python.org/pypi/restq\n.. _restq/issues: https://github.com/provoke-vagueness/restq/issues\n.. |coverage| image:: https://coveralls.io/repos/provoke-vagueness/restq/badge.png?branch=master\n :target: https://coveralls.io/r/provoke-vagueness/restq?branch=master\n :alt: Latest PyPI version\n.. |pypi_version| image:: https://pypip.in/v/restq/badge.png\n :target: https://crate.io/packages/restq/\n :alt: Latest PyPI version\n.. |build_status| image:: https://secure.travis-ci.org/provoke-vagueness/restq.png?branch=master\n :target: http://travis-ci.org/#!/provoke-vagueness/restq", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/provoke-vagueness/restq", "keywords": null, "license": "Apache Software Licence", "maintainer": null, "maintainer_email": null, "name": "restq", "package_url": "https://pypi.org/project/restq/", "platform": "cygwin,win,linux", "project_url": "https://pypi.org/project/restq/", "project_urls": { "Download": "UNKNOWN", "Homepage": "https://github.com/provoke-vagueness/restq" }, "release_url": "https://pypi.org/project/restq/0.1.2/", "requires_dist": null, "requires_python": null, "summary": "All-in-memory job queue with RESTful interface.", "version": "0.1.2" }, "last_serial": 849039, "releases": { "0.0.2": [ { "comment_text": "", "digests": { "md5": "8f37831df7dc008ae9f52b9732449fd3", "sha256": "a17dde0be4ece3293efa8ed54ae61708b279d31bd785e2e70efec02db4ce66e4" }, "downloads": -1, "filename": "restq-0.0.2.tar.gz", "has_sig": false, "md5_digest": "8f37831df7dc008ae9f52b9732449fd3", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15961, "upload_time": "2013-06-06T10:09:25", "url": "https://files.pythonhosted.org/packages/90/51/e48ec0977c5ad0d039f9d987483de3c927d18cecf6d875739e427e8b330a/restq-0.0.2.tar.gz" } ], "0.0.3": [ { "comment_text": "", "digests": { "md5": "14187041ed5cdc6789b7c65b97bcbeb5", "sha256": "c6d4e1be0a3a439a5fbc0ec2fb13aa8692ad9e9b2023753d8b347219dbcab140" }, "downloads": -1, "filename": "restq-0.0.3.tar.gz", "has_sig": false, "md5_digest": "14187041ed5cdc6789b7c65b97bcbeb5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 16319, "upload_time": "2013-06-06T12:31:02", "url": "https://files.pythonhosted.org/packages/1c/a6/274e6744b87ee48a93ab95d674d10ef33038a3140d7abba4cdae019c3f36/restq-0.0.3.tar.gz" } ], "0.0.4": [ { "comment_text": "", "digests": { "md5": "61c9bdd717e338ebf7f58dcd80cf589b", "sha256": "f4a574892745d08d6ac261d89628521c3e9e30f09c2efebc88c64e914af7fbce" }, "downloads": -1, "filename": "restq-0.0.4.tar.gz", "has_sig": false, "md5_digest": "61c9bdd717e338ebf7f58dcd80cf589b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18656, "upload_time": "2013-06-09T08:00:43", "url": "https://files.pythonhosted.org/packages/16/86/9f0d734ce11f24f972041ae3def9d244cc5e35f59a485d287a4d2ddf149a/restq-0.0.4.tar.gz" } ], "0.1.0": [ { "comment_text": "", "digests": { "md5": "af740d3928ede1b6b9aae5c8d1d4a3fc", "sha256": "e18bad4ec1a447cdb51c6ae34e59c5547853db5eab8470961721646ddfec23fa" }, "downloads": -1, "filename": "restq-0.1.0.tar.gz", "has_sig": false, "md5_digest": "af740d3928ede1b6b9aae5c8d1d4a3fc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 19043, "upload_time": "2013-07-17T13:10:13", "url": "https://files.pythonhosted.org/packages/c6/f5/b56635a3d088c871f31dd0fcb1bc2ceb99efcd7ef05657adab3715b756a6/restq-0.1.0.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "822eeeecd1b3f73deb9ac53f5abf95a4", "sha256": "1c6d9fe404d0a2233750ab90ea2979c2bd301c5c65fc306575c05ea0bb215789" }, "downloads": -1, "filename": "restq-0.1.1.tar.gz", "has_sig": false, "md5_digest": "822eeeecd1b3f73deb9ac53f5abf95a4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 21156, "upload_time": "2013-07-20T10:24:36", "url": "https://files.pythonhosted.org/packages/82/34/b7c3546710375fb9a1da704535a69cab797c0ec460ab6ccd26a0fe01f692/restq-0.1.1.tar.gz" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "277230b48f2db94b8550b408549fd69d", "sha256": "be3942faa725f0c0ac250c2ebf22f04f6a8e94149373e2f6a8716d623f11b45c" }, "downloads": -1, "filename": "restq-0.1.2.tar.gz", "has_sig": false, "md5_digest": "277230b48f2db94b8550b408549fd69d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 21769, "upload_time": "2013-08-26T10:53:57", "url": "https://files.pythonhosted.org/packages/b7/00/1c4c9c52326eadacc521be1b341229bf90db392402596fe8a0457ab4a42b/restq-0.1.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "277230b48f2db94b8550b408549fd69d", "sha256": "be3942faa725f0c0ac250c2ebf22f04f6a8e94149373e2f6a8716d623f11b45c" }, "downloads": -1, "filename": "restq-0.1.2.tar.gz", "has_sig": false, "md5_digest": "277230b48f2db94b8550b408549fd69d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 21769, "upload_time": "2013-08-26T10:53:57", "url": "https://files.pythonhosted.org/packages/b7/00/1c4c9c52326eadacc521be1b341229bf90db392402596fe8a0457ab4a42b/restq-0.1.2.tar.gz" } ] }