{ "info": { "author": "Nicolas Rodriguez", "author_email": "nrodriguez@jbox-web.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Environment :: Web Environment", "Framework :: Django", "Intended Audience :: Developers", "Intended Audience :: System Administrators", "Natural Language :: English", "Operating System :: OS Independent", "Programming Language :: Python", "Topic :: Software Development :: Libraries :: Python Modules" ], "description": "django-gearman-jbox\n===================\n\n*django-gearman-jbox* is a convenience wrapper for the [Gearman][Gearman]\n[Python Bindings][python-gearman].\n\nWith django-gearman-jbox, you can code workers as well as clients in a Django project\nwith minimal overhead in your application. Server connections etc. all take\nplace in django-gearman-jbox and don't unnecessarily clog your application code.\n\nThis library is based in large part on Fred Wenzel's [django-gearman] and Jozef \u0160ev\u010d\u00edk's [django-gearman-commands].\n- [django-gearman][django-gearman] for the 'decorator way' to create workers (very cool)\n- [django-gearman-commands][django-gearman-commands] for the 'gearman_server_info' command (very nice)\n\nBut there are some modifications :\n\nWorkers are now launched individually, so you have to pass 2 mandatory parameters to start the worker :\n - the Django App name where workers reside with the `-a` parameter\n - the worker's name with `-n`\n\nThe `-q` parameter is still here and has the same function than in [django-gearman].\n\n- I removed the ability of lauching many wokers at once (`-w` parameter in [django-gearman]).\nI prefer having one process for each worker and Supervisord managing process. (screen is not an option)\n- I also added a signal handler to catch SIGTERM signals send by Supervisord and SIGINT when you worker is attached to the console.\nThis gives you the possibility of executing code just before the worker terminates.\n(See `django_gearman_jbox\\management\\commands\\gearman_worker.py`, line 116)\n\n[Gearman]: http://gearman.org\n[python-gearman]: http://github.com/samuel/python-gearman\n[django-gearman]: https://github.com/fwenzel/django-gearman\n[django-gearman-commands]: https://github.com/CodeScaleInc/django-gearman-commands\n\n\nInstallation\n------------\nIt's the same for both the client and worker instances of your django project :\n\n $ pip install django-gearman-jbox\n\nAdd ``django_gearman_jbox`` to the `INSTALLED_APPS` section of `settings.py`.\n\nSpecify the following setting in your local settings.py file:\n\n # One or more gearman servers\n GEARMAN_CLIENT_SERVERS = ['127.0.0.1']\n GEARMAN_WORKER_SERVERS = ['127.0.0.1']\n\nWorkers\n-------\n### Registering workers\nCreate a directory `gearman_workers` in any of your django apps, and define as many\nworkers as you like, one worker per file. Create an empty `__init__.py` so the directory will be\nloaded as a package.\n\nExample :\n\n my_django_app\n |_ models.py\n |_ gearman_workers\n |_ __init__.py\n |_ worker_foo.py\n |_ worker_bar.py\n\n\n### Registering tasks\nIn the worker file, you can define as many tasks as functions as you like.\nThe function must accept a single argument as passed by the caller and must\nreturn the result of the operation, if applicable. (Note : It must accept an argument, even if you don't use it).\n\nMark each of these functions as gearman tasks by decorating them with :\n\n import django_gearman_jbox.decorators.gearman_task\n\n @gearman_task()\n def my_task_function(foo):\n pass\n\n### Task naming\nThe tasks are given a default name of their import path, with the phrase\n`gearman_task` stripped out of them, for readability reasons. You can override\nthe task name by specifying `name` parameter of the decorator. Here's how :\n\n import django_gearman_jbox.decorators.gearman_task\n\n @gearman_task(name='my-task-name')\n def my_task_function(foo):\n pass\n\n### Task parameters\nThe gearman docs specify that the task function can accept only one parameter\n(usually refered to as the ``data`` parameter). Additionally, that parameter\nmay only be a string. Sometimes that may not be enough. What if you would like\nto pass an array or a dict? You would need to serialize and deserialize them.\nFortunately, django-gearman-jbox can take care of this, so that you can spend\nall of your time on coding the actual task.\n\n @gearman_task(name='my-task-name')\n def my_task_function(foo):\n pass\n\n client.submit_job('my-task-name', {'foo': 'becomes', 'this': 'dict'})\n client.submit_job('my-task-name', Decimal(1.0))\n\n### Tasks with more than one parameter\n\nYou can pass as many arguments as you want, of whatever (serializable) type\nyou like. Here's an example job definition :\n\n @gearman_task(name='my-task-name')\n def my_task_function(one, two, three):\n pass\n\nYou can execute this function in two different ways :\n\n client.submit_job('my-task-name', one=1, two=2, three=3)\n client.submit_job('my-task-name', args=[1, 2, 3])\n\nUnfortunately, executing it like this:\n\n client.submit_job('my-task-name', 1, 2, 3)\n\nwould produce the error, because ``submit_job`` from Gearman's Python bindings\ncontains __a lot__ of arguments and it's much easier to specify them via\nkeyword names or a special ``args`` keyword than to type something like seven\n``None``s instead :\n\n client.submit_job('my-task-name', None, None, None, None, None, None, None, 1, 2, 3)\n\nThe only limitation that you have are gearman reserved keyword parameters. As of\nGearman 2.0.2 these are :\n\n * data\n * unique\n * priority\n * background\n * wait_until_complete\n * max_retries\n * poll_timeout\n\nSo, if you want your task definition to have, for example, ``unique`` or\n``background`` keyword parameters, you need to execute the task in a special,\nmore verbose way. Here's an example of such a task and its execution :\n\n @gearman_task(name='my-task-name')\n def my_task_function(background, unique):\n pass\n\n client.submit_job('my-task-name', kwargs={\"background\": True, \"unique\": False})\n client.submit_job('my-task-name', args=[True, False])\n\nFinally:\n\n client.submit_job('my-task-name', background=True, unique=True, kwargs={\"background\": False, \"unique\": False})\n\nDon't panic, your task is safe! That's because you're using ``kwargs``\ndirectly. Therefore, Gearman's bindings would receive ``True`` for\n``submit_job`` function, while your task would receive ``False``.\n\nAlways remember to double-check your parameter names with the reserved words\nlist.\n\n### Starting a worker\nTo start a worker, run `python manage.py gearman_worker -a -n `. It will start\nserving all registered tasks for that worker.\n\nExample :\n\n $ python manage.py gearman_worker -a django_app_name -n worker_foo\n $ python manage.py gearman_worker -a django_app_name -n worker_bar\n\nTo spawn more than one worker see Supervisord configuration below.\n\n### Task queues\nQueues are a virtual abstraction layer built on top of gearman tasks. An\neasy way to describe it is the following example: Imagine you have a task\nfor fetching e-mails from the server, another task for sending the emails\nand one more task for sending SMS via an SMS gateway. A problem you may\nencounter is that the email fetching tasks may effectively \"block\" the worker\n(there could be so many of them, it could be so time-consuming, that no other\ntask would be able to pass through). Of course, one solution would be to add\nmore workers (via the Supervisord), but that would only temporarily\nsolve the problem. This is where queues come in.\n\nThe first thing to do is to pass a queue name into the job description, like\nthis :\n\n @gearman_task(name=\"task_foo\", queue=\"foo\")\n def function_foo(some_arg):\n pass\n\n @gearman_task(name=\"task_bar\", queue=\"bar\")\n def function_bar(some_arg):\n pass\n\n @gearman_task(name=\"task_babar\", queue=\"bar\")\n def function_babar(some_arg):\n pass\n\nYou may then proceed to start the tasks that are bound to a specific\nqueue :\n\n python manage.py gearman_worker -a -n -q bar\n\nBe aware of the fact that if you don't specify the queue name, the worker\nwill load all tasks.\n\n### Start workers with Supervisord\nSupervisor - http://supervisord.org/ is babysitter for processes.\nIt allows you to launch, restart and monitor running processes. In our case it will be workers.\nTo do so, create one config file by worker and adjust the number of workers you want with the 'numprocs' parameter :\n\n`worker_foo.conf` :\n\n [program:worker_foo]\n command = /path-to-your-virtualenv/bin/python /path-to-your-project/manage.py gearman_worker -a -n %(program_name)s\n process_name = %(program_name)s_%(process_num)02d\n numprocs = 1\n autostart = true\n autorestart = true\n user = myapp\n directory = /home/myapp/\n environment = HOME='/home/myapp',USER='myapp',LOGNAME='myapp',\n\n`worker_bar.conf` :\n\n [program:worker_bar]\n command = /path-to-your-virtualenv/bin/python /path-to-your-project/manage.py gearman_worker -a -n %(program_name)s -q bar\n process_name = %(program_name)s_%(process_num)02d\n numprocs = 2\n autostart = true\n autorestart = true\n user = myapp\n directory = /home/myapp/\n environment = HOME='/home/myapp',USER='myapp',LOGNAME='myapp',\n\nYou can also create a `groups.conf` file with this content :\n\n [group:foo]\n programs=worker_foo, worker_foo2\n\n [group:bar]\n programs=worker_bar, worker_bar2\n\n\nThis will create process 'group' and allows you to reload all workers related to this group at once when you redeploy new code.\n\nOnce you're config files are created, do `/etc/init.d/supervisord start` to start Supervisord and `supervisorctl reload` if you modify config or\n\n supervisorctl reread\n supervisorctl update\n supervisorctl restart foo:*\n supervisorctl restart bar:*\n\n### Execute code when workers die\nWorkers catch SIGTERM and SIGINT signals to kill themselves with a `sys.exit(0)` in a callback function.\nAt this point in the code you can add your own function(s) which will be executed before the `sys.exit(0)`\nSee `django_gearman_jbox\\management\\commands\\gearman_worker.py`, line 116\n\nNote that this will impact all workers as it resides in the `gearman_worker.py` script which is global for all workers.\n\n\nClients\n-------\nTo make your workers work, you need a client app passing data to them.\nCreate and instance of the `django_gearman_jbox.GearmanClient` class and execute submit_job with it :\n\n from django_gearman_jbox import GearmanClient\n\n sentence = \"The quick brown fox jumps over the lazy dog.\"\n\n client = GearmanClient()\n res = client.submit_job(\"foo\", kwargs={\"sentence\": sentence})\n print \"Result: '%s'\" % res\n\nDispatching a background event without waiting for the result is easy as well :\n\n client.submit_job(\"foo\", background=True, kwargs={\"sentence\": sentence})\n\nGearman Server Infos\n-------------------\n\n`python manage.py gearman_server_info` outputs current status of Gearman servers.\nIf you installed Prettytable dependency, here is how output looks like :\n\n $ python manage.py gearman_server_info\n +---------------------+------------------------+\n | Gearman Server Host | Gearman Server Version |\n +---------------------+------------------------+\n | 127.0.0.1:4730 | OK 0.29 |\n +---------------------+------------------------+.\n\n +---------------+---------------+--------------+-------------+\n | Task Name | Total Workers | Running Jobs | Queued Jobs |\n +---------------+---------------+--------------+-------------+\n | data_unlock | 1 | 0 | 0 |\n | data_import | 1 | 1 | 0 |\n | cache_cleanup | 1 | 0 | 0 |\n +---------------+---------------+--------------+-------------+.\n\n +-----------+------------------+-----------+-----------------+\n | Worker IP | Registered Tasks | Client ID | File Descriptor |\n +-----------+------------------+-----------+-----------------+\n | 127.0.0.1 | data_unlock | - | 35 |\n | 127.0.0.1 | data_import | - | 36 |\n | 127.0.0.1 | cache_cleanup | - | 37 |\n +-----------+------------------+-----------+-----------------+\n\nIf you have a lot of workers, you can filter output using command argument (case-sensitive):\n\n $ python manage.py gearman_server_info cleanup\n +---------------------+------------------------+--------------------+\n | Gearman Server Host | Gearman Server Version | Ping Response Time |\n +---------------------+------------------------+--------------------+\n | 127.0.0.1:4730 | OK 1.1.3 | 0.0006871223449707 |\n +---------------------+------------------------+--------------------+.\n\n +---------------+---------------+--------------+-------------+\n | Task Name | Total Workers | Running Jobs | Queued Jobs |\n +---------------+---------------+--------------+-------------+\n | cache_cleanup | 1 | 0 | 0 |\n +---------------+---------------+--------------+-------------+.\n\n +-----------+------------------+-----------+-----------------+\n | Worker IP | Registered Tasks | Client ID | File Descriptor |\n +-----------+------------------+-----------+-----------------+\n | 127.0.0.1 | cache_cleanup | - | 37 |\n +-----------+------------------+-----------+-----------------+\n\nLicensing\n---------\nThis software is licensed under the [Mozilla Tri-License][MPL]:\n\n ***** BEGIN LICENSE BLOCK *****\n Version: MPL 1.1/GPL 2.0/LGPL 2.1\n\n The contents of this file are subject to the Mozilla Public License Version\n 1.1 (the \"License\"); you may not use this file except in compliance with\n the License. You may obtain a copy of the License at\n http://www.mozilla.org/MPL/\n\n Software distributed under the License is distributed on an \"AS IS\" basis,\n WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License\n for the specific language governing rights and limitations under the\n License.\n\n The Original Code is django-gearman.\n\n The Initial Developer of the Original Code is Mozilla.\n Portions created by the Initial Developer are Copyright (C) 2010\n the Initial Developer. All Rights Reserved.\n\n Contributor(s):\n Frederic Wenzel >\n Jeff Balogh \n Jonas \n Jozef \u0160ev\u010d\u00edk \n Nicolas Rodriguez \n\n Alternatively, the contents of this file may be used under the terms of\n either the GNU General Public License Version 2 or later (the \"GPL\"), or\n the GNU Lesser General Public License Version 2.1 or later (the \"LGPL\"),\n in which case the provisions of the GPL or the LGPL are applicable instead\n of those above. If you wish to allow use of your version of this file only\n under the terms of either the GPL or the LGPL, and not to allow others to\n use your version of this file under the terms of the MPL, indicate your\n decision by deleting the provisions above and replace them with the notice\n and other provisions required by the GPL or the LGPL. If you do not delete\n the provisions above, a recipient may use your version of this file under\n the terms of any one of the MPL, the GPL or the LGPL.\n\n ***** END LICENSE BLOCK *****\n\n[MPL]: http://www.mozilla.org/MPL/", "description_content_type": null, "docs_url": null, "download_url": "http://jbox-web.github.com/django-gearman-jbox/", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://jbox-web.github.com/django-gearman-jbox/", "keywords": "django,gearman,jobs,tasks,asynchronous", "license": "MPL", "maintainer": null, "maintainer_email": null, "name": "django-gearman-jbox", "package_url": "https://pypi.org/project/django-gearman-jbox/", "platform": "Linux", "project_url": "https://pypi.org/project/django-gearman-jbox/", "project_urls": { "Download": "http://jbox-web.github.com/django-gearman-jbox/", "Homepage": "http://jbox-web.github.com/django-gearman-jbox/" }, "release_url": "https://pypi.org/project/django-gearman-jbox/0.1/", "requires_dist": null, "requires_python": null, "summary": "A convenience wrapper for Gearman clients and workers in Django/Python", "version": "0.1" }, "last_serial": 789722, "releases": { "0.1": [ { "comment_text": "", "digests": { "md5": "c2470bae122e229cbeace10a42cc5b34", "sha256": "9bbd303f24cb1f07bee89d362a79385a8bac375e0ac7d723d8e5173eb4c99caa" }, "downloads": -1, "filename": "django-gearman-jbox-0.1.tar.gz", "has_sig": false, "md5_digest": "c2470bae122e229cbeace10a42cc5b34", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15498, "upload_time": "2013-01-31T03:29:56", "url": "https://files.pythonhosted.org/packages/ac/e7/be32ca57c2128937a75b10e24b615e45077723aa343ff766cb519543399f/django-gearman-jbox-0.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "c2470bae122e229cbeace10a42cc5b34", "sha256": "9bbd303f24cb1f07bee89d362a79385a8bac375e0ac7d723d8e5173eb4c99caa" }, "downloads": -1, "filename": "django-gearman-jbox-0.1.tar.gz", "has_sig": false, "md5_digest": "c2470bae122e229cbeace10a42cc5b34", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15498, "upload_time": "2013-01-31T03:29:56", "url": "https://files.pythonhosted.org/packages/ac/e7/be32ca57c2128937a75b10e24b615e45077723aa343ff766cb519543399f/django-gearman-jbox-0.1.tar.gz" } ] }