{
    "info": {
        "author": "Brett Langdon",
        "author_email": "brett@blangdon.com",
        "bugtrack_url": null,
        "classifiers": [
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python",
            "Programming Language :: Python :: 2.6",
            "Programming Language :: Python :: 2.7",
            "Topic :: Utilities"
        ],
        "description": "qw\n==\n\nqw (or QueueWorker) is used to run worker processes which listen on a redis list for jobs to process.\n\n## Setup\n### pip\n\n`pip install qw`\n\n### git\n\n```\ngit clone git://github.com/brettlangdon/qw.git\ncd ./qw\npython setup.py install\n```\n\n## Design\n### Manager\nThe manager is simply a process manager. It's job is to start/stop worker sub-processes.\n\n### Worker\nThe workers are processes which sit and listen for jobs on a few queues and then process\nthose jobs.\n\n### Target\nThe worker/manager take a `target` which can be either a function or a string (importable function).\n\n```python\ndef target(job_id, job_data):\n    pass\n\nmanager = Manager(target)\n# OR\nmanager = Manager('__main__.target')\n```\n### Queues\nThere are a few different queues that are used. The job queues are just redis lists, manager/worker lists are sets and jobs are hashes.\n\nA worker picks up a job from either `all:jobs`, `<manager>:jobs` or `<worker>:jobs`, pulls the corresponding `job:<job_id>` key and\nprocesses it with the provided `target`, after processing it will then remove the `job:<job_id>` key as well as the job id from\nthe `<worker>:jobs` queue.\n\n* `all:managers` - a set of all managers\n* `all:jobs` - a queue that all workers can pull jobs from, the values are just the job ids\n* `job:<job_id>` - a hash of the job data\n* `<manager>:workers` - a set of all workers belonging to a given manager\n* `<manager>:jobs` - a queue of jobs for a specific manager, workers will try to pull from here before `all:jobs`, the values are just the job ids\n* `<worker>:jobs` - a queue of jobs for a specific worker, this is meant as a in progress queue for each worker, the workers will pull jobs into this queue from either `<manager>:jobs` or `all:jobs`, the values are just the job ids\n\n## Basic Usage\n\n```python\nfrom qw.manager import Manager\n\n\ndef job_printer(job_id, job_data):\n    print job_id\n    print job_data\n\n\nmanager = Manager(job_printer)\nmanager.start()\nmanager.join()\n```\n\n## API\n### Manager(object)\n* `__init__(self, target, host=\"localhost\", port=6379, db=0, num_workers=None, name=None)`\n* `start(self)`\n* `stop(self)`\n* `join(self)`\n\n### Worker(multiprocess.Process)\n* `__init__(self, client, target, manager_name=None, timeout=10)`\n* `run(self)`\n* `shutdown(self)`\n\n### Client(redis.StrictRedi)\n* `__init__(self, host=\"localhost\", port=6379, db=0)`\n* `register_manager(self, name)`\n* `deregister_manager(self, name)`\n* `register_worker(self, manager, name)`\n* `deregister_worker(self, manager, name)`\n* `queue_job(self, job_data, manager=None, worker=None)`\n* `fetch_next_job(self, manager, worker, timeout=10)`\n* `finish_job(self, job_id, worker_name)`\n* `get_all_managers(self)`\n* `get_manager_workers(self, manager_name)`\n* `get_worker_pending_jobs(self, worker_name)`\n* `get_manager_queued_jobs(self, manager_name)`\n* `get_all_queued_jobs(self)`\n* `get_all_pending_jobs(self)`\n\n## CLI Tools\n### qw-manager\nThe `qw-manager` tool is used to start a new manager process with the provided `target` string, which gets run\nfor every job processed by a worker.\n```\n$ qw-manager --help\nUsage:\n  qw-manager [--level=<log-level>] [--workers=<num-workers>] [--name=<name>] [--host=<host>] [--port=<port>] [--db=<db>] <target>\n  qw-manager (--help | --version)\n\nOptions:\n  --help                       Show this help message\n  --version                    Show version information\n  -l --level=<log-level>       Set the log level (debug,info,warn,error) [default: info]\n  -w --workers=<num-workers>   Set the number of workers to start, defaults to number of cpus\n  -n --name=<name>             Set the manager name, defaults to hostname\n  -h --host=<host>             Set the redis host to use [default: localhost]\n  -p --port=<port>             Set the redis port to use [default: 6379]\n  -d --db=<db>                 Set the redis db number to use [default: 0]\n```\n### qw-client\nThe `qw-client` command is useful to look at basic stats of running managers, workers and job queues\nas well as to push json data in the form of a string or a file to the main queue or a manager specific queue.\n```\n$ qw-client --help\nUsage:\n  qw-client [--host=<host>] [--port=<port>] [--db=<db>] managers\n  qw-client [--host=<host>] [--port=<port>] [--db=<db>] workers [<manager>]\n  qw-client [--host=<host>] [--port=<port>] [--db=<db>] jobs [<manager>]\n  qw-client [--host=<host>] [--port=<port>] [--db=<db>] queue string <data> [<manager>]\n  qw-client [--host=<host>] [--port=<port>] [--db=<db>] queue file <file> [<manager>]\n  qw-client (--help | --version)\n\nOptions:\n  --help                       Show this help message\n  --version                    Show version information\n  -h --host=<host>             Set the redis host to use [default: localhost]\n  -p --port=<port>             Set the redis port to use [default: 6379]\n  -d --db=<db>                 Set the redis db number to use [default: 0]\n```\n",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/brettlangdon/qw",
        "keywords": null,
        "license": "MIT",
        "maintainer": null,
        "maintainer_email": null,
        "name": "qw",
        "package_url": "https://pypi.org/project/qw/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/qw/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "https://github.com/brettlangdon/qw"
        },
        "release_url": "https://pypi.org/project/qw/0.1.1/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "Python Distributed Redis Queue Workers",
        "version": "0.1.1"
    },
    "last_serial": 1268555,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "45fca2c8999df7d2fe9cbc0858743ab8",
                    "sha256": "08930584b7083510c9ecac64ece69bc01d2378ab26ed24a034d73064b8107cae"
                },
                "downloads": -1,
                "filename": "qw-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "45fca2c8999df7d2fe9cbc0858743ab8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5540,
                "upload_time": "2014-10-09T22:04:14",
                "url": "https://files.pythonhosted.org/packages/5c/5b/afd4b48235035c1d65c1474f209455e5a74b3d676b4b59051f0932f1b0c4/qw-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ced3cd0042b5702051ce7558bb1ad27f",
                    "sha256": "567e4056017f5e876329170b17578f52d2d783c230f3c11607d4ee961f8c6bf4"
                },
                "downloads": -1,
                "filename": "qw-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "ced3cd0042b5702051ce7558bb1ad27f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5623,
                "upload_time": "2014-10-13T13:01:47",
                "url": "https://files.pythonhosted.org/packages/9b/74/407f1d1541ab9c12247405b35176c1eed73b3d2a1bd925222ac6f8e91119/qw-0.1.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "ced3cd0042b5702051ce7558bb1ad27f",
                "sha256": "567e4056017f5e876329170b17578f52d2d783c230f3c11607d4ee961f8c6bf4"
            },
            "downloads": -1,
            "filename": "qw-0.1.1.tar.gz",
            "has_sig": false,
            "md5_digest": "ced3cd0042b5702051ce7558bb1ad27f",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 5623,
            "upload_time": "2014-10-13T13:01:47",
            "url": "https://files.pythonhosted.org/packages/9b/74/407f1d1541ab9c12247405b35176c1eed73b3d2a1bd925222ac6f8e91119/qw-0.1.1.tar.gz"
        }
    ]
}