{
    "info": {
        "author": "Sanhe Hu",
        "author_email": "husanhe@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Natural Language :: English",
            "Operating System :: MacOS",
            "Operating System :: Microsoft :: Windows",
            "Operating System :: Unix",
            "Programming Language :: Python",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3.4",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6"
        ],
        "description": ".. image:: https://travis-ci.org/MacHu-GWU/pytq-project.svg?branch=master\n    :target: https://travis-ci.org/MacHu-GWU/pytq-project?branch=master\n\n.. image:: https://codecov.io/gh/MacHu-GWU/pytq-project/branch/master/graph/badge.svg\n  :target: https://codecov.io/gh/MacHu-GWU/pytq-project\n\n.. image:: https://img.shields.io/pypi/v/pytq.svg\n    :target: https://pypi.python.org/pypi/pytq\n\n.. image:: https://img.shields.io/pypi/l/pytq.svg\n    :target: https://pypi.python.org/pypi/pytq\n\n.. image:: https://img.shields.io/pypi/pyversions/pytq.svg\n    :target: https://pypi.python.org/pypi/pytq\n\n.. image:: https://img.shields.io/badge/Star_Me_on_GitHub!--None.svg?style=social\n    :target: https://github.com/MacHu-GWU/pytq-project\n\n\nWelcome to ``pytq`` Documentation\n==============================================================================\n\n``pytq`` (Python Task Queue) is a task scheduler library.\n\nProblem we solve:\n\n1. You have N task to do.\n2. each task has ``input_data``, and after been processed, we got ``output_data``.\n\n``pytq`` provide these feature out-of-the-box (And it's all customizable).\n\n1. Save output_data to data-persistence system.\n2. Filter out duplicate input data.\n3. Built-in Multi thread processor boost the speed.\n4. Nice built-in log system.\n5. And its easy to define how you gonna:\n    - process your input_data\n    - integrate with your data persistence system\n    - filter duplicates input_data\n    - retrive output_data\n\n\nExample\n------------------------------------------------------------------------------\n\nSuppose you have some url to crawl, and you don't want to crawl those url you\nsuccessfully crawled, and also you want to save your crawled data in database.\n\n.. code-block:: python\n\n    #!/usr/bin/env python\n    # -*- coding: utf-8 -*-\n\n    \"\"\"\n    This script implement multi-thread safe, a sqlite backed task queue scheduler.\n    \"\"\"\n\n    from pytq import SqliteDictScheduler\n\n\n    # Define your input_data model\n    class UrlRequest(object):\n        def __init__(self, url, context_data=None):\n            self.url = url # your have url to crawl\n            self.context_data = context_data # and maybe some context data to use\n\n\n    class Scheduler(SqliteDictScheduler):\n        # (Required) define how you gonna process your data\n        def user_process(self, input_data):\n            # you need to implement get_html_from_url yourself\n            html = get_html_from_url(input_data.url)\n\n            # you need to implement parse_html yourself\n            output_data = parse_html(html)\n            return output_data\n\n    s = Scheduler(user_db_path=\"example.sqlite\")\n\n    # let's have some urls\n    input_data_queue = [\n        UrlRequest(url=\"https://pypi.python.org/pypi/pytq\"),\n        UrlRequest(url=\"https://pypi.python.org/pypi/crawlib\"),\n        UrlRequest(url=\"https://pypi.python.org/pypi/loggerFactory\"),\n    ]\n\n    # execute multi thread process\n    s.do(input_data_queue, multiprocess=True)\n\n    # print output\n    for id, outpupt_data in s.items():\n        ...\n\nCustomize:\n\n.. code-block:: python\n\n    class Scheduler(SqliteDictScheduler):\n        # (Optional) define the identifier of input_data (for duplicate)\n        def user_hash_input(self, input_data):\n            return input_data.url\n\n        # (Optional) define how do you save output_data to database\n        # Here we just use the default one\n        def user_post_process(self, task):\n            self._default_post_process(task)\n\n        # (Optional) define how do you skip crawled url\n        # Here we just use the default one\n        def user_is_duplicate(self, task):\n            return self._default_is_duplicate(task)\n\n\nTODO: more example is coming.\n\n\nQuick Links\n------------------------------------------------------------------------------\n\n- .. image:: https://img.shields.io/badge/Link-Document-red.svg\n      :target: https://pytq.readthedocs.io/index.html\n\n- .. image:: https://img.shields.io/badge/Link-API_Reference_and_Source_Code-red.svg\n      :target: https://pytq.readthedocs.io/py-modindex.html\n\n- .. image:: https://img.shields.io/badge/Link-Install-red.svg\n      :target: `install`_\n\n- .. image:: https://img.shields.io/badge/Link-GitHub-blue.svg\n      :target: https://github.com/MacHu-GWU/pytq-project\n\n- .. image:: https://img.shields.io/badge/Link-Submit_Issue_and_Feature_Request-blue.svg\n      :target: https://github.com/MacHu-GWU/pytq-project/issues\n\n- .. image:: https://img.shields.io/badge/Link-Download-blue.svg\n      :target: https://pypi.python.org/pypi/pytq#downloads\n\n\n.. _install:\n\nInstall\n------------------------------------------------------------------------------\n\n``pytq`` is released on PyPI, so all you need is:\n\n.. code-block:: console\n\n    $ pip install pytq\n\nTo upgrade to latest version:\n\n.. code-block:: console\n\n    $ pip install --upgrade pytq\n\n",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "https://pypi.python.org/pypi/pytq/0.0.7#downloads",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/MacHu-GWU/",
        "keywords": "",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "pytq",
        "package_url": "https://pypi.org/project/pytq/",
        "platform": "Windows",
        "project_url": "https://pypi.org/project/pytq/",
        "project_urls": {
            "Download": "https://pypi.python.org/pypi/pytq/0.0.7#downloads",
            "Homepage": "https://github.com/MacHu-GWU/"
        },
        "release_url": "https://pypi.org/project/pytq/0.0.7/",
        "requires_dist": [
            "attrs",
            "pymongo",
            "mongomock",
            "sqlalchemy"
        ],
        "requires_python": "",
        "summary": "A Task Queue Scheduler Framework.",
        "version": "0.0.7"
    },
    "last_serial": 3376300,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "375208f420fd840a33591b8937dd5581",
                    "sha256": "56aa1f26dda8a848a59778541267ac73ac27dda8e13075a1561abf38a9af342a"
                },
                "downloads": -1,
                "filename": "pytq-0.0.1-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "375208f420fd840a33591b8937dd5581",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 107813,
                "upload_time": "2017-10-12T00:46:45",
                "url": "https://files.pythonhosted.org/packages/5a/7c/f3ff101ccd794987a35c6512bee8b1f01f25e023cd0c6fbccf44110cd0a1/pytq-0.0.1-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ddef2eafa6c481e9adf226c6f83f0935",
                    "sha256": "cc9cb2c0c0d9c5242f01ce0a375169f12b88f6e2a29e6e6a3dc9ff25c371a1c4"
                },
                "downloads": -1,
                "filename": "pytq-0.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "ddef2eafa6c481e9adf226c6f83f0935",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 83960,
                "upload_time": "2017-10-12T00:46:46",
                "url": "https://files.pythonhosted.org/packages/1d/f5/cd3ec135b2e18fd61366394335c7d68fb380a541748b0dbe92d8ee4502dd/pytq-0.0.1.tar.gz"
            }
        ],
        "0.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "349a5d025c6d5b61368f27a426436db3",
                    "sha256": "aa95511dedc4e1f86c9ca953007cbbdf44674beb23aca0740499866a496a5140"
                },
                "downloads": -1,
                "filename": "pytq-0.0.2-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "349a5d025c6d5b61368f27a426436db3",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 52953,
                "upload_time": "2017-10-15T01:13:18",
                "url": "https://files.pythonhosted.org/packages/42/3a/d198ea3991470dd0b99eb0b9d69cf4280e87ae104a4ece9104362d5e148c/pytq-0.0.2-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ce9c7c3f1bb09afb06058f92617cba00",
                    "sha256": "9494d091ffb90fcb86f12a5be90b390f183df32d24a049dd8d22041c962aca42"
                },
                "downloads": -1,
                "filename": "pytq-0.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "ce9c7c3f1bb09afb06058f92617cba00",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 38812,
                "upload_time": "2017-10-15T01:13:20",
                "url": "https://files.pythonhosted.org/packages/b5/26/ebb317c08ae96106ae312d1168a84de626a0f31905867615f29aefead85e/pytq-0.0.2.tar.gz"
            }
        ],
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f822cc7df3a3fa4e3da310078cfaab8c",
                    "sha256": "691193b5b97b73626da2fe753d27888caa9cf697ff08515732ab91aa5dd64752"
                },
                "downloads": -1,
                "filename": "pytq-0.0.3-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "f822cc7df3a3fa4e3da310078cfaab8c",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 56870,
                "upload_time": "2017-10-16T02:20:06",
                "url": "https://files.pythonhosted.org/packages/db/f2/c1e70fd762fe7b9c9698e09bcf56a1bcae4a8314bae5f124e807a55afc0b/pytq-0.0.3-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e0164fed67adcc0ed421a13e4baf758d",
                    "sha256": "6ea1cd44690d6606dd41959970eb6c17927ac5a4b1fb77beab7da081b89427e1"
                },
                "downloads": -1,
                "filename": "pytq-0.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "e0164fed67adcc0ed421a13e4baf758d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 40653,
                "upload_time": "2017-10-16T02:20:08",
                "url": "https://files.pythonhosted.org/packages/40/06/e9b9fb5eb7e7c92ae0d9cd99cd820eacace404d21ffd3ea8921642cae301/pytq-0.0.3.tar.gz"
            }
        ],
        "0.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "428166ee281d0c8a3cfddeeafda25390",
                    "sha256": "ed76c94880140389a03085cd8e7c46780980f2fea4341b28108349501c226228"
                },
                "downloads": -1,
                "filename": "pytq-0.0.4-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "428166ee281d0c8a3cfddeeafda25390",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 57341,
                "upload_time": "2017-11-02T20:23:17",
                "url": "https://files.pythonhosted.org/packages/f6/99/1c6699a58321beb1243573b9575db4fd1ef70d6c2faaeffe3f22a3f819ed/pytq-0.0.4-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "1d4042747de849e190faed0ea6d2f50d",
                    "sha256": "4aae0fec073935023b95cd778eafa3e9ecb75cfc61cba88a0cb69f331ad822cd"
                },
                "downloads": -1,
                "filename": "pytq-0.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "1d4042747de849e190faed0ea6d2f50d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 41091,
                "upload_time": "2017-11-02T20:23:18",
                "url": "https://files.pythonhosted.org/packages/fb/f2/1d5106ef6635a3345bc3ccc0ad93c6d5853a5af69096e96b44df92b11e08/pytq-0.0.4.tar.gz"
            }
        ],
        "0.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "520210b35bfa19c8c31fa68bb0bd9f14",
                    "sha256": "d1c48a4cf2d7d7541cb45cf316850a8f55cbc0e258eb20777dc8ca14a287fa2d"
                },
                "downloads": -1,
                "filename": "pytq-0.0.5-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "520210b35bfa19c8c31fa68bb0bd9f14",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 65891,
                "upload_time": "2017-11-16T10:39:01",
                "url": "https://files.pythonhosted.org/packages/20/6b/ff91a870dea001d575f92b2333c40f4d75988ebf7a994414f09110a68cd7/pytq-0.0.5-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "7b0a72b7b450d360606e155707cfce2f",
                    "sha256": "92d02f894d289a647f8d2ca5f3d9585e36326b7e91301be9b52238773e30bdef"
                },
                "downloads": -1,
                "filename": "pytq-0.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "7b0a72b7b450d360606e155707cfce2f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 45896,
                "upload_time": "2017-11-16T10:39:03",
                "url": "https://files.pythonhosted.org/packages/bf/89/d420b90cb0f41f558f721eec810bea7a84a017e14fc5da246245d13873a5/pytq-0.0.5.tar.gz"
            }
        ],
        "0.0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b6b095b756231df81b81f15afe8d8f65",
                    "sha256": "3528c53a4c1a26efe277147ecc590f79f7bf9f032e4174641dc34cfa02702373"
                },
                "downloads": -1,
                "filename": "pytq-0.0.6-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "b6b095b756231df81b81f15afe8d8f65",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 67090,
                "upload_time": "2017-11-23T18:05:10",
                "url": "https://files.pythonhosted.org/packages/b4/82/c4e77a51f382c0263dc2a1ef8117b77a1dff00496657935324be81406427/pytq-0.0.6-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "f882f17430f5437ec85a0c5e66909884",
                    "sha256": "52d5a2c56ba243ab3ceb114d96bd2d6fe2115e1cfb27406b3923f75d689f5492"
                },
                "downloads": -1,
                "filename": "pytq-0.0.6.tar.gz",
                "has_sig": false,
                "md5_digest": "f882f17430f5437ec85a0c5e66909884",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 46397,
                "upload_time": "2017-11-23T18:05:12",
                "url": "https://files.pythonhosted.org/packages/6a/36/2979cb5dea2f350bda30975092b17e1cf18fb344307d18c50fc92d2200f1/pytq-0.0.6.tar.gz"
            }
        ],
        "0.0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b27bc5014c8da24b2c7042f4963876c5",
                    "sha256": "5b8fb61a6f78438868ae3e4b9eef2f4df97ad7b15c3ff727332fcf6ed19d5d0d"
                },
                "downloads": -1,
                "filename": "pytq-0.0.7-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "b27bc5014c8da24b2c7042f4963876c5",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 67238,
                "upload_time": "2017-11-30T04:28:58",
                "url": "https://files.pythonhosted.org/packages/cd/7e/1a6a6027b80d5b41c8982eef03c0b1f975ed5dcb03cafc3cca391a228825/pytq-0.0.7-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "46f614fb8552ac2d46930f33ae2013e9",
                    "sha256": "a8595a8e60e9f1f85f3526f3cdeccb7fe94c753f998af0016c4dc87ccd0f1452"
                },
                "downloads": -1,
                "filename": "pytq-0.0.7.tar.gz",
                "has_sig": false,
                "md5_digest": "46f614fb8552ac2d46930f33ae2013e9",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 46569,
                "upload_time": "2017-11-30T04:29:00",
                "url": "https://files.pythonhosted.org/packages/30/08/19fe1ea16ab2182cf6538f6fc7b177b80c8069b3dc35dc132fab5d22d5df/pytq-0.0.7.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "b27bc5014c8da24b2c7042f4963876c5",
                "sha256": "5b8fb61a6f78438868ae3e4b9eef2f4df97ad7b15c3ff727332fcf6ed19d5d0d"
            },
            "downloads": -1,
            "filename": "pytq-0.0.7-py2-none-any.whl",
            "has_sig": false,
            "md5_digest": "b27bc5014c8da24b2c7042f4963876c5",
            "packagetype": "bdist_wheel",
            "python_version": "py2",
            "requires_python": null,
            "size": 67238,
            "upload_time": "2017-11-30T04:28:58",
            "url": "https://files.pythonhosted.org/packages/cd/7e/1a6a6027b80d5b41c8982eef03c0b1f975ed5dcb03cafc3cca391a228825/pytq-0.0.7-py2-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "46f614fb8552ac2d46930f33ae2013e9",
                "sha256": "a8595a8e60e9f1f85f3526f3cdeccb7fe94c753f998af0016c4dc87ccd0f1452"
            },
            "downloads": -1,
            "filename": "pytq-0.0.7.tar.gz",
            "has_sig": false,
            "md5_digest": "46f614fb8552ac2d46930f33ae2013e9",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 46569,
            "upload_time": "2017-11-30T04:29:00",
            "url": "https://files.pythonhosted.org/packages/30/08/19fe1ea16ab2182cf6538f6fc7b177b80c8069b3dc35dc132fab5d22d5df/pytq-0.0.7.tar.gz"
        }
    ]
}