{
    "info": {
        "author": "Roy Binux",
        "author_email": "roy@binux.me",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Environment :: Web Environment",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: Apache Software License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 2",
            "Programming Language :: Python :: 2.6",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.3",
            "Programming Language :: Python :: 3.4",
            "Programming Language :: Python :: 3.5",
            "Topic :: Internet :: WWW/HTTP",
            "Topic :: Software Development :: Libraries :: Application Frameworks",
            "Topic :: Software Development :: Libraries :: Python Modules"
        ],
        "description": "pyspider [![Build Status]][Travis CI] [![Coverage Status]][Coverage] [![Try]][Demo]\n========\n\nA Powerful Spider(Web Crawler) System in Python. **[TRY IT NOW!][Demo]**\n\n- Write script in Python\n- Powerful WebUI with script editor, task monitor, project manager and result viewer\n- [MySQL](https://www.mysql.com/), [MongoDB](https://www.mongodb.org/), [Redis](http://redis.io/), [SQLite](https://www.sqlite.org/), [Elasticsearch](https://www.elastic.co/products/elasticsearch); [PostgreSQL](http://www.postgresql.org/) with [SQLAlchemy](http://www.sqlalchemy.org/) as database backend\n- [RabbitMQ](http://www.rabbitmq.com/), [Beanstalk](http://kr.github.com/beanstalkd/), [Redis](http://redis.io/) and [Kombu](http://kombu.readthedocs.org/) as message queue\n- Task priority, retry, periodical, recrawl by age, etc...\n- Distributed architecture, Crawl Javascript pages, Python 2.{6,7}, 3.{3,4,5,6} support, etc...\n\nTutorial: [http://docs.pyspider.org/en/latest/tutorial/](http://docs.pyspider.org/en/latest/tutorial/)  \nDocumentation: [http://docs.pyspider.org/](http://docs.pyspider.org/)  \nRelease notes: [https://github.com/binux/pyspider/releases](https://github.com/binux/pyspider/releases)  \n\nSample Code \n-----------\n\n```python\nfrom pyspider.libs.base_handler import *\n\n\nclass Handler(BaseHandler):\n    crawl_config = {\n    }\n\n    @every(minutes=24 * 60)\n    def on_start(self):\n        self.crawl('http://scrapy.org/', callback=self.index_page)\n\n    @config(age=10 * 24 * 60 * 60)\n    def index_page(self, response):\n        for each in response.doc('a[href^=\"http\"]').items():\n            self.crawl(each.attr.href, callback=self.detail_page)\n\n    def detail_page(self, response):\n        return {\n            \"url\": response.url,\n            \"title\": response.doc('title').text(),\n        }\n```\n\n[![Demo][Demo Img]][Demo]\n\n\nInstallation\n------------\n\n* `pip install pyspider`\n* run command `pyspider`, visit [http://localhost:5000/](http://localhost:5000/)\n\n**WARNING:** WebUI is open to the public by default, it can be used to execute any command which may harm your system. Please use it in an internal network or [enable `need-auth` for webui](http://docs.pyspider.org/en/latest/Command-Line/#-config).\n\nQuickstart: [http://docs.pyspider.org/en/latest/Quickstart/](http://docs.pyspider.org/en/latest/Quickstart/)\n\nContribute\n----------\n\n* Use It\n* Open [Issue], send PR\n* [User Group]\n* [\u4e2d\u6587\u95ee\u7b54](http://segmentfault.com/t/pyspider)\n\n\nTODO\n----\n\n### v0.4.0\n\n- [ ] a visual scraping interface like [portia](https://github.com/scrapinghub/portia)\n\n\nLicense\n-------\nLicensed under the Apache License, Version 2.0\n\n\n[Build Status]:         https://img.shields.io/travis/binux/pyspider/master.svg?style=flat\n[Travis CI]:            https://travis-ci.org/binux/pyspider\n[Coverage Status]:      https://img.shields.io/coveralls/binux/pyspider.svg?branch=master&style=flat\n[Coverage]:             https://coveralls.io/r/binux/pyspider\n[Try]:                  https://img.shields.io/badge/try-pyspider-blue.svg?style=flat\n[Demo]:                 http://demo.pyspider.org/\n[Demo Img]:             https://github.com/binux/pyspider/blob/master/docs/imgs/demo.png\n[Issue]:                https://github.com/binux/pyspider/issues\n[User Group]:           https://groups.google.com/group/pyspider-users\n",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/binux/pyspider",
        "keywords": "scrapy crawler spider webui",
        "license": "Apache License, Version 2.0",
        "maintainer": "",
        "maintainer_email": "",
        "name": "pyspider",
        "package_url": "https://pypi.org/project/pyspider/",
        "platform": "",
        "project_url": "https://pypi.org/project/pyspider/",
        "project_urls": {
            "Homepage": "https://github.com/binux/pyspider"
        },
        "release_url": "https://pypi.org/project/pyspider/0.3.10/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "A Powerful Spider System in Python",
        "version": "0.3.10"
    },
    "last_serial": 3775550,
    "releases": {
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "16ddf2a993ea143b77eaf81a1c80ea16",
                    "sha256": "77e07ae17c21a6375c1ad6355a7fc1e31924194c8820c7ad74e4b65707457c81"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "16ddf2a993ea143b77eaf81a1c80ea16",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 75760,
                "upload_time": "2015-01-11T05:39:16",
                "url": "https://files.pythonhosted.org/packages/89/ae/905a5cd507766960100b01be69e80af7647f13a4121b30a3c744688e25a0/pyspider-0.3.0.tar.gz"
            }
        ],
        "0.3.0.dev4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "be7ed3273d6670d3a165a050b6a914ea",
                    "sha256": "9753775dd0afd6cb63482ac7bdb0205b5b520886107bcbd706d53397aebc7f22"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.0.dev4.tar.gz",
                "has_sig": false,
                "md5_digest": "be7ed3273d6670d3a165a050b6a914ea",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 65661,
                "upload_time": "2014-11-24T17:09:34",
                "url": "https://files.pythonhosted.org/packages/df/fd/cb78bcf3586315321652d9680e395810c496103cfec42ec3369a01851463/pyspider-0.3.0.dev4.tar.gz"
            }
        ],
        "0.3.0.dev5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "06b31cbea8d813ff7d6234ce7988c926",
                    "sha256": "0cfa258f263ba83db1bed5ba7fc2384aa9fd61fa811006b6fe9c4e4c004b2cb2"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.0.dev5.tar.gz",
                "has_sig": false,
                "md5_digest": "06b31cbea8d813ff7d6234ce7988c926",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 65657,
                "upload_time": "2014-11-24T17:13:10",
                "url": "https://files.pythonhosted.org/packages/22/72/fc7cf35491da47be108fd1e434c49d00b4f31155983f5f6ada57a9ec6d68/pyspider-0.3.0.dev5.tar.gz"
            }
        ],
        "0.3.0.dev6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5c6dcbe973a9c04b8ae7e2a7557deec3",
                    "sha256": "99a934aa61ea633ddcbf1c1a9025338ed230b5eaf7a70760d0a11085a5a78a8d"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.0.dev6.tar.gz",
                "has_sig": false,
                "md5_digest": "5c6dcbe973a9c04b8ae7e2a7557deec3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 68415,
                "upload_time": "2014-12-05T05:31:27",
                "url": "https://files.pythonhosted.org/packages/87/32/40e0271cc8a62fd223bf68410f97e41996096401c7a08fd3eacac56d1471/pyspider-0.3.0.dev6.tar.gz"
            }
        ],
        "0.3.0.dev7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "69bdc7581ab779f7b4a03edd53dba831",
                    "sha256": "5875300e7f25b61c2ea04103bc1cb3d23aff20dc79b65cbd80bc4df61cfe5835"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.0.dev7.tar.gz",
                "has_sig": false,
                "md5_digest": "69bdc7581ab779f7b4a03edd53dba831",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 71524,
                "upload_time": "2014-12-24T13:38:58",
                "url": "https://files.pythonhosted.org/packages/21/91/9bc5bfc0577609fbc9439dba3547c4fad9711dd487d6fadbd112bd09e8b3/pyspider-0.3.0.dev7.tar.gz"
            }
        ],
        "0.3.0a1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "901e907404a856faf2f76f77083f2975",
                    "sha256": "8c394606eb2cf254f10001ca7e3ed8f841eaaa8bedad09553f86dc51e2c68897"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.0a1.tar.gz",
                "has_sig": false,
                "md5_digest": "901e907404a856faf2f76f77083f2975",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 71829,
                "upload_time": "2014-12-27T05:10:28",
                "url": "https://files.pythonhosted.org/packages/4b/44/15b54628edd57e6e857da8ba69f2bd2b50591d2f133be4848595fc8956d3/pyspider-0.3.0a1.tar.gz"
            }
        ],
        "0.3.0b1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "cd25e34dd3e6ef6050ee709bb1982015",
                    "sha256": "3e02bdee25e1232a61421489d21ba98087d779aa118f422a72288e306f77f624"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.0b1.tar.gz",
                "has_sig": false,
                "md5_digest": "cd25e34dd3e6ef6050ee709bb1982015",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 74851,
                "upload_time": "2015-01-06T09:14:45",
                "url": "https://files.pythonhosted.org/packages/2f/2b/326b6e01bc36a5cacb03ac32dff9a8f31b213aec458a2fd2455438de2cd5/pyspider-0.3.0b1.tar.gz"
            }
        ],
        "0.3.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "22d4b27b88ec9387361ef2ce967ae40d",
                    "sha256": "b348e51de4311566867c4c1f8eac281f0bb2430a49cf4f939d0bcb578e74f647"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.1.tar.gz",
                "has_sig": false,
                "md5_digest": "22d4b27b88ec9387361ef2ce967ae40d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 78870,
                "upload_time": "2015-01-22T16:00:04",
                "url": "https://files.pythonhosted.org/packages/a1/92/4c5c25640c1f99787c3adb41f70f8ca2ea40e54cae0839fc731172768155/pyspider-0.3.1.tar.gz"
            }
        ],
        "0.3.10": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bc064430ba86f117cb3b418f4b6e9f2b",
                    "sha256": "0148e5db79e743c8a096d260ae279e77cf4bf28736b593702ef006b9bb471cbb"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.10.tar.gz",
                "has_sig": false,
                "md5_digest": "bc064430ba86f117cb3b418f4b6e9f2b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 110883,
                "upload_time": "2018-04-18T04:17:18",
                "url": "https://files.pythonhosted.org/packages/d0/97/d6062c928f53d899ff2a8538fed11d4d425ba3d27c96248a2c601c1c9fef/pyspider-0.3.10.tar.gz"
            }
        ],
        "0.3.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a4dc377e4425ed0d000a8fc37c69a9ea",
                    "sha256": "e303a7bd17962f376d8182ad9c4e88a39f95d4bddb0fbb2d91d818800d6a2308"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.2.tar.gz",
                "has_sig": false,
                "md5_digest": "a4dc377e4425ed0d000a8fc37c69a9ea",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 79972,
                "upload_time": "2015-02-11T16:34:15",
                "url": "https://files.pythonhosted.org/packages/b0/85/42db16f6fb7deb5f18e2121e7eea849af10625c3335e070c64804fceeeba/pyspider-0.3.2.tar.gz"
            }
        ],
        "0.3.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "eb3e3f45e8567fb42f56a0a960366bec",
                    "sha256": "447b288286d36d240db6cb5660fdb69d931ae4ca2cb156b99275593bbed820f9"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.3.tar.gz",
                "has_sig": false,
                "md5_digest": "eb3e3f45e8567fb42f56a0a960366bec",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 87871,
                "upload_time": "2015-03-08T13:51:09",
                "url": "https://files.pythonhosted.org/packages/dd/6b/9432ddb5c4aeafbe4f9251606cc2136656a4f448d420340bb6cca782ad57/pyspider-0.3.3.tar.gz"
            }
        ],
        "0.3.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ab76928f19827926aea5e843d163ee8d",
                    "sha256": "cc1e4c8f98bc3dd1fefcea5a08e57f5cba601f97efe86602b3c94a9d28d58057"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.4.tar.gz",
                "has_sig": false,
                "md5_digest": "ab76928f19827926aea5e843d163ee8d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 91514,
                "upload_time": "2015-04-21T15:04:11",
                "url": "https://files.pythonhosted.org/packages/7b/6f/1dddd8b5cec5e495d6e3e38469fcc2904135397efe2c1ec55ef82d1f3c65/pyspider-0.3.4.tar.gz"
            }
        ],
        "0.3.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ed015e19439a766ac5b698fce37a83e8",
                    "sha256": "703a1543b9f8acca7c88d7f5353a342c31fd2bb734da8c83602c013eaf5daf5e"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.5.tar.gz",
                "has_sig": false,
                "md5_digest": "ed015e19439a766ac5b698fce37a83e8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 94503,
                "upload_time": "2015-05-22T16:02:43",
                "url": "https://files.pythonhosted.org/packages/13/9a/bab0c731ff144b76872ed7a3771f00a2ffde6e7500a47b75cae11cc26b17/pyspider-0.3.5.tar.gz"
            }
        ],
        "0.3.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4c2b1496bc8a85ab8efb965943d3235d",
                    "sha256": "918c5916fe16fb7b5eded4f3b313577758e51de71ee8bfbf2777521dc622d956"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.6.tar.gz",
                "has_sig": false,
                "md5_digest": "4c2b1496bc8a85ab8efb965943d3235d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 99654,
                "upload_time": "2015-11-10T00:33:58",
                "url": "https://files.pythonhosted.org/packages/06/fe/3a397cf310bd2ab6fa30f2409fd17729c2090e9ba2217c31bf43bc7afe96/pyspider-0.3.6.tar.gz"
            }
        ],
        "0.3.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "20b89fab7ac0b89cbfb869e735da443f",
                    "sha256": "05e9c488119aed0a20daba578a2876d4f612fad7f13b680ccb414a971b5aa7cc"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.7.tar.gz",
                "has_sig": false,
                "md5_digest": "20b89fab7ac0b89cbfb869e735da443f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 105564,
                "upload_time": "2016-04-20T20:10:12",
                "url": "https://files.pythonhosted.org/packages/a9/d8/b36d83cac39e7cd3264bd737473ffb5e5284045976fd81f21c4dbf316551/pyspider-0.3.7.tar.gz"
            }
        ],
        "0.3.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9de728ca790010e8cd9fe2bc8014a5c4",
                    "sha256": "61ea59b19ea2bca3d595e028ebc3901cdb65b6f6807f8a55b0979490e00f360b"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.8.tar.gz",
                "has_sig": false,
                "md5_digest": "9de728ca790010e8cd9fe2bc8014a5c4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 106899,
                "upload_time": "2016-08-18T20:07:36",
                "url": "https://files.pythonhosted.org/packages/06/34/fbc3def68cdfbb23c38c94352d0a516a06614994c2fc0796c0e0954f2af8/pyspider-0.3.8.tar.gz"
            }
        ],
        "0.3.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5916fc5d34e96aa4ff54a75527413f4f",
                    "sha256": "eae362d9fd940a3adc31e2f82f3a82b213b79b8808b3cec0e4ccb0998c7c872c"
                },
                "downloads": -1,
                "filename": "pyspider-0.3.9.tar.gz",
                "has_sig": false,
                "md5_digest": "5916fc5d34e96aa4ff54a75527413f4f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 109807,
                "upload_time": "2017-03-18T21:00:42",
                "url": "https://files.pythonhosted.org/packages/df/c3/a045f1814374ef983216f3a5c0023febe0ae8203d40016d14919f6f45411/pyspider-0.3.9.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "bc064430ba86f117cb3b418f4b6e9f2b",
                "sha256": "0148e5db79e743c8a096d260ae279e77cf4bf28736b593702ef006b9bb471cbb"
            },
            "downloads": -1,
            "filename": "pyspider-0.3.10.tar.gz",
            "has_sig": false,
            "md5_digest": "bc064430ba86f117cb3b418f4b6e9f2b",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 110883,
            "upload_time": "2018-04-18T04:17:18",
            "url": "https://files.pythonhosted.org/packages/d0/97/d6062c928f53d899ff2a8538fed11d4d425ba3d27c96248a2c601c1c9fef/pyspider-0.3.10.tar.gz"
        }
    ]
}