{ "info": { "author": "SpazioDati", "author_email": "hello@spaziodati.eu", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Environment :: Console", "Intended Audience :: Developers", "License :: OSI Approved :: BSD License", "Operating System :: OS Independent", "Programming Language :: Python", "Topic :: Utilities" ], "description": "scrapyrwiki\n===========\n\nA collection of helpers for running scrapers built with\n`Scrapy `_ in `ScraperWiki `_\n\n\nLaunch scraper without scrapy CLI\n---------------------------------\n\nExample:\n\n.. code:: python\n\n from scrapy.conf import settings\n from scrapyrwiki import run_spider\n\n def main():\n run_spider(MySpider(), settings)\n\n if __name__ == '__main__':\n main()\n\n\nSave produced data to ScraperWiki\n---------------------------------\n\nJust add \"scrapyrwiki.pipelines.ScraperWikiPipeline\" to ITEM_PIPELINES\n\nExample:\n\n.. code:: python\n\n from scrapy.conf import settings\n from scrapyrwiki import run_spider\n\n def scraperwiki():\n options = {\n 'SW_SAVE_BUFFER': 5,\n 'SW_UNIQUE_KEYS': {\"MyItem\": ['url']},\n 'ITEM_PIPELINES': ['scrapyrwiki.pipelines.ScraperWikiPipeline'],\n }\n settings.overrides.update(options)\n run_spider(MySpider(), settings)\n\n\n if __name__ == 'scraper':\n scraperwiki()\n\n\nCheck spider contracts in CI\n----------------------------\n\nJust launch spider with run_tests\n\nExample:\n\n.. code:: python\n\n from scrapyrwiki import run_tests\n from scrapy.conf import settings\n\n run_tests(MySpider(), \"output.xml\", settings)\n\nNote: For testing the HTTP cache is used. In the directory where the script is\nlaunched there must be a scrapy.cfg (needed by Scrapy to identify that's a scraper\ndirectory) and a .scrapy directory with the HTTP cache db.\n\nThe output is in XUnit format, tested on `Jenkins `_\n\n\nLog scraper errors to Sentry\n----------------------------\n\nInstall `scrapy-sentry `_ and set the\nenvironment variable SENTRY_DSN with the Sentry key. Scrapyrwiki will handle\neverything for you.", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://github.com/SpazioDati/scrapyrwiki", "keywords": null, "license": "BSD", "maintainer": null, "maintainer_email": null, "name": "scrapyrwiki", "package_url": "https://pypi.org/project/scrapyrwiki/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/scrapyrwiki/", "project_urls": { "Download": "UNKNOWN", "Homepage": "http://github.com/SpazioDati/scrapyrwiki" }, "release_url": "https://pypi.org/project/scrapyrwiki/0.2/", "requires_dist": null, "requires_python": null, "summary": "A collection of helpers for running Scrapy in ScraperWiki", "version": "0.2" }, "last_serial": 799339, "releases": { "0.1": [ { "comment_text": "", "digests": { "md5": "7791e8c63c2e1bccd3add0dfa0610d23", "sha256": "9b81189ad4bbc56061e4c211adb9963669a6ca7780f6ecef35ead176881d05d6" }, "downloads": -1, "filename": "scrapyrwiki-0.1.tar.gz", "has_sig": false, "md5_digest": "7791e8c63c2e1bccd3add0dfa0610d23", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3353, "upload_time": "2013-01-28T14:39:56", "url": "https://files.pythonhosted.org/packages/d5/b9/8c7de81a92f45267638c5f402e25a34e907f833238c1a2fde00d26887103/scrapyrwiki-0.1.tar.gz" } ], "0.2": [ { "comment_text": "", "digests": { "md5": "edcec4d73d677c3f89507aebde4d5edd", "sha256": "0afe100bdbc403955228309d9942066a13278d99843630a89a0e25ec72f4dcd8" }, "downloads": -1, "filename": "scrapyrwiki-0.2.tar.gz", "has_sig": false, "md5_digest": "edcec4d73d677c3f89507aebde4d5edd", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3525, "upload_time": "2013-02-27T09:38:19", "url": "https://files.pythonhosted.org/packages/56/69/f6486c5083066040f0461e24ae39cc371c43728230cfb9fd91618207b9b7/scrapyrwiki-0.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "edcec4d73d677c3f89507aebde4d5edd", "sha256": "0afe100bdbc403955228309d9942066a13278d99843630a89a0e25ec72f4dcd8" }, "downloads": -1, "filename": "scrapyrwiki-0.2.tar.gz", "has_sig": false, "md5_digest": "edcec4d73d677c3f89507aebde4d5edd", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3525, "upload_time": "2013-02-27T09:38:19", "url": "https://files.pythonhosted.org/packages/56/69/f6486c5083066040f0461e24ae39cc371c43728230cfb9fd91618207b9b7/scrapyrwiki-0.2.tar.gz" } ] }