{ "info": { "author": "Alex Wang", "author_email": "", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Environment :: Plugins", "Framework :: Scrapy", "Intended Audience :: Developers", "License :: OSI Approved :: BSD License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Topic :: Internet :: WWW/HTTP", "Topic :: Software Development :: Libraries :: Python Modules" ], "description": "====================\nScrapy-HTTPCache\n====================\n\n.. image:: https://img.shields.io/pypi/v/scrapy-httpcache.svg\n :target: https://pypi.python.org/pypi/scrapy-httpcache\n :alt: PyPI Version\n\n.. image:: https://img.shields.io/travis/xiaowangwindow/scrapy-httpcache/master.svg\n :target: http://travis-ci.org/xiaowangwindow/scrapy-httpcache\n :alt: Build Status\n\nOverview\n========\n\nscrapy-httpcache is a scrapy middleware to save http cache in mongodb.\nBesides, scrapy-httpcache contains two extra storage plugin,\nincluding request_error_storage and banned_storage.\nrequest_error_storage can save Request which occur error.\nbanned_storage can save Banned Request whose block_checker can be override.\n\n\nRequirements\n============\n\n* Python 3.3+\n* Works on Linux, Windows, Mac OSX, BSD\n\nInstall\n=======\n\nThe quick way::\n\n pip install scrapy-httpcache\n\nOR copy this middleware to your scrapy project.\n\nDocumentation\n=============\n\nIn settings.py, for example::\n\n # -----------------------------------------------------------------------------\n # SCRAPY HTTPCACHE SETTINGS\n # -----------------------------------------------------------------------------\n DOWNLOADER_MIDDLEWARES.update({\n 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': None,\n 'scrapy_httpcache.downloadermiddlewares.httpcache.AsyncHttpCacheMiddleware': 900,\n })\n\n HTTPCACHE_ENABLED = True\n HTTPCACHE_IGNORE_HTTP_CODES = [301, 302, 500, 503]\n HTTPCACHE_STORAGE = 'scrapy_httpcache.extensions.httpcache_storage.MongoDBCacheStorage'\n HTTPCACHE_MONGODB_HOST = '127.0.0.1'\n HTTPCACHE_MONGODB_PORT = 27017\n HTTPCACHE_MONGODB_USERNAME = 'root'\n HTTPCACHE_MONGODB_PASSWORD = 'password'\n HTTPCACHE_MONGODB_CONNECTION_POOL_KWARGS = {}\n HTTPCACHE_MONGODB_AUTH_DB = 'admin'\n HTTPCACHE_MONGODB_DB = 'cache_storage'\n HTTPCACHE_MONGODB_COLL = 'cache'\n\n # -----------------------------------------------------------------------------\n # SCRAPY HTTPCACHE BANNED SETTINGS (optional)\n # -----------------------------------------------------------------------------\n BANNED_STORAGE = 'scrapy_httpcache.extensions.banned_storage.MongoBannedStorage'\n\n # -----------------------------------------------------------------------------\n # SCRAPY HTTPCACHE REQUEST ERROR SETTINGS (optional)\n # -----------------------------------------------------------------------------\n REQUEST_ERROR_STORAGE = 'scrapy_httpcache.extensions.request_error_storage.MongoRequestErrorStorage'\n\nIf you want to remove banned response, use `send_catch_log_deferred` to send signal to `scrapy_httpcache.signals.remove_banned`\nwith kwargs contains (spider, response, exception), which callback function return a Deferred.", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/xiaowangwindow/scrapy-httpcache", "keywords": "", "license": "BSD", "maintainer": "Alex Wang", "maintainer_email": "xiaowangwindow@163.com", "name": "scrapy-httpcache", "package_url": "https://pypi.org/project/scrapy-httpcache/", "platform": "", "project_url": "https://pypi.org/project/scrapy-httpcache/", "project_urls": { "Homepage": "https://github.com/xiaowangwindow/scrapy-httpcache" }, "release_url": "https://pypi.org/project/scrapy-httpcache/0.0.5/", "requires_dist": null, "requires_python": "", "summary": "A middleware to cache http response for Scrapy", "version": "0.0.5" }, "last_serial": 3847011, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "a15d38846d39736484291716abd63d7c", "sha256": "2415e74225b31271750226acb3d981330960e6e460d3c6e960ae07ec2b75a9ee" }, "downloads": -1, "filename": "scrapy-httpcache-0.0.1.tar.gz", "has_sig": false, "md5_digest": "a15d38846d39736484291716abd63d7c", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12409, "upload_time": "2017-10-20T08:30:52", "url": "https://files.pythonhosted.org/packages/57/34/de03fe267740a3cd14e6b931fb689bb46e26ebfdab36ecdd8656d3b75160/scrapy-httpcache-0.0.1.tar.gz" } ], "0.0.2": [ { "comment_text": "", "digests": { "md5": "e10e9f5d78418a7cfa2afe2a370ed17c", "sha256": "67af28fab197e609ab59166829d8693bf5c003786d2c731d8c79db5d2ce14c2f" }, "downloads": -1, "filename": "scrapy-httpcache-0.0.2.tar.gz", "has_sig": false, "md5_digest": "e10e9f5d78418a7cfa2afe2a370ed17c", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13327, "upload_time": "2017-10-24T08:18:18", "url": "https://files.pythonhosted.org/packages/32/00/fa178e77c419ac454329ff90c4f244f4d721fe2c420ec96395de1bdb99c1/scrapy-httpcache-0.0.2.tar.gz" } ], "0.0.3": [ { "comment_text": "", "digests": { "md5": "3056f5fb02f3f7a47770ce1d018d3fac", "sha256": "e1555d3dfd5016d9a99a427183ff11e2a34c1b5a2d8d99db7d41a0276d759e9c" }, "downloads": -1, "filename": "scrapy-httpcache-0.0.3.tar.gz", "has_sig": false, "md5_digest": "3056f5fb02f3f7a47770ce1d018d3fac", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13288, "upload_time": "2017-10-24T08:53:15", "url": "https://files.pythonhosted.org/packages/79/e4/489bb5491dcdaf2cc286d8a96c290f1224f64bf8ae677b79b90ea02810f1/scrapy-httpcache-0.0.3.tar.gz" } ], "0.0.4": [ { "comment_text": "", "digests": { "md5": "8e375b1d8bd584d1a3abd21e72ec52cd", "sha256": "00acf2154f8b4a437ff64f8189c4759ccf26f06500a2f9e7015bb808251b049f" }, "downloads": -1, "filename": "scrapy-httpcache-0.0.4.tar.gz", "has_sig": false, "md5_digest": "8e375b1d8bd584d1a3abd21e72ec52cd", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13299, "upload_time": "2017-10-30T06:32:51", "url": "https://files.pythonhosted.org/packages/11/b3/5bb26d01cfa9d6935a55204369d1fe14b9b8e662418d9b34b3ca2775a937/scrapy-httpcache-0.0.4.tar.gz" } ], "0.0.5": [ { "comment_text": "", "digests": { "md5": "c7eb4292d6fefe6fc254152d2a08ee0b", "sha256": "6df37c945e69a1374ebbcec188eea7c01b3640bcfce46bffc1d51c1fd03caa3d" }, "downloads": -1, "filename": "scrapy-httpcache-0.0.5.tar.gz", "has_sig": false, "md5_digest": "c7eb4292d6fefe6fc254152d2a08ee0b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13476, "upload_time": "2018-05-09T10:00:07", "url": "https://files.pythonhosted.org/packages/71/97/103414578b9a7a58ad6e0aff38969844c6b50066aff3cdb7d79b631d896a/scrapy-httpcache-0.0.5.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "c7eb4292d6fefe6fc254152d2a08ee0b", "sha256": "6df37c945e69a1374ebbcec188eea7c01b3640bcfce46bffc1d51c1fd03caa3d" }, "downloads": -1, "filename": "scrapy-httpcache-0.0.5.tar.gz", "has_sig": false, "md5_digest": "c7eb4292d6fefe6fc254152d2a08ee0b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13476, "upload_time": "2018-05-09T10:00:07", "url": "https://files.pythonhosted.org/packages/71/97/103414578b9a7a58ad6e0aff38969844c6b50066aff3cdb7d79b631d896a/scrapy-httpcache-0.0.5.tar.gz" } ] }