{ "info": { "author": "Lhassan Baazzi", "author_email": "baazzilhassan@gmail.com", "bugtrack_url": null, "classifiers": [], "description": "# Scrapy MongoDB Queue\nMongoDB-based components for scrapy that allows distributed crawling\n\n# Available Scrapy components:\n* Scheduler\n* Duplication Filter\n\nInstallation\n------------\n\nFrom `pypi`::\n\n $ pip install scrapy-mongodb-queue\n\nFrom `github`::\n\n $ git clone https://github.com/jbinfo/scrapy-mongodb-queue.git\n $ cd scrapy-mongodb-queue\n $ python setup.py install\n\nUsage\n-----\n\nEnable the components in your `settings.py`:\n\n.. code-block:: python\n\n # Enables scheduling storing requests queue in redis.\n SCHEDULER = \"scrapy_mongodb_queue.scheduler.Scheduler\"\n\n # Don't cleanup mongodb queues, allows to pause/resume crawls.\n MONGODB_QUEUE_PERSIST = True\n\n # Specify the host and port to use when connecting to Redis (optional).\n MONGODB_SERVER = 'localhost'\n MONGODB_PORT = 27017\n MONGODB_DB = \"my_db\"\n\n # MongoDB collection name\n MONGODB_QUEUE_NAME = \"my_queue\"\n\nAuthor\n------\n\nThis project is maintained by Lhassan Baazzi ([GitHub](https://github.com/jbinfo) | [Twitter](https://twitter.com/baazzilhassan) | [LinkedIn](https://ma.linkedin.com/pub/lhassan-baazzi/49/606/a70))", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/jbinfo/scrapy-mongodb", "keywords": null, "license": "MIT", "maintainer": null, "maintainer_email": null, "name": "scrapy-mongodb-queue", "package_url": "https://pypi.org/project/scrapy-mongodb-queue/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/scrapy-mongodb-queue/", "project_urls": { "Download": "UNKNOWN", "Homepage": "https://github.com/jbinfo/scrapy-mongodb" }, "release_url": "https://pypi.org/project/scrapy-mongodb-queue/0.1.0/", "requires_dist": null, "requires_python": null, "summary": "MongoDB-based components for Scrapy", "version": "0.1.0" }, "last_serial": 1654668, "releases": { "0.1.0": [] }, "urls": [] }