{
    "info": {
        "author": "Shi Tao",
        "author_email": "shitao0418@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7",
            "Programming Language :: Python :: 3.8",
            "Programming Language :: Python :: 3.9"
        ],
        "description": "\n# scrapy-redis \u96c6\u7fa4\u7248\n\n![PyPI](https://img.shields.io/pypi/v/scrapy-redis-sentinel)\n![PyPI - License](https://img.shields.io/pypi/l/scrapy-redis-sentinel)\n![GitHub last commit](https://img.shields.io/github/last-commit/crawlmap/scrapy-redis-sentinel)\n![PyPI - Downloads](https://img.shields.io/pypi/dw/scrapy-redis-sentinel)\n\n\u672c\u9879\u76ee\u57fa\u4e8e\u539f\u9879\u76ee [scrapy-redis](https://github.com/rmax/scrapy-redis)\n\n\u8fdb\u884c\u4fee\u6539\uff0c\u4fee\u6539\u5185\u5bb9\u5982\u4e0b\uff1a\n\n1. \u6dfb\u52a0\u4e86 `Redis` \u54e8\u5175\u8fde\u63a5\u652f\u6301\n2. \u6dfb\u52a0\u4e86 `Redis` \u96c6\u7fa4\u8fde\u63a5\u652f\u6301\n3. \u6dfb\u52a0\u4e86 `Bloomfilter` \u53bb\u91cd\n\n## \u5b89\u88c5\n\n```bash\npip install scrapy-redis-sentinel --user\n```\n\n## \u914d\u7f6e\u793a\u4f8b\n\n> \u539f\u7248\u672c scrapy-redis \u7684\u6240\u6709\u914d\u7f6e\u90fd\u652f\u6301, \u4f18\u5148\u7ea7\uff1a\u54e8\u5175\u6a21\u5f0f > \u96c6\u7fa4\u6a21\u5f0f > \u5355\u673a\u6a21\u5f0f\n\n```python\n# ----------------------------------------Bloomfilter \u914d\u7f6e-------------------------------------\n# \u4f7f\u7528\u7684\u54c8\u5e0c\u51fd\u6570\u6570\uff0c\u9ed8\u8ba4\u4e3a 6\nBLOOMFILTER_HASH_NUMBER = 6\n\n# Bloomfilter \u4f7f\u7528\u7684 Redis \u5185\u5b58\u4f4d\uff0c30 \u8868\u793a 2 ^ 30 = 128MB\uff0c\u9ed8\u8ba4\u4e3a 30   (2 ^ 22 = 1MB \u53ef\u53bb\u91cd 130W URL)\nBLOOMFILTER_BIT = 30\n\n# \u662f\u5426\u5f00\u542f\u53bb\u91cd\u8c03\u8bd5\u6a21\u5f0f \u9ed8\u8ba4\u4e3a False \u5173\u95ed\nDUPEFILTER_DEBUG = False\n\n# ----------------------------------------Redis \u5355\u673a\u6a21\u5f0f-------------------------------------\n# Redis \u5355\u673a\u5730\u5740\nREDIS_HOST = \"172.25.2.25\"\nREDIS_PORT = 6379\n\n# REDIS \u5355\u673a\u6a21\u5f0f\u914d\u7f6e\u53c2\u6570\nREDIS_PARAMS = {\n    \"password\": \"password\",\n    \"db\": 0\n}\n\n# ----------------------------------------Redis \u54e8\u5175\u6a21\u5f0f-------------------------------------\n\n# Redis \u54e8\u5175\u5730\u5740\nREDIS_SENTINELS = [\n    ('172.25.2.25', 26379),\n    ('172.25.2.26', 26379),\n    ('172.25.2.27', 26379)\n]\n\n# REDIS_SENTINEL_PARAMS \u54e8\u5175\u6a21\u5f0f\u914d\u7f6e\u53c2\u6570\u3002\nREDIS_SENTINEL_PARAMS = {\n    \"service_name\": \"mymaster\",\n    \"password\": \"password\",\n    \"db\": 0\n}\n\n# ----------------------------------------Redis \u96c6\u7fa4\u6a21\u5f0f-------------------------------------\n\n# Redis \u96c6\u7fa4\u5730\u5740\nREDIS_STARTUP_NODES = [\n    {\"host\": \"172.25.2.25\", \"port\": \"6379\"},\n    {\"host\": \"172.25.2.26\", \"port\": \"6379\"},\n    {\"host\": \"172.25.2.27\", \"port\": \"6379\"},\n]\n\n# REDIS_CLUSTER_PARAMS \u96c6\u7fa4\u6a21\u5f0f\u914d\u7f6e\u53c2\u6570\nREDIS_CLUSTER_PARAMS = {\n    \"password\": \"password\"\n}\n\n# ----------------------------------------Scrapy \u5176\u4ed6\u53c2\u6570-------------------------------------\n\n# \u5728 redis \u4e2d\u4fdd\u6301 scrapy-redis \u7528\u5230\u7684\u5404\u4e2a\u961f\u5217\uff0c\u4ece\u800c\u5141\u8bb8\u6682\u505c\u548c\u6682\u505c\u540e\u6062\u590d\uff0c\u4e5f\u5c31\u662f\u4e0d\u6e05\u7406 redis queues\nSCHEDULER_PERSIST = True\n# \u8c03\u5ea6\u961f\u5217  \nSCHEDULER = \"scrapy_redis_sentinel.scheduler.Scheduler\"\n# \u57fa\u7840\u53bb\u91cd\nDUPEFILTER_CLASS = \"scrapy_redis_sentinel.dupefilter.RedisDupeFilter\"\n# BloomFilter\n# DUPEFILTER_CLASS = \"scrapy_redis_sentinel.dupefilter.RedisBloomFilter\"\n\n# \u542f\u7528\u57fa\u4e8e Redis \u7edf\u8ba1\u4fe1\u606f\nSTATS_CLASS = \"scrapy_redis_sentinel.stats.RedisStatsCollector\"\n\n# \u6307\u5b9a\u6392\u5e8f\u722c\u53d6\u5730\u5740\u65f6\u4f7f\u7528\u7684\u961f\u5217\n# \u9ed8\u8ba4\u7684 \u6309\u4f18\u5148\u7ea7\u6392\u5e8f( Scrapy \u9ed8\u8ba4)\uff0c\u7531 sorted set \u5b9e\u73b0\u7684\u4e00\u79cd\u975e FIFO\u3001LIFO \u65b9\u5f0f\u3002\n# SCHEDULER_QUEUE_CLASS = 'scrapy_redis_sentinel.queue.SpiderPriorityQueue'\n# \u53ef\u9009\u7684 \u6309\u5148\u8fdb\u5148\u51fa\u6392\u5e8f\uff08FIFO\uff09\n# SCHEDULER_QUEUE_CLASS = 'scrapy_redis_sentinel.queue.SpiderStack'\n# \u53ef\u9009\u7684 \u6309\u540e\u8fdb\u5148\u51fa\u6392\u5e8f\uff08LIFO\uff09\n# SCHEDULER_QUEUE_CLASS = 'scrapy_redis_sentinel.queue.SpiderStack'\n```\n\n> \u6ce8\uff1a\u5f53\u4f7f\u7528\u96c6\u7fa4\u65f6\u5355\u673a\u4e0d\u751f\u6548\n\n## spiders \u4f7f\u7528\n\n**\u4fee\u6539 RedisSpider \u5f15\u5165\u65b9\u5f0f**\n\n\u539f\u7248\u672c `scrapy-redis` \u4f7f\u7528\u65b9\u5f0f\n\n```python\nfrom scrapy_redis.spiders import RedisSpider\n\n\nclass Spider(RedisSpider):\n    ...\n\n```\n\n`scrapy-redis-sentinel` \u4f7f\u7528\u65b9\u5f0f\n\n```python\nfrom scrapy_redis_sentinel.spiders import RedisSpider\n\n\nclass Spider(RedisSpider):\n    ...\n\n```\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/crawlmap/scrapy-redis-sentinel.git",
        "keywords": "scrapy-redis,scrapy-redis-sentinel,scrapy-redis-cluster",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "scrapy-redis-sentinel",
        "package_url": "https://pypi.org/project/scrapy-redis-sentinel/",
        "platform": "",
        "project_url": "https://pypi.org/project/scrapy-redis-sentinel/",
        "project_urls": {
            "Documentation": "https://crawlaio.com/scrapy-redis-sentinel/",
            "Homepage": "https://github.com/crawlmap/scrapy-redis-sentinel.git"
        },
        "release_url": "https://pypi.org/project/scrapy-redis-sentinel/0.7.2/",
        "requires_dist": [
            "redis (==3.5.3)",
            "redis-py-cluster (==2.1.3)",
            "Scrapy"
        ],
        "requires_python": "",
        "summary": "Redis Cluster for Scrapy.",
        "version": "0.7.2",
        "yanked": false,
        "yanked_reason": null
    },
    "last_serial": 11788737,
    "releases": {
        "0.1.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6f9d2893f7987f319f0ac7c36ceafe12",
                    "sha256": "7cf05d6b0170f90892a375d5562c52c35a579e085a8af88891e153a2173092bd"
                },
                "downloads": -1,
                "filename": "scrapy-redis-sentinel-0.1.6.tar.gz",
                "has_sig": false,
                "md5_digest": "6f9d2893f7987f319f0ac7c36ceafe12",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10939,
                "upload_time": "2019-08-13T09:10:25",
                "upload_time_iso_8601": "2019-08-13T09:10:25.706978Z",
                "url": "https://files.pythonhosted.org/packages/7c/19/cafa1c774de34088e86f017f687d84401d08f007808097026a42db831d9f/scrapy-redis-sentinel-0.1.6.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.1.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2bd69d39701b0ada7997f1720a2ce05c",
                    "sha256": "077f1fe7b72b905a69fd8e95e2978f1dbc34553b5981a8c95ecaeadd8f06ffd1"
                },
                "downloads": -1,
                "filename": "scrapy-redis-sentinel-0.1.7.tar.gz",
                "has_sig": false,
                "md5_digest": "2bd69d39701b0ada7997f1720a2ce05c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11723,
                "upload_time": "2019-09-05T09:27:24",
                "upload_time_iso_8601": "2019-09-05T09:27:24.530841Z",
                "url": "https://files.pythonhosted.org/packages/9d/d4/600822ab903009cc4fdcdf56927566650f0b404c981d19ead20dbf898200/scrapy-redis-sentinel-0.1.7.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "efa44f52f1e6adb057fadbd12b37cd46",
                    "sha256": "a0c694ea886a69a115f39af3aed51f2212679cd9135ae20a4f0dd0aa48325102"
                },
                "downloads": -1,
                "filename": "scrapy-redis-sentinel-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "efa44f52f1e6adb057fadbd12b37cd46",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12663,
                "upload_time": "2019-10-29T01:38:20",
                "upload_time_iso_8601": "2019-10-29T01:38:20.037752Z",
                "url": "https://files.pythonhosted.org/packages/e7/f3/4b9c3295cede3057ad3d5ef71f3ef65f66ab078959ce10a1652a951d6020/scrapy-redis-sentinel-0.2.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4e640ab9951ca9a55f080dd2f502647b",
                    "sha256": "ccceec4d2b3676312464e2e514977341989feaa631d37a769212a3ed681056cd"
                },
                "downloads": -1,
                "filename": "scrapy-redis-sentinel-0.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "4e640ab9951ca9a55f080dd2f502647b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12654,
                "upload_time": "2019-11-01T06:57:36",
                "upload_time_iso_8601": "2019-11-01T06:57:36.349639Z",
                "url": "https://files.pythonhosted.org/packages/87/32/1d08bdd3c1800a483bfd7017a05084577dec9f0db2e55efc94a3e65816ee/scrapy-redis-sentinel-0.2.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.2.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b4c8baa41f6bd22fe210c99a133909ae",
                    "sha256": "4963eacf5423b4ef686968b153da4c7a4ba49e3cca5485ce3e47fd04cf76b403"
                },
                "downloads": -1,
                "filename": "scrapy_redis_sentinel-0.2.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "b4c8baa41f6bd22fe210c99a133909ae",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 15683,
                "upload_time": "2020-06-18T15:01:38",
                "upload_time_iso_8601": "2020-06-18T15:01:38.430940Z",
                "url": "https://files.pythonhosted.org/packages/5c/7e/a130f45741617d85c4bee39dd485b44801741e8b9a5f175acfd7b0c908db/scrapy_redis_sentinel-0.2.2-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d24322c165a63956597e3037eed843d6",
                    "sha256": "ac94f96f66a33dda56fbdb088d7035630866565e21caa4c5301af1c01afbf31e"
                },
                "downloads": -1,
                "filename": "scrapy-redis-sentinel-0.2.2.tar.gz",
                "has_sig": false,
                "md5_digest": "d24322c165a63956597e3037eed843d6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12532,
                "upload_time": "2020-06-18T15:01:40",
                "upload_time_iso_8601": "2020-06-18T15:01:40.618215Z",
                "url": "https://files.pythonhosted.org/packages/3a/72/942ac58390101d6d6a70db764c1bf17f784453f61ca759d50587b19443d1/scrapy-redis-sentinel-0.2.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.2.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "1801f3e86707daaf5684754525830fcf",
                    "sha256": "329df69f281a32e9b9c03be4711a51c7c5164e4048bbf86697bbd4554b504a31"
                },
                "downloads": -1,
                "filename": "scrapy_redis_sentinel-0.2.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "1801f3e86707daaf5684754525830fcf",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 15691,
                "upload_time": "2020-06-18T15:05:14",
                "upload_time_iso_8601": "2020-06-18T15:05:14.544921Z",
                "url": "https://files.pythonhosted.org/packages/e9/1a/0b678cf9861a2119befcc5f5f0bb8df2078ec620119535e50988f1c9f825/scrapy_redis_sentinel-0.2.3-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "4383a94d421a5a4aaabed40bbeee2859",
                    "sha256": "d53ddcacdf1aaa8a7b9529661c8df702d79ccb55b31076b157880daf7a30622f"
                },
                "downloads": -1,
                "filename": "scrapy-redis-sentinel-0.2.3.tar.gz",
                "has_sig": false,
                "md5_digest": "4383a94d421a5a4aaabed40bbeee2859",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12537,
                "upload_time": "2020-06-18T15:05:17",
                "upload_time_iso_8601": "2020-06-18T15:05:17.197966Z",
                "url": "https://files.pythonhosted.org/packages/63/f4/0e2d5a168424c9fdb9ae4dd751de83f250c089d2883a82266138b38b830f/scrapy-redis-sentinel-0.2.3.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.7.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "914dab8ea5b8a6ac2dff7ce4bfd4d25d",
                    "sha256": "e986fa617edebb411a17b9670716ce46758bfff987b4bac7bdd785b1f5398482"
                },
                "downloads": -1,
                "filename": "scrapy_redis_sentinel-0.7.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "914dab8ea5b8a6ac2dff7ce4bfd4d25d",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 17497,
                "upload_time": "2021-09-07T05:50:34",
                "upload_time_iso_8601": "2021-09-07T05:50:34.131271Z",
                "url": "https://files.pythonhosted.org/packages/c7/1f/5c8fc823e5e46bd3de525fe019d26706c9da46b79663ca36b59c18b6ee86/scrapy_redis_sentinel-0.7.1-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d4c4db671df9f5da6086d55cbf92011a",
                    "sha256": "dc8e21a36ce66da1e077e59c5525f1a49187e5dfb430c2a27a58ea096787e491"
                },
                "downloads": -1,
                "filename": "scrapy-redis-sentinel-0.7.1.tar.gz",
                "has_sig": false,
                "md5_digest": "d4c4db671df9f5da6086d55cbf92011a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15565,
                "upload_time": "2021-09-07T05:50:35",
                "upload_time_iso_8601": "2021-09-07T05:50:35.592157Z",
                "url": "https://files.pythonhosted.org/packages/a7/7e/1a0b9e092d5ff4281ed21a2c7ed921deab58c9f9b928b7121b6b31cd012f/scrapy-redis-sentinel-0.7.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.7.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3018d3bdee71b29e6e2511fc6785230f",
                    "sha256": "394507acc94c23be45d95c6206743014122683f6ace3832069104134941e3a4d"
                },
                "downloads": -1,
                "filename": "scrapy_redis_sentinel-0.7.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "3018d3bdee71b29e6e2511fc6785230f",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 17500,
                "upload_time": "2021-10-21T05:54:17",
                "upload_time_iso_8601": "2021-10-21T05:54:17.248903Z",
                "url": "https://files.pythonhosted.org/packages/99/7b/f90943c6c0827ade2b9de8a625a0930b954e912264558026829fb9c50bbe/scrapy_redis_sentinel-0.7.2-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c4a761423260b8f8ee1a3716a4ab3086",
                    "sha256": "02c12eae22777b5c622f57440dae95d1342c56c5cf65bbc9693a99be9ed58b65"
                },
                "downloads": -1,
                "filename": "scrapy-redis-sentinel-0.7.2.tar.gz",
                "has_sig": false,
                "md5_digest": "c4a761423260b8f8ee1a3716a4ab3086",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15554,
                "upload_time": "2021-10-21T05:54:19",
                "upload_time_iso_8601": "2021-10-21T05:54:19.824902Z",
                "url": "https://files.pythonhosted.org/packages/67/18/d4a8b495982d36679a16591cb0b9c38eab45adad579a2ca02a19a09fda6d/scrapy-redis-sentinel-0.7.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "3018d3bdee71b29e6e2511fc6785230f",
                "sha256": "394507acc94c23be45d95c6206743014122683f6ace3832069104134941e3a4d"
            },
            "downloads": -1,
            "filename": "scrapy_redis_sentinel-0.7.2-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "3018d3bdee71b29e6e2511fc6785230f",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 17500,
            "upload_time": "2021-10-21T05:54:17",
            "upload_time_iso_8601": "2021-10-21T05:54:17.248903Z",
            "url": "https://files.pythonhosted.org/packages/99/7b/f90943c6c0827ade2b9de8a625a0930b954e912264558026829fb9c50bbe/scrapy_redis_sentinel-0.7.2-py2.py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "c4a761423260b8f8ee1a3716a4ab3086",
                "sha256": "02c12eae22777b5c622f57440dae95d1342c56c5cf65bbc9693a99be9ed58b65"
            },
            "downloads": -1,
            "filename": "scrapy-redis-sentinel-0.7.2.tar.gz",
            "has_sig": false,
            "md5_digest": "c4a761423260b8f8ee1a3716a4ab3086",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 15554,
            "upload_time": "2021-10-21T05:54:19",
            "upload_time_iso_8601": "2021-10-21T05:54:19.824902Z",
            "url": "https://files.pythonhosted.org/packages/67/18/d4a8b495982d36679a16591cb0b9c38eab45adad579a2ca02a19a09fda6d/scrapy-redis-sentinel-0.7.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "vulnerabilities": []
}