{
    "info": {
        "author": "Tishacy",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7"
        ],
        "description": "# MSpider\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)  [![Pyversion](https://img.shields.io/pypi/pyversions/mspider.svg?color=#)](https://pypi.org/project/mspider/) [![Version](https://img.shields.io/pypi/v/mspider.svg?color=red)](https://pypi.org/project/mspider)\n\nA Multi-threaded Spider wrapper that could make your spider multi-threaded easily, helping you crawl website faster. :zap:\n\n*Note that this is for python3 only.*\n\n## Install\n\nMSpider could be easily installed using pip:\n\n```bash\npip install mspider\n```\n\n## Quick Start\n\n### Automatically create a `MSpider`\n\n1. `cd` to the folder you\u2019d like to create a `MSpider` in terminal or cmd, then type `genspider -b <template based> <your spider name>`, such as:\n\n   ```bash\n   $ genspider -b MSpider test\n   ```\n\n   where `-b` is to choose the template of spider you based, you could choose 'MSpider' (Default if not given) or 'Crawler', and `test` is the spider name.\n\n   A file `test.py` that contains a `MSpider` is created successfully if seeing the following information.\n\n   ```bash\n   create a spider named test.\n   ```\n\n2. Open the spider file `test.py`. Find `self.source = []` in line 8 (or line 15 if your spider template is 'Crawler'), and replacing it by the sources (usually a list of urls) you\u2019d like to handle by the spider, such as:\n\n   ```python\n   self.source = ['http://www.github.com',\n                  'http://www.baidu.com']\n   ```\n\n   Each element of the `self.source` is called `src_item`, and the index of `src_item` is called `index`.\n\n3. Find the function `basic_func`, where you could define your spider function, such as:\n\n   ```python\n   def basic_func(self, index, src_item):\n       url = src_item\n       res = self.sess.get(url)\n       html = res.content.decode('utf-8')\n       # deal with the html\n       # save the extracted information\n   ```\n\n4. Run the spider to start crawling.\n\n   ```bash\n   $ python3 test.py\n   ```\n\n   You just input the number of source items handled by each thread (BATCH SIZE) in the terminal or cmd, then return it, and then the MSpider will crawl your sources in a multi-threaded manner.\n\n   ```bash\n   [INFO]: MSpider is ready.\n   [INFO]: 2 urls in total.\n   [INPUT]: BATCH SIZE: 1\n   [INFO]: Open threads: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 2/2 [00:00<00:00, 356.36it/s]\n   [INFO]: Task done.\n   [INFO]: The task costs 1.1157 sec.\n   [INFO]: 0 urls failed.\n   ```\n\n### Mannually create a `MSpider`\n\n1. Standard import the MSpider.\n\n   ```python\n   from mspider.spider import MSpider\n   ```\n\n2. Define the function of your single threaded spider.\n\n   Note that this function must has two parameters.\n\n   - `index`: the index of source item\n   - `src_item`: the source item you are going to deal with in this function, which is usually an url or anything you need to process, such as a tuple like `(name, url)`.\n\n   ```python\n   def spi_func(index, src_item):\n       name, url = src_item\n       res = mspider.sess.get(url)\n       html = res.content.decode('utf-8')\n       # deal with the html\n       # save the extracted information\n   ```\n\n3. Now comes the key part. Create an instance of `MSpider` and pass it your spider function and sources you\u2019d crawl.\n\n   ```python\n   sources = [('github', 'http://www.github.com'),\n              ('baidu', 'http://www.baidu.com')]\n   mspider = MSpider(spi_func, sources)\n   ```\n\n4. Start to crawl!\n\n   ```python\n   mspider.crawl()\n   ```\n\n   Then you will see the following information in your terminal or cmd. You just input the BATCH SIZE, and then the MSpider will crawl your sources in a multi-threaded manner.\n\n   ```bash\n   [INFO]: MSpider is ready.\n   [INFO]: 2 urls in total.\n   [INPUT]: BATCH SIZE: 1\n   [INFO]: Open threads: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 2/2 [00:00<00:00, 356.36it/s]\n   [INFO]: Task done.\n   [INFO]: The task costs 1.1157 sec.\n   [INFO]: 0 urls failed.\n   ```\n\n## Usages\n\nThe `mspider` package has three main modules, `pp`, `mtd` and `spider`\n\n- `pp`  has a class of `ProxyPool`, which helps you get the proxy IP pool from xici free IPs.\n\n  **Note that there are few free IPs could work, so try not to use this module. If you\u2019d like to use proxy IP for your spider, this code may be helpful for you to write your own proxy pool.**\n\n- `mtd` has two classes, `Crawler` and `Downloader`\n\n  - `Crawler` helps you make your spider multi-threaded.\n  - `Downloader` helps you download things multi-threadedly as long as you pass your urls in the form of `list(zip(names, urls)) ` in it.\n\n- `spider` has the class of `MSpider`, which uses the `Crawler` in module `mtd`, and has some basic configurations of `Crawler`, so this is an easier way to turn your spider into a multi-threaded spider.\n\n### Usage of `pp.ProxyPool`\n\n```python\nfrom mspider.pp import ProxyPool\n\npool = ProxyPool()\n\n# Once an instance of ProxyPool is initialized,\n# it will has an attribute named ip_list, which\n# has a list of IPs crawled from xici free IPs.\nprint(pool.ip_list)\n\"\"\"\n{'http': ['HTTP://211.162.70.229:3128',\n          'HTTP://124.207.82.166:8008',\n          'HTTP://121.69.37.6:9797',\n          'HTTP://1.196.160.94:9999',\n          'HTTP://59.44.247.194:9797',\n          'HTTP://14.146.92.72:9797',\n          'HTTP://223.166.247.206:9000',\n          'HTTP://182.111.129.37:53281',\n          'HTTP://58.243.50.184:53281',\n          'HTTP://218.28.58.150:53281'],\n 'https': ['HTTPS://113.140.1.82:53281',\n           'HTTPS://14.23.58.58:443',\n           'HTTPS://122.136.212.132:53281']}\n\"\"\"\n# Randomly choose an IP\nprotocol = \"http\" # or \"https\"\nip = pool.random_choose_ip(protocol)\nprint(ip)\n\"\"\"\n'HTTP://59.44.247.194:9797'\n\"\"\"\n\n# Update the IP list\npool.get_ip_list()\npool.check_all_ip()\n\n# Request an url using proxy by 'GET'\nurl = \"http://www.google.com\"\nres = pool.open_url(url)\nprint(res.status_code)\n\"\"\"\n200\n\"\"\"\n\n# Request an url using post by 'POST'\nurl = \"http://www.google.com\"\ndata = {'key':'value'}\nres = pool.post(url, data)\nprint(res.status_code)\n\"\"\"\n200\n\"\"\"\n```\n\n### Usage of `mtd.Downloader`\n\n```python\nfrom mspider.mtd import Downloader\n\n# Prepare source data that need download\nnames = ['a', 'b', 'c']\nurls = ['https://www.baidu.com/img/baidu_resultlogo@2.png',\n        'https://www.baidu.com/img/baidu_resultlogo@2.png',\n        'https://www.baidu.com/img/baidu_resultlogo@2.png']\nsource = list(zip(names, urls))\n\n# Download them!\ndl = Downloader(source)\ndl.download(out_folder='test', engine='wget')\n\"\"\"\n[INFO]: 3 urls in total.\n[INPUT]: BATCH SIZE: 1\n[INFO]: Open threads: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 3/3 [00:00<00:00, 3167.90it/s]\n[INFO]: Task done.\n[INFO]: The task costs 0.3324 sec.\n[INFO]: 0 urls failed.\n\"\"\"\n```\n\n### Usage of `spider.MSpider`\n\nSee this in  [**Quick Start**](#quick-start).\n\n## Feature\n- v2.0.5:\n  - Add spider templates. One is based on `spider.MSpider`, the other is based on `mtd.Crawler`.\n  - Add the argument `batch_size` to `spider.MSpider` and `mtd.Crawler`.\n\n## License\n\nCopyright (c) 2019 tishacy.\n\nLicensed under the [MIT License](./LICENSE).\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/Tishacy/MSpider",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "mspider",
        "package_url": "https://pypi.org/project/mspider/",
        "platform": "",
        "project_url": "https://pypi.org/project/mspider/",
        "project_urls": {
            "Homepage": "https://github.com/Tishacy/MSpider"
        },
        "release_url": "https://pypi.org/project/mspider/0.2.5/",
        "requires_dist": [
            "numpy (>=1.14.4)",
            "requests (>=2.18.4)",
            "pandas (>=0.24.1)",
            "tqdm (>=4.30.0)",
            "wget (>=3.2)",
            "beautifulsoup4 (>=4.7.1)"
        ],
        "requires_python": "",
        "summary": "Make your spider multi-threaded.",
        "version": "0.2.5"
    },
    "last_serial": 5110173,
    "releases": {
        "0.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c4e8b2f74de4135701042f394f2ddc09",
                    "sha256": "eb80064a3894d3a1dcaf8f4f383abdd4d220f081bd48b9b2d17d762fb7754df5"
                },
                "downloads": -1,
                "filename": "mspider-0.2.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "c4e8b2f74de4135701042f394f2ddc09",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10091,
                "upload_time": "2019-03-22T09:13:04",
                "url": "https://files.pythonhosted.org/packages/cb/61/22dcfae91c8e388d0167d0267b923ad3c095137aee4dbe00a8d06c5d4ceb/mspider-0.2.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d00f0b9e409afa9d4211dd34895d8a1f",
                    "sha256": "1fed767401ac21ac342c35a55d3d68849acd00e7776e107f250c8393405a6c1f"
                },
                "downloads": -1,
                "filename": "mspider-0.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "d00f0b9e409afa9d4211dd34895d8a1f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7633,
                "upload_time": "2019-03-22T09:13:09",
                "url": "https://files.pythonhosted.org/packages/fb/06/14c7f20f3272eceb923434dce92215538fcab2700b85625a438dec5209c0/mspider-0.2.1.tar.gz"
            }
        ],
        "0.2.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fc1fa1d43a25b71e5e344a9935e40e8d",
                    "sha256": "9a400af43b09ea147ffe5e6768f4d145e423b31656e16cd032d2f3ff4f3a438f"
                },
                "downloads": -1,
                "filename": "mspider-0.2.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "fc1fa1d43a25b71e5e344a9935e40e8d",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10125,
                "upload_time": "2019-03-22T15:52:08",
                "url": "https://files.pythonhosted.org/packages/70/65/106e6420c4071e61e889db34b53debda143daed882b77d31b7342187d464/mspider-0.2.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "da3fd5c8cc1675957655083343cfb1c9",
                    "sha256": "f1daa698460f8ed14320c754086021b799e647ba7e2a911ec75d178044c5ab29"
                },
                "downloads": -1,
                "filename": "mspider-0.2.2.tar.gz",
                "has_sig": false,
                "md5_digest": "da3fd5c8cc1675957655083343cfb1c9",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7761,
                "upload_time": "2019-03-22T15:52:11",
                "url": "https://files.pythonhosted.org/packages/8f/a2/b8272c880862194b7df61ab277bb953f0c55ba6b5dd49739b960093c09e5/mspider-0.2.2.tar.gz"
            }
        ],
        "0.2.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "eae73c312cb0318e04351947aab986f8",
                    "sha256": "0f4e2188b46799a8d155184600faad07425f2d3471925ec2aad6b58097cf36e2"
                },
                "downloads": -1,
                "filename": "mspider-0.2.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "eae73c312cb0318e04351947aab986f8",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10900,
                "upload_time": "2019-03-25T14:09:35",
                "url": "https://files.pythonhosted.org/packages/e5/66/803ed99602a74df3cafdc58460e6705f61272bfdc736c8a42bd4552a6c24/mspider-0.2.3-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "02902f6296bd16ec351c0d1f94a6c061",
                    "sha256": "b6ddbd5d4f97e0255051ecf5f64b527c04de5ef4e273e0f88c8351e27e641d77"
                },
                "downloads": -1,
                "filename": "mspider-0.2.3.tar.gz",
                "has_sig": false,
                "md5_digest": "02902f6296bd16ec351c0d1f94a6c061",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8488,
                "upload_time": "2019-03-25T14:09:39",
                "url": "https://files.pythonhosted.org/packages/45/e8/2d6c93461b45697d8a33ea2bd22a02dd1d458e86d0bcba04737af68f0590/mspider-0.2.3.tar.gz"
            }
        ],
        "0.2.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "45c61b8a8daa50213667c4a2a4b9b04c",
                    "sha256": "b1a5a50d5cb1ff76b7b9aba9af39c163027c9ca1c53e9d6d1bdace3e40d1d7e6"
                },
                "downloads": -1,
                "filename": "mspider-0.2.5-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "45c61b8a8daa50213667c4a2a4b9b04c",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 11997,
                "upload_time": "2019-04-07T14:44:12",
                "url": "https://files.pythonhosted.org/packages/24/db/7601857b3071abb7f8b3fd2d21610f55480469c0b59bad25acebbc278390/mspider-0.2.5-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "277bc54299da484088bff7aebfa7b3ae",
                    "sha256": "5d9881caf957580668afe86470f18758426da04a3f7a1fce83db76b8a75f373e"
                },
                "downloads": -1,
                "filename": "mspider-0.2.5.tar.gz",
                "has_sig": false,
                "md5_digest": "277bc54299da484088bff7aebfa7b3ae",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9091,
                "upload_time": "2019-04-07T14:44:18",
                "url": "https://files.pythonhosted.org/packages/ce/3f/dd5e8d70b571d589f3efc79351053bef0db82ac46afe099926ede188fadf/mspider-0.2.5.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "45c61b8a8daa50213667c4a2a4b9b04c",
                "sha256": "b1a5a50d5cb1ff76b7b9aba9af39c163027c9ca1c53e9d6d1bdace3e40d1d7e6"
            },
            "downloads": -1,
            "filename": "mspider-0.2.5-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "45c61b8a8daa50213667c4a2a4b9b04c",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 11997,
            "upload_time": "2019-04-07T14:44:12",
            "url": "https://files.pythonhosted.org/packages/24/db/7601857b3071abb7f8b3fd2d21610f55480469c0b59bad25acebbc278390/mspider-0.2.5-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "277bc54299da484088bff7aebfa7b3ae",
                "sha256": "5d9881caf957580668afe86470f18758426da04a3f7a1fce83db76b8a75f373e"
            },
            "downloads": -1,
            "filename": "mspider-0.2.5.tar.gz",
            "has_sig": false,
            "md5_digest": "277bc54299da484088bff7aebfa7b3ae",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 9091,
            "upload_time": "2019-04-07T14:44:18",
            "url": "https://files.pythonhosted.org/packages/ce/3f/dd5e8d70b571d589f3efc79351053bef0db82ac46afe099926ede188fadf/mspider-0.2.5.tar.gz"
        }
    ]
}