{ "info": { "author": "Liangrui He", "author_email": "i@nyankosama.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Environment :: Console", "Environment :: Web Environment", "Intended Audience :: Developers", "Intended Audience :: End Users/Desktop", "License :: OSI Approved :: MIT License", "Natural Language :: English", "Operating System :: OS Independent", "Programming Language :: Python :: 2.7", "Topic :: Internet", "Topic :: Internet :: WWW/HTTP", "Topic :: System :: Networking", "Topic :: Utilities" ], "description": "crawl-me\n========\n\n`\u4e2d\u6587README `_.\n\nCrawl-me is a light-weight fast plugin based web picture crawler. You\ncan download your favorite pictures via the plugin if the website is\nsupported. For now, the plugins include gamersky and pixiv. If you want\nto contribute, please just feel free to contact with me.\n\nFork me on Github :)\n`https://github.com/nyankosama/crawl-me `_\n\nFeatures\n========\n\n- Crawl-me core supports muti-thread downloading using http\n range-headers, so it's very fast.\n- It's plugin based, so you can free add any plugin you want.\n\nAvailable plugins\n=================\n\n- pixiv : This plugin allows you to download any author's paintings in\n `pixiv `_ site.\n- gamersky : This plugin supports downloading all pictures in special\n topic from `gamersky `_ site.\n\nInstallation\n============\n\ninstall via pip\n---------------\n\nMake sure you have already installed\n`python2.7 `_ and\n`pip `_.\n\nDue to the fact that package relies on lxml, if your platform is linux,\nplease make sure you have installed lib libxslt-devel libxml2-devel. And\nfor windows please select a suitable `lxml\ninstaller `_ to\ninstall.\n\nAnd then:\n\n::\n\n $ pip install crawl-me\n\nFor windows, please add {$python-home}/Scripts/ to systempath\n\ninstall via git\n---------------\n\n1. Ubuntu\n~~~~~~~~~\n\nInstall the prerequisite library first:\n\n::\n\n sudo apt-get install libxml2-dev\n sudo apt-get install libxslt1-dev \n\nAnd then you should install\n`setuptools `_ in\norder to run the setup.py file\n\n::\n\n sudo apt-get install python-setuptools\n\nFinally, git clone the source, and install:\n\n::\n\n $ git clone https://github.com/nyankosama/crawl-me.git\n $ cd crawl-me/\n $ sudo python setup.py install\n\n2. Windows\n~~~~~~~~~~\n\nMake sure you have already installed\n`python2.7 `_ and\n`pip `_\n\nYou can install python2.7 via windows installer. You can install pip via\ndownloading the `get-pip.py `_,\nand run it via python:\n\n::\n\n python get-pip.py\n\nAnd then install the prerequisite library lxml. please select a suitable\n`lxml installer `_ to\ninstall.\n\nFinally git clone the source, and install:\n\n::\n\n $ git clone https://github.com/nyankosama/crawl-me.git\n $ cd crawl-me/\n $ sudo python setup.py install\n\nFor windows, please add {$python-home}/Scripts/ to systempath\n\nUsage\n=====\n\nExamples\n--------\n\n1. Download 10 pages pictures at the url of\n http://www.gamersky.com/ent/201404/352055.shtml in gamersky site, and\n store the pictures into local direcotry.\n\n ::\n\n crawl-me gamersky http://www.gamersky.com/ent/201404/352055.shtml ./gamersky-crawl 1 10\n\n2. Download all the paintings of \u85e4\u539f(Fujiwara, Pixiv ID=27517), and\n store them into local directory.\n\n ::\n\n crawl-me pixiv 27517 ./pixiv-crawl \n\nCommand line options\n--------------------\n\n1. general help\n\n ::\n\n $ crawl-me -h \n\n usage: crawl-me [-h] plugin\n\n positional arguments:\n plugin plugin the crawler uses\n\n optional arguments:\n -h, --help show this help message and exit\n\n available plugins:\n ----gamersky\n ----pixiv\n\n2. gamersky\n\n ::\n\n $ crawl-me gamersky -h\n\n usage: crawl-me [-h] plugin url savePath beginPage endPage\n\n positional arguments:\n plugin plugin the crawler uses\n url your url to crawl\n savePath the path where the imgs ars saved\n beginPage the page where we start crawling\n endPage the page where we end crawling \n\n optional arguments:\n -h, --help show this help message and exit\n\n3. pixiv\n\n ::\n\n $ crawl-me pixiv -h\n\n usage: crawl-me [-h] plugin authorId savePath pixivId password\n\n positional arguments:\n plugin plugin the crawler uses\n authorId the author id you want to crawl\n savePath the path where the imgs ars saved\n pixivId your pixiv login id\n password your pixiv login password\n\n optional arguments:\n -h, --help show this help message and exit\n\nTODO\n====\n\n- Functions:\n\n - support breakpoint resume\n\n- Plugins:\n\n - weibo\n - qq zone\n\nLicenses\n========\n\n`MIT `_\n\n\nChangeLog\n=========\n\n0.1.8\n-----\n\n*Date: 2014-06-15*\n\n- add English README\n\n0.1.8dev-20140615\n-----------------\n\n*Date: 2014-06-15*\n\n- bug fix:-v --version option load project.json fail\n\n0.1.8dev-20140612\n-----------------\n\n*Date: 2014-06-12*\n\n- add -v --version option for main runnable file to show the package\n version\n\n0.1.7\n-----\n\n*Date: 2014-06-11*\n\n- add the http range header support auto-check\n\n0.1.6\n-----\n\n*Date: 2014-06-11*\n\n- bug fix: terminal without colour doesnt display syslog prefix\n\n0.1.5\n-----\n\n*Date: 2014-06-11*\n\n- bug fix:pip install bug in windows platform\n\n0.1.5dev-20140611\n-----------------\n\n*Date: 2014-06-11*\n\n- bug fix:pypi data\\_files\n\n0.1.4\n-----\n\n*Date: 2014-06-11*\n\n- the latest release\n\n0.1.4dev-20140611\n-----------------\n\n*Date: 2014-06-11*\n\n- modify the README.md. Now it is consistent with rst format to display\n on pypi\n\n0.1.4dev-20140610\n-----------------\n\n*Date: 2014-06-10*\n\n- add support for installing from pip\n\n0.1.4dev3\n---------\n\n*Date: 2014-06-10*\n\n- bug fix\n\n- fix the binary write problem in windows platform\n\n0.1.4dev2\n---------\n\n*Date: 2014-06-10*\n\n- add setuptools install support\n\n0.1.4dev1\n---------\n\n*Date: 2014-06-09*\n\n- bug fix\n\n- rangedownloader:http range-headers may not be supported\n\n0.1.3\n-----\n\n*Date: 2014-06-07*\n\n- do some refactory\n\n- add conf dictionary\n\n0.1.2\n-----\n\n*Date: 2014-06-06*\n\n- add plugin\n\n- pixiv\n\n0.1.1\n-----\n\n*Date: 2014-06-05*\n\n- add plugin\n\n- gamersky\n\n0.0.1\n-----\n\n*Date: 2014-06-05*\n\n- init the project", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/nyankosama/crawl-me", "keywords": "crawler downloader picture pixiv", "license": "MIT", "maintainer": null, "maintainer_email": null, "name": "crawl-me", "package_url": "https://pypi.org/project/crawl-me/", "platform": "any", "project_url": "https://pypi.org/project/crawl-me/", "project_urls": { "Download": "UNKNOWN", "Homepage": "https://github.com/nyankosama/crawl-me" }, "release_url": "https://pypi.org/project/crawl-me/0.1.8/", "requires_dist": null, "requires_python": null, "summary": "This is a plugin based picture crawler. Now pixiv/gamersky plugin available!", "version": "0.1.8" }, "last_serial": 1130578, "releases": { "0.1.4": [ { "comment_text": "", "digests": { "md5": "10127970908e2d66c11375fac89348e9", "sha256": "f266ff1c0f7eb05f7dbaf2a9ff13b2754583652a3c98ca06c50898b1a1819ec0" }, "downloads": -1, "filename": "crawl-me-0.1.4.tar.gz", "has_sig": false, "md5_digest": "10127970908e2d66c11375fac89348e9", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11260, "upload_time": "2014-06-10T16:14:29", "url": "https://files.pythonhosted.org/packages/15/af/66bbffffa781119756250e21a4925ace771d2c62606b2a3d773a19980a35/crawl-me-0.1.4.tar.gz" } ], "0.1.4a1": [ { "comment_text": "", "digests": { "md5": "a500120a4e438980a0ecd745fda8e595", "sha256": "8ecded16b1913d3dbdfd599f8b4e5738f4c15d159b0156a34cf828005821a7f3" }, "downloads": -1, "filename": "crawl-me-0.1.4a1.tar.gz", "has_sig": false, "md5_digest": "a500120a4e438980a0ecd745fda8e595", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11285, "upload_time": "2014-06-10T16:28:56", "url": "https://files.pythonhosted.org/packages/2b/26/3d402a344bbec1204eae8791445154a39863982ef955426013af4661dd26/crawl-me-0.1.4a1.tar.gz" } ], "0.1.4a2": [ { "comment_text": "built for Linux-3.8.0-29-generic-x86_64-with-glibc2.4", "digests": { "md5": "30af60265caed7cfd2932231260b1b66", "sha256": "9a71faa425cf48ec310d7f2525039d4afa89711b4286f25f190fcdca35899bab" }, "downloads": -1, "filename": "crawl-me-0.1.4a2.linux-x86_64.tar.gz", "has_sig": false, "md5_digest": "30af60265caed7cfd2932231260b1b66", "packagetype": "bdist_dumb", "python_version": "any", "requires_python": null, "size": 21835, "upload_time": "2014-06-10T16:54:05", "url": "https://files.pythonhosted.org/packages/50/40/888279211b07335c28f8389ac2d25ccf3b5cb23d30838d8c7016b1f80864/crawl-me-0.1.4a2.linux-x86_64.tar.gz" }, { "comment_text": "", "digests": { "md5": "7ca925862752897e6a453a63aabb20e7", "sha256": "b9fec4854fbac61771523a9942a9fa90bfd7837f2536b1d46f70300184debb1d" }, "downloads": -1, "filename": "crawl-me-0.1.4a2.tar.gz", "has_sig": false, "md5_digest": "7ca925862752897e6a453a63aabb20e7", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11291, "upload_time": "2014-06-10T16:54:02", "url": "https://files.pythonhosted.org/packages/0b/c5/3584b041fc02fcf68846eeab1e93d510871fcd8de05a4b0a2b42cfb6bbc7/crawl-me-0.1.4a2.tar.gz" } ], "0.1.4a3": [ { "comment_text": "", "digests": { "md5": "211343a90788707c2c380bc39e94f79c", "sha256": "d337c7940815cc2413a9706c0ad4ca4caf29b557c71c312243397697820ac107" }, "downloads": -1, "filename": "crawl-me-0.1.4a3.tar.gz", "has_sig": false, "md5_digest": "211343a90788707c2c380bc39e94f79c", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13607, "upload_time": "2014-06-10T17:00:34", "url": "https://files.pythonhosted.org/packages/c3/3d/3a52570ede5a22e50f3839690c863bfaf546346f2ed742d70632caa71e1e/crawl-me-0.1.4a3.tar.gz" } ], "0.1.4dev-20140610": [ { "comment_text": "", "digests": { "md5": "5f87d49221e90e98ab96912a0f0aae73", "sha256": "fa42f08e65daccdebb39864e08f68fb35548f6a85a704fa2ecce66af13566329" }, "downloads": -1, "filename": "crawl-me-0.1.4dev-20140610.tar.gz", "has_sig": false, "md5_digest": "5f87d49221e90e98ab96912a0f0aae73", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11366, "upload_time": "2014-06-10T14:46:18", "url": "https://files.pythonhosted.org/packages/dd/0e/8870c3b27a1ea8e1e5cc07464f72c7f9f77893869555d0b480d4e4ae5f00/crawl-me-0.1.4dev-20140610.tar.gz" } ], "0.1.4dev-20140611": [ { "comment_text": "", "digests": { "md5": "6db1d43fe2253d974694bdf256ea374d", "sha256": "233afbf28103a22d3bde0b77e2ed33afdb698711b2f493f56a1aa7c1f29f8bbb" }, "downloads": -1, "filename": "crawl-me-0.1.4dev-20140611.tar.gz", "has_sig": false, "md5_digest": "6db1d43fe2253d974694bdf256ea374d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11250, "upload_time": "2014-06-10T16:08:41", "url": "https://files.pythonhosted.org/packages/ae/31/067b0d41eb27a63fd6a62df00543885aeaa56aa9528bd4f498115b0c0ce5/crawl-me-0.1.4dev-20140611.tar.gz" } ], "0.1.5": [ { "comment_text": "", "digests": { "md5": "a236d61443f3acf77b30e4ff7b51a89a", "sha256": "388a571b4e8cef9132e37b01887c2f62f0cb50fb0f11114b761e3bea655004ed" }, "downloads": -1, "filename": "crawl-me-0.1.5.tar.gz", "has_sig": false, "md5_digest": "a236d61443f3acf77b30e4ff7b51a89a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11820, "upload_time": "2014-06-11T01:59:27", "url": "https://files.pythonhosted.org/packages/10/af/5b95fee41de8613de2844514c177a05b4dca8fc8191da9d38d51e2c8a5bf/crawl-me-0.1.5.tar.gz" } ], "0.1.5dev-20140611": [ { "comment_text": "", "digests": { "md5": "09e9096cdbf343bac8681665b69687a5", "sha256": "265697cf95bf24c068486e96bd6059548329b4df9b05d40682f7704ed17e9d66" }, "downloads": -1, "filename": "crawl-me-0.1.5dev-20140611.tar.gz", "has_sig": false, "md5_digest": "09e9096cdbf343bac8681665b69687a5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13616, "upload_time": "2014-06-10T17:04:15", "url": "https://files.pythonhosted.org/packages/85/38/9e3f0f15c8ddaf18aefa90ebf08568424480299756020e1e91ea597d66c5/crawl-me-0.1.5dev-20140611.tar.gz" } ], "0.1.5dev-20140611-1": [ { "comment_text": "", "digests": { "md5": "3a02e59d13ed532d303b2a2d86c7baec", "sha256": "d55c33c855205c6a9dbb3c67cd2c13ed7a99fbef3ea369f40c32a726a0cb5f14" }, "downloads": -1, "filename": "crawl-me-0.1.5dev-20140611-1.tar.gz", "has_sig": false, "md5_digest": "3a02e59d13ed532d303b2a2d86c7baec", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11806, "upload_time": "2014-06-11T01:50:20", "url": "https://files.pythonhosted.org/packages/22/92/61ed9ab42ad943551638c2a3b90f396014313b1e9bf1f15266da9e8a3994/crawl-me-0.1.5dev-20140611-1.tar.gz" } ], "0.1.6": [ { "comment_text": "", "digests": { "md5": "cd5db0bf903529c1195ca9f37365f6e9", "sha256": "5baf0cfc8483e9430779f95264bb093e791f5a72368cbf674cfbc0d36cb6364b" }, "downloads": -1, "filename": "crawl-me-0.1.6.tar.gz", "has_sig": false, "md5_digest": "cd5db0bf903529c1195ca9f37365f6e9", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12028, "upload_time": "2014-06-11T13:02:55", "url": "https://files.pythonhosted.org/packages/cb/d1/63bad8278813fc718cf6e47f025808dd07ac3f604c136eeab87fbaa85aba/crawl-me-0.1.6.tar.gz" } ], "0.1.7": [ { "comment_text": "", "digests": { "md5": "3ae5281ac5d162b63523fc40caacf768", "sha256": "97161660e33b1328356cc59960263876c22b489d7ae7650cafd0f1a29513e569" }, "downloads": -1, "filename": "crawl-me-0.1.7.tar.gz", "has_sig": false, "md5_digest": "3ae5281ac5d162b63523fc40caacf768", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12231, "upload_time": "2014-06-11T13:34:14", "url": "https://files.pythonhosted.org/packages/7a/b4/04d18b73ab29936c562985ffa92f2591c4d91ab0b4adba2d16f6a6d0b0ca/crawl-me-0.1.7.tar.gz" } ], "0.1.8": [ { "comment_text": "", "digests": { "md5": "af3580ee755abc5fa30a76d6cdaed2c0", "sha256": "dcff48b08c34e67fb2a27a3abfbefb06e943fb9be0deda613e0b9b323f1d99e4" }, "downloads": -1, "filename": "crawl-me-0.1.8.tar.gz", "has_sig": false, "md5_digest": "af3580ee755abc5fa30a76d6cdaed2c0", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11940, "upload_time": "2014-06-15T08:54:05", "url": "https://files.pythonhosted.org/packages/0c/89/bbc708c8443446b5c0d87475889e0fba6f02549385842f89bd69006cf6b2/crawl-me-0.1.8.tar.gz" } ], "0.1.8dev-20140612": [ { "comment_text": "", "digests": { "md5": "696fc2b90d39a7d2992c5b43ed92ca21", "sha256": "90452567f6193e314fd7bf729a1eaedc8c2b13ac1e9a9d0ab1338dd2db1952bf" }, "downloads": -1, "filename": "crawl-me-0.1.8dev-20140612.tar.gz", "has_sig": false, "md5_digest": "696fc2b90d39a7d2992c5b43ed92ca21", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12383, "upload_time": "2014-06-12T14:10:00", "url": "https://files.pythonhosted.org/packages/d8/75/981320e08dee8f3035ef5dd08256af72db3ab0c17a07c6f9f682a8ac132c/crawl-me-0.1.8dev-20140612.tar.gz" } ], "0.1.8dev-20140615": [ { "comment_text": "", "digests": { "md5": "22790f2b6cbd88f76459e4871eef0bb8", "sha256": "a1ab2afceb7056aef243e7152ddd247bce593511f45ea230d309ca465e2a02ae" }, "downloads": -1, "filename": "crawl-me-0.1.8dev-20140615.tar.gz", "has_sig": false, "md5_digest": "22790f2b6cbd88f76459e4871eef0bb8", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12331, "upload_time": "2014-06-15T07:53:47", "url": "https://files.pythonhosted.org/packages/b8/6b/f49c0a6b4040f2f5104a8c71bf632761f42f3016443dad4049540bb6dc27/crawl-me-0.1.8dev-20140615.tar.gz" } ], "0.1.8dev-20140615-1": [ { "comment_text": "", "digests": { "md5": "3c7fafa0938aa4b7b93ed847eb2ef215", "sha256": "2da424adf6cc2120bce2b983935134f128c93def6c70819c027ebb1324b22ccd" }, "downloads": -1, "filename": "crawl-me-0.1.8dev-20140615-1.tar.gz", "has_sig": false, "md5_digest": "3c7fafa0938aa4b7b93ed847eb2ef215", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11946, "upload_time": "2014-06-15T08:51:15", "url": "https://files.pythonhosted.org/packages/84/44/08dbdb1c0e04448ab5787f79f3f690eda146ea947ded6d8e7e2d008ffd02/crawl-me-0.1.8dev-20140615-1.tar.gz" } ], "0.1.9dev-20140617-1": [ { "comment_text": "", "digests": { "md5": "5a4cba468ee0b3656222aac6e9b86d0d", "sha256": "5ba607bfc10dd6e55d3c706af2b676b63fb8f4aaa5cc09407b27a32102369520" }, "downloads": -1, "filename": "crawl-me-0.1.9dev-20140617-1.tar.gz", "has_sig": false, "md5_digest": "5a4cba468ee0b3656222aac6e9b86d0d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12156, "upload_time": "2014-06-17T14:11:59", "url": "https://files.pythonhosted.org/packages/af/60/796e4352dc7b6f2f22c128cfea58ef5e02dc3b032d078d5c3bebbe483fac/crawl-me-0.1.9dev-20140617-1.tar.gz" } ], "0.1.9dev-20140619-1": [ { "comment_text": "", "digests": { "md5": "2a6412252d5245e8ffcd3688d5d004a9", "sha256": "895980cc1db5c06271e5a98756b4713fbd30539c6b705031f6449275d6be2085" }, "downloads": -1, "filename": "crawl-me-0.1.9dev-20140619-1.tar.gz", "has_sig": false, "md5_digest": "2a6412252d5245e8ffcd3688d5d004a9", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12229, "upload_time": "2014-06-19T08:24:00", "url": "https://files.pythonhosted.org/packages/0b/92/34212e27b38b2ae4bccf57ad999c966980147a73d0566dc382420419f4e6/crawl-me-0.1.9dev-20140619-1.tar.gz" } ], "0.1.9dev-20140619-2": [ { "comment_text": "", "digests": { "md5": "0ee32f380acaead0e2d20cf35c210fe2", "sha256": "1c170a7feaa36fc926565d066e8b1782cd40b1b713603bf334efe5b005f204c7" }, "downloads": -1, "filename": "crawl-me-0.1.9dev-20140619-2.tar.gz", "has_sig": false, "md5_digest": "0ee32f380acaead0e2d20cf35c210fe2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12241, "upload_time": "2014-06-19T09:07:30", "url": "https://files.pythonhosted.org/packages/39/5a/1d4efe625df4ec2eb38002abd2c2cc9d30d269b3a2fb03de5a5785921084/crawl-me-0.1.9dev-20140619-2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "af3580ee755abc5fa30a76d6cdaed2c0", "sha256": "dcff48b08c34e67fb2a27a3abfbefb06e943fb9be0deda613e0b9b323f1d99e4" }, "downloads": -1, "filename": "crawl-me-0.1.8.tar.gz", "has_sig": false, "md5_digest": "af3580ee755abc5fa30a76d6cdaed2c0", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11940, "upload_time": "2014-06-15T08:54:05", "url": "https://files.pythonhosted.org/packages/0c/89/bbc708c8443446b5c0d87475889e0fba6f02549385842f89bd69006cf6b2/crawl-me-0.1.8.tar.gz" } ] }