{ "info": { "author": "helixcs", "author_email": "zhangjian12424@gmail.com", "bugtrack_url": null, "classifiers": [ "License :: OSI Approved :: MIT License", "Programming Language :: Python", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy" ], "description": "\n# Weibo Scraper\n\n[![PyPI](https://img.shields.io/pypi/v/weibo-scraper.svg)](https://pypi.org/project/weibo-scraper/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/weibo-scraper.svg)](https://docs.python.org/3/whatsnew/3.6.html)\n[![Build Status](https://travis-ci.org/Xarrow/weibo-scraper.svg?branch=master)](https://travis-ci.org/Xarrow/weibo-scraper)\n[![codecov](https://codecov.io/gh/Xarrow/weibo-scraper/branch/master/graph/badge.svg)](https://codecov.io/gh/Xarrow/weibo-scraper)\n\n----\n\nSimple weibo tweet scraper . Crawl weibo tweets without authorization.\nThere are many limitations in official API .\nIn general , we can inspect mobile site which has it's own API by Chrome.\n\n----\n\n# Why\n\n1. Crawl weibo data in order to research big data .\n\n2. Back up data for weibo's shameful blockade . \n\n\n----\n# Installation\n\n\n### pip\n\n```shell\n\n$ pip install weibo-scraper\n\n```\n\nOr Upgrade it.\n\n\n```shell\n\n$ pip install --upgrade weibo-scraper\n\n```\n\n### pipenv\n\n```shell\n\n$ pipenv install weibo-scraper\n\n```\nOr Upgrade it.\n\n```shell\n$ pipenv update --outdated # show packages which are outdated\n\n$ pipenv update weibo-scraper # just update weibo-scraper\n\n```\n\n\n\n\nOnly Python 3.6+ is supported\n\n----\n# Usage\n\n1. Firstly , you can get weibo profile by `name` or `uid` .\n\n```python\n>>> from weibo_scraper import get_weibo_profile\n>>> weibo_profile = get_weibo_profile(name='\u6765\u53bb\u4e4b\u95f4',)\n>>> ....\n```\nYou will get weibo profile response which is type of `weibo_base.UserMeta`, and this response include fields as below\n\nfield|chinese|type|sample|ext\n---|---|---|---|---\nid|\u7528\u6237id|str||\nscreen_name|\u5fae\u535a\u6635\u79f0|Option[str]||\navatar_hd|\u9ad8\u6e05\u5934\u50cf|Option[str]|'https://ww2.sinaimg.cn/orj480/4242e8adjw8elz58g3kyvj20c80c8myg.jpg'|\ncover_image_phone|\u624b\u673a\u7248\u5c01\u9762|Option[str]|'https://tva1.sinaimg.cn/crop.0.0.640.640.640/549d0121tw1egm1kjly3jj20hs0hsq4f.jpg'|\ndescription| \u63cf\u8ff0|Option[str]||\nfollow_count|\u5173\u6ce8\u6570|Option[int]|3568|\nfollower_count|\u88ab\u5173\u6ce8\u6570|Option[int]|794803|\ngender|\u6027\u522b|Option[str]|'m'/'f'|\nraw_user_response|\u539f\u59cb\u8fd4\u56de|Option[dict]||\n\n\n2. Secondly , via `tweet_container_id` to get weibo tweets is a rare way to use but it also works well .\n\n```python\n>>> from weibo_scraper import get_weibo_tweets\n>>> for tweet in get_weibo_tweets(tweet_container_id='1076033637346297',pages=1):\n>>> print(tweet)\n>>> ....\n\n```\n\n3. Of Course , you can also get raw weibo tweets by nick name which is exist . And the param of `pages` is optional .\n\n```python\n>>> from weibo_scraper import get_weibo_tweets_by_name\n>>> for tweet in get_weibo_tweets_by_name(name='\u563b\u7ea2\u8c46', pages=1):\n>>> print(tweet)\n>>> ....\n```\n\n3. If you want to get all tweets , you can set the param of `pages` as `None`\n\n```python\n>>> from weibo_scraper import get_weibo_tweets_by_name\n>>> for tweet in get_weibo_tweets_by_name(name='\u563b\u7ea2\u8c46', pages=None):\n>>> print(tweet)\n>>> ....\n```\n\n4. There is a giant update since 1.0.5 \ud83c\udf70!\n\nYou can also get formatted tweets via api of `weibo_scrapy.get_formatted_weibo_tweets_by_name`,\n\n```python\n>>> from weibo_scraper import get_formatted_weibo_tweets_by_name\n>>> result_iterator = get_formatted_weibo_tweets_by_name(name='\u563b\u7ea2\u8c46', pages=None)\n>>> for user_meta in result_iterator:\n>>> for tweetMeta in user_meta.cards_node:\n>>> print(tweetMeta.mblog.text)\n>>> ....\n```\n\n\n![img](https://raw.githubusercontent.com/Xarrow/weibo-scraper/master/weibo_tweets.png)\n\n----\n\n# Weibo Flasgger\n\n\n[Weibo Flasgger](https://github.com/Xarrow/weibo-scraper/blob/search_name/samples/weibo_flasgger/FLASGGER_README.md) is a web api document for weibo scraper , and powered by flasgger .\n\n![img](https://raw.githubusercontent.com/rochacbruno/flasgger/master/docs/flasgger.png)\n\n# P.S\n1. Inspiration from [Twitter-Scraper](https://github.com/kennethreitz/twitter-scraper) .\n\n2. For 'XIHONGDOU' .\n\n3. Welcome To Fork Me .\n\n----\n# LICENSE\n\nMIT\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/Xarrow/weibo-scraper", "keywords": "weibo scraper crawl", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "weibo-scraper", "package_url": "https://pypi.org/project/weibo-scraper/", "platform": "", "project_url": "https://pypi.org/project/weibo-scraper/", "project_urls": { "Homepage": "https://github.com/Xarrow/weibo-scraper" }, "release_url": "https://pypi.org/project/weibo-scraper/1.0.6/", "requires_dist": [ "requests" ], "requires_python": ">=3.6", "summary": "", "version": "1.0.6" }, "last_serial": 4614995, "releases": { "1.0.2": [ { "comment_text": "", "digests": { "md5": "e6496cf6e774bae8fdb78e2b4085a207", "sha256": "6786fedf215b3709c2d4fc023987a4a610c9d228e46d6853705ebb4d20420386" }, "downloads": -1, "filename": "weibo_scraper-1.0.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "e6496cf6e774bae8fdb78e2b4085a207", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": ">=3.6", "size": 2791, "upload_time": "2018-05-10T19:55:12", "url": "https://files.pythonhosted.org/packages/d5/4c/af7faca3101fbca2b21828ad334a3d919c5059f7c4f7e8c80424f5889120/weibo_scraper-1.0.2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "1280370d3208a4ebf1d5ad1a328de649", "sha256": "e6f448202bfa30173a3da925a6a420e5b345d20c26d9d56dfc99dd7c43200e9b" }, "downloads": -1, "filename": "weibo-scraper-1.0.2.tar.gz", "has_sig": false, "md5_digest": "1280370d3208a4ebf1d5ad1a328de649", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 2993, "upload_time": "2018-05-10T19:55:16", "url": "https://files.pythonhosted.org/packages/91/cf/ad0a93e3ff6a0e6f07995170280cae067eb2547d98649999f23d60533b73/weibo-scraper-1.0.2.tar.gz" } ], "1.0.4": [ { "comment_text": "", "digests": { "md5": "34f1732cd24156e516583ce9136b7196", "sha256": "c20a239709b17bf26b61577f47daace9351871c4a378c18f4e834d400376fba5" }, "downloads": -1, "filename": "weibo_scraper-1.0.4-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "34f1732cd24156e516583ce9136b7196", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": ">=3.6", "size": 10172, "upload_time": "2018-05-21T18:20:58", "url": "https://files.pythonhosted.org/packages/b3/f0/747251a8403d3302fbb3202a58f95f00c1c6df6567798bbfc0a75d781e9f/weibo_scraper-1.0.4-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "f638c7a91b2c6c992a3b053e12ed66ff", "sha256": "d0ae4645d15c8c5e2a4367b368845d80bf30a9f354f3ca1e6f0d153f180ef121" }, "downloads": -1, "filename": "weibo-scraper-1.0.4.tar.gz", "has_sig": false, "md5_digest": "f638c7a91b2c6c992a3b053e12ed66ff", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 13930, "upload_time": "2018-05-21T18:21:07", "url": "https://files.pythonhosted.org/packages/9f/bf/7a807e512d3809c9a54189774be64e9623cd08783b5a1cfddb7b46266da2/weibo-scraper-1.0.4.tar.gz" } ], "1.0.6": [ { "comment_text": "", "digests": { "md5": "a638d292600fc5c245615fc3aab97492", "sha256": "b2cf9a7b0d88e32df97cb83928c72e50b079ba9a72c387976c3a4f4b337e6653" }, "downloads": -1, "filename": "weibo_scraper-1.0.6-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "a638d292600fc5c245615fc3aab97492", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": ">=3.6", "size": 8498, "upload_time": "2018-06-09T12:16:54", "url": "https://files.pythonhosted.org/packages/ea/fc/bb8fb49af5ccc52e00cac26bfa270d20920fc4b8575580863e468d0afc3e/weibo_scraper-1.0.6-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "745738a5581fff1d1e1fe762616e265d", "sha256": "02ee046aa163cd9c35eb92f1dd03adaccee3902bc17bd6856de9a828d958a6c7" }, "downloads": -1, "filename": "weibo-scraper-1.0.6.tar.gz", "has_sig": false, "md5_digest": "745738a5581fff1d1e1fe762616e265d", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 7847, "upload_time": "2018-06-09T12:16:55", "url": "https://files.pythonhosted.org/packages/47/c0/77ea5373f5e404f0234d523c6cda1354f7537bf04d7e2131d1a51dcdfda3/weibo-scraper-1.0.6.tar.gz" } ], "1.0.7b0": [ { "comment_text": "", "digests": { "md5": "df13c267bc09413447ce824626b1167b", "sha256": "e703f23286fb8bd440ac4c75e783fe5e081292e0aa82f8d82d14af5ed9c2168e" }, "downloads": -1, "filename": "weibo_scraper-1.0.7b0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "df13c267bc09413447ce824626b1167b", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": ">=3.6", "size": 22037, "upload_time": "2018-12-19T04:05:51", "url": "https://files.pythonhosted.org/packages/22/9b/39b263f13fb5cb6defa5aa7c93809aeea7bfcbf2baf081170bad7b414ca9/weibo_scraper-1.0.7b0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "e1934798e7855dc2a581cd881f550934", "sha256": "aceedbacd810d6b14b03047cf623a6aae6c9f57989681954f04b0f45107766b6" }, "downloads": -1, "filename": "weibo-scraper-1.0.7b0.tar.gz", "has_sig": false, "md5_digest": "e1934798e7855dc2a581cd881f550934", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 19164, "upload_time": "2018-12-19T04:05:53", "url": "https://files.pythonhosted.org/packages/5b/fe/206e8a30cbe1e7b9f8f9c0f731679e78a748cf86b590d6ea40898d63ae8e/weibo-scraper-1.0.7b0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "a638d292600fc5c245615fc3aab97492", "sha256": "b2cf9a7b0d88e32df97cb83928c72e50b079ba9a72c387976c3a4f4b337e6653" }, "downloads": -1, "filename": "weibo_scraper-1.0.6-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "a638d292600fc5c245615fc3aab97492", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": ">=3.6", "size": 8498, "upload_time": "2018-06-09T12:16:54", "url": "https://files.pythonhosted.org/packages/ea/fc/bb8fb49af5ccc52e00cac26bfa270d20920fc4b8575580863e468d0afc3e/weibo_scraper-1.0.6-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "745738a5581fff1d1e1fe762616e265d", "sha256": "02ee046aa163cd9c35eb92f1dd03adaccee3902bc17bd6856de9a828d958a6c7" }, "downloads": -1, "filename": "weibo-scraper-1.0.6.tar.gz", "has_sig": false, "md5_digest": "745738a5581fff1d1e1fe762616e265d", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 7847, "upload_time": "2018-06-09T12:16:55", "url": "https://files.pythonhosted.org/packages/47/c0/77ea5373f5e404f0234d523c6cda1354f7537bf04d7e2131d1a51dcdfda3/weibo-scraper-1.0.6.tar.gz" } ] }