{ "info": { "author": "sloft", "author_email": "nomail@example.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Environment :: Console", "Intended Audience :: Developers", "License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)", "Operating System :: OS Independent", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Topic :: Internet", "Topic :: Internet :: WWW/HTTP", "Topic :: Internet :: WWW/HTTP :: Browsers", "Topic :: Internet :: WWW/HTTP :: Indexing/Search", "Topic :: Internet :: WWW/HTTP :: Site Management", "Topic :: Internet :: WWW/HTTP :: Site Management :: Link Checking", "Topic :: Software Development :: Libraries", "Topic :: Software Development :: Libraries :: Python Modules" ], "description": "# Pynav2\n## Headless programmatic web browser on top of Requests and Beautiful Soup\n\n### Requirements\nPython 3.4+\n\nUnittest tested from Python 3.4 to 3.7\n\n### Installation\nIf python3 is the default python binary\n```bash\npip install pynav2\n```\nIf python2 is the default python binary\n```bash\npip3 install pynav2\n```\n### Licence\nGNU LGPLv3 (GNU Lesser General Public License Version 3)\n\n### Interactive mode examples\nRequired for all examples\n```python\nfrom pynav2 import Browser\nb = Browser()\n```\n\n#### HTTP GET request and print the response\nGet http://example.com (use https if available on server)\n```python\n>>> b.get('example.com')\n\n>>> b.text # alias for b.response.text\n'\\n\\n\\n\\nexample.com...'\n```\n\n#### HTTP GET request and print the json response\nGet http://example.com/user-agent/json wich return a the json-encoded content of a response if nay\n```python\n>>> b.get('example.com/user-agent/json')\n\n>>> b.json # alias for b.response.json()\n{'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0'}\n```\n\n#### HTTP POST request and print the response\n```python\n>>> data = {'q': 'python'}\n>>> b.post('example.com/search', data=data)\n\n>>> b.text\n'\\n\\n\\n\\nexample.com...'\n```\n\n#### HTTP POST json request and print the json response\n```python\n>>> import json\n>>> data = {'login': 'user', 'password': 'pass'}\n>>> b.post('example.com/login', json=json.dumps(data)) # json to send in the body of the request\n\n>>> b.json\n{'login': 'success'}\n```\n\n#### HTTP HEAD request and print response headers\n```python\n>>> b.head('example.com')\n\n>>> b.response.headers\n{'Server': 'nginx', 'Content-Type': 'text/html; charset=utf-8', 'Content-Length': '48842', 'Age': '3154', 'Connection': 'keep-alive'}\n```\n\n#### HTTP PUT request and print the json response\n```python\n>>> data = {'version': '2.1', 'licence': 'LGPL'}\n>>> b.put('example.com/api/about/', data=data)\n\n>>> b.json\n{'update': 'success'}\n```\n\n#### HTTP PATCH request and print the json response\n```python\n>>> data = {'version': '2.1'}\n>>> b.patch('example.com/api/about/', data=data)\n\n>>> b.json\n{'patch': 'success'}\n```\n\n#### HTTP DELETE request and print the json response\n```python\n>>> b.delete('example.com/api/user/102')\n\n>>> b.json\n{'delete': 'success'}\n```\n\n#### HTTP OPTIONS request and print the json response\n```python\n>>> b.options('example.com/api/user')\n\n>>> b.json\n{'options': '...'}\n```\n\n#### Get all links\n```python\n>>> b.get('example.com')\n\n>>> b.links\n['http://example.com/news', 'http://example.com/forum', 'http://example.com/contact']\n>>> for link in b.links:\n... print(link)\n...\nhttp://example.com/news\nhttp://example.com/forum\nhttp://example.com/contact\n\n```\n\n#### Filter links\nAny beautifulSoup.find_all() parameter can be added, see [Beautiful Soup documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n```python\n>>> import re\n>>> b.get('example.com')\n\n>>> b.get_links(text='Python Events') # regular expression\n>>> b.get_links(class_=\"jump-link\") # no regular expression for class attribute\n>>> b.get_links(href=\"windows\") # regular expression\n>>> b.get_links(title=re.compile('success')) # manual regular expression\n```\n\n#### Get all images\n```python\n>>> b.get('example.com')\n\n>>> b.images\n['http://example.com/img/logo.png', 'http://example.com/img/picture.jpg', 'http://there.com/news.gif']\n```\n\n#### Filter images\nAny beautifulSoup.find_all() parameter can be added, see [Beautiful Soup documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n```python\n>>> b.get('example.com')\n\n>>> b.get_images(src='logo') # regular expression\n>>> b.get_images(class_='python-logo') # no regular expression for class attribute\n>>> b.get_images(alt='yth') # regular expression\n```\n\n#### Download file\n```python\n>>> b.verbose=True\n>>> b.download('http://example.com/ubuntu-amd64', '/tmp') # it will follow redirect and look for headers content-disposition to find filename\ndownloading ubuntu-18.04.1-desktop-amd64.iso (1.8 GB) to: /tmp/ubuntu-18.04.1-desktop-amd64.iso\ndownload completed in 12 minutes 5 seconds (1.8 GB)\n\n```\n\n#### Handle referer\n```python\n>>> b.handle_referer = True\n>>> b.get('somewhere.com')\n>>> b.get('example.com') # request headers will have http://somewhere.com as referer\n>>> b.get('there.com') # request headers will have http://example.com as referer\n```\n\n#### Set referer manually \n```python\n>>> b.referer = 'http://www.here.com'\n>>> b.get('example.com') # request headers will have http://here.com as referer\n```\n\n#### Set user-agent \nuseragent module include a list of user-agents :\n\nfirefox_windows, chrome_windows, edge_windows, ie_windows, firefox_linux, chrome_linux, safari_mac\n\nDefault user-agent is firefox_windows \n```python\n>>> from pynav2 import useragent\n>>> b.user_agent = useragent.firefox_linux\n>>> b.get('example.com') # request headers will have 'Mozilla/5.0 (X11; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0' as User-Agent\n>>> b.user_agent = 'my_app/v1.0'\n>>> b.get('example.com') # request headers will have my_app/v1.0 as User-Agent \n```\n\n#### Set sleep time before a request \n```python\n>>> b.set_sleep_time(0.5, 1.5) # random x seconds between 0.5 to 1.5 seconds and wait x before each request\n>>> b.get('example.com') # wait x seconds before request\n```\n\n#### Define request timeout\n10 seconds timeout\n```python\n>>> b.timeout = 10\n```\n\n#### Close all opened TCP sessions\n```python\n>>> b.get('example1.com')\n>>> b.get('example2.com')\n>>> b.get('example3.com')\n>>> b.session.close()\n```\n\n#### Set HTTP proxy working with HTTPS request for one request\nFor SOCKS proxies see [Requests documentation](http://docs.python-requests.org/en/master/user/advanced/#socks)\n```python\n>>> b.get('https://httpbin.org/ip').json()['origin']\n111.111.111.111\n>>> proxies = {'https':'10.0.0.0:1234'}\n>>> b.timeout = 10 # could be useful to wait 10 seconds if proxies are slow\n>>> b.get('https://httpbin.org/ip', proxies=proxies).json()['origin']\n10.0.0.0\n```\n\n#### Set HTTP proxy working with HTTPS request for all requests\nFor SOCKS proxies see [Requests documentation](http://docs.python-requests.org/en/master/user/advanced/#socks)\n```python\n>>> b.get('https://httpbin.org/ip').json()['origin']\n111.111.111.111\n>>> b.proxies = {'https':'10.0.0.0:1234'}\n>>> b.timeout = 10 # could be useful to wait 10 seconds if proxies are slow\n>>> b.get('https://httpbin.org/ip').json()['origin']\n10.0.0.0\n```\n\n#### Set HTTP proxy working with HTTPS request for all request and another proxy for a specific domain\nFor SOCKS proxies see [Requests documentation](http://docs.python-requests.org/en/master/user/advanced/#socks)\n```python\n>>> b.get('https://httpbin.org/ip').json()['origin']\n111.111.111.111\n>>> b.proxies = {'https':'10.0.0.0:1234', 'https://specific-domain.com' : '10.11.12.13:1234'}\n>>> b.timeout = 10 # could be useful to wait 10 seconds if proxies are slow\n>>> b.get('https://httpbin.org/ip').json()['origin']\n10.0.0.0\n>>> b.get('https://specific-domain.com/ip').json()['origin']\n10.11.12.13\n```\n\n#### Get beautifulsoup instance\nAfter a get or post request, Browser.bs (beautifulsoup) is automatically initiated with b.response.text\n\nSee [Beautifll Soup documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) \n```python\n>>> b.get('example.com')\n>>> b.bs.find_all('a')\n```\n\n#### Get requests objects instances\n\nSee [Requests documentation](http://docs.python-requests.org/en/master/) \n```python\n>>> b.get('example.com')\n>>> b.session\n>>> b.request\n>>> b.response\n```\n\n#### Get browser history\n```python\n>>> b.get('example1.com')\n>>> b.get('example2.com')\n>>> b.get('example3.com')\n>>> print b.history\n['example1.com', 'example2.com', 'example3.com']\n```\n\n#### Disable \"InsecureRequestWarning: Unverified HTTPS request is being made\"\n```python\n>>> import urllib3\n>>> urllib3.disable_warnings()\n>>> b.get('example.com') # no warnings \n```\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "https://pypi.org/project/pynav2/#files", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/sloft/pynav2/", "keywords": "programmatic,web,browser", "license": "GNU Lesser General Public License Version 3 (LGPLv3)", "maintainer": "", "maintainer_email": "", "name": "pynav2", "package_url": "https://pypi.org/project/pynav2/", "platform": "", "project_url": "https://pypi.org/project/pynav2/", "project_urls": { "Download": "https://pypi.org/project/pynav2/#files", "Homepage": "https://github.com/sloft/pynav2/" }, "release_url": "https://pypi.org/project/pynav2/2.1/", "requires_dist": [ "requests", "beautifulsoup4" ], "requires_python": ">=3.4", "summary": "Headless programmatic web browser on top of Requests and Beautiful Soup", "version": "2.1" }, "last_serial": 4741957, "releases": { "2.0": [ { "comment_text": "", "digests": { "md5": "41f40226a9959a4e00ec1b938991f8b5", "sha256": "c46328a0e01edadd4dd88830875b532ccf1fbe26fa1b41fa81ad1cd3e07fcab7" }, "downloads": -1, "filename": "pynav2-2.0-py3-none-any.whl", "has_sig": false, "md5_digest": "41f40226a9959a4e00ec1b938991f8b5", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.4", "size": 10735, "upload_time": "2018-11-12T21:40:23", "url": "https://files.pythonhosted.org/packages/ac/80/a45077e31e381110b350d85ade0f21ad197b790538a48e39774e09d1bc8b/pynav2-2.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c083e6a4ac1068e565e51296df58012f", "sha256": "f16e6d1b86a8258dc8ef1a172ebc194387e6ef562e4ec0debffce0e57afdb347" }, "downloads": -1, "filename": "pynav2-2.0.tar.gz", "has_sig": false, "md5_digest": "c083e6a4ac1068e565e51296df58012f", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.4", "size": 8514, "upload_time": "2018-11-12T21:40:26", "url": "https://files.pythonhosted.org/packages/fc/22/903bde239801d46fca71db4b5af0144e3bc2a46ca347daefeef984007814/pynav2-2.0.tar.gz" } ], "2.1": [ { "comment_text": "", "digests": { "md5": "a757633dc00d99b388cfc32370e9d8cb", "sha256": "c8c3c5be9821280bc418a88ab036ee9e38ebee22ef50a5663c3f8f862f5ef5a0" }, "downloads": -1, "filename": "pynav2-2.1-py3-none-any.whl", "has_sig": false, "md5_digest": "a757633dc00d99b388cfc32370e9d8cb", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.4", "size": 11471, "upload_time": "2019-01-25T21:39:57", "url": "https://files.pythonhosted.org/packages/ba/e0/5be9087f82aae9a0ca988cef6b6c375c64a4da01ce118c0639d8a13cdd2e/pynav2-2.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "35dea6598ee3933c0a08833a5b03147b", "sha256": "403222943708fa79ecc4453dfe81ffa05ec841056295a581667e94d8045fc5c8" }, "downloads": -1, "filename": "pynav2-2.1.tar.gz", "has_sig": false, "md5_digest": "35dea6598ee3933c0a08833a5b03147b", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.4", "size": 12336, "upload_time": "2019-01-25T21:39:58", "url": "https://files.pythonhosted.org/packages/41/17/31b4fd7f4474911b9b5d0d50bad87d163539d35835e3c0f22ddcc75a21c4/pynav2-2.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "a757633dc00d99b388cfc32370e9d8cb", "sha256": "c8c3c5be9821280bc418a88ab036ee9e38ebee22ef50a5663c3f8f862f5ef5a0" }, "downloads": -1, "filename": "pynav2-2.1-py3-none-any.whl", "has_sig": false, "md5_digest": "a757633dc00d99b388cfc32370e9d8cb", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.4", "size": 11471, "upload_time": "2019-01-25T21:39:57", "url": "https://files.pythonhosted.org/packages/ba/e0/5be9087f82aae9a0ca988cef6b6c375c64a4da01ce118c0639d8a13cdd2e/pynav2-2.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "35dea6598ee3933c0a08833a5b03147b", "sha256": "403222943708fa79ecc4453dfe81ffa05ec841056295a581667e94d8045fc5c8" }, "downloads": -1, "filename": "pynav2-2.1.tar.gz", "has_sig": false, "md5_digest": "35dea6598ee3933c0a08833a5b03147b", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.4", "size": 12336, "upload_time": "2019-01-25T21:39:58", "url": "https://files.pythonhosted.org/packages/41/17/31b4fd7f4474911b9b5d0d50bad87d163539d35835e3c0f22ddcc75a21c4/pynav2-2.1.tar.gz" } ] }