{ "info": { "author": "James Mills", "author_email": "James Mills, j dot mills at griffith dot edu dot au", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Environment :: Console", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Natural Language :: English", "Operating System :: MacOS :: MacOS X", "Operating System :: POSIX :: Linux", "Programming Language :: Python :: 2.7", "Topic :: Internet :: WWW/HTTP :: Indexing/Search" ], "description": ".. _Python Programming Language: http://www.python.org/\n.. _Python Standard Library: http://docs.python.org/library/\n.. _restclient: http://pypi.python.org/pypi/restclient\n.. _cssselect: http://pypi.python.org/pypi/cssselect\n.. _lxml: http://pypi.python.org/pypi/lxml/3.0.2\n.. _url: http://pypi.python.org/pypi/url\n.. _nltk: https://pypi.python.org/pypi/nltk\n.. _calais: https://bitbucket.org/prologic/calais\n.. _BeautifulSoup: https://pypi.python.org/pypi/BeautifulSoup\n.. _Griffith University: http://www.griffith.edu.au/\n.. _Project Website: http://bitbucket.org/prologic/spyda\n.. _PyPi Page: http://pypi.python.org/pypi/spyda\n.. _Read the Docs: http://spyda.readthedocs.org/en/latest/\n.. _Downloads Page: https://bitbucket.org/prologic/spyda/downloads\n\n\nspyda is a simple tool and library written in the `Python Programming Language`_ to crawl a given url whilst allowing you to restrict results to a specified\ndomain and optionally also perform pattern matching against URLs crawled. spyda will report on any URLs it was unable to crawl along with their status code\nand store successfully crawled links and their content in a directory structure that matches the domain and URLs searched.\n\nspyda was developed at `Griffith University`_ as a tool and library to assist with web crawling tasks and data extraction and has been used to help\nmatch researcher names against publications as well as extract data and links from external sources of data.\n\n- Visit the `Project Website`_\n- `Read the Docs`_\n- Download it from the `Downloads Page`_\n\n.. image:: https://pypip.in/v/spyda/badge.png\n :target: https://crate.io/packages/spyda/\n :alt: Latest PyPI version\n\n.. image:: https://pypip.in/d/spyda/badge.png\n :target: https://crate.io/packages/spyda/\n :alt: Number of PyPI downloads\n\n.. image:: https://jenkins.shiningpanda-ci.com/prologic/job/spyda/badge/icon\n :target: https://jenkins.shiningpanda-ci.com/prologic/job/spyda/\n :alt: Build Status\n\n\nRequirements\n------------\n\n- `restclient`_\n- `cssselect`_\n- `lxml`_\n- `url`_\n- `nltk`_\n- `calais`_\n- `BeautifulSoup`_\n\nspyda also comes basic documentation and a full comprehensive unit test suite which require the following:\n\nTo build the docs:\n\n- `sphinx `_\n- `sphinxcontrib-bitbucket `_\n\nTo run the unit tests:\n\n- `pytest `_\n- `circuits `_\n\n\nInstallation\n------------\n\nThe simplest and recommended way to install spyda is with pip.\nYou may install the latest stable release from PyPI with pip::\n\n > pip install spyda\n\nIf you do not have pip, you may use easy_install::\n\n > easy_install spyda\n\nAlternatively, you may download the source package from the\n`PyPI Page`_ or the `Downloads page`_ on the `Project Website`_;\nextract it and install using::\n\n > python setup.py install\n\nYou can also install the\n`latest-development version `_ by using ``pip`` or ``easy_install``::\n \n > pip install spyda==dev\n\nor::\n \n > easy_install spyda==dev\n\n\nFor further information see the `spyda documentation `_.\n\n\nSupported Platforms\n-------------------\n\n- Linux, FreeBSD, Mac OS X\n- Python 2.6, 2.7\n\n**Windows**: We acknowledge that Windows exists and make reasonable efforts\n to maintain compatibility. Unfortunately we cannot guarantee\n support at this time.\n\n\nChanges\n-------\n\n\nspyda 0.0.2 (2013-11-19)\n........................\n\n- Updated the README\n- Added build status\n- Added install instructions\n\n\nspyda 0.0.1 (2013-11-19)\n........................\n\n- Initiali Public Release", "description_content_type": null, "docs_url": "https://pythonhosted.org/spyda/", "download_url": "https://bitbucket.org/prologic/spyda/downloads/", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://bitbucket.org/prologic/spyda", "keywords": "Python Spider Web Crawling and Extraction Tool and Library", "license": "MIT", "maintainer": null, "maintainer_email": null, "name": "spyda", "package_url": "https://pypi.org/project/spyda/", "platform": "POSIX", "project_url": "https://pypi.org/project/spyda/", "project_urls": { "Download": "https://bitbucket.org/prologic/spyda/downloads/", "Homepage": "https://bitbucket.org/prologic/spyda" }, "release_url": "https://pypi.org/project/spyda/0.0.2/", "requires_dist": null, "requires_python": null, "summary": "Spyda - Python Spider Tool and Library", "version": "0.0.2" }, "last_serial": 923300, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "d1d5dc41d1fcf01459d1659e74d7f9c1", "sha256": "118f0123c6c3da70c21820d1903986543fe868925c108de7a8e6ba66d513742b" }, "downloads": -1, "filename": "spyda-0.0.1-py2.7.egg", "has_sig": false, "md5_digest": "d1d5dc41d1fcf01459d1659e74d7f9c1", "packagetype": "bdist_egg", "python_version": "2.7", "requires_python": null, "size": 51067, "upload_time": "2013-11-19T04:36:07", "url": "https://files.pythonhosted.org/packages/1d/9b/8fdb047e2fe1f5f0510203b4543312c4ffead615547bea99eba78bb1491c/spyda-0.0.1-py2.7.egg" }, { "comment_text": "", "digests": { "md5": "cc836dd0d0cd696f3475d04149754605", "sha256": "caff23f3ddd524245ae0823a63b67f5f8017cbae54d2a558f5cb1f5226918806" }, "downloads": -1, "filename": "spyda-0.0.1.tar.gz", "has_sig": false, "md5_digest": "cc836dd0d0cd696f3475d04149754605", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 167272, "upload_time": "2013-11-19T04:36:02", "url": "https://files.pythonhosted.org/packages/05/21/e59bac420c48721d227ee1e031fa87d579b03207ed2cc068b7260443873a/spyda-0.0.1.tar.gz" } ], "0.0.2": [ { "comment_text": "", "digests": { "md5": "92f7265a9b72cfc8b76246fe88fb9f50", "sha256": "c2a50998aab9281f25944e3fc66ea490b9b65dd0e254ff205cdc0dd96fc33dfd" }, "downloads": -1, "filename": "spyda-0.0.2-py2.7.egg", "has_sig": false, "md5_digest": "92f7265a9b72cfc8b76246fe88fb9f50", "packagetype": "bdist_egg", "python_version": "2.7", "requires_python": null, "size": 51626, "upload_time": "2013-11-19T04:46:42", "url": "https://files.pythonhosted.org/packages/72/a6/984641cc494464a0ff3cec91f3f31cd58b80d8cad0dced95f267c704579a/spyda-0.0.2-py2.7.egg" }, { "comment_text": "", "digests": { "md5": "dc6d98feda96282b3d254db75e86cc0e", "sha256": "6cbace90a424420dc60d4f59091c44d182165ec91a09f8a28803bb814b9c161c" }, "downloads": -1, "filename": "spyda-0.0.2.tar.gz", "has_sig": false, "md5_digest": "dc6d98feda96282b3d254db75e86cc0e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 172712, "upload_time": "2013-11-19T04:46:37", "url": "https://files.pythonhosted.org/packages/bf/4d/0144e9b1d0b04b9653db11dd690a9ad6d030067c7fa8aabb7eb817af9c4b/spyda-0.0.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "92f7265a9b72cfc8b76246fe88fb9f50", "sha256": "c2a50998aab9281f25944e3fc66ea490b9b65dd0e254ff205cdc0dd96fc33dfd" }, "downloads": -1, "filename": "spyda-0.0.2-py2.7.egg", "has_sig": false, "md5_digest": "92f7265a9b72cfc8b76246fe88fb9f50", "packagetype": "bdist_egg", "python_version": "2.7", "requires_python": null, "size": 51626, "upload_time": "2013-11-19T04:46:42", "url": "https://files.pythonhosted.org/packages/72/a6/984641cc494464a0ff3cec91f3f31cd58b80d8cad0dced95f267c704579a/spyda-0.0.2-py2.7.egg" }, { "comment_text": "", "digests": { "md5": "dc6d98feda96282b3d254db75e86cc0e", "sha256": "6cbace90a424420dc60d4f59091c44d182165ec91a09f8a28803bb814b9c161c" }, "downloads": -1, "filename": "spyda-0.0.2.tar.gz", "has_sig": false, "md5_digest": "dc6d98feda96282b3d254db75e86cc0e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 172712, "upload_time": "2013-11-19T04:46:37", "url": "https://files.pythonhosted.org/packages/bf/4d/0144e9b1d0b04b9653db11dd690a9ad6d030067c7fa8aabb7eb817af9c4b/spyda-0.0.2.tar.gz" } ] }