{
    "info": {
        "author": "James Arnold",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: BSD License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3"
        ],
        "description": "# flexes-feed\n[![Build Status](https://travis-ci.org/lanl-ansi/flexes-feed.svg?branch=master)](https://travis-ci.org/lanl-ansi/flexes-feed) \n[![codecov](https://codecov.io/gh/lanl-ansi/flexes-feed/branch/master/graph/badge.svg)](https://codecov.io/gh/lanl-ansi/flexes-feed)\n\nGeneric structure for retrieving and processing regularly updated data from the web \n\n## Scraper Usage\nTo create a new scraper simply create a class that inherits from the `Scraper` class \nand override the `check()` method.\n\nHere is a quick pseudo example:\n\n```python\nimport requests\nfrom data_feed.scraper import NewFile, Scraper\n\nclass MyScraper(Scraper):\n  def check(self):\n    response = requests.get(self.channel)\n    # Parse content from page \n    # If the file has changed return a NewFile object\n    return [NewFile(file_url, self.s3_folder)]\n    \ndef run_scraper():\n  s3_folder = 's3://bucket/path/to/store/data'\n  channel = 'http://somedata.com'\n  scraper = MyScraper(s3_folder, channel)\n  scraper.run()\n  \nif __name__ == '__main__':\n  run_scraper()\n```\n\nSee [examples/noaa_wind_scraper.py](noaa_wind_scraper.py) for a real example. The example\nalso requires that `BeautifulSoup4` and `lxml` are installed. To install, simply\nrun `pip install BeautifulSoup4 lxml`.\n\n## Subscriber Usage\nTo create a new subscriber simply create a class that inherits from the `Subscriber` \nclass and override the `process()` method.\n\nHere is a quick pseduo example:\n\n```python\nfrom data_feed.subscriber import Subscriber\n\nclass MySubscriber(Subscriber):\n  def process(self, s3_uri):\n    # Process file(s) in s3_uri\n    \ndef subscribe():\n  channel = 'http://somedata.com'\n  sub = MySubscriber(channel)\n  sub.subscribe()\n  \nif __name__ == '__main__':\n  subscribe()\n```\n  \nSee [examples/noaa_wind_subscriber.py](noaa_wind_subscriber.py) for a real example using the \nlanlytics API.",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "flexes-feed",
        "package_url": "https://pypi.org/project/flexes-feed/",
        "platform": "",
        "project_url": "https://pypi.org/project/flexes-feed/",
        "project_urls": null,
        "release_url": "https://pypi.org/project/flexes-feed/0.1.0/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Library for setting up data processing pipelines from remote data feeds",
        "version": "0.1.0"
    },
    "last_serial": 4969292,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b1ba8bb655dfb45edb8fdc72db5946be",
                    "sha256": "da305e7ecefb941c1c149043063b67e9e45f3d8698c8845585052323f52f8c47"
                },
                "downloads": -1,
                "filename": "flexes_feed-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "b1ba8bb655dfb45edb8fdc72db5946be",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5578,
                "upload_time": "2019-03-21T17:53:58",
                "url": "https://files.pythonhosted.org/packages/12/56/1176a6d77da56540a398e5bca92ed82a017eed5f7e649d557dd8d4fff240/flexes_feed-0.1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "b1ba8bb655dfb45edb8fdc72db5946be",
                "sha256": "da305e7ecefb941c1c149043063b67e9e45f3d8698c8845585052323f52f8c47"
            },
            "downloads": -1,
            "filename": "flexes_feed-0.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "b1ba8bb655dfb45edb8fdc72db5946be",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 5578,
            "upload_time": "2019-03-21T17:53:58",
            "url": "https://files.pythonhosted.org/packages/12/56/1176a6d77da56540a398e5bca92ed82a017eed5f7e649d557dd8d4fff240/flexes_feed-0.1.0.tar.gz"
        }
    ]
}