{
    "info": {
        "author": "Gabriel Falcao",
        "author_email": "gabriel@nacaolivre.org",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 3 - Alpha",
            "Intended Audience :: Developers",
            "Operating System :: OS Independent",
            "Programming Language :: Python",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.1",
            "Programming Language :: Python :: 3.2",
            "Programming Language :: Python :: 3.3",
            "Topic :: Software Development :: Build Tools"
        ],
        "description": "|Build Status| # LineUp - Distributed Pipeline Framework for Python\n\nLineup is a redis-based\n`pipeline <http://en.wikipedia.org/wiki/Pipeline_(software)>`__\nframework that turns horizontal scalling seamless.\n\nIt's currently providing parallelism through python threads and is\npretty useful for writing systems where the workers make lots of network\nI/O.\n\nIt scales horizontally, so you can simply run more workers in as many\nmachines you want.\n\nInstalling\n----------\n\n.. code:: bash\n\n    easy_install curdling\n    curd install lineup\n\nPhilosophy\n----------\n\nLineup focus in:\n\n1. Simplicity: easy to create otherwise complex pipelines\n2. Easy-scale: just run more workers and you're good.\n\nDefining steps\n--------------\n\nSteps must always implement the method ``consume(self, instructions)``\nand always call ``self.produce()`` with it's portion of work.\n\nExample: a pipeline that downloads files\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis is a copy of the `examples <examples>`__ folder.\n\n.. code:: python\n\n    import re\n    import codecs\n    import requests\n\n    from lineup import Step\n    from lineup import Pipeline\n\n    class Download(Step):\n        def after_consume(self, instructions):\n            self.log(\n                \"Done downloading %s\",\n                instructions['url'],\n            )\n\n        def before_consume(self):\n            self.log(\"The downloader is ready\")\n\n        def consume(self, instructions):\n            url = instructions['url']\n            method = instructions.get('method', 'get').lower()\n\n            http_request = getattr(requests, method)\n            response = http_request(url)\n            instructions['download'] = {\n                'content': response.content,\n                'headers': dict(response.headers),\n                'status_code': response.status_code,\n            }\n            self.produce(instructions)\n\n\n    class Cache(Step):\n        def after_consume(self, instructions):\n            self.log(\"Done caching %s\", instructions.keys())\n\n        def before_consume(self):\n            self.log(\"The cacher is also ready\")\n\n        def get_slug(self, url):\n            url = re.sub(r'^https?://', '', url)\n            url = re.sub(r'\\W+', '-', url)\n            return url\n\n        def consume(self, instructions):\n            url = instructions['url']\n            name = self.get_slug(url)\n            with codecs.open(name, 'wb', 'utf-8') as fd:\n                fd.write(instructions['download']['content'])\n                fd.close()\n                instructions['cached_at'] = {\n                    'filename': fd.name,\n                }\n            self.produce(instructions)\n\n\n\n    # Defining pipelines\n\n    class SimpleUrlDownloader(Pipeline):\n        name = 'downloader'\n        steps = [Download, Cache]\n\nCommand line\n------------\n\nWhen running from the command line, lineup will recursively try to\nimport all python files given as ``--working-dir`` argument, which\ndefaults to the relative equivalent to ``os.getcwd()``.\n\nIn other words it will find your stuff automatically in dir you run the\npipeline from, or from the ``--working-dir`` arg.\n\nType ``lineup --help`` for more info.\n\nListing available pipelines\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nLineUp scans the given working directory recursively for pipeline class\ndeclarations, here is how you can list all the pipelines that lineup can\nfind:\n\n.. code:: bash\n\n    lineup list pipelines\n\nRunning a pipeline in foreground\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. code:: bash\n\n    # Lineup will connect to this redis\n    export LINEUP_REDIS_URI='redis://0@localhost:6379'\n\n    lineup downloader run --output=rpush@example-output\n\n.. figure:: example/run.png\n   :alt: example/run.png\n\n   example/run.png\nFeeding a pipeline through command line\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. code:: bash\n\n    lineup downloader push {\"url\": \"http://github.com/gabrielfalcao.keys\"}\n\n.. figure:: example/push.png\n   :alt: example/run.png\n\n   example/run.png\nFeeding a pipeline programatically\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. code:: python\n\n    from lineup import JSONRedisBackend\n    from example.pipelines import SimpleUrlDownloader\n\n    pipeline = SimpleUrlDownloader(JSONRedisBackend)\n    pipeline.input.put({\n        \"url\": \"http://github.com/gabrielfalcao.keys\"\n    })\n\nStopping all running pipelines\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis broadcasts a \"stop\" signal in the backend.\n\n.. code:: bash\n\n    lineup downloader stop\n\nContributing\n------------\n\nInstall Dependencies\n~~~~~~~~~~~~~~~~~~~~\n\n.. code:: bash\n\n    curd install -r development.txt\n\nRun tests\n~~~~~~~~~\n\n.. code:: bash\n\n    make test\n\n.. |Build Status| image:: https://travis-ci.org/weedlabs/lineup.png?branch=master\n   :target: https://travis-ci.org/weedlabs/lineup",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/weedlabs/lineup",
        "keywords": null,
        "license": "UNKNOWN",
        "maintainer": null,
        "maintainer_email": null,
        "name": "lineup",
        "package_url": "https://pypi.org/project/lineup/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/lineup/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "https://github.com/weedlabs/lineup"
        },
        "release_url": "https://pypi.org/project/lineup/0.1.8/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "Distributed Pipeline Framework for Python",
        "version": "0.1.8"
    },
    "last_serial": 1709692,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "271498b8d9d3bdca7a33a6fabe5c9249",
                    "sha256": "dcc62d7a6b35c7d62d0eda9dc7d26039f6f32782e616285fa5e5fd111e5f4ce0"
                },
                "downloads": -1,
                "filename": "lineup-0.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "271498b8d9d3bdca7a33a6fabe5c9249",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8284,
                "upload_time": "2013-12-23T02:48:17",
                "url": "https://files.pythonhosted.org/packages/47/6e/024ca6570c135a79f0e4ed274824d633f5b340e0d047a4186b62d2d8d86e/lineup-0.0.1.tar.gz"
            }
        ],
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8c2b0cf5f9590fd8c6a931f77c653c61",
                    "sha256": "1d4d4702abd292428964d7e12b8a96374a7a78d303d151d9ec7ece3692f63760"
                },
                "downloads": -1,
                "filename": "lineup-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "8c2b0cf5f9590fd8c6a931f77c653c61",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12725,
                "upload_time": "2014-02-02T14:51:26",
                "url": "https://files.pythonhosted.org/packages/c7/8c/6363cec3b450b4a6bb3ebdc23407eadb87ca85118cb3f084c1a2694e0e37/lineup-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "54a0f19b3214c440b274a95229b36332",
                    "sha256": "26bd1b3ed175cf7d8ab9b105d02a61311c755927d2a9535c9537c99e0750ff5e"
                },
                "downloads": -1,
                "filename": "lineup-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "54a0f19b3214c440b274a95229b36332",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12759,
                "upload_time": "2014-02-02T15:16:58",
                "url": "https://files.pythonhosted.org/packages/02/7e/af4e4fd4ae5b9d81eeb123d40fa2f6cec5b07e6dbcfc1360e033113785ee/lineup-0.1.1.tar.gz"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0c5b381864191723ed408fd6ece6aa38",
                    "sha256": "e3b131f46dfd65116e67ceb1fd65064f53a1388f9b34857a4def759868be80b0"
                },
                "downloads": -1,
                "filename": "lineup-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "0c5b381864191723ed408fd6ece6aa38",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12826,
                "upload_time": "2014-02-05T06:24:15",
                "url": "https://files.pythonhosted.org/packages/c1/5c/52762741cc0851b2da46271ed73406334b20d5816a18ea1aecb4671e5e23/lineup-0.1.2.tar.gz"
            }
        ],
        "0.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d663fadd438904871c90e965a2697bb6",
                    "sha256": "018c39c72bfaad0cca9004144b8d1d77e42debe6488e49baaddd3a9e4ba97070"
                },
                "downloads": -1,
                "filename": "lineup-0.1.3.tar.gz",
                "has_sig": false,
                "md5_digest": "d663fadd438904871c90e965a2697bb6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13164,
                "upload_time": "2014-02-05T07:12:33",
                "url": "https://files.pythonhosted.org/packages/d2/41/2764bcc2a1c4d7a301c617340d863aff71a791e2a12c20b21aa53c245524/lineup-0.1.3.tar.gz"
            }
        ],
        "0.1.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f46b6b4ee0c163cd31669e3a9d2df5e7",
                    "sha256": "0fb6acc7cce14d696dc0cbdd0b906d94dae6e7fc0b2e9fec2c430dc7cd65745b"
                },
                "downloads": -1,
                "filename": "lineup-0.1.4.tar.gz",
                "has_sig": false,
                "md5_digest": "f46b6b4ee0c163cd31669e3a9d2df5e7",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13181,
                "upload_time": "2014-02-05T22:17:51",
                "url": "https://files.pythonhosted.org/packages/f4/36/e4c2d03637fbc4e20bddf94bb92170a34814356f0711fc4a3baa5e4c7ee8/lineup-0.1.4.tar.gz"
            }
        ],
        "0.1.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "32dbfc09c33d003311b8ad3471c1ebdf",
                    "sha256": "14c633391c4c6cf34f8fe58175f118bae48f1b8d3635b39abbfaadee03f0cb60"
                },
                "downloads": -1,
                "filename": "lineup-0.1.5.tar.gz",
                "has_sig": false,
                "md5_digest": "32dbfc09c33d003311b8ad3471c1ebdf",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13177,
                "upload_time": "2015-06-24T18:54:23",
                "url": "https://files.pythonhosted.org/packages/ab/1d/af90a5d00a8994fc54d2b80291623957d251239cba243e4638c706292e04/lineup-0.1.5.tar.gz"
            }
        ],
        "0.1.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a3f6bb1037f47d7ed13091f2abfa464c",
                    "sha256": "583cac839ddbd494f5c1e74d0ef8f34c6a58648dc90ac913c8b8d42264b593e4"
                },
                "downloads": -1,
                "filename": "lineup-0.1.6.tar.gz",
                "has_sig": false,
                "md5_digest": "a3f6bb1037f47d7ed13091f2abfa464c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13211,
                "upload_time": "2015-06-24T18:57:37",
                "url": "https://files.pythonhosted.org/packages/d5/d3/27d802322dc1b0265886ed09ed4c5a9fbe115214dcacbecedc73bffc1930/lineup-0.1.6.tar.gz"
            }
        ],
        "0.1.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8a02ad53e4ac394011384116c8bdba12",
                    "sha256": "4cc14fbafee5e0b5a674d88ef47d779b52f65f16f3a8c97ed6db71c8f0a7582c"
                },
                "downloads": -1,
                "filename": "lineup-0.1.7.tar.gz",
                "has_sig": false,
                "md5_digest": "8a02ad53e4ac394011384116c8bdba12",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13201,
                "upload_time": "2015-06-24T19:49:43",
                "url": "https://files.pythonhosted.org/packages/d3/56/5e7abefd770528f1ca35dc59dbc86f7fc64addd3b67e8c1651b399725508/lineup-0.1.7.tar.gz"
            }
        ],
        "0.1.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "1442131135e9b49eb32408ee3fb71490",
                    "sha256": "9ce267e3acbac8425f304de159d0706a8fd61ff5d81534b6e538f3f928939cfd"
                },
                "downloads": -1,
                "filename": "lineup-0.1.8.tar.gz",
                "has_sig": false,
                "md5_digest": "1442131135e9b49eb32408ee3fb71490",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13222,
                "upload_time": "2015-09-05T21:45:18",
                "url": "https://files.pythonhosted.org/packages/6c/e5/8a1b3b454d345594e2e7dbaaf3d600021efdafb13a9eea26edeb83e4d161/lineup-0.1.8.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "1442131135e9b49eb32408ee3fb71490",
                "sha256": "9ce267e3acbac8425f304de159d0706a8fd61ff5d81534b6e538f3f928939cfd"
            },
            "downloads": -1,
            "filename": "lineup-0.1.8.tar.gz",
            "has_sig": false,
            "md5_digest": "1442131135e9b49eb32408ee3fb71490",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 13222,
            "upload_time": "2015-09-05T21:45:18",
            "url": "https://files.pythonhosted.org/packages/6c/e5/8a1b3b454d345594e2e7dbaaf3d600021efdafb13a9eea26edeb83e4d161/lineup-0.1.8.tar.gz"
        }
    ]
}