{
    "info": {
        "author": "Jeff Klukas",
        "author_email": "klukas@simple.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Environment :: Console",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: Apache Software License",
            "Natural Language :: English",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 2.6",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3.3",
            "Programming Language :: Python :: 3.4",
            "Topic :: Software Development :: Libraries :: Python Modules"
        ],
        "description": "Pipewelder\n==========\n\n.. figure:: welder.jpg\n   :alt: A worker welding a pipe\n\nPipewelder is a framework that provides a command-line tool and Python\nAPI to manage `AWS Data\nPipeline <http://aws.amazon.com/datapipeline/>`__ jobs from flat files.\nSimple uses it as a cron-like job scheduler.\n\nSource\n  https://github.com/SimpleFinance/pipewelder\n\nDocumentation\n  http://pipewelder.readthedocs.org\n\nPyPI\n  https://pypi.python.org/pypi/pipewelder\n\nOverview\n--------\n\nPipewelder aims to ease the task of scheduling jobs by defining very\nsimple pipelines which are little more than an execution schedule,\noffloading most of the execution logic to files in S3. Pipewelder uses\nData Pipeline's concept of `data\nstaging <http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-concepts-staging.html>`__\nto pull input files from S3 at the beginning of execution and to upload\noutput files back to S3 at the end of execution.\n\nIf you follow Pipewelder's directory structure, all of your pipeline\nlogic can live in version-controlled flat files. The included\ncommand-line interface gives you simple commands to validate your\npipeline definitions, upload task definitions to S3, and activate your\npipelines.\n\nInstallation\n------------\n\nPipewelder is available from `PyPI <https://pypi.python.org/pypi>`__ via\n``pip`` and is compatible with Python 2.6, 2.7, 3.3, and 3.4:\n\n::\n\n    pip install pipewelder\n\nThe easiest way to get started is to clone the project from GitHub, copy\nthe example project from Pipewelder's tests, and then modify to suit:\n\n.. code:: bash\n\n    git clone https://github.com/SimpleFinance/pipewelder.git\n    cp -r pipewelder/tests/test_data my-pipewelder-project\n\nIf you're setting up Pipewelder and need help, feel free to email the\nauthor.\n\nDevelopment\n-----------\n\nTo do development on Pipewelder, clone the repository and run ``make``\nto install dependencies and run tests.\n\nDirectory Structure\n-------------------\n\nTo use Pipewelder, you provide a template pipeline definition along with\none or more directories that correspond to particular pipeline\ninstances. The directory structure looks like this (see\n`test\\_data <tests/test_data>`__ for a working example):\n\n::\n\n    pipeline_definition.json\n    pipewelder.json <- optional configuration file\n    my_first_pipeline/\n        run\n        values.json\n        tasks/\n            task1.sh\n            task2.sh\n    my_second_pipeline/\n    ...\n\nThe ``values.json`` file in each pipeline directory specifies parameter\nvalues that are used modify the template definition including the S3\npaths for inputs, outputs, and logs. Some of these values are used\ndirectly by Pipewelder as well.\n\nA\n```ShellCommandActivity`` <http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-shellcommandactivity.html>`__\nin the template definition simply looks for an executable file named\n``run`` and executes it. ``run`` is the entry point for whatever work\nyou want your pipeline to do.\n\nOften, your ``run`` executable will be a wrapper script to execute a\nvariety of similar tasks. When that's the case, use the ``tasks``\nsubdirectory to hold these definitions. These tasks could be text files,\nshell scripts, SQL code, or whatever else your ``run`` file expects.\nPipewelder gives ``tasks`` folder special treatment in that the CLI will\nmake sure to remove existing task definitions when uploading files.\n\nUsing the Command-Line Interface\n--------------------------------\n\nThe Pipewelder CLI should always be invoked from the top-level directory\nof your definitions (the directory where ``pipeline_definition.json``\nlives). If your directory structure matches Pipewelder's expectations,\nit should work without further configuration.\n\nAs you make changes to your template definition or ``values.json``\nfiles, it can be useful to check whether AWS considers your definitions\nvalid:\n\n::\n\n    $ pipewelder validate\n\nOnce you've defined your pipelines, you'll need to upload the files to\nS3:\n\n::\n\n    $ pipewelder upload\n\nFinally, activate your pipelines:\n\n::\n\n    $ pipewelder activate\n\nAny time you change the ``values.json`` or ``pipeline_definition.json``,\nyou'll need to run the ``activate`` subcommand again. Because active\npipelines can't be modified, the ``activate`` command will delete the\nexisting pipeline and create a new one in its place. The run history for\nthe previous pipeline will be discarded.\n\nAcknowledgments\n---------------\n\nPipewelder's package structure is based on\n`python-project-template <https://github.com/seanfisk/python-project-template>`__.",
        "description_content_type": null,
        "docs_url": null,
        "download_url": null,
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://github.com/jklukas/pipewelder",
        "keywords": null,
        "license": null,
        "maintainer": null,
        "maintainer_email": null,
        "name": "pipewelder",
        "package_url": "https://pypi.org/project/pipewelder/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/pipewelder/",
        "project_urls": {
            "Homepage": "http://github.com/jklukas/pipewelder"
        },
        "release_url": "https://pypi.org/project/pipewelder/0.1.4/",
        "requires_dist": [
            "boto",
            "six"
        ],
        "requires_python": null,
        "summary": "Scheduled task execution on top of AWS Data Pipeline",
        "version": "0.1.4"
    },
    "last_serial": 1454177,
    "releases": {
        "0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "554abdf631041183ca5cfa452be611b2",
                    "sha256": "e5518fb95318cc26e1038b86f3ba28663923cdda5b625570c6884dc8f3cdcc84"
                },
                "downloads": -1,
                "filename": "pipewelder-0.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "554abdf631041183ca5cfa452be611b2",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 16963,
                "upload_time": "2015-02-26T21:45:07",
                "url": "https://files.pythonhosted.org/packages/6a/8f/d7a755db9296aad3f3938143a50bd194c6783d72f727336eeb17e8a749ec/pipewelder-0.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "5a3c2d225a6b4a84fa63cacf18947ae9",
                    "sha256": "3a7ef6e200786af3e21b8f97906350cde8ea3599bcab86ce86d1d8d186ac2231"
                },
                "downloads": -1,
                "filename": "pipewelder-0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "5a3c2d225a6b4a84fa63cacf18947ae9",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 27208,
                "upload_time": "2015-02-26T21:45:10",
                "url": "https://files.pythonhosted.org/packages/44/e9/7fb44358e0a19c9fa32b5fee282877a82feaebb67b1a942d4259f4ab951d/pipewelder-0.1.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "13498529628c609da30e2ad2d7ce3211",
                    "sha256": "d0d1556b628464e33cdfbc481ee716608de4855e1829d9c6b42b0960c4927ecc"
                },
                "downloads": -1,
                "filename": "pipewelder-0.1.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "13498529628c609da30e2ad2d7ce3211",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 17122,
                "upload_time": "2015-02-26T21:53:06",
                "url": "https://files.pythonhosted.org/packages/4e/49/170b34d7cb8b5bfd57b55d18b07fafa16146bd046eb6de7b1a7c39fb070f/pipewelder-0.1.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d2ade490dc73fe644925e7ed8cd8f5e5",
                    "sha256": "2543f7f56da587066202e2613b067ad6e22cccba8e6020372747c1b29ed4b896"
                },
                "downloads": -1,
                "filename": "pipewelder-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "d2ade490dc73fe644925e7ed8cd8f5e5",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 27396,
                "upload_time": "2015-02-26T21:53:09",
                "url": "https://files.pythonhosted.org/packages/de/a3/e4c894aca425e7f44167047fbd02cdaf429bb73e921d01feafd4a53b6ab7/pipewelder-0.1.1.tar.gz"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "eac70db71320b10012f9c90e583e9bb5",
                    "sha256": "f026bd7ab5d3b7f29e98ab885117242b8a30a85f3df41579d4da86a4e5205dd2"
                },
                "downloads": -1,
                "filename": "pipewelder-0.1.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "eac70db71320b10012f9c90e583e9bb5",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 17119,
                "upload_time": "2015-02-26T22:03:15",
                "url": "https://files.pythonhosted.org/packages/79/65/3aaad9d4f46c1faababb62c1c3034c04b4e23cd6fa65a349523332a0ecaf/pipewelder-0.1.2-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d7f373514a0ebeb360010aa19885d1e1",
                    "sha256": "e3f73faa56a81185048892a0a03189e4642e4dd2d7f383534450b848f1aa883d"
                },
                "downloads": -1,
                "filename": "pipewelder-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "d7f373514a0ebeb360010aa19885d1e1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 356851,
                "upload_time": "2015-02-26T22:03:18",
                "url": "https://files.pythonhosted.org/packages/54/94/798bae39ac4ae21793968580974a5f8dc0e92c0b543a782c91530a3eafb6/pipewelder-0.1.2.tar.gz"
            }
        ],
        "0.1.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "659f84aacd38692b28222b142bfe2a26",
                    "sha256": "27143e50afb0b1487fd552eeb7e2076724211a38854b5b2cbde41f28555c603a"
                },
                "downloads": -1,
                "filename": "pipewelder-0.1.4-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "659f84aacd38692b28222b142bfe2a26",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 17773,
                "upload_time": "2015-03-09T14:40:52",
                "url": "https://files.pythonhosted.org/packages/16/ce/0bf8ef9549372a741aa9c469b6fd57048fdb8e86a04a3ec297efe5bb0c9e/pipewelder-0.1.4-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ebdacf6ccca534a91cd3855cf26aa76f",
                    "sha256": "480936b6e08b4cd1a628b2bb9a129c5a9c60f558ea74a0f71cde661b1f120541"
                },
                "downloads": -1,
                "filename": "pipewelder-0.1.4.tar.gz",
                "has_sig": false,
                "md5_digest": "ebdacf6ccca534a91cd3855cf26aa76f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 357558,
                "upload_time": "2015-03-09T14:40:56",
                "url": "https://files.pythonhosted.org/packages/cf/07/700280dd6a5880c4f3e52aaf855f3268d5318f7d6bd0ef992c3dffd4a68c/pipewelder-0.1.4.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "659f84aacd38692b28222b142bfe2a26",
                "sha256": "27143e50afb0b1487fd552eeb7e2076724211a38854b5b2cbde41f28555c603a"
            },
            "downloads": -1,
            "filename": "pipewelder-0.1.4-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "659f84aacd38692b28222b142bfe2a26",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 17773,
            "upload_time": "2015-03-09T14:40:52",
            "url": "https://files.pythonhosted.org/packages/16/ce/0bf8ef9549372a741aa9c469b6fd57048fdb8e86a04a3ec297efe5bb0c9e/pipewelder-0.1.4-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "ebdacf6ccca534a91cd3855cf26aa76f",
                "sha256": "480936b6e08b4cd1a628b2bb9a129c5a9c60f558ea74a0f71cde661b1f120541"
            },
            "downloads": -1,
            "filename": "pipewelder-0.1.4.tar.gz",
            "has_sig": false,
            "md5_digest": "ebdacf6ccca534a91cd3855cf26aa76f",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 357558,
            "upload_time": "2015-03-09T14:40:56",
            "url": "https://files.pythonhosted.org/packages/cf/07/700280dd6a5880c4f3e52aaf855f3268d5318f7d6bd0ef992c3dffd4a68c/pipewelder-0.1.4.tar.gz"
        }
    ]
}