{ "info": { "author": "Florian Finkernagel", "author_email": "finkernagel@imt.uni-marburg.de", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "Programming Language :: Python" ], "description": "# pypipegraph \n\n| Build status: | [![Build Status](https://travis-ci.com/TyberiusPrime/pypipegraph.svg?branch=master)](https://travis-ci.com/TyberiusPrime/pypipegraph)|\n|---------------|-----------------------------------------------------------------------------|\n| Documentation | https://pypipegraph.readthedocs.io/en/latest/\n| Code style | ![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)\n\n## Introduction\n\n[pypipegraph](https://github.com/IMTMarburg/pypipegraph): is an\nMIT-licensed library for constructing a workflow piece by piece and\nexecuting just the parts of it that need to be (re-)done. It supports\nusing multiple cores (SMP) and (eventually, alpha code right now)\nmachines (cluster) and is a hybrid between a dependency tracker (think\n'make') and a cluster engine.\n\nMore specifically, you construct Jobs, which encapsulate output (i.e.\nstuff that needs to be done), invariants (which force re-evaluation of\noutput jobs if they change), and stuff inbetween (e.g. load data from\ndisk).\n\nFrom your point of view, you create a pypipegraph, you create jobs,\nchain them together, then ask the pypipegraph to run. It examines all\njobs for their need to run (either because the have not been finished,\nor because they have been invalidated), distributes them across multiple\npython instances, and get's them executed in a sensible order.\n\nIt is robust against jobs dying for whatever reason (only the failed job\nand everything 'downstream' will be affected, independend jobs will\ncontinue running), allows you to resume at any point 'in between' jobs,\nand isolates jobs against each other.\n\npypipegraph supports Python 3 only.\n\n## 30 second summary\n\n```python\n pypipegraph.new_pipeline()\n output_filenameA = 'sampleA.txt'\n def do_the_work():\n op = open(output_filename, 'wb').write(\"hello world\")\n jobA = pypipegraph.FileGeneratingJob(output_filenameA, do_the_work)\n output_filenameB = 'sampleB.txt'\n def do_the_work():\n op = open(output_filenameB, 'wb').write(open(output_filenameA, 'rb').read() + \", once again\")\n jobB = pypipegraph.FileGeneratingJob(output_filenameB, do_the_work)\n jobB.depends_on(jobA)\n pypipegraph.run()\n print('the pipegraph is done and has returned control to you.')\n print('sampleA.txt contains \"hello world\"')\n print('sampleB.txt contains \"hello world, once again\")\n```", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://https://github.com/IMTMarburg/pypipegraph", "keywords": "", "license": "mit", "maintainer": "", "maintainer_email": "", "name": "pypipegraph", "package_url": "https://pypi.org/project/pypipegraph/", "platform": "any", "project_url": "https://pypi.org/project/pypipegraph/", "project_urls": { "Homepage": "http://https://github.com/IMTMarburg/pypipegraph" }, "release_url": "https://pypi.org/project/pypipegraph/0.187/", "requires_dist": null, "requires_python": ">=3.6", "summary": "A workflow (job) engine/pipeline for bioinformatics and scientific computing.", "version": "0.187" }, "last_serial": 5741599, "releases": { "0.126": [ { "comment_text": "", "digests": { "md5": "b6ec42a60b898c6b065dac89f3fc13da", "sha256": "adc1564ae5957f397d5f9d486a8e5d513cbd5e32a5e514c13fe7d44c04766013" }, "downloads": -1, "filename": "pypipegraph-0.126.tar.gz", "has_sig": false, "md5_digest": "b6ec42a60b898c6b065dac89f3fc13da", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 50921, "upload_time": "2012-01-30T15:22:45", "url": "https://files.pythonhosted.org/packages/c1/ff/c6f771327f03ddc7e66783b01e71d483ae91d0fd763c3a0142c204288802/pypipegraph-0.126.tar.gz" } ], "0.128": [ { "comment_text": "", "digests": { "md5": "b2a1ac16d9d8b0de0257063467cb3194", "sha256": "cf723ed49e290c2f068b709fb2c53eb9dfae45b0a584718b5ccccb2184128fb5" }, "downloads": -1, "filename": "pypipegraph-0.128.tar.gz", "has_sig": false, "md5_digest": "b2a1ac16d9d8b0de0257063467cb3194", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 47555, "upload_time": "2012-01-30T15:30:36", "url": "https://files.pythonhosted.org/packages/af/33/8fb04dcc9deb780fe82a0fb1b70a3e05e892944f98aa46032f9d65bb5c10/pypipegraph-0.128.tar.gz" } ], "0.151": [ { "comment_text": "", "digests": { "md5": "76661b015c968290389480fd062bbe4f", "sha256": "c325f60d391ab243fa544fb04a7083ea9534015c46b3db91a8c4b0e0eafa55db" }, "downloads": -1, "filename": "pypipegraph-0.151.tar.gz", "has_sig": false, "md5_digest": "76661b015c968290389480fd062bbe4f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 49714, "upload_time": "2012-04-12T14:38:50", "url": "https://files.pythonhosted.org/packages/dc/ab/c4788b666b5db3b082e29ed610fde2b58e234b1dd8b564e9b6a838e3dd45/pypipegraph-0.151.tar.gz" } ], "0.156": [ { "comment_text": "", "digests": { "md5": "3381fef2b9552dbd4a83c21323586f77", "sha256": "e5610d74467181335e6448cabc51811d4614a472a1c4f5b798d85a49b55ab4ef" }, "downloads": -1, "filename": "pypipegraph-0.156.tar.gz", "has_sig": false, "md5_digest": "3381fef2b9552dbd4a83c21323586f77", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 53186, "upload_time": "2012-04-20T10:24:33", "url": "https://files.pythonhosted.org/packages/f3/17/9065ca8340ac9cc61b58442c3cee2c7e4bc8cfd727e94e09322fb54d8d70/pypipegraph-0.156.tar.gz" } ], "0.157": [ { "comment_text": "", "digests": { "md5": "6ad9dccadd313eea6e3b04c7ff2ec333", "sha256": "b852718bd72c937fb8423b68741ab7233628ea53b04def0f90b1b133e9b95cfb" }, "downloads": -1, "filename": "pypipegraph-0.157.tar.gz", "has_sig": false, "md5_digest": "6ad9dccadd313eea6e3b04c7ff2ec333", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 50463, "upload_time": "2012-05-22T09:32:24", "url": "https://files.pythonhosted.org/packages/ad/95/848aa22f6de6260edb43e0a6527107b7c66a3f3d09e8ee7ca15352eb9657/pypipegraph-0.157.tar.gz" } ], "0.158": [ { "comment_text": "", "digests": { "md5": "5906b5ca97865d0f715fd11d44ea66ff", "sha256": "ab49a9d2831b21441b983526601c799740b5b7faf17fcdcaca5f620a9d9a8849" }, "downloads": -1, "filename": "pypipegraph-0.158.tar.gz", "has_sig": false, "md5_digest": "5906b5ca97865d0f715fd11d44ea66ff", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 47421, "upload_time": "2017-11-22T13:40:10", "url": "https://files.pythonhosted.org/packages/f8/b6/99826689285bbcac6901f7abe6c88d82f34c3ea45a7dbf0abc29354f3c3b/pypipegraph-0.158.tar.gz" } ], "0.159": [ { "comment_text": "", "digests": { "md5": "1a71b9d60c79fb2f52f78debaf0d3c29", "sha256": "7c9c0b583a20afb87a080e3fae90df198dcd8fc3bc1586ec612fc492702a16ef" }, "downloads": -1, "filename": "pypipegraph-0.159.tar.gz", "has_sig": false, "md5_digest": "1a71b9d60c79fb2f52f78debaf0d3c29", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 50984, "upload_time": "2017-12-11T11:08:55", "url": "https://files.pythonhosted.org/packages/5f/da/5741f4b3499859fa09dead1895fb7447eeea2bc0a80e7630a308bf4e2671/pypipegraph-0.159.tar.gz" } ], "0.160": [ { "comment_text": "", "digests": { "md5": "e5c9403340c14cfad4cee26cef105ff4", "sha256": "d2941487b2bb00bf937498449c6586d0167c448b25ce56a12f6f10c9c953af45" }, "downloads": -1, "filename": "pypipegraph-0.160.tar.gz", "has_sig": false, "md5_digest": "e5c9403340c14cfad4cee26cef105ff4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 47683, "upload_time": "2018-05-14T12:29:13", "url": "https://files.pythonhosted.org/packages/88/ff/4505a38e3ccec040735beeaaad8381ae60e287f5730c7fbff6a997158541/pypipegraph-0.160.tar.gz" } ], "0.170": [ { "comment_text": "", "digests": { "md5": "336e2eb782ddc20e08ed580a349aea74", "sha256": "21d518f10eec06a39891d56470968caac7a52f0084799b2aa8f073b7d3fe9e01" }, "downloads": -1, "filename": "pypipegraph-0.170.tar.gz", "has_sig": false, "md5_digest": "336e2eb782ddc20e08ed580a349aea74", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 48701, "upload_time": "2018-11-15T11:04:50", "url": "https://files.pythonhosted.org/packages/ca/84/00945de0499e2dcebaed85fcaeab2be72fa935e1ae0ef8de48ba38201970/pypipegraph-0.170.tar.gz" } ], "0.171": [ { "comment_text": "", "digests": { "md5": "0282d2805c26358932bae23e3d90d163", "sha256": "0ab61ea06d8bd7e7e51ce437826803edeb8ae40da1c389e7107f77e9ddfbbc01" }, "downloads": -1, "filename": "pypipegraph-0.171.tar.gz", "has_sig": false, "md5_digest": "0282d2805c26358932bae23e3d90d163", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 49385, "upload_time": "2018-11-23T10:32:01", "url": "https://files.pythonhosted.org/packages/bf/8e/e45167fc275694a4927cf3bdba7f4866f8ee8f6850c5df993bf5f45c322b/pypipegraph-0.171.tar.gz" } ], "0.172": [ { "comment_text": "", "digests": { "md5": "92fcd5007cccbbab0009c1b46e508ca4", "sha256": "0fed0cf49761ff3234044bc101ad7d5c4a5f8cba320200ba8e62413701c07294" }, "downloads": -1, "filename": "pypipegraph-0.172.tar.gz", "has_sig": false, "md5_digest": "92fcd5007cccbbab0009c1b46e508ca4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 49378, "upload_time": "2018-11-23T10:35:06", "url": "https://files.pythonhosted.org/packages/61/16/467d0bb635613833f3af852d7681b544e15be7d2c155e3a8b0e52188271d/pypipegraph-0.172.tar.gz" } ], "0.173": [ { "comment_text": "", "digests": { "md5": "bc4e9e855c6562cae35024e4301b183e", "sha256": "2302677876ee53bd3e3026f7d866752846051519c8dac8ee1c15ba09fc7eb97f" }, "downloads": -1, "filename": "pypipegraph-0.173.tar.gz", "has_sig": false, "md5_digest": "bc4e9e855c6562cae35024e4301b183e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 50715, "upload_time": "2018-11-23T10:38:08", "url": "https://files.pythonhosted.org/packages/2e/e8/f11a58e30c14dbe13a4614bc343dcea77b5f1166fe667a45cff5f3312a32/pypipegraph-0.173.tar.gz" } ], "0.174": [ { "comment_text": "", "digests": { "md5": "02bca885c97a59aed5a58851d42a539e", "sha256": "32d841352de82010426d7782f37dc3744cee4c807a130a91198f8cc916f5ac98" }, "downloads": -1, "filename": "pypipegraph-0.174.tar.gz", "has_sig": false, "md5_digest": "02bca885c97a59aed5a58851d42a539e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 50752, "upload_time": "2018-11-23T12:19:01", "url": "https://files.pythonhosted.org/packages/42/80/8c164e3415cc898481b9c0342cb2011f97d0ea1d7a6b125630591c0703f1/pypipegraph-0.174.tar.gz" } ], "0.175": [ { "comment_text": "", "digests": { "md5": "e560c685a62b5bc748ac7352bd2aaf14", "sha256": "9ddc03bdc10e72ddcad0cf4f95344696ad753bb7d74fb5368f47a6be9677324c" }, "downloads": -1, "filename": "pypipegraph-0.175.tar.gz", "has_sig": false, "md5_digest": "e560c685a62b5bc748ac7352bd2aaf14", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 50839, "upload_time": "2019-01-04T09:17:29", "url": "https://files.pythonhosted.org/packages/f2/3e/2665e5e0483fe184255fb35f2efeae8edf78d14c76ce1d7f89ecceae25d9/pypipegraph-0.175.tar.gz" } ], "0.177": [ { "comment_text": "", "digests": { "md5": "91e76fd8eaed9b609e945bbf57f31d08", "sha256": "19d717f7890350c876da2ee06863843b4674ff20b992570d8447235dee9175bb" }, "downloads": -1, "filename": "pypipegraph-0.177.tar.gz", "has_sig": false, "md5_digest": "91e76fd8eaed9b609e945bbf57f31d08", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 96955, "upload_time": "2019-01-09T14:23:08", "url": "https://files.pythonhosted.org/packages/d3/fe/a81075c55521aab706f7bf26a10c894bdf3fc145969c1bc4320e82862ca6/pypipegraph-0.177.tar.gz" } ], "0.178": [ { "comment_text": "", "digests": { "md5": "11a95c2d19e45f694cc5173b8f6276b1", "sha256": "033b5de032c8227ec8b961374e6405fb529ac0f4d19b01526056e4f0fd8b9a6a" }, "downloads": -1, "filename": "pypipegraph-0.178.tar.gz", "has_sig": false, "md5_digest": "11a95c2d19e45f694cc5173b8f6276b1", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 130976, "upload_time": "2019-01-11T14:59:19", "url": "https://files.pythonhosted.org/packages/8c/ad/5aea60ef61b252d58ed29a6840ed6c83937a9e8f38f971e97ed234f43dac/pypipegraph-0.178.tar.gz" } ], "0.180": [ { "comment_text": "", "digests": { "md5": "5068ee145d08bba449f824f135e07c08", "sha256": "0a7e818c2b748b033fee5fbe37bc02857a689fc236168c303238975304615d89" }, "downloads": -1, "filename": "pypipegraph-0.180.tar.gz", "has_sig": false, "md5_digest": "5068ee145d08bba449f824f135e07c08", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 128779, "upload_time": "2019-01-23T12:05:37", "url": "https://files.pythonhosted.org/packages/44/79/0c4e8f10ca2468619ba6dc468b571972a2380cf3c1de0277b0b0953aefd0/pypipegraph-0.180.tar.gz" } ], "0.181": [ { "comment_text": "", "digests": { "md5": "03b9205fb5d990a27c806d28663c978d", "sha256": "58b99288b815149216e83836f7d0a1e81c50f4f2bdc00620132b52df2f8f43a5" }, "downloads": -1, "filename": "pypipegraph-0.181.tar.gz", "has_sig": false, "md5_digest": "03b9205fb5d990a27c806d28663c978d", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 131281, "upload_time": "2019-03-11T10:14:32", "url": "https://files.pythonhosted.org/packages/28/67/feab534fe6f4384fbf8aa1772076af37f2dcfaa48ae62cf8ee5a360118cc/pypipegraph-0.181.tar.gz" } ], "0.182": [ { "comment_text": "", "digests": { "md5": "3fe6a95b792b130fe87c5b5e23da8bd7", "sha256": "f6697906ab71c8186bf127c4a2874f66a4c9142ec25b7fc1ad827b4c22c06405" }, "downloads": -1, "filename": "pypipegraph-0.182.tar.gz", "has_sig": false, "md5_digest": "3fe6a95b792b130fe87c5b5e23da8bd7", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 131385, "upload_time": "2019-03-13T15:31:39", "url": "https://files.pythonhosted.org/packages/46/0d/02fa9a21e991cddf7b4006e7236bb43b1f83d169a49880a1a3bee93a50e0/pypipegraph-0.182.tar.gz" } ], "0.183": [ { "comment_text": "", "digests": { "md5": "2fcf9aa85726c446fad7fa6d0b200f81", "sha256": "2abfa9b7f643df81c1c94f7027a8f9e4c3d452ec3710792ca3e72e88b5ce8d28" }, "downloads": -1, "filename": "pypipegraph-0.183.tar.gz", "has_sig": false, "md5_digest": "2fcf9aa85726c446fad7fa6d0b200f81", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 131571, "upload_time": "2019-03-13T16:00:44", "url": "https://files.pythonhosted.org/packages/db/88/c74733d4268e118d32b5722401d3a68fcd364553652094f9b04446b7726b/pypipegraph-0.183.tar.gz" } ], "0.184": [ { "comment_text": "", "digests": { "md5": "b83c55e0dbb0d9278ae1e5db4ada352f", "sha256": "a24b7e726443a1d2b697013dcd12be9a8cf6bd6aadab92a48063b809eb4c2857" }, "downloads": -1, "filename": "pypipegraph-0.184.tar.gz", "has_sig": false, "md5_digest": "b83c55e0dbb0d9278ae1e5db4ada352f", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 105322, "upload_time": "2019-05-07T13:36:16", "url": "https://files.pythonhosted.org/packages/43/d8/b37ba9334deb51b10cb1163b13f2aea68ddf83fb7ad139f6ea338bb1b144/pypipegraph-0.184.tar.gz" } ], "0.185": [ { "comment_text": "", "digests": { "md5": "6cc3fb3a6b07b0b51baaa9f03af6f9cc", "sha256": "a57ccd42f4a2a32591af1250a4b158b5165dc20ad7ea16229acecf37180d8752" }, "downloads": -1, "filename": "pypipegraph-0.185.tar.gz", "has_sig": false, "md5_digest": "6cc3fb3a6b07b0b51baaa9f03af6f9cc", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 105563, "upload_time": "2019-05-07T13:44:13", "url": "https://files.pythonhosted.org/packages/3a/1b/b2b3e3446b1f75e7f8f7fb4c42c992928374489bbdec41bcf7cd19b66733/pypipegraph-0.185.tar.gz" } ], "0.187": [ { "comment_text": "", "digests": { "md5": "4f833cee6b5eed5494c5f1703ca412ab", "sha256": "9f4428c60bf6237198ed0b10eebb0d7e1c77cb288b8bc5942060ba8f7be975ea" }, "downloads": -1, "filename": "pypipegraph-0.187.tar.gz", "has_sig": false, "md5_digest": "4f833cee6b5eed5494c5f1703ca412ab", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 108018, "upload_time": "2019-08-28T09:42:25", "url": "https://files.pythonhosted.org/packages/73/32/cfaa8f2d433377bd9151e4dc205d1172f2df78a6c8f6c9210a60ca88d405/pypipegraph-0.187.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "4f833cee6b5eed5494c5f1703ca412ab", "sha256": "9f4428c60bf6237198ed0b10eebb0d7e1c77cb288b8bc5942060ba8f7be975ea" }, "downloads": -1, "filename": "pypipegraph-0.187.tar.gz", "has_sig": false, "md5_digest": "4f833cee6b5eed5494c5f1703ca412ab", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 108018, "upload_time": "2019-08-28T09:42:25", "url": "https://files.pythonhosted.org/packages/73/32/cfaa8f2d433377bd9151e4dc205d1172f2df78a6c8f6c9210a60ca88d405/pypipegraph-0.187.tar.gz" } ] }