{
"info": {
"author": "python-paipa contributors",
"author_email": "python-tribe@stylight.com",
"bugtrack_url": null,
"classifiers": [
"Intended Audience :: Developers",
"License :: OSI Approved",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Topic :: Software Development :: Libraries :: Python Modules"
],
"description": "paipa - the threaded data pipelining framework\n==============================================\n\nPython pipeline library developed by `STYLIGHT `_.\n\n.. image:: https://travis-ci.org/stylight/python-paipa.svg?branch=master\n :target: https://travis-ci.org/stylight/python-paipa\n\n\nInstallation\n------------\n\n pip install paipa\n\nQuick summary\n-------------\n\nThe lib allows for *threaded pipelines* as well as so-called *co-routine\npipelines*. The main use-case is limiting peak memory usage when doing complex\noperations on large-ish (or small-ish) data-sets.\n\nQuick teaser (for more examples, see docs_):\n\n >>> pipeline = paipa.Pipeline(\n ... [\n ... (DownloadImage, 4),\n ... (StoreDatabase, 1),\n ... ],\n ... )\n >>> pipeline.run()\n\nThis example will create 5 threads, 4 for downloading images, one for storing\nstuff to the database. The outputs of all ``DownloadImage`` steps will be\nforwarded to the one ``StoreDatabase`` thread via a ``Queue``.\n\nThread startup and tear-down is handled by the library and doesn't concern the\nprogrammer at all. All (known) failure cases lead to either a re-spawning of\nthe failed thread or a controlled shutdown of the system. In no case the system\nshould block and do nothing, if it does then it's definitely a bug and needs\nto be reported.\n\nPipeline ingestion can be done via a separate thread or by consuming an\niterable. In the case of using an iterable, an ingestion thread is created\nwhich consumes the iterable in a controlled manner.\n\n.. _docs:\n\nDocumentation\n-------------\n\n - Threaded Pipeline\n - `Introduction `__\n - `More ways to use pipelines `__\n - `How to write steps `__\n - Coroutine Pipeline\n - `How to write a coroutine pipeline `__\n\n\nFeatures and non-features\n-------------------------\n\nWhat it does:\n\n - process stuff concurrently, but one step after another\n - allows class based, iterator based and function based steps\n - can scale steps independent of each other (manually)\n - don't ever do deadlocks\n - never expose the developer to the concept of a thread\n (if she doesn't like to)\n - run in finite batches where all threads are terminated at the end\n - run in continuous mode while being fed through a queue\n - automatically rate limit each step to minimize memory usage\n - terminate the pipeline in case of an Exception and propagate the error\n to the developer, handling graceful shutdown of all threads involved\n\nWhat it explicitly doesn't do (if it doesn't, it's not a bug!):\n\n - pipelines with multiple different parents. Multiple parents *of\n the same type* are handled automatically though.\n - auto-scale threads\n - bake bread (haha)\n\nWhat it shouldn't do (if it does, it's a bug!):\n\n - confuse the user/developer\n - hang on termination\n - hog resources\n",
"description_content_type": null,
"docs_url": null,
"download_url": "",
"downloads": {
"last_day": -1,
"last_month": -1,
"last_week": -1
},
"home_page": "https://github.com/stylight/python-paipa",
"keywords": "",
"license": "Apache Software License 2.0",
"maintainer": "",
"maintainer_email": "",
"name": "paipa",
"package_url": "https://pypi.org/project/paipa/",
"platform": "",
"project_url": "https://pypi.org/project/paipa/",
"project_urls": {
"Homepage": "https://github.com/stylight/python-paipa"
},
"release_url": "https://pypi.org/project/paipa/0.3.3/",
"requires_dist": null,
"requires_python": "",
"summary": "Python pipeline library. Maori: (noun) pipe.",
"version": "0.3.3"
},
"last_serial": 3481234,
"releases": {
"0.0.2": [
{
"comment_text": "",
"digests": {
"md5": "060a6f2d9dc22bc7fbc6e1b7bbe78d9c",
"sha256": "3659b87592ef2ae1088afb3d38c0e11d274182a1357b302a25e83e8173e89f3f"
},
"downloads": -1,
"filename": "paipa-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "060a6f2d9dc22bc7fbc6e1b7bbe78d9c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 13875,
"upload_time": "2016-01-19T16:25:34",
"url": "https://files.pythonhosted.org/packages/73/eb/6a1e078056bf34ecc666f92be1d40d88507eb1366fa183be36afa141881f/paipa-0.0.2.tar.gz"
}
],
"0.1.0": [
{
"comment_text": "",
"digests": {
"md5": "7dea2b3f636da9e9e776bdfa2e1ef99b",
"sha256": "60c1565b5c9a2eed39ffc9fa1d16cd8877c5b8ed5684b1c41295f722fa82441b"
},
"downloads": -1,
"filename": "paipa-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "7dea2b3f636da9e9e776bdfa2e1ef99b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 14447,
"upload_time": "2015-10-27T11:26:40",
"url": "https://files.pythonhosted.org/packages/d4/f6/226de6efaa227031c6902d681dde176dc62b2445af0d31a04995ffbbb23f/paipa-0.1.0.tar.gz"
}
],
"0.1.1": [
{
"comment_text": "",
"digests": {
"md5": "7cf8f8568281f8d8d4716c078b50b412",
"sha256": "676c18eda4fa1f478fb25a95145f19f38f34365ae960a5da060f580f1f639210"
},
"downloads": -1,
"filename": "paipa-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "7cf8f8568281f8d8d4716c078b50b412",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 15941,
"upload_time": "2016-05-10T13:50:20",
"url": "https://files.pythonhosted.org/packages/c5/5f/33fb863e69f27a1262fedb2a3e57e8be58a8a6912b6848f7a2a8e09bfb93/paipa-0.1.1.tar.gz"
}
],
"0.2.0": [
{
"comment_text": "",
"digests": {
"md5": "a3368fb6032cf355975100f96edbf477",
"sha256": "69bdc901833195a20e001f4809f3df1941ffa09782c8c2e5db889833934a163e"
},
"downloads": -1,
"filename": "paipa-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "a3368fb6032cf355975100f96edbf477",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 20346,
"upload_time": "2017-05-26T09:54:28",
"url": "https://files.pythonhosted.org/packages/45/83/f14308c61e8536c53b51e809a167ac0feb78ea6ba9b571e79f851475ec5a/paipa-0.2.0.tar.gz"
}
],
"0.2.1": [
{
"comment_text": "",
"digests": {
"md5": "08e976ff2b750c6d2c502e0fbced25ca",
"sha256": "a4d42a8a279a38b4ff619357e12302acbbec9238c681060ca12106eb29c66c3d"
},
"downloads": -1,
"filename": "paipa-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "08e976ff2b750c6d2c502e0fbced25ca",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 20600,
"upload_time": "2017-05-26T10:49:43",
"url": "https://files.pythonhosted.org/packages/c0/c7/f15e9a37a973638f940601a3820bc5dc7fdc056376630a083b010e728845/paipa-0.2.1.tar.gz"
}
],
"0.3.0": [
{
"comment_text": "",
"digests": {
"md5": "c9bb372f7259386ea84198148bc2ec19",
"sha256": "318f4618b6680f36ed95c520325cacaadd564d13c9908d9f0b4c17bc228eb00a"
},
"downloads": -1,
"filename": "paipa-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "c9bb372f7259386ea84198148bc2ec19",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 79530,
"upload_time": "2017-06-16T11:43:04",
"url": "https://files.pythonhosted.org/packages/69/73/a6bc9c4c9ceed5c08175b38bb4da6d0eba0aa7eb52aec3feac234e9f4e3f/paipa-0.3.0.tar.gz"
}
],
"0.3.1": [
{
"comment_text": "",
"digests": {
"md5": "47939a4b6945d1d69c249e8bd318fd86",
"sha256": "3c27732a7b39d5ebef12f27b559a0b3767a27cbef3ea3d117d473b66c209bf9b"
},
"downloads": -1,
"filename": "paipa-0.3.1-py2.7-linux-x86_64.egg",
"has_sig": false,
"md5_digest": "47939a4b6945d1d69c249e8bd318fd86",
"packagetype": "bdist_egg",
"python_version": "2.7",
"requires_python": null,
"size": 229153,
"upload_time": "2017-07-10T14:45:02",
"url": "https://files.pythonhosted.org/packages/bd/c8/7f4419b1317ccfc14970a80149e405dbaf1122f1b715ba2eb6b491fbf1aa/paipa-0.3.1-py2.7-linux-x86_64.egg"
},
{
"comment_text": "",
"digests": {
"md5": "7e2e2bb2fabac91ef881d328b6fc3eb6",
"sha256": "8eac674601e631322a00058fc8baa800a092ef06510095605aa51f34672521b5"
},
"downloads": -1,
"filename": "paipa-0.3.1.tar.gz",
"has_sig": false,
"md5_digest": "7e2e2bb2fabac91ef881d328b6fc3eb6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 19335,
"upload_time": "2017-07-10T14:46:06",
"url": "https://files.pythonhosted.org/packages/04/14/79e073a4c95ae1eddc155cbc49e49edf85197362362ac85836fee0f9072f/paipa-0.3.1.tar.gz"
}
],
"0.3.2": [
{
"comment_text": "",
"digests": {
"md5": "0edd2fa1412d6b9ce35444fe4bdaf238",
"sha256": "5c672a29ba28d78ab352ddad777104a1bec8ce45220263b365edb6bc43010841"
},
"downloads": -1,
"filename": "paipa-0.3.2.tar.gz",
"has_sig": false,
"md5_digest": "0edd2fa1412d6b9ce35444fe4bdaf238",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 20420,
"upload_time": "2018-01-09T14:55:44",
"url": "https://files.pythonhosted.org/packages/1b/8e/4e74f00b3488670ba3661eb9f815abff57b96d19fce60fb2eeb8bdd925c5/paipa-0.3.2.tar.gz"
}
],
"0.3.3": [
{
"comment_text": "",
"digests": {
"md5": "48b5612b27db3159f4983a2877457873",
"sha256": "48db7fa3cc1e766cd087b9977bb411f5badffcb56baf33491d3ca0577f9d0638"
},
"downloads": -1,
"filename": "paipa-0.3.3.tar.gz",
"has_sig": false,
"md5_digest": "48b5612b27db3159f4983a2877457873",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 20516,
"upload_time": "2018-01-11T15:34:25",
"url": "https://files.pythonhosted.org/packages/82/8e/b3f9c8c7a7648d3dece778831daa109a5be39fd1d8d9708eff77dbb21f24/paipa-0.3.3.tar.gz"
}
]
},
"urls": [
{
"comment_text": "",
"digests": {
"md5": "48b5612b27db3159f4983a2877457873",
"sha256": "48db7fa3cc1e766cd087b9977bb411f5badffcb56baf33491d3ca0577f9d0638"
},
"downloads": -1,
"filename": "paipa-0.3.3.tar.gz",
"has_sig": false,
"md5_digest": "48b5612b27db3159f4983a2877457873",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 20516,
"upload_time": "2018-01-11T15:34:25",
"url": "https://files.pythonhosted.org/packages/82/8e/b3f9c8c7a7648d3dece778831daa109a5be39fd1d8d9708eff77dbb21f24/paipa-0.3.3.tar.gz"
}
]
}