{
    "info": {
        "author": "daniyall",
        "author_email": "dev.daniyall@gmail.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "A parallel pipelining framework for Python. Developers can create nodes\nand chain them together to create pipelines.\n\nClasses that extend ``Node`` must implement ``run`` method that will be\ncalled whenever new data is available.\n\nTable of Contents\n-----------------\n\n-  `Installation <#installation>`__\n-  `Example Usage <#example-usage>`__\n-  `Parallel Execution <#parallel-execution>`__\n-  `Stream Names <#stream-names>`__\n-  `Progress Updates <#progress-updates>`__\n-  `Projects Using PyPiper <#projects-using-pypiper>`__\n\nInstallation\n------------\n\n.. code:: bash\n\n   pip install pypiper\n\nExample Usage\n-------------\n\n.. code:: python\n\n   from pyPiper import Node, Pipeline\n\n   class Generate(Node):\n       def setup(self, size):\n           self.size = size\n           self.pos = 0\n\n       def run(self, data):\n           if self.pos < self.size:\n               self.emit(self.pos)\n               self.pos = self.pos + 1\n           else:\n               self.close()\n\n   class Square(Node):\n       def run(self, data):\n           self.emit(data**2)\n\n\n   pipeline = Pipeline(Generate(\"gen\", size=10) | Square(\"square\"))\n   print(pipeline)\n   pipeline.run()\n\nNodes can also specify a batch size that dictates how much data should\nbe pushed to the node. For example, building on the previous example. In\nthis case ``batch_size`` is specified in the nodes ``setup`` method.\nAlternatively, it can be set when creating the node (ex.\n``Printer(\"print\", batch_size=5)``)\n\n.. code:: python\n\n   class Printer(Node):\n       def setup(self):\n           self.batch_size = Node.BATCH_SIZE_ALL\n\n       def run(self, data):\n           print(data)\n\n   pipeline = Pipeline(Generate(\"gen\", size=10) | Square(\"square\") | Printer(\"print\"))\n   print(pipeline)\n   pipeline.run()\n\nParallel Execution\n------------------\n\nTo process pipelines in parallel, pass ``n_threads`` > 1 when creating\nthe pipeline. Parallel execution is done using ``multiprocessing`` and\nis well suited to CPU intensive tasks such as audio processing and\nfeature extraction. For example:\n\n.. code:: python\n\n   class Generate(Node):\n       def setup(self, size):\n           self.pos = 0\n\n       def run(self, data):\n           if self.pos < self.size:\n               self.emit(self.pos)\n               self.pos = self.pos + 1\n           else:\n               self.close()\n\n   pipeline = Pipeline(Generate(\"gen\", size=10) | Square(\"square\") | Printer(\"print\"), n_threads=2)\n   print(pipeline)\n   pipeline.run()\n\nStream Names\n------------\n\nYou can also name input and output streams. For example:\n\n.. code:: python\n\n   gen = EvenOddGenerate(\"gen\", size=20, out_streams=[\"even\", \"odd\"])\n   double = Double(\"double\", out_streams=\"num\", in_streams=\"even\")\n   square = Square(\"square\", out_streams=\"num\", in_streams=\"odd\")\n\n   printer1 = Printer(\"p1\", in_streams=\"num\", batch_size=Node.BATCH_SIZE_ALL)\n   printer2 = Printer(\"p2\", in_streams=\"num\", batch_size=Node.BATCH_SIZE_ALL)\n\n   p = Pipeline(gen | [double | printer1, square | printer2], quiet=False)\n\n   p.run()\n\nEvenOddGenerate generates a pair of numbers. using the ``out_streams``\nparameter, we name the first number even and second number odd. When\ninitializing the double and square nodes, we tell double to take the\neven number and square to take the odd number.\n\nIf multiple output streams are passed into a node, by default, they will\nbe come into the node as a list. For example,\n\n.. code:: python\n\n   gen = EvenOddGenerate(\"gen\", size=10, out_streams=[\"even\", \"odd\"])\n   printer = Printer(\"p1\", batch_size=1)\n\n   p = Pipeline(gen | printer, quiet=False)\n   p.run()\n\nWill output\n\n.. code:: python\n\n   [0,1],\n   [2,3],\n   ...\n\nHowever, if you can split the streams by specifying their names in the\ninput streams ``in_streams`` parameter. So, \\```python gen =\nEvenOddGenerate(\u201cgen\u201d, size=20, out_streams=[\u201ceven\u201d, \u201codd\u201d])\n\nprinter = Printer(\u201cp1\u201d, in_streams=[\u201ceven\u201d, \u201codd\u201d], batch_size=1)\n\np = Pipeline(gen \\| printer, quiet=False) p.run() \\``\\`\n\nWill generate:\n\n.. code:: python\n\n   0,\n   1,\n   2,\n   3,\n   ...\n\nProgress Updates\n----------------\n\nWhen calling ``pipeline.run()``, you can provide a callback function for\nprogress updates. Whenever the pipelines makes progress, it calls this\nfunction with the number of items that have been processed so far and\nthe total number of items that need to be processed. For example, if you\nwere using a tqdm progress bar, you could use the following code:\n\n.. code:: python\n\n   from tqdm import tqdm\n\n   class TqdmUpdate(tqdm):\n       def update(self, done, total_size=None):\n           if total_size is not None:\n               self.total = total_size\n           self.n = done\n           super().refresh()\n\n   if __name__ == '__main__':\n       gen = Generate(\"gen\", size=10)\n       double = Double(\"double\")\n       sleeper = Sleep(\"sleep\")\n\n       p = Pipeline(gen | [double, sleeper], n_threads=4, quiet=True)\n       with TqdmUpdate(desc=\"Progress\") as pbar:\n           p.run(update_callback=pbar.update)\n\nProjects Using PyPiper\n----------------------\n\n-  `COVFEFE <https://github.com/SPOClab-ca/COVFEFE>`__: A feature\n   extraction tool focusing on lexical, syntactic and pragmatic features\n   from text and audio features from audio.",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "https://github.com/daniyall/pyPiper/archive/0.5.3.tar.gz",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/daniyall/pyPiper",
        "keywords": "data-science,pipelining,stream-processing,data-analysis",
        "license": "LICENSE",
        "maintainer": "",
        "maintainer_email": "",
        "name": "pyPiper",
        "package_url": "https://pypi.org/project/pyPiper/",
        "platform": "",
        "project_url": "https://pypi.org/project/pyPiper/",
        "project_urls": {
            "Download": "https://github.com/daniyall/pyPiper/archive/0.5.3.tar.gz",
            "Homepage": "https://github.com/daniyall/pyPiper"
        },
        "release_url": "https://pypi.org/project/pyPiper/0.5.3/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "A pipelining framework designed for data analysis but can be useful to other applications",
        "version": "0.5.3"
    },
    "last_serial": 4821734,
    "releases": {
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4bd6d7f92c5f919fafe9b61ec1de4de2",
                    "sha256": "942282ec8b8bfc8dc219930c40702c2996d63d9be0de511426a88ef4c4864289"
                },
                "downloads": -1,
                "filename": "pyPiper-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "4bd6d7f92c5f919fafe9b61ec1de4de2",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2816,
                "upload_time": "2018-01-07T20:22:59",
                "url": "https://files.pythonhosted.org/packages/2b/d4/d4fcac077cd3ac1cbf401c8218165343b6cd2a30508d9193447fdd12d0dc/pyPiper-0.2.0.tar.gz"
            }
        ],
        "0.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "117d8ed2473521996ccace650a42affa",
                    "sha256": "16cc11d288288472253d8dc6883e14f8482c0889cce45edcb2b653669087b896"
                },
                "downloads": -1,
                "filename": "pyPiper-0.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "117d8ed2473521996ccace650a42affa",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2921,
                "upload_time": "2018-01-07T20:30:43",
                "url": "https://files.pythonhosted.org/packages/25/fc/8387a62385fe6efccf82a67c6f9b70860f9f865e2158b7f730315faf8d3b/pyPiper-0.2.1.tar.gz"
            }
        ],
        "0.2.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2b0b99a6262352beef3190741ba85bd7",
                    "sha256": "ad9f3974246cd92b3ae4c0e085e9076ca4f51b47ee2c34975d997dd111e10234"
                },
                "downloads": -1,
                "filename": "pyPiper-0.2.2.tar.gz",
                "has_sig": false,
                "md5_digest": "2b0b99a6262352beef3190741ba85bd7",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16465,
                "upload_time": "2018-01-08T23:03:05",
                "url": "https://files.pythonhosted.org/packages/e8/e4/5de57a78a739fb16e228df2da7e78ca3bfecbe1c835a610b68f7e5bfb87f/pyPiper-0.2.2.tar.gz"
            }
        ],
        "0.2.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "41d138f2d2c65953878f0cf1ecc4cfa4",
                    "sha256": "65f45949ada3ef4ae1b7461a2e8798623540b2fa6b9c68717a629c2bbcab4160"
                },
                "downloads": -1,
                "filename": "pyPiper-0.2.3.tar.gz",
                "has_sig": false,
                "md5_digest": "41d138f2d2c65953878f0cf1ecc4cfa4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16463,
                "upload_time": "2018-01-08T23:15:05",
                "url": "https://files.pythonhosted.org/packages/62/ab/4cf4bd64a528fc731147cf55eff55d9de8861cbc74680b25b2bb77901102/pyPiper-0.2.3.tar.gz"
            }
        ],
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5c87fbb7cf299376c90ef208a84c9ca4",
                    "sha256": "0ef2c7266b6810ba3b28b20062a9c0b60bee3d6797c534a714b49220054923f3"
                },
                "downloads": -1,
                "filename": "pyPiper-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "5c87fbb7cf299376c90ef208a84c9ca4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16484,
                "upload_time": "2018-01-08T23:58:40",
                "url": "https://files.pythonhosted.org/packages/26/38/c1fe691aa90ad964719e84deb2de0410744a0b67aea3fe6fd5a013dc777a/pyPiper-0.3.0.tar.gz"
            }
        ],
        "0.3.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6d13bb957e7adbc115061c5feaa77a47",
                    "sha256": "feed088d7660791eb234e07983ea80fe99fd0fc15f8f70600e4e97110a11851b"
                },
                "downloads": -1,
                "filename": "pyPiper-0.3.1.tar.gz",
                "has_sig": false,
                "md5_digest": "6d13bb957e7adbc115061c5feaa77a47",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16484,
                "upload_time": "2018-01-09T00:02:45",
                "url": "https://files.pythonhosted.org/packages/27/25/75d785da7765cdf3897b17fac903b6278211f62bc316c402b7f4dec2bd69/pyPiper-0.3.1.tar.gz"
            }
        ],
        "0.3.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b7dbc4bfb80f6fc9d3b32183d06a485c",
                    "sha256": "71bbb9fe01533e8d0caebc588b1e1be228e55e4d16dfffb0b6c6d357a3f74b30"
                },
                "downloads": -1,
                "filename": "pyPiper-0.3.2.tar.gz",
                "has_sig": false,
                "md5_digest": "b7dbc4bfb80f6fc9d3b32183d06a485c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16624,
                "upload_time": "2018-01-09T02:20:08",
                "url": "https://files.pythonhosted.org/packages/91/e9/dd64d6868523bd6ec14dafc7a42609d39619de1935cb3c0e9720c1537568/pyPiper-0.3.2.tar.gz"
            }
        ],
        "0.3.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "149d160d2e09aaa2f7ba1d91ded319e4",
                    "sha256": "61b37dc943ba06926c956b87ca3083b0866a9454e097d2c58093dcd17d794e5a"
                },
                "downloads": -1,
                "filename": "pyPiper-0.3.3.tar.gz",
                "has_sig": false,
                "md5_digest": "149d160d2e09aaa2f7ba1d91ded319e4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17082,
                "upload_time": "2018-01-09T23:16:36",
                "url": "https://files.pythonhosted.org/packages/cc/59/4b4b6b44c3836a23c00ca9d6e70a4a31cefec6f113bc18a19f8a961ab8ff/pyPiper-0.3.3.tar.gz"
            }
        ],
        "0.3.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a8785731d16b55f16aaf4aaff821b32d",
                    "sha256": "1df488a4776f7f1b81923f7a2cf499907f142972258d472f8957ae6d79695701"
                },
                "downloads": -1,
                "filename": "pyPiper-0.3.4.tar.gz",
                "has_sig": false,
                "md5_digest": "a8785731d16b55f16aaf4aaff821b32d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16974,
                "upload_time": "2018-02-01T17:04:09",
                "url": "https://files.pythonhosted.org/packages/c0/e0/5c1fb9059bc332174272e56eb3c584830be8a739cac6842d3649ce94e904/pyPiper-0.3.4.tar.gz"
            }
        ],
        "0.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "17233df10ed8ad6d025e481af8de3d81",
                    "sha256": "d7a77f495beafea1b638dff4b33995ac79194349c59e726b09512e77555e3399"
                },
                "downloads": -1,
                "filename": "pyPiper-0.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "17233df10ed8ad6d025e481af8de3d81",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19040,
                "upload_time": "2018-07-16T18:58:00",
                "url": "https://files.pythonhosted.org/packages/a3/14/c682c061cdb8c64fd2abd7ad714810e4808d675808ee9c5009dc746d808f/pyPiper-0.4.0.tar.gz"
            }
        ],
        "0.4.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8fd24ebb02c4b01cbed1544bc036792a",
                    "sha256": "61278b6795c2ddfb3b85d103d822994d851d09d55a5bd33615ec5662be78376c"
                },
                "downloads": -1,
                "filename": "pyPiper-0.4.1.tar.gz",
                "has_sig": false,
                "md5_digest": "8fd24ebb02c4b01cbed1544bc036792a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19042,
                "upload_time": "2018-07-16T20:16:43",
                "url": "https://files.pythonhosted.org/packages/bd/58/4a13d5f18a91c30138d45ea86ba5027eb0b80ccd8de9212380718c020e5e/pyPiper-0.4.1.tar.gz"
            }
        ],
        "0.5.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2439c50cd475bafd36940eacc5d3bc09",
                    "sha256": "32f125408217a234e846bf67ce2adc16b5cd2156f0be11b201575d4a9e2f2ca4"
                },
                "downloads": -1,
                "filename": "pyPiper-0.5.0.tar.gz",
                "has_sig": false,
                "md5_digest": "2439c50cd475bafd36940eacc5d3bc09",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 20189,
                "upload_time": "2018-08-23T15:11:57",
                "url": "https://files.pythonhosted.org/packages/1d/5c/4ba5908d37e7ec989948a1de323b85acd98205ad93c319b2151b8349a970/pyPiper-0.5.0.tar.gz"
            }
        ],
        "0.5.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9ce98b7f6646bd8c14424f6f7f931cb9",
                    "sha256": "df55b2c0a0fa835f24eb8caf4f06012f50ff638556ee5ccd4ea9f4c958a2b77c"
                },
                "downloads": -1,
                "filename": "pyPiper-0.5.1.tar.gz",
                "has_sig": false,
                "md5_digest": "9ce98b7f6646bd8c14424f6f7f931cb9",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 21380,
                "upload_time": "2018-10-10T03:08:16",
                "url": "https://files.pythonhosted.org/packages/9c/bf/c6624a0dbc596dbb5e6ddc35338087930ad18f3294a9439bdaf62392a068/pyPiper-0.5.1.tar.gz"
            }
        ],
        "0.5.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2c6e76c57e01ac4f26d6ac87e5ed2ff8",
                    "sha256": "6d954ffe5d8fec729455b10e6325e41566c8043ee07be4177e3e380e81d8ba24"
                },
                "downloads": -1,
                "filename": "pyPiper-0.5.2.tar.gz",
                "has_sig": false,
                "md5_digest": "2c6e76c57e01ac4f26d6ac87e5ed2ff8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 21378,
                "upload_time": "2018-11-19T21:14:07",
                "url": "https://files.pythonhosted.org/packages/59/d8/91f30a2a230a4411c18ff927eef2cb9e1045d8da94a1e02e51a23ce994fc/pyPiper-0.5.2.tar.gz"
            }
        ],
        "0.5.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5cd4c8f6e17861866692ef31bd97eaa7",
                    "sha256": "bdf39c8d590922252dd30e3acdba93542622f11651436b2c942f3bb02a13f114"
                },
                "downloads": -1,
                "filename": "pyPiper-0.5.3.tar.gz",
                "has_sig": false,
                "md5_digest": "5cd4c8f6e17861866692ef31bd97eaa7",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 22188,
                "upload_time": "2019-02-14T19:02:11",
                "url": "https://files.pythonhosted.org/packages/9d/53/0f5a4e815b27f40a73eb5f65d9b1a2c847ecf6e163077716d54b42c5160b/pyPiper-0.5.3.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "5cd4c8f6e17861866692ef31bd97eaa7",
                "sha256": "bdf39c8d590922252dd30e3acdba93542622f11651436b2c942f3bb02a13f114"
            },
            "downloads": -1,
            "filename": "pyPiper-0.5.3.tar.gz",
            "has_sig": false,
            "md5_digest": "5cd4c8f6e17861866692ef31bd97eaa7",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 22188,
            "upload_time": "2019-02-14T19:02:11",
            "url": "https://files.pythonhosted.org/packages/9d/53/0f5a4e815b27f40a73eb5f65d9b1a2c847ecf6e163077716d54b42c5160b/pyPiper-0.5.3.tar.gz"
        }
    ]
}