{
    "info": {
        "author": "Ewald R. de Wit",
        "author_email": "ewald.de.wit@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: BSD License",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3 :: Only"
        ],
        "description": "|Build| |PyVersion| |Status| |PyPiVersion| |License| |Docs|\n\nIntroduction\n============\n\nDistex offers a distributed process pool to utilize multiple CPUs or machines.\nIt uses\n`asyncio <https://docs.python.org/3.6/library/asyncio.html>`_\nto efficiently manage the worker processes.\n\nFeatures:\n\n* Scales from 1 to 1000's of processors;\n* Can handle in the order of 50.000 small tasks per second;\n* Easy to use with SSH (secure shell) hosts;\n* Full async support;\n* Maps over unbounded iterables;\n* Compatible with\n  `concurrent.futures.ProcessPool <https://docs.python.org/3/library/concurrent.futures.html>`_\n  (or PEP3148_).\n\n\nInstallation\n------------\n\n::\n\n    pip3 install -U distex\n\nWhen using remote hosts then distex must be installed on those too.\nMake sure that the ``distex_proc`` script can be found in the path.\n\nFor SSH hosts: Authentication should be done with SSH keys since there is\nno support for passwords. The remote installation  can be tested with::\n\n    ssh <host> distex_proc\n\nDependencies:\n\n* Python_ version 3.6 or higher;\n* On Unix the ``uvloop`` package is recommended: ``pip3 install uvloop``\n* SSH client and server (optional).\n\nExamples\n--------\n\nA process pool can have local and remote workers.\nHere is a pool that uses 4 local workers:\n\n.. code-block:: python\n\n    from distex import Pool\n\n    def f(x):\n        return x*x\n\n    pool = Pool(4)\n    for y in pool.map(f, range(100)):\n        print(y)\n\nTo create a pool that also uses 8 workers on host ``maxi``, using ssh:\n\n.. code-block:: python\n\n    pool = Pool(4, 'ssh://maxi/8')\n\nTo use a pool in combination with\n`eventkit <https://github.com/erdewit/eventkit>`_:\n\n.. code-block:: python\n\n    from distex import Pool\n    import eventkit as ev\n    import bz2\n\n    pool = Pool()\n    # await pool  # un-comment in Jupyter\n    data = [b'A' * 1000000] * 1000\n\n    pipe = ev.Sequence(data).poolmap(pool, bz2.compress).map(len).mean().last()\n\n    print(pipe.run())  # in Jupyter: print(await pipe)\n    pool.shutdown()\n\nThere is full support for every asynchronous construct imaginable:\n\n.. code-block:: python\n\n    import asyncio\n    from distex import Pool\n\n    def init():\n        # pool initializer: set the start time for every worker\n        import time\n        import builtins\n        builtins.t0 = time.time()\n\n    async def timer(i=0):\n        # async code running in the pool\n        import time\n        import asyncio\n        await asyncio.sleep(1)\n        return time.time() - t0\n\n    async def ait():\n        # async iterator running on the user side\n        for i in range(20):\n            await asyncio.sleep(0.1)\n            yield i\n\n    async def main():\n        async with Pool(4, initializer=init, qsize=1) as pool:\n            async for t in pool.map_async(timer, ait()):\n                print(t)\n            print(await pool.run_on_all_async(timer))\n\n\n    loop = asyncio.get_event_loop()\n    loop.run_until_complete(main())\n\n\nHigh level architecture\n-----------------------\n\nDistex does not use remote 'task servers'.\nInstead it is done the other way around: A local\nserver is started first; Then the local and remote workers are started\nand each of them will connect on its own back to the server. When all\nworkers have connected then the pool is ready for duty.\n\nEach worker consists of a single-threaded process that is running\nan asyncio event loop. This loop is used both for communication and for\nrunning asynchronous tasks. Synchronous tasks are run in a blocking fashion.\n\nWhen using ssh, a remote (or 'reverse') tunnel is created from a remote Unix socket\nto the local Unix socket that the local server is listening on.\nMultiple workers on a remote machine will use the same Unix socket and\nshare the same ssh tunnel.\n\nThe plain ``ssh`` executable is used instead of much nicer solutions such\nas `AsyncSSH <https://github.com/ronf/asyncssh>`_. This is to keep the\nCPU usage of encrypting/decrypting outside of the event loop and offload\nit to the ``ssh`` process(es).\n\nDocumentation\n-------------\n\n`Distex documentation <http://rawgit.com/erdewit/distex/master/docs/html/api.html>`_\n\n\n:author: Ewald de Wit <ewald.de.wit@gmail.com>\n\n.. _Python: http://www.python.org\n\n.. _ssh-keygen: https://linux.die.net/man/1/ssh-keygen\n\n.. _ssh-copy-id: https://linux.die.net/man/1/ssh-copy-id\n\n.. _PEP3148: https://www.python.org/dev/peps/pep-3148\n\n.. |PyPiVersion| image:: https://img.shields.io/pypi/v/distex.svg\n   :alt: PyPi\n   :target: https://pypi.python.org/pypi/distex\n\n.. |Build| image:: https://travis-ci.org/erdewit/distex.svg?branch=master\n   :alt: Build\n   :target: https://travis-ci.org/erdewit/distex\n\n.. |PyVersion| image:: https://img.shields.io/badge/python-3.6+-blue.svg\n   :alt:\n\n.. |Status| image:: https://img.shields.io/badge/status-beta-green.svg\n   :alt:\n\n.. |License| image:: https://img.shields.io/badge/license-BSD-blue.svg\n   :alt:\n\n.. |Docs| image:: https://readthedocs.org/projects/distex/badge/?version=latest\n   :alt: Documentation\n   :target: https://distex.readthedocs.io/\n\n\n",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/erdewit/distex",
        "keywords": "python asyncio parallel distributed computing process pool task queue",
        "license": "BSD",
        "maintainer": "",
        "maintainer_email": "",
        "name": "distex",
        "package_url": "https://pypi.org/project/distex/",
        "platform": "",
        "project_url": "https://pypi.org/project/distex/",
        "project_urls": {
            "Homepage": "https://github.com/erdewit/distex"
        },
        "release_url": "https://pypi.org/project/distex/0.7.0/",
        "requires_dist": [
            "dill",
            "cloudpickle",
            "eventkit"
        ],
        "requires_python": "",
        "summary": "Async distributed process pool using asyncio",
        "version": "0.7.0"
    },
    "last_serial": 5967453,
    "releases": {
        "0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "29bee88baaa23e986bca851177a66473",
                    "sha256": "2bc7fe3036335e60760fae5dbd39a417ae7aca312a1f0ff448e4f453b8fc8d60"
                },
                "downloads": -1,
                "filename": "distex-0.5-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "29bee88baaa23e986bca851177a66473",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 18078,
                "upload_time": "2017-11-27T10:50:36",
                "url": "https://files.pythonhosted.org/packages/5a/0a/06d2a138db70e0aae2f7b4757edfa6c28b1a008719c719feeeb7233657f6/distex-0.5-py3-none-any.whl"
            }
        ],
        "0.5.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9019eaf5f52a5386685270778e060833",
                    "sha256": "5223efa397c1fd73609a94e8aa7045111a9f9f3f41060aca4c2632db022e3f3e"
                },
                "downloads": -1,
                "filename": "distex-0.5.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "9019eaf5f52a5386685270778e060833",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 18312,
                "upload_time": "2017-11-29T11:09:38",
                "url": "https://files.pythonhosted.org/packages/83/f8/2eb5e4874b5c75252269e40ebd4034db4f7ec988f04b4ec29f4b91052e2c/distex-0.5.1-py3-none-any.whl"
            }
        ],
        "0.5.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "890081f9162c706d1db7f28c171c8748",
                    "sha256": "ec022596816b26caead3578e28ebfabe6a65092c43102d804901f12fee39b494"
                },
                "downloads": -1,
                "filename": "distex-0.5.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "890081f9162c706d1db7f28c171c8748",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 18644,
                "upload_time": "2017-12-03T11:52:51",
                "url": "https://files.pythonhosted.org/packages/4a/dd/a3e467bf07bd57dfe45efd4253da97df6a6821c0a76ba93d6d3f6b012bb1/distex-0.5.2-py3-none-any.whl"
            }
        ],
        "0.5.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5d4d406bb39a29d0901d2dd524176f54",
                    "sha256": "70d6ff2249234cb345198ce45f69b19072b0a82bc2911b5aab20106139d07f93"
                },
                "downloads": -1,
                "filename": "distex-0.5.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "5d4d406bb39a29d0901d2dd524176f54",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 18693,
                "upload_time": "2017-12-04T14:49:16",
                "url": "https://files.pythonhosted.org/packages/80/42/113aff2055270c802a416e49f0d56da07bdbc57497ea13dfb3003ef30500/distex-0.5.3-py3-none-any.whl"
            }
        ],
        "0.5.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "662f9c4ea77028b5ff53a7ab48934576",
                    "sha256": "d00bdcf84b28d8ed0f77ba4e4f9075c7ba60a480eae2d3795b759c75009e2699"
                },
                "downloads": -1,
                "filename": "distex-0.5.4-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "662f9c4ea77028b5ff53a7ab48934576",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 19048,
                "upload_time": "2017-12-12T11:18:34",
                "url": "https://files.pythonhosted.org/packages/52/82/7bde6952c34e4ddd2f9d005f14e2815f05f5908f3ef21f39136dd6c6947f/distex-0.5.4-py3-none-any.whl"
            }
        ],
        "0.5.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "dc1d7f922f9176b6fb5c5e08307c9002",
                    "sha256": "8eeed7218871073ffc71911b9e6554e9e0305d7eadf9bb1feb7966325ba3681e"
                },
                "downloads": -1,
                "filename": "distex-0.5.5-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "dc1d7f922f9176b6fb5c5e08307c9002",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 19125,
                "upload_time": "2018-01-18T13:43:45",
                "url": "https://files.pythonhosted.org/packages/d2/0b/6d8b182c594e3534f346d232ea7e06dfa0bd48a38595da03208347e39a1d/distex-0.5.5-py3-none-any.whl"
            }
        ],
        "0.5.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "29fe0a324a352d19587cc8c6353872c2",
                    "sha256": "cb4d6268dc638efed613112d035f5e33aca4863d0db799653455ff9880729ee6"
                },
                "downloads": -1,
                "filename": "distex-0.5.6-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "29fe0a324a352d19587cc8c6353872c2",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 19356,
                "upload_time": "2018-01-25T21:53:42",
                "url": "https://files.pythonhosted.org/packages/40/7b/b174d1448f58eb32061bacd6db5655be714aac7b08e904536cdae610569b/distex-0.5.6-py3-none-any.whl"
            }
        ],
        "0.5.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "cd20844e7e256c4a6b0e7017f58de931",
                    "sha256": "2c6b905f9cdf4db35f5d1aa3e20035ababef0d2fe8dec15d38dc9aba477f5b3b"
                },
                "downloads": -1,
                "filename": "distex-0.5.7-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "cd20844e7e256c4a6b0e7017f58de931",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 16817,
                "upload_time": "2018-05-19T22:05:34",
                "url": "https://files.pythonhosted.org/packages/5b/1e/631c019f53304988463919312d5aec60a10653391068e1dc049fd581006c/distex-0.5.7-py3-none-any.whl"
            }
        ],
        "0.5.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "71439afec65482cef072996199dab7e9",
                    "sha256": "f0dde55524b1b0731d38ceb4eba30a0776e98f81b587cd0080822aa65b211032"
                },
                "downloads": -1,
                "filename": "distex-0.5.8-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "71439afec65482cef072996199dab7e9",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 16845,
                "upload_time": "2018-08-11T10:36:00",
                "url": "https://files.pythonhosted.org/packages/5c/8a/a69d8d2249dcc5e3af8d84db611f21ab13b274b985cdf5a6919527ac642e/distex-0.5.8-py3-none-any.whl"
            }
        ],
        "0.5.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "799005442b1b292f55e187c3b8e82286",
                    "sha256": "ed082c0276a311b62eb324a467bcd54b63fc51507859a166fd1757482ba94d85"
                },
                "downloads": -1,
                "filename": "distex-0.5.9.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "799005442b1b292f55e187c3b8e82286",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 30585,
                "upload_time": "2018-08-19T07:18:56",
                "url": "https://files.pythonhosted.org/packages/93/ea/18b9ce48a0ef82af619c445aede405e9df5231ac491768581306e8c2c56f/distex-0.5.9.linux-x86_64.tar.gz"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "fae474cb9ecec00a4c3eb41403d88b37",
                    "sha256": "1fe3e414a76d05aab4b7d9055c0c0d16a5b9899838ff7daa267d588826b91412"
                },
                "downloads": -1,
                "filename": "distex-0.5.9-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "fae474cb9ecec00a4c3eb41403d88b37",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 16917,
                "upload_time": "2018-08-19T07:18:54",
                "url": "https://files.pythonhosted.org/packages/eb/3f/5fd2f742af1f86db6e5641f1a84073ca34b4f0d6cd71e41d4907bce932bf/distex-0.5.9-py3-none-any.whl"
            }
        ],
        "0.6.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3aa0f08948973223abc27d124b331537",
                    "sha256": "63bf61d8f9f3da5b4c0040ea10d482feb0dcadf008813bfaa9d374129a537bdb"
                },
                "downloads": -1,
                "filename": "distex-0.6.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "3aa0f08948973223abc27d124b331537",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 18902,
                "upload_time": "2019-03-16T13:23:08",
                "url": "https://files.pythonhosted.org/packages/34/68/a922108b202347326001f628d2f96c211ee2e84fcac726a8085c8c75f77b/distex-0.6.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "071028a258801b01cb6eb63c5bc6a423",
                    "sha256": "be4ef4e0dc57db61c9cfd1076e783ad5b6b4231930f0620d5c5d0409373519fd"
                },
                "downloads": -1,
                "filename": "distex-0.6.0.tar.gz",
                "has_sig": false,
                "md5_digest": "071028a258801b01cb6eb63c5bc6a423",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16918,
                "upload_time": "2019-03-16T13:23:10",
                "url": "https://files.pythonhosted.org/packages/95/aa/97d7a364517b4c0539e0808fc8940c5f59440610f3c118cea1ebb80a794f/distex-0.6.0.tar.gz"
            }
        ],
        "0.7.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "26aca67baf1d89a6f429f3c3bed5c83e",
                    "sha256": "3e3fdd8e2aa4c2165b0cc6050c5d32a9e95b8311280b1b7bae8fb400c3723c1f"
                },
                "downloads": -1,
                "filename": "distex-0.7.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "26aca67baf1d89a6f429f3c3bed5c83e",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 19148,
                "upload_time": "2019-10-13T13:52:44",
                "url": "https://files.pythonhosted.org/packages/54/e7/96167111cfdf399191030c65e83ce562a7ea0eb49402c47768737aa3ce25/distex-0.7.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "07398f5dc87776788ff2acbba9ce6fe1",
                    "sha256": "2fdf336f53e9c834f36139e5071fa9d0258e0f911cda2225a9e2cc3e5bb36bd6"
                },
                "downloads": -1,
                "filename": "distex-0.7.0.tar.gz",
                "has_sig": false,
                "md5_digest": "07398f5dc87776788ff2acbba9ce6fe1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17140,
                "upload_time": "2019-10-13T13:52:46",
                "url": "https://files.pythonhosted.org/packages/c9/ac/54fc96c0040f96e2d9fefdfa3fec05130f52eefb51deff8505ac6f993884/distex-0.7.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "26aca67baf1d89a6f429f3c3bed5c83e",
                "sha256": "3e3fdd8e2aa4c2165b0cc6050c5d32a9e95b8311280b1b7bae8fb400c3723c1f"
            },
            "downloads": -1,
            "filename": "distex-0.7.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "26aca67baf1d89a6f429f3c3bed5c83e",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 19148,
            "upload_time": "2019-10-13T13:52:44",
            "url": "https://files.pythonhosted.org/packages/54/e7/96167111cfdf399191030c65e83ce562a7ea0eb49402c47768737aa3ce25/distex-0.7.0-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "07398f5dc87776788ff2acbba9ce6fe1",
                "sha256": "2fdf336f53e9c834f36139e5071fa9d0258e0f911cda2225a9e2cc3e5bb36bd6"
            },
            "downloads": -1,
            "filename": "distex-0.7.0.tar.gz",
            "has_sig": false,
            "md5_digest": "07398f5dc87776788ff2acbba9ce6fe1",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 17140,
            "upload_time": "2019-10-13T13:52:46",
            "url": "https://files.pythonhosted.org/packages/c9/ac/54fc96c0040f96e2d9fefdfa3fec05130f52eefb51deff8505ac6f993884/distex-0.7.0.tar.gz"
        }
    ]
}