{ "info": { "author": "Sergio Oller", "author_email": "sergioller@gmail.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.2", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6" ], "description": "parmap\n======\n\n.. image:: https://travis-ci.org/zeehio/parmap.svg?branch=master\n :target: https://travis-ci.org/zeehio/parmap\n\n.. image:: https://readthedocs.org/projects/parmap/badge/?version=latest\n :target: https://readthedocs.org/projects/parmap/?badge=latest\n :alt: Documentation Status\n\n.. image:: https://codecov.io/github/zeehio/parmap/coverage.svg?branch=master\n :target: https://codecov.io/github/zeehio/parmap?branch=master\n\n.. image:: https://codeclimate.com/github/zeehio/parmap/badges/gpa.svg\n :target: https://codeclimate.com/github/zeehio/parmap\n :alt: Code Climate\n\n\nThis small python module implements four functions: ``map`` and\n``starmap``, and their async versions ``map_async`` and ``starmap_async``.\n\nWhat does parmap offer?\n-----------------------\n\n- Provide an easy to use syntax for both ``map`` and ``starmap``.\n- Parallelize transparently whenever possible.\n- Pass additional positional and keyword arguments to parallelized functions.\n- Show a progress bar (requires `tqdm` as optional package)\n\nInstallation:\n-------------\n\n::\n\n \u00a0pip install tqdm # for progress bar support\n pip install parmap\n\n\nUsage:\n------\n\nHere are some examples with some unparallelized code parallelized with\nparmap:\n\nSimple parallelization example:\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n import parmap\n # You want to do:\n mylist = [1,2,3]\n argument1 = 3.14\n argument2 = True\n y = [myfunction(x, argument1, mykeyword=argument2) for x in mylist]\n # In parallel:\n y = parmap.map(myfunction, mylist, argument1, mykeyword=argument2)\n\n\nShow a progress bar:\n~~~~~~~~~~~~~~~~~~~~~\n\nRequires ``pip install tqdm``\n\n::\n\n # You want to do:\n y = [myfunction(x) for x in mylist]\n # In parallel, with a progress bar\n y = parmap.map(myfunction, mylist, pm_pbar=True)\n\n\nPassing multiple arguments:\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n # You want to do:\n z = [myfunction(x, y, argument1, argument2, mykey=argument3) for (x,y) in mylist]\n # In parallel:\n z = parmap.starmap(myfunction, mylist, argument1, argument2, mykey=argument3)\n\n # You want to do:\n listx = [1, 2, 3, 4, 5, 6]\n listy = [2, 3, 4, 5, 6, 7]\n param = 3.14\n param2 = 42\n listz = []\n for (x, y) in zip(listx, listy):\n listz.append(myfunction(x, y, param1, param2))\n # In parallel:\n listz = parmap.starmap(myfunction, zip(listx, listy), param1, param2)\n\n\nAdvanced: Multiple parallel tasks running in parallel\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIn this example, Task1 uses 5 cores, while Task2 uses 3 cores. Both tasks start\nto compute simultaneously, and we print a message as soon as any of the tasks\nfinishes, retreiving the result.\n\n::\n\n import parmap\n def task1(item):\n return 2*item\n\n def task2(item):\n return 2*item + 1\n\n items1 = range(500000)\n items2 = range(500)\n\n with parmap.map_async(task1, items1, pm_processes=5) as result1:\n with parmap.map_async(task2, items2, pm_processes=3) as result2:\n data_task1 = None\n data_task2 = None\n task1_working = True\n task2_working = True\n while task1_working or task2_working:\n result1.wait(0.1)\n if task1_working and result1.ready():\n print(\"Task 1 has finished!\")\n data_task1 = result1.get()\n task1_working = False\n result2.wait(0.1)\n if task2_working and result2.ready():\n print(\"Task 2 has finished!\")\n data_task2 = result2.get()\n task2_working = False\n #Further work with data_task1 or data_task2\n\n\nmap and starmap already exist. Why reinvent the wheel?\n---------------------------------------------------------\n\nThe existing functions have some usability limitations:\n\n- The built-in python function ``map`` [#builtin-map]_\n is not able to parallelize.\n- ``multiprocessing.Pool().starmap`` [#multiproc-starmap]_\n is only available in python-3.3 and later versions.\n- ``multiprocessing.Pool().map`` [#multiproc-map]_\n does not allow any additional argument to the mapped function.\n- ``multiprocessing.Pool().starmap`` allows passing multiple arguments,\n but in order to pass a constant argument to the mapped function you\n will need to convert it to an iterator using\n ``itertools.repeat(your_parameter)`` [#itertools-repeat]_\n\n``parmap`` aims to overcome this limitations in the simplest possible way.\n\nAdditional features in parmap:\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- Create a pool for parallel computation automatically if possible.\n- ``parmap.map(..., ..., pm_parallel=False)`` # disables parallelization\n- ``parmap.map(..., ..., pm_processes=4)`` # use 4 parallel processes\n- ``parmap.map(..., ..., pm_pbar=True)`` # show a progress bar (requires tqdm)\n- ``parmap.map(..., ..., pm_pool=multiprocessing.Pool())`` # use an existing\n pool, in this case parmap will not close the pool.\n- ``parmap.map(..., ..., pm_chunksize=3)`` # size of chunks (see\n multiprocessing.Pool().map)\n\nLimitations:\n-------------\n\n``parmap.map()`` and ``parmap.starmap()`` (and their async versions) have their own \narguments (``pm_parallel``, ``pm_pbar``...). Those arguments are never passed\nto the underlying function. In the following example, ``myfun`` will receive \n``myargument``, but not ``pm_parallel``. Do not write functions that require\nkeyword arguments starting with ``pm_``, as ``parmap`` may need them in the future.\n\n::\n\n parmap.map(myfun, mylist, pm_parallel=True, myargument=False)\n\nAdditionally, there are other keyword arguments that should be avoided in the\nfunctions you write, because of parmap backwards compatibility reasons. The list\nof conflicting arguments is: ``parallel``, ``chunksize``, ``pool``,\n``processes``, ``callback``, ``error_callback`` and ``parmap_progress``.\n\n\n\nAcknowledgments:\n----------------\n\nThis package started after `this question `__, \nwhen I offered this `answer `__, \ntaking the suggestions of J.F. Sebastian for his `answer `__\n\nKnown works using parmap\n---------------------------\n\n- Davide Gerosa, Michael Kesden, \"PRECESSION. Dynamics of spinning black-hole\n binaries with python.\" `arXiv:1605.01067 `__, 2016\n- Thibault de Boissiere, `Implementation of Deep learning papers `__, 2017\n - Wasserstein Generative Adversarial Networks `arXiv:1701.07875 `__\n - pix2pix `arXiv:1611.07004 `__\n - Improved Techniques for Training Generative Adversarial Networks `arXiv:1606.03498 `__\n - Colorful Image Colorization `arXiv:1603.08511 `__\n - Deep Feature Interpolation for Image Content Changes `arXiv:1611.05507 `__\n - InfoGAN `arXiv:1606.03657 `__\n- Geoscience Australia, `SIFRA, a System for Infrastructure Facility Resilience Analysis `__, 2017\n- Andr\u00e9 F. Rendeiro, Christian Schmidl, Jonathan C. Strefford, Renata Walewska, Zadie Davis, Matthias Farlik, David Oscier, Christoph Bock \"Chromatin accessibility maps of chronic lymphocytic leukemia identify subtype-specific epigenome signatures and transcription regulatory networks\" Nat. Commun. 7:11938 doi: 10.1038/ncomms11938 (2016). `Paper `__, `Code `__\n\n\nReferences\n-----------\n\n.. [#builtin-map] http://docs.python.org/dev/library/functions.html#map\n.. [#multiproc-starmap] http://docs.python.org/dev/library/multiprocessing.html#multiprocessing.pool.Pool.starmap\n.. [#multiproc-map] http://docs.python.org/dev/library/multiprocessing.html#multiprocessing.pool.Pool.map\n.. [#itertools-repeat] http://docs.python.org/2/library/itertools.html#itertools.repeat\n\n\n\n", "description_content_type": "text/x-rst", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/zeehio/parmap", "keywords": "", "license": "APACHE-2.0", "maintainer": "", "maintainer_email": "", "name": "parmap", "package_url": "https://pypi.org/project/parmap/", "platform": "", "project_url": "https://pypi.org/project/parmap/", "project_urls": { "Homepage": "https://github.com/zeehio/parmap" }, "release_url": "https://pypi.org/project/parmap/1.5.2/", "requires_dist": [ "tqdm (>=4.8.4) ; extra == 'progress_bar'" ], "requires_python": "", "summary": "map and starmap implementations passing additional arguments and parallelizing if possible", "version": "1.5.2" }, "last_serial": 5296256, "releases": { "1.2.0": [ { "comment_text": "", "digests": { "md5": "0df2bc147c852a2b116f7bce3eadab03", "sha256": "0e625b8c026a665e5339b954496ed71095d8e48d9c814fc98156c1d49c26bad8" }, "downloads": -1, "filename": "parmap-1.2.0.tar.gz", "has_sig": false, "md5_digest": "0df2bc147c852a2b116f7bce3eadab03", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7652, "upload_time": "2014-01-23T11:48:13", "url": "https://files.pythonhosted.org/packages/18/0c/062dfee2e27e0feb7f03b00a0f94bdbee86f5bb7f81f1ce8fad24bfcc91c/parmap-1.2.0.tar.gz" } ], "1.2.1": [ { "comment_text": "", "digests": { "md5": "74619438db1214f11c802d5dc75c492e", "sha256": "7b16cee8f65f90735531ce7f7520ee9f9cf223b854ad8292601c7aed3990fe56" }, "downloads": -1, "filename": "parmap-1.2.1.tar.gz", "has_sig": false, "md5_digest": "74619438db1214f11c802d5dc75c492e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7714, "upload_time": "2014-05-16T11:26:17", "url": "https://files.pythonhosted.org/packages/f7/6c/1fd276ed57c2fd58091fc87d84bbb2ebc8e08c84a13b95e000e986a0e8b7/parmap-1.2.1.tar.gz" } ], "1.2.2": [ { "comment_text": "", "digests": { "md5": "6bb34f24fefa4ce94d2a2da8cadcdb55", "sha256": "89273ad0c4745b3655c0119e85ff2a30c858a35a65df48ff6c397a83b3d1150d" }, "downloads": -1, "filename": "parmap-1.2.2.tar.gz", "has_sig": false, "md5_digest": "6bb34f24fefa4ce94d2a2da8cadcdb55", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7673, "upload_time": "2015-04-23T07:02:02", "url": "https://files.pythonhosted.org/packages/ee/f6/2ee4563496e3005d66ce4231d4f3ec7cc2d4b1bbb540818e8d369318c721/parmap-1.2.2.tar.gz" } ], "1.2.3": [ { "comment_text": "", "digests": { "md5": "44179108f54015dabf7e46d17ee61851", "sha256": "7437566648f505d63b00429cb65538f08b9a4e45c6ec958af9e77a69512588bd" }, "downloads": -1, "filename": "parmap-1.2.3.tar.gz", "has_sig": true, "md5_digest": "44179108f54015dabf7e46d17ee61851", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15593, "upload_time": "2015-08-31T15:39:12", "url": "https://files.pythonhosted.org/packages/e2/7c/fa3106ea0f3e4624a356b0155a8141f3ffd69e24aefa7d608f2baf438ba4/parmap-1.2.3.tar.gz" } ], "1.2dev": [ { "comment_text": "", "digests": { "md5": "fc68122df4cb02891988e866d3c9dd6a", "sha256": "ecea0b137f4a3d73b941d8f0daecfe32f4c78ea130393ac54713da7a31a9785f" }, "downloads": -1, "filename": "parmap-1.2dev.tar.gz", "has_sig": false, "md5_digest": "fc68122df4cb02891988e866d3c9dd6a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7657, "upload_time": "2014-01-23T11:45:49", "url": "https://files.pythonhosted.org/packages/50/26/664b53d918c89a78bec58223291b418484b0f8256db307093ce0bef900b2/parmap-1.2dev.tar.gz" } ], "1.3.0": [ { "comment_text": "", "digests": { "md5": "4a0783a420e11d5dc1bc840974c48edc", "sha256": "9d34552df102d08a99a37ea2afe5aaabafd988db5bde5cb7b975a0531ea8d477" }, "downloads": -1, "filename": "parmap-1.3.0-py2.py3-none-any.whl", "has_sig": true, "md5_digest": "4a0783a420e11d5dc1bc840974c48edc", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 7420, "upload_time": "2016-05-30T14:30:15", "url": "https://files.pythonhosted.org/packages/1e/ee/4b87cbd23ae86e80e524b159baa13826d37b751c78c1fbe4dc1093cc7ed7/parmap-1.3.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "6029610257fafe29b1db633ef4102101", "sha256": "40089d4ae24fd5da91bd9460ed5db874fdd35e222617a56f24ddd226e4eb67ea" }, "downloads": -1, "filename": "parmap-1.3.0.tar.gz", "has_sig": true, "md5_digest": "6029610257fafe29b1db633ef4102101", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18425, "upload_time": "2016-05-30T14:29:58", "url": "https://files.pythonhosted.org/packages/da/41/05c40d3955e0ba6464369bad9da63c44fb4447d28bfd19dcce12018b723f/parmap-1.3.0.tar.gz" } ], "1.4.0": [ { "comment_text": "", "digests": { "md5": "c8e94e527bb064adb9aa8e24874ec641", "sha256": "c205a09725f26f9be0ebb62d709632ac3f3d05e2de9f9f2cd762750752843206" }, "downloads": -1, "filename": "parmap-1.4.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "c8e94e527bb064adb9aa8e24874ec641", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 7992, "upload_time": "2017-07-19T09:39:43", "url": "https://files.pythonhosted.org/packages/a1/a3/8dbcec81fe039e79dc2608859fd59ea62e208ae4ef172b4439a8cb139bdc/parmap-1.4.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "6e01f80f6b38b950e471e0cf61bcaaae", "sha256": "09f66abe7235de942ebb25cbb3c4786de9beedccefeef4fefb59bb5c40777f6a" }, "downloads": -1, "filename": "parmap-1.4.0.tar.gz", "has_sig": true, "md5_digest": "6e01f80f6b38b950e471e0cf61bcaaae", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 19085, "upload_time": "2017-07-19T09:39:44", "url": "https://files.pythonhosted.org/packages/6e/d2/400781417eaf123e5ff394ce428dc3e4175f6065b1f20ed5f6d6a3307210/parmap-1.4.0.tar.gz" } ], "1.5.0": [ { "comment_text": "", "digests": { "md5": "f6e1ad3121d962926e7d64dc6f1233fa", "sha256": "c9a87ae5399f13820616758c8a256d06f2b89f11159ff3ff0a02ed9d0c387d91" }, "downloads": -1, "filename": "parmap-1.5.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "f6e1ad3121d962926e7d64dc6f1233fa", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 11381, "upload_time": "2017-09-21T10:29:13", "url": "https://files.pythonhosted.org/packages/5c/24/d2b50d27f36f02cbf70dee6b7650c7294920a0eb28289b98ce8fc4667c2e/parmap-1.5.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "373163e995029556398201085fba67c5", "sha256": "73bf656e80842a872a855ca6aabd1ea69d26ae04b6d20f3485b131ec62f5645e" }, "downloads": -1, "filename": "parmap-1.5.0.tar.gz", "has_sig": false, "md5_digest": "373163e995029556398201085fba67c5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 20692, "upload_time": "2017-09-21T10:28:38", "url": "https://files.pythonhosted.org/packages/df/54/b2d3241ce0ea6dbb211e5835bd5bed1c786b3f756e9d230a2c1ff5f57f30/parmap-1.5.0.tar.gz" } ], "1.5.1": [ { "comment_text": "", "digests": { "md5": "c78bebf6b8d13f925f5f228f8ec4bc3f", "sha256": "b459525a076ddba31c55e4721c79c5f4693f1908fbdb2cb9090b2ed9d8cf03ef" }, "downloads": -1, "filename": "parmap-1.5.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "c78bebf6b8d13f925f5f228f8ec4bc3f", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 11385, "upload_time": "2017-09-21T11:46:52", "url": "https://files.pythonhosted.org/packages/33/8f/608aefca30002d69d18535cf65c3dba9ce1610f01ab112138e1cd3e31f76/parmap-1.5.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "1bbceeb5b9f0453c6ca48d4fd043dbdc", "sha256": "793ab5a5fe58d6b110b4e5cec72606c5666ec9384dbceb431084bd43b575343e" }, "downloads": -1, "filename": "parmap-1.5.1.tar.gz", "has_sig": false, "md5_digest": "1bbceeb5b9f0453c6ca48d4fd043dbdc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 20758, "upload_time": "2017-09-21T11:46:54", "url": "https://files.pythonhosted.org/packages/b7/ca/12da4e44931135d2b76e758adb3def95423c08d76b744cb10e6781c3603b/parmap-1.5.1.tar.gz" } ], "1.5.2": [ { "comment_text": "", "digests": { "md5": "fdcb581d464db7a6605e38bc9bdb915d", "sha256": "1724d30b66c05ce462990fb667e421c27b65331599738dd11e899e4758a88d76" }, "downloads": -1, "filename": "parmap-1.5.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "fdcb581d464db7a6605e38bc9bdb915d", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 15796, "upload_time": "2019-05-21T07:25:06", "url": "https://files.pythonhosted.org/packages/39/2d/6754856b5d0fdff6651c1230f0a971d4d5c08a2b91361d4e7c91303c2f60/parmap-1.5.2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "574a505aaee3a01fc55370b88b0749fb", "sha256": "a15f33fa0e2a454f29aecfdd5a825c780b7d26d876eb6093476ddf5c47c8f9b9" }, "downloads": -1, "filename": "parmap-1.5.2.tar.gz", "has_sig": false, "md5_digest": "574a505aaee3a01fc55370b88b0749fb", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24110, "upload_time": "2019-05-21T07:25:07", "url": "https://files.pythonhosted.org/packages/c5/57/0fafe2d6a750ae94bc4d0191eb60002d16a1393fd343041c768eb54ceaf9/parmap-1.5.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "fdcb581d464db7a6605e38bc9bdb915d", "sha256": "1724d30b66c05ce462990fb667e421c27b65331599738dd11e899e4758a88d76" }, "downloads": -1, "filename": "parmap-1.5.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "fdcb581d464db7a6605e38bc9bdb915d", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 15796, "upload_time": "2019-05-21T07:25:06", "url": "https://files.pythonhosted.org/packages/39/2d/6754856b5d0fdff6651c1230f0a971d4d5c08a2b91361d4e7c91303c2f60/parmap-1.5.2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "574a505aaee3a01fc55370b88b0749fb", "sha256": "a15f33fa0e2a454f29aecfdd5a825c780b7d26d876eb6093476ddf5c47c8f9b9" }, "downloads": -1, "filename": "parmap-1.5.2.tar.gz", "has_sig": false, "md5_digest": "574a505aaee3a01fc55370b88b0749fb", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24110, "upload_time": "2019-05-21T07:25:07", "url": "https://files.pythonhosted.org/packages/c5/57/0fafe2d6a750ae94bc4d0191eb60002d16a1393fd343041c768eb54ceaf9/parmap-1.5.2.tar.gz" } ] }