{
    "info": {
        "author": "Jelle Aalbers",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "npshmex\n=======\n\n[![Build Status](https://travis-ci.org/JelleAalbers/npshmex.svg?branch=master)](https://travis-ci.org/JelleAalbers/npshmex)\n\nNpshmex provides a drop-in replacement for concurrent.futures.ProcessPoolExecutor,\nusing shared memory provided by [SharedArray](https://gitlab.com/tenzing/shared-array) \n(rather than pickle) to transfer numpy arrays between processes.\n\nSynopsis:\n```python\nimport numpy as np\nfrom npshmex import ProcessPoolExecutor\n\ndef add_one(x):\n    return x + 1\n\nex = ProcessPoolExecutor()\nbig_data = np.ones(int(2e7))\n\nf = ex.submit(add_one, big_data)\nprint(f.result()[0])           # 2.0\n```\nThe last two lines take about ~290 ms on my laptop, but ~1250 ms using \n`concurrent.futures.ProcessPoolExecutor`: more than a factor four difference.\nTo run this example, npshmex claims only half as much memory as the standard library\n`ProcessPoolExecutor`, based on the threshold at which I get a MemoryError.\n\nFor the trivial `add_one` task, multiprocessing overhead is dominant even when\nspawning a single child process (a bare `add_one(big_data)` takes ~55 ms).\nHowever, since part of the multiprocessing overhead is in the parent process, \nit will also bottleneck more complex tasks when they are scaled over enough processes.\n\nHow it works\n--------------\n\nPython multiprocessing uses pickle to serialize data for transfer between processes.\nWhen passing around large numpy arrays, this can quickly become a bottleneck. \n\nNpshmex's ProcessPoolExecutor-replacement instead transfers input and output numpy arrays\nusing shared memory (`/dev/shm`). \nDictionary outputs with numpy arrays as values are also supported.\nOnly the shared-memory `filenames' are actually transferred between processes.\n\nNote that npshmex copies data from numpy arrays into shared memory\nto transfer them. It doesn't copy it again on retrieval; it just creates the\nnumpy array with the shared memory backing it.\nStill, if you are transferring the same array back and forth, \nthis amounts to two unnecessary memory copies.\nYou can avoid these, and the use of npshmex, by managing the shared memory yourself:\n```python\nfrom concurrent.futures import ProcessPoolExecutor\nimport SharedArray\n\ndef add_one(shm_key):\n    x = SharedArray.attach(shm_key)    \n    x += 1\n\nshm_key = 'shm://test'\nex = ProcessPoolExecutor()\n\nbig_data = SharedArray.create(shm_key, int(2e7))\nbig_data += 1\n\nf = ex.submit(add_one, shm_key)\nf.result()\nSharedArray.delete(shm_key)\nprint(x[0])     # 2.0\n```\nThe last four lines now only take ~130 ms on my laptop, which is over\ntwice as fast as npshmex. However, as you can see, it involves \na more substantial rewrite of your code.\n\nNpshmex also supports numpy structured arrays: it transfers the content as a `np.void` array, \nand the dtype (encoded to bytes) using a separate array.\n\n\nClearing shared memory\n------------------------\n\nNpshmex tells SharedArray to mark shared memory for deletion as soon as it has created\nnumpy arrays back from it. As explained in the \n[SharedArray](https://gitlab.com/tenzing/shared-array) documention, \nyou'll keep the numpy array until you lose the last reference to it \n(as with regular python objects).\n\nIf your program exits while data is being transfered between processes, \nsome shared files will remain in `/dev/shm`. You can manually clear all npshmex-associated\nshared memory from all processes on the machine with `npshmex.shm_clear()`. \nOtherwise, it will be up to you, your operating system, or your system administrator\nto clean up the mess...\n\nv0.1.0\n--------\n  * Initial release",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/JelleAalbers/npshmex",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "npshmex",
        "package_url": "https://pypi.org/project/npshmex/",
        "platform": "",
        "project_url": "https://pypi.org/project/npshmex/",
        "project_urls": {
            "Homepage": "https://github.com/JelleAalbers/npshmex"
        },
        "release_url": "https://pypi.org/project/npshmex/0.1.2/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "ProcessPoolExecutor that passes numpyarrays through shared memory",
        "version": "0.1.2"
    },
    "last_serial": 5156747,
    "releases": {
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "54a6ba3b0601f871e7a3194af26cf155",
                    "sha256": "1b03955f962750ce738a9ca8ef826078e1dec266a4e8f9ff457ea62a91c52809"
                },
                "downloads": -1,
                "filename": "npshmex-0.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "54a6ba3b0601f871e7a3194af26cf155",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2342,
                "upload_time": "2019-04-17T14:33:10",
                "url": "https://files.pythonhosted.org/packages/2f/f4/dea486bf7a2520b78c0546a8b3eb42ec1d3cba4eafca8ac95a78884468d1/npshmex-0.0.3.tar.gz"
            }
        ],
        "0.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "dcc23221a66bfac6e7d22fcf01976dc0",
                    "sha256": "c0cbd7fada50890700870117991c29c9e9bb59956c789867061387df48150636"
                },
                "downloads": -1,
                "filename": "npshmex-0.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "dcc23221a66bfac6e7d22fcf01976dc0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2749,
                "upload_time": "2019-04-17T14:38:35",
                "url": "https://files.pythonhosted.org/packages/92/7c/9435e13be746544fa087f7b78dc6c5442616dadc07d7fff6286260eb1a88/npshmex-0.0.4.tar.gz"
            }
        ],
        "0.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ef6841b0d6f7cce8a6ab72e647869783",
                    "sha256": "41dbd67752e11873f51d4476d1653355c9d526b2a6579d5915f03cfa266fbb0b"
                },
                "downloads": -1,
                "filename": "npshmex-0.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "ef6841b0d6f7cce8a6ab72e647869783",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2777,
                "upload_time": "2019-04-17T14:43:30",
                "url": "https://files.pythonhosted.org/packages/71/15/b22f6592cf0cb74d7f5723a137e0ac29df42352f34f40c8f3ac0dcdfa06a/npshmex-0.0.5.tar.gz"
            }
        ],
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4dfccd53c74e04463747f5a196751bb7",
                    "sha256": "41505dca0f8c88914430ef27c5f26b3a69a9c30825964d22977528c70694452c"
                },
                "downloads": -1,
                "filename": "npshmex-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "4dfccd53c74e04463747f5a196751bb7",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2945,
                "upload_time": "2019-04-17T15:10:45",
                "url": "https://files.pythonhosted.org/packages/8e/18/4253dbb87683d9c7d67960225369eeb78f71274360cf1a09d38546a10e2b/npshmex-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fcbf7bde862ee28dac66dd5fb8f42056",
                    "sha256": "7e7399fbf641e545402496e326d0c4ef6c68526228c38a1e1ca4771119527b20"
                },
                "downloads": -1,
                "filename": "npshmex-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "fcbf7bde862ee28dac66dd5fb8f42056",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 4437,
                "upload_time": "2019-04-17T19:11:38",
                "url": "https://files.pythonhosted.org/packages/3d/e9/a9bdfe45dfc07a6619a0e605ee6156586ff2f647dff09ff16695f8b7e59d/npshmex-0.1.1.tar.gz"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "215eac22d0ca9731d7ebbe336b0c56fb",
                    "sha256": "6882a249f69a6f371341280588b8bd7a2bbaccb77ece2fca644512bf7301dcdb"
                },
                "downloads": -1,
                "filename": "npshmex-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "215eac22d0ca9731d7ebbe336b0c56fb",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6418,
                "upload_time": "2019-04-17T19:20:16",
                "url": "https://files.pythonhosted.org/packages/f1/19/e06c47bfe6d8f9d3b20a680e5be3f157f9d3a0f800e7079f4a0bb2370c56/npshmex-0.1.2.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "215eac22d0ca9731d7ebbe336b0c56fb",
                "sha256": "6882a249f69a6f371341280588b8bd7a2bbaccb77ece2fca644512bf7301dcdb"
            },
            "downloads": -1,
            "filename": "npshmex-0.1.2.tar.gz",
            "has_sig": false,
            "md5_digest": "215eac22d0ca9731d7ebbe336b0c56fb",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 6418,
            "upload_time": "2019-04-17T19:20:16",
            "url": "https://files.pythonhosted.org/packages/f1/19/e06c47bfe6d8f9d3b20a680e5be3f157f9d3a0f800e7079f4a0bb2370c56/npshmex-0.1.2.tar.gz"
        }
    ]
}