{ "info": { "author": "Mathieu Mirmont", "author_email": "mat@parad0x.org", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "Intended Audience :: Science/Research", "License :: OSI Approved :: GNU General Public License v2 (GPLv2)", "Operating System :: POSIX", "Operating System :: Unix", "Programming Language :: C", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Topic :: Scientific/Engineering" ], "description": "SharedArray python/numpy extension\n==================================\n\nThis is a simple python extension that lets you share numpy arrays with\nother processes on the same computer. It uses either shared files or\nPOSIX shared memory as data stores and therefore should work on most\noperating systems.\n\nExample\n-------\n\nHere\u2019s a simple example to give an idea of how it works. This example\ndoes everything from a single python interpreter for the sake of\nclarity, but the real point is to share arrays between python\ninterpreters.\n\n.. code:: python\n\n import numpy as np\n import SharedArray as sa\n\n # Create an array in shared memory.\n a = sa.create(\"shm://test\", 10)\n\n # Attach it as a different array. This can be done from another\n # python interpreter as long as it runs on the same computer.\n b = sa.attach(\"shm://test\")\n\n # See how they are actually sharing the same memory.\n a[0] = 42\n print(b[0])\n\n # Destroying a does not affect b.\n del a\n print(b[0])\n\n # See how \"test\" is still present in shared memory even though we\n # destroyed the array a. This method only works on Linux.\n sa.list()\n\n # Now destroy the array \"test\" from memory.\n sa.delete(\"test\")\n\n # The array b is still there, but once you destroy it then the\n # data is gone for real.\n print(b[0])\n\nFunctions\n---------\n\nSharedArray.create(name, shape, dtype=float)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis function creates an array in shared memory and returns a numpy\narray that uses the shared memory as data backend.\n\nThe shared memory is identified by ``name``, which can use the\n``file://`` prefix to indicate that the data backend will be a file, or\n``shm://`` to indicate that the data backend shall be a POSIX shared\nmemory object. For backward compatibility ``shm://`` is assumed when no\nprefix is given. Most operating systems implement strong file caching so\nusing a file as a data backend won\u2019t usually affect performance.\n\nThe ``shape`` and ``dtype`` arguments are identical to those of the\nnumpy function ``numpy.zeros()``, and the returned array is indeed\ninitialized to zeros.\n\nThe content of the array lives in shared memory and/or in a file and\nwon\u2019t be lost when the numpy array is deleted, nor when the python\ninterpreter exits. To delete a shared array and reclaim system resources\nuse the ``SharedArray.delete()`` function.\n\nSharedArray.attach(name)\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis function attaches a previously created array in shared memory\nidentified by ``name``, which can use the ``file://`` prefix to indicate\nthat the array is stored as a file, or ``shm://`` to indicate that the\narray is stored as a POSIX shared memory object. For backward\ncompatibility ``shm://`` is assumed when no prefix is given.\n\nAn array may be simultaneously attached from multiple different\nprocesses (i.e.\u00a0python interpreters).\n\nThe content of the array lives in shared memory and/or in a file and\nwon\u2019t be lost when the numpy array is deleted, nor when the python\ninterpreter exits. To delete a shared array reclaim system resources use\nthe ``SharedArray.delete()`` function.\n\nSharedArray.delete(name)\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis function destroys the previously created array identified by\n``name``, which can use the ``file://`` prefix to indicate that the\narray is stored as a file, or ``shm://`` to indicate that the array is\nstored as a POSIX shared memory object. For backward compatibility\n``shm://`` is assumed when no prefix is given\n\nAfter calling ``delete``, the array will not be attachable anymore, but\nexisting attachments will remain valid until they are themselves\ndestroyed. The data is reclaimed by the system when the very last\nattachment is deleted.\n\nSharedArray.list()\n~~~~~~~~~~~~~~~~~~\n\nThis function returns a list of previously created arrays stored as\nPOSIX SHM objects, along with their name, data type and dimensions. This\nfunction only works on Linux because it directly accesses files exposed\nunder ``/dev/shm``. There doesn\u2019t seem to be a portable method of\nachieving this.\n\nConstants\n---------\n\nSharedArray.MS_ASYNC\n~~~~~~~~~~~~~~~~~~~~\n\nFlag for the ``msync()`` method of the base object of the returned numpy\narray (see below). Specifies that an update be scheduled, but the call\nreturns immediately.\n\nSharedArray.MS_SYNC\n~~~~~~~~~~~~~~~~~~~\n\nFlag for the ``msync()`` method of the base object of the returned numpy\narray (see below). Requests an update and waits for it to complete.\n\nSharedArray.MS_INVALIDATE\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFlag for the ``msync()`` method of the base object of the returned numpy\narray (see below). Asks to invalidate other mappings of the same file\n(so that they can be updated with the fresh values just written).\n\nBase object\n-----------\n\nSharedArray registers its own python object as the\n`base `__\nobject of the returned numpy array. This base object exposes the\nfollowing methods and attributes:\n\nmsync(array, flags)\n~~~~~~~~~~~~~~~~~~~\n\nThis method is a wrapper around ``msync(2)`` and is only useful when\nusing file-backed arrays (i.e.\u00a0not POSIX shared memory). msync(2)\nflushes the mapped memory region back to the filesystem. The ``flags``\nare exported as constants in the module definition (see above) and are a\n1:1 map of the ``msync(2)`` flags, please refer to the manual page of\n``msync(2)`` for details.\n\nmlock(array)\n~~~~~~~~~~~~\n\nThis method is a wrapper around ``mlock(2)``: lock the memory map into\nRAM, preventing that memory from being paged to the swap area.\n\nmunlock(array)\n~~~~~~~~~~~~~~\n\nThis method is a wrapper around ``munlock(2)``: unlock the memory map,\nallowing that memory to be paged to the swap area.\n\nname\n~~~~\n\nThis constant string is the name of the array as passed to\n``SharedArray.create()`` or ``SharedArray.attach()``. It may be passed\nto ``SharedArray.delete()``.\n\naddr\n~~~~\n\nBase address of the array in memory.\n\nsize\n~~~~\n\nSize of the array in memory.\n\nRequirements\n------------\n\n- Python 2.7 or 3+\n- Numpy 1.8+\n- Posix shared memory interface\n\nSharedArray uses the posix shm interface (``shm_open`` and\n``shm_unlink``) and so should work on most POSIX operating systems\n(Linux, BSD, etc.). It has been reported to work on macOS, and it is\nunlikely to work on Windows.\n\nInstallation\n------------\n\nThe extension uses the ``distutils`` python package that should be\nfamiliar to most python users. To test the extension directly from the\nsource tree, without installing, type:\n\n.. code:: sh\n\n python setup.py build_ext --inplace\n\nTo build and install the extension system-wide, type:\n\n.. code:: sh\n\n python setup.py build\n sudo python setup.py install\n\nThe package is also available on PyPI and can be installed using the pip\ntool.\n\nFAQ\n---\n\nOn Linux, I get segfaults when working with very large arrays.\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nA few people have reported segfaults with very large arrays using POSIX\nshared memory. This is not a bug in SharedArray but rather an indication\nthat the system ran out of POSIX shared memory.\n\nOn Linux a ``tmpfs`` virtual filesystem is used to provide POSIX shared\nmemory, and by default it is given only about 20% of the total available\nmemory, depending on the distribution. That amount can be changed by\nre-mounting the ``tmpfs`` filesystem with the ``size=100%`` option:\n\n.. code:: sh\n\n sudo mount -o remount,size=100% /run/shm\n\nAlso you can make the change permanent, on next boot, by setting\n``SHM_SIZE=100%`` in ``/etc/defaults/tmpfs`` on recent Debian\ninstallations.\n\nOn Linux, I get \u201cCannot allocate memory\u201d when creating many arrays.\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSharedArray uses one memory map per array that is attached (or created).\nBy default the maximum number of memory maps per process is set by the\nLinux kernel to 65530. If you want to create more arrays than that you\nneed to tune the kernel parameter ``vm.max_map_count`` and set it to a\nhigher value.\n\n.. code:: sh\n\n /sbin/sysctl vm.max_map_count=655300\n\nNote that for the change to be permanent you need to add this line to\n``/etc/sysctl.conf``:\n\n.. code:: sh\n\n vm.max_map_count=655300\n\nI can\u2019t attach old (pre 0.4) arrays anymore.\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSince version 0.4 all arrays are now page aligned in memory, to be used\nwith SIMD instructions (e.g.\u00a0fftw library). As a side effect, arrays\ncreated with a previous version of SharedArray aren\u2019t compatible with\nthe new version (the location of the metadata changed). Save your work\nbefore upgrading.\n\nContact\n-------\n\nThis package is hosted on `GitLab `__ at:\nhttps://gitlab.com/tenzing/shared-array\n\nPackages are also available on PyPi at:\nhttps://pypi.python.org/pypi/SharedArray\n\nFor bug reports, feature requests, suggestions, patches and everything\nelse related to SharedArray, feel free to raise issues on the `project\npage `__. You can also contact\nthe maintainer directly by email at mat@parad0x.org.", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://gitlab.com/tenzing/shared-array", "keywords": "numpy array shared memory shm", "license": "", "maintainer": "", "maintainer_email": "", "name": "SharedArray", "package_url": "https://pypi.org/project/SharedArray/", "platform": "", "project_url": "https://pypi.org/project/SharedArray/", "project_urls": { "Homepage": "https://gitlab.com/tenzing/shared-array" }, "release_url": "https://pypi.org/project/SharedArray/3.2.0/", "requires_dist": null, "requires_python": "", "summary": "Share numpy arrays between processes", "version": "3.2.0" }, "last_serial": 5777222, "releases": { "0.1": [ { "comment_text": "", "digests": { "md5": "15d257fba864e943370590f024487e6d", "sha256": "50375a23ad58fe9ae6343d1f35eb087d154d277c61850bc06ddbd9d5e51553b7" }, "downloads": -1, "filename": "SharedArray-0.1.tar.gz", "has_sig": true, "md5_digest": "15d257fba864e943370590f024487e6d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8329, "upload_time": "2014-12-15T17:53:34", "url": "https://files.pythonhosted.org/packages/48/5d/800370dfce6d87bcee7f298bc8acf13ec2b58209ab30c6eb12d1f527e049/SharedArray-0.1.tar.gz" } ], "0.2": [ { "comment_text": "", "digests": { "md5": "7c36aca5a78d54898bd14a8842754e98", "sha256": "9ff4952783ce4865b86337463878aa29187377060ffb37a68008987f113c695c" }, "downloads": -1, "filename": "SharedArray-0.2.tar.gz", "has_sig": true, "md5_digest": "7c36aca5a78d54898bd14a8842754e98", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8339, "upload_time": "2015-01-09T10:38:43", "url": "https://files.pythonhosted.org/packages/c3/d2/ce5e2e0ca5fb853ea003ccfddc4c999671b01c097549edf3ef5ad998445f/SharedArray-0.2.tar.gz" } ], "0.3": [ { "comment_text": "", "digests": { "md5": "ef24203e25a90c94f2373f2e718c6f6a", "sha256": "03f141a06366489b63e7941e8058740ef861aa9e064778ce21c5fe480af0cfa6" }, "downloads": -1, "filename": "SharedArray-0.3.tar.gz", "has_sig": false, "md5_digest": "ef24203e25a90c94f2373f2e718c6f6a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8376, "upload_time": "2015-08-23T11:17:57", "url": "https://files.pythonhosted.org/packages/24/47/d4e082499556fac76d442263f19a44a128c7c45f2821383b938d6d18f56a/SharedArray-0.3.tar.gz" } ], "0.4": [ { "comment_text": "", "digests": { "md5": "cbf53fee20e4e882d885b5de6833ad36", "sha256": "c11fcc0fa98adbff7a93c57198f4776b8757408534bf7955a1d78820b7223201" }, "downloads": -1, "filename": "SharedArray-0.4.tar.gz", "has_sig": false, "md5_digest": "cbf53fee20e4e882d885b5de6833ad36", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7886, "upload_time": "2016-02-06T20:33:55", "url": "https://files.pythonhosted.org/packages/19/20/de622226d3a45d16cb5f259727bba1212807592be5ad2831d9c81d101cf3/SharedArray-0.4.tar.gz" } ], "0.5": [ { "comment_text": "", "digests": { "md5": "35cb000626287c2e1df4a15f6617340c", "sha256": "37f26776f3d951e9e92bbfe846e3313c9c462127090ad408b89cb68c62be6a75" }, "downloads": -1, "filename": "SharedArray-0.5.tar.gz", "has_sig": false, "md5_digest": "35cb000626287c2e1df4a15f6617340c", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9379, "upload_time": "2016-03-14T16:11:41", "url": "https://files.pythonhosted.org/packages/42/7d/a3b68b3742bc485bb3fcefe4e7625473cf462d0db8d5e92093cbdb3ddfa2/SharedArray-0.5.tar.gz" } ], "1.0": [ { "comment_text": "", "digests": { "md5": "fe4ab851536f08bd6f1c338c69e90c68", "sha256": "5370284a51507cb8c326f18aca9d92ec463e74eb961cd309cc002078dc38c4d6" }, "downloads": -1, "filename": "SharedArray-1.0.tar.gz", "has_sig": false, "md5_digest": "fe4ab851536f08bd6f1c338c69e90c68", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8334, "upload_time": "2016-04-15T20:49:00", "url": "https://files.pythonhosted.org/packages/d3/a8/e8449f55d0de59614f546df1938729348e900913d9b5924d895df5c5e7de/SharedArray-1.0.tar.gz" } ], "2.0": [ { "comment_text": "", "digests": { "md5": "d938730e068003887218a69c020311de", "sha256": "3ae2059e24e564ca0e485075086b3646434e8c377d2328089b878e60d951d067" }, "downloads": -1, "filename": "SharedArray-2.0.tar.gz", "has_sig": false, "md5_digest": "d938730e068003887218a69c020311de", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 10672, "upload_time": "2016-11-20T21:44:54", "url": "https://files.pythonhosted.org/packages/e0/e4/44f55a1c01dff3ef9b0c95bce201527a493ead215dc3d938f99c1de54fa1/SharedArray-2.0.tar.gz" } ], "2.0.1": [ { "comment_text": "", "digests": { "md5": "551991d354de4f5b26a2b9025dd3b2b9", "sha256": "1dfc525d37cbfdc76a7b5064373a5ea2b612ffa299b8e0d6b5c66960cba8d8fb" }, "downloads": -1, "filename": "SharedArray-2.0.1.tar.gz", "has_sig": false, "md5_digest": "551991d354de4f5b26a2b9025dd3b2b9", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 10864, "upload_time": "2016-11-20T22:15:02", "url": "https://files.pythonhosted.org/packages/9f/66/d85d2c6b2a1e6169c00786f4867e9af1ebff650f4ded67d70857093b3e2f/SharedArray-2.0.1.tar.gz" } ], "2.0.2": [ { "comment_text": "", "digests": { "md5": "c44395d600cdb1addf2960035f7c0f2f", "sha256": "14a3d98be242b5fed26992c82b1e7d05ea8ea13545443e3ebf072109022b8768" }, "downloads": -1, "filename": "SharedArray-2.0.2.tar.gz", "has_sig": false, "md5_digest": "c44395d600cdb1addf2960035f7c0f2f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12248, "upload_time": "2016-11-22T10:20:51", "url": "https://files.pythonhosted.org/packages/7b/64/24f5876511af54eadea7622ea60f63cf05ccc61caa22e1f2b612a09c27f3/SharedArray-2.0.2.tar.gz" } ], "2.0.3": [ { "comment_text": "", "digests": { "md5": "d419e5ca9fe734f3d54cb28995973351", "sha256": "f2c1496926276798dad10e2d554855352a24022b483dbc8d57ae2e4a6a6f99d3" }, "downloads": -1, "filename": "SharedArray-2.0.3.tar.gz", "has_sig": false, "md5_digest": "d419e5ca9fe734f3d54cb28995973351", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11302, "upload_time": "2017-10-08T13:59:23", "url": "https://files.pythonhosted.org/packages/a0/27/57837b8f1abb910c68b81caff0333ccf403c2f2db18cb3d04be7f3796f48/SharedArray-2.0.3.tar.gz" } ], "2.0.4": [ { "comment_text": "", "digests": { "md5": "21778896f8caa293f01cf2f5723db8c3", "sha256": "fc753ee6f9db7b09e1ecf26f2d1ab6801c4381223230edbf37e83a9c06e0389b" }, "downloads": -1, "filename": "SharedArray-2.0.4.tar.gz", "has_sig": false, "md5_digest": "21778896f8caa293f01cf2f5723db8c3", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12833, "upload_time": "2017-12-18T20:27:16", "url": "https://files.pythonhosted.org/packages/34/c6/6af0801fbbb00a498da65adb99e2fd6af2060021dc4c84451392a0151ad9/SharedArray-2.0.4.tar.gz" } ], "3.0.0": [ { "comment_text": "", "digests": { "md5": "5c5e402bed390cea0e419e7b99090cc8", "sha256": "5730bdce87fcde4d28e134f2777dc42c0eb4e3d15c7f468534ff431dc25ae813" }, "downloads": -1, "filename": "SharedArray-3.0.0.tar.gz", "has_sig": false, "md5_digest": "5c5e402bed390cea0e419e7b99090cc8", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 19755, "upload_time": "2018-04-25T21:47:02", "url": "https://files.pythonhosted.org/packages/52/47/07d5573e5f388aa694b997459b0c254b24c7e91b9c5aaa3cbe71930b26e7/SharedArray-3.0.0.tar.gz" } ], "3.1.0": [ { "comment_text": "", "digests": { "md5": "8282a9073e82994085ef622d9e4018ed", "sha256": "c7a47d3d2aacf07428438e938dee609d2306859ff5e2ed20df7fd28bc6bea3db" }, "downloads": -1, "filename": "SharedArray-3.1.0.tar.gz", "has_sig": false, "md5_digest": "8282a9073e82994085ef622d9e4018ed", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 21747, "upload_time": "2018-10-14T12:51:37", "url": "https://files.pythonhosted.org/packages/cf/58/da95daa37d635e3094eb56ac655a22ea95b4236e166278e9f07ff444712a/SharedArray-3.1.0.tar.gz" } ], "3.2.0": [ { "comment_text": "", "digests": { "md5": "2aa4a176a0bda0c3e4578b935a36924b", "sha256": "a1f3aad1f1f40e84cbfda0e0a33d3f5148551f5a65a0d274d04d8988e69a20ac" }, "downloads": -1, "filename": "SharedArray-3.2.0.tar.gz", "has_sig": false, "md5_digest": "2aa4a176a0bda0c3e4578b935a36924b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 22243, "upload_time": "2019-09-03T17:44:47", "url": "https://files.pythonhosted.org/packages/e5/2b/f9e5c32411eeaf8be0561aae75c397d55414e8c0e9e47973cd67bc42d5d6/SharedArray-3.2.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "2aa4a176a0bda0c3e4578b935a36924b", "sha256": "a1f3aad1f1f40e84cbfda0e0a33d3f5148551f5a65a0d274d04d8988e69a20ac" }, "downloads": -1, "filename": "SharedArray-3.2.0.tar.gz", "has_sig": false, "md5_digest": "2aa4a176a0bda0c3e4578b935a36924b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 22243, "upload_time": "2019-09-03T17:44:47", "url": "https://files.pythonhosted.org/packages/e5/2b/f9e5c32411eeaf8be0561aae75c397d55414e8c0e9e47973cd67bc42d5d6/SharedArray-3.2.0.tar.gz" } ] }