{
    "info": {
        "author": "Praekelt Consulting",
        "author_email": "dev@praekelt.com",
        "bugtrack_url": null,
        "classifiers": [
            "Intended Audience :: Developers",
            "License :: OSI Approved :: BSD License",
            "Operating System :: OS Independent",
            "Programming Language :: Python",
            "Topic :: Internet :: WWW/HTTP :: Dynamic Content"
        ],
        "description": "Python Multicore\n================\n**A module that makes it easy to parallelize Python code.**\n\n.. figure:: https://travis-ci.org/praekelt/multicore.svg?branch=develop\n   :align: center\n   :alt: Travis\n\n.. contents:: Contents\n    :depth: 5\n\nInstallation\n------------\n\n#. Install or add ``multicore`` to your Python path.\n\nOverview\n--------\n\nPython supports multi-threading but the global interpreter lock (GIL) prevents\nus from utilising all CPU cores for CPU heavy tasks. The recommended approach\nis to use Python's multiprocessing library to work around the GIL, but that has its own set of challenges, notably\nthe ability to share data between sub-processes is limited.\n\nThe goal of the multicore library is to make it as simple as possible to parallelize code\nwhile incurring the *least* amount of overhead.\n\nFeatures\n--------\n\n#. Persistent pool of workers enabling persistent database connections.\n#. Memory maps for inter process communication. *Much* faster than multiprocess's own IPC or even pipes.\n#. Can take system load average into account to decide whether parallelization\n   is worth it at any given time.\n\nArchitecture\n------------\n\nPython Multicore is effectively an in-memory queue that is processed by a fixed\nset of workers. It uses memory mapping to avoid the latency imposed by using a\nqueing system such as celery.\n\nUsage\n-----\n\nLet's render 100 users. Always break a large task into smaller tasks, but not\ntoo small! If the ranges are too small then tasks aren't worth the effort\nbecause the overhead becomes too much.::\n\n    import time\n\n    from multicore import initialize, shutdown, Task\n    from multicore.utils import ranges\n\n\n    # Note the scoping of the \"items\" variable and the functions\n    items = range(100)\n\n\n    def as_string(item):\n        return str(item)\n\n\n    def expensive_as_string(item):\n        time.sleep(0.01)\n        return str(item)\n\n\n    def multi_expensive_as_string(start, end):\n        return \",\".join([expensive_as_string(item) for item in items[start:end]])\n\n\n    if __name__ == \"__main__\":\n\n        # Needs to be called only once for lifetime of process\n        initialize()\n\n        # Example 1: trivial (and slightly pointless) usage\n        task = Task()\n        for i in range(20):\n            task.run(as_string, i)\n        print(\", \".join(task.get()))\n\n        # Example 2: divide job optimally using ranges function\n        task = Task()\n        for start, end in ranges(items):\n            # Note we don't pass items because pickling is expensive and defeats\n            # the purpose of the exercize.\n            task.run(multi_expensive_as_string, start, end)\n        print(\", \".join(task.get()))\n\n        # Stop the multicore workers\n        shutdown()\n\nThe ``Task`` constructor accepts an optional parameter ``max_load_average``. If\nthe load average for the last minute is larger than a defined threshold then\n``None`` is returned and your code must cater for the sequential code path.\nNote that the threshold is specified as for a single core machine, so typically\nless than one.\n\nThe ``run`` method accepts an optional parameter ``serialization_format`` with value\n``pickle`` (the default), ``json`` or ``string``. Pickle is slow and safe. If you\nknow what type of data you have (you should!) set this as appropriate.\n\nThe ``run`` method also accepts an optional parameter ``use_dill`` with default\nvalue ``False``. Dill is a library that can often pickle things that can't be\npickled by the standard pickler but it is slightly slower.\n\nFAQ's\n-----\n\nWill it try to execute hundreds of pieces of code in parallel?\n**************************************************************\n\nNo. The worker pool has a fixed size and can only execute number-of-cores\ntasks in parallel. You may also set `max_load_average` as a further guard.\n\nWhy didn't you use multiprocessing.Pool?\n****************************************\n\nIt just has too many issues with eg. Django when it comes to scoping. Even pipes\nand sockets introduce too much overhead, so memory mapping is used.\n\nDo you have any benchmarks?\n***************************\n\nNo, because this is just an interface, not a collection of parallel code.\n\nIn general the code scales nearly linearly if you don't access the database.\nMulticore itself adds about 5 milliseconds overhead on my machine.\n\nThe memory map is too small for my data structures\n**************************************************\n\nA future version will address this through dynamic memory map scaling.\n\nAuthors\n=======\n\nPraekelt Consulting\n-------------------\n* Hedley Roos\n\nChangelog\n=========\n\n0.1.1\n-----\n#. Update readme with a working example.\n#. Guard against attempting to re-use a completed task.\n\n0.1\n---\n#. Initial release.\n\n",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://github.com/praekelt/multicore",
        "keywords": "",
        "license": "BSD",
        "maintainer": "",
        "maintainer_email": "",
        "name": "multicore",
        "package_url": "https://pypi.org/project/multicore/",
        "platform": "",
        "project_url": "https://pypi.org/project/multicore/",
        "project_urls": {
            "Homepage": "http://github.com/praekelt/multicore"
        },
        "release_url": "https://pypi.org/project/multicore/0.1.1/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Simpler Python multiprocess coding. Persistent workers, memory maps for minimum overhead.",
        "version": "0.1.1"
    },
    "last_serial": 4711139,
    "releases": {
        "0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "16374c34d0c43b1ffa544c9bd412359c",
                    "sha256": "776cc0fa4c1ae44e75e29de8c8fc0227f5170c3f560fac00cc978ac89c0466fa"
                },
                "downloads": -1,
                "filename": "multicore-0.1-py2.7.egg",
                "has_sig": false,
                "md5_digest": "16374c34d0c43b1ffa544c9bd412359c",
                "packagetype": "bdist_egg",
                "python_version": "2.7",
                "requires_python": null,
                "size": 15466,
                "upload_time": "2017-08-07T15:29:58",
                "url": "https://files.pythonhosted.org/packages/e0/ec/2c58a5b993d7a798879f8748bf87cdcbeb61a4b098673224968d95bb9c9b/multicore-0.1-py2.7.egg"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "b0cdc0169e79d19c5f7df544d9474119",
                    "sha256": "aa52d6ae79ea9e9ac21aae02c095b26edd2fa30d1a4a1991923f9f0b46dd4f8e"
                },
                "downloads": -1,
                "filename": "multicore-0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "b0cdc0169e79d19c5f7df544d9474119",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 22590,
                "upload_time": "2017-08-07T15:29:55",
                "url": "https://files.pythonhosted.org/packages/3b/51/ad00107b9b32d58019701d3e082b12152d4ae77407d32f63e2fab140176d/multicore-0.1.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "abac231f12d6466a3353a489fe413088",
                    "sha256": "b87846c0ad7881590827041e4a27008f1708a1bdc90650d6eba5d5a5ac70959e"
                },
                "downloads": -1,
                "filename": "multicore-0.1.1-py2.7.egg",
                "has_sig": false,
                "md5_digest": "abac231f12d6466a3353a489fe413088",
                "packagetype": "bdist_egg",
                "python_version": "2.7",
                "requires_python": null,
                "size": 15650,
                "upload_time": "2017-08-13T09:19:53",
                "url": "https://files.pythonhosted.org/packages/53/d9/b3c89d9c2c93d367589ff79de3f27c2820b105bcffffe0ce1f68f52ba9c8/multicore-0.1.1-py2.7.egg"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "5fc435be02bb933b43bc592b06e0f73c",
                    "sha256": "c45d3f594afcdeb162cc38db2a16567cc82bfad04d346956ea42088fc62b6186"
                },
                "downloads": -1,
                "filename": "multicore-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "5fc435be02bb933b43bc592b06e0f73c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 23095,
                "upload_time": "2017-08-13T09:19:50",
                "url": "https://files.pythonhosted.org/packages/dd/08/aa2a4f2d602d0c2e32e26fc0e2e7f4e97b0aeda6bf4c7fa74b6aaff9f2fb/multicore-0.1.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "abac231f12d6466a3353a489fe413088",
                "sha256": "b87846c0ad7881590827041e4a27008f1708a1bdc90650d6eba5d5a5ac70959e"
            },
            "downloads": -1,
            "filename": "multicore-0.1.1-py2.7.egg",
            "has_sig": false,
            "md5_digest": "abac231f12d6466a3353a489fe413088",
            "packagetype": "bdist_egg",
            "python_version": "2.7",
            "requires_python": null,
            "size": 15650,
            "upload_time": "2017-08-13T09:19:53",
            "url": "https://files.pythonhosted.org/packages/53/d9/b3c89d9c2c93d367589ff79de3f27c2820b105bcffffe0ce1f68f52ba9c8/multicore-0.1.1-py2.7.egg"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "5fc435be02bb933b43bc592b06e0f73c",
                "sha256": "c45d3f594afcdeb162cc38db2a16567cc82bfad04d346956ea42088fc62b6186"
            },
            "downloads": -1,
            "filename": "multicore-0.1.1.tar.gz",
            "has_sig": false,
            "md5_digest": "5fc435be02bb933b43bc592b06e0f73c",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 23095,
            "upload_time": "2017-08-13T09:19:50",
            "url": "https://files.pythonhosted.org/packages/dd/08/aa2a4f2d602d0c2e32e26fc0e2e7f4e97b0aeda6bf4c7fa74b6aaff9f2fb/multicore-0.1.1.tar.gz"
        }
    ]
}