{
    "info": {
        "author": "gatoatigrado",
        "author_email": "gatoatigrado@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 3 - Alpha",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: Apache Software License",
            "Operating System :: OS Independent",
            "Programming Language :: Python",
            "Topic :: Software Development :: Libraries :: Python Modules"
        ],
        "description": "|Build Status|\n\nVariations on imap, not in C\n============================\n\nThe ``vimap`` package is designed to provide a more flexible alternative\nfor ``multiprocessing.imap_unordered``. (You should read\n``multiprocessing`` documentation if you haven't already, or else this\nREADME won't make sense!) It aspires to support HTTP-like clients\nprocessing data, though contains nothing client-specific.\n\nWhat in particular makes it more flexible?\n\n-  It facilitates custom initialization of processes (one could connect\n   to a HTTP client, etc.), and cleanup.\n-  It facilitates configuration passing to processes -- often, with\n   multiprocessing, one ends up [wastefully] tupling configuration\n   values with every input.\n-  It facilitates passing non-serializable objects (before processes are\n   forked)\n-  Allows timeouts and retrying of failed tasks or processes\n\nWhat do we aspire to do better than the regular ``multiprocessing``\nlibrary?\n\n-  You don't have to hack to achieve custom process initialization [\n   c.f. http://stackoverflow.com/a/10118250/81636 ]\n-  The API helps prevent dumb mistakes, like initializing a pool before\n   you've defined relevant functions. [\n   http://stackoverflow.com/q/2782961/81636 ]\n-  Aims to have better worker exception handling -- multiprocessing will\n   leave around dead worker processes; we aim not to.\n-  Collection of common use cases (reading from files, etc.)\n\nOther features / design decisions:\n\n-  Attempts to keep all workers busy by pre-emptively enqueuing tasks\n-  Also, it *probably* doesn't affect you, but this library can also\n   work around a bug in Python 2.6.7 [\n   http://stackoverflow.com/q/16684900/81636 ].\n\nDefining worker functions\n-------------------------\n\n``vimap`` provides its custom initialization and such via decorated\nfunctions. If your inputs are HTTP requests, and you want to get\nresponses from any of a set of servers, you could express your program\nas such (using the ``requests`` HTTP library -- it's intuitive so you\nprobably don't need to read its documentation),\n\n::\n\n    import vimap.worker_process\n    @vimap.worker_process.worker\n    def send_reqests_worker(requests, server):\n        s = requests.Session()\n        for request in requests:\n            yield s.post('http://{0}{1}'.format(server, request.uri), data=request.data)\n        s.close()\n\nWhat is happening? When the worker processes start up, a new session is\nopened. Each request (some pickleable object containing a .uri and a\n.data), sent by the parent process, is posted to the server. Then, the\nworker yields a *single* response, and this response is sent back to the\nparent process.\n\nimapping data from the parent process\n-------------------------------------\n\nLet's continue the example,\n\n::\n\n    import vimap.pool\n    pool = vimap.pool.fork(send_requests_worker.init_args(server=server) for server in my_servers)\n\nThis initializes a pool of workers. Each one gets a bound argument\n``server``. When the worker processes start up, they start running until\nthey try to pull an element off of the ``requests`` iterator; then they\nmust pause for the parent process to send data. The parent process can\nsend data like so,\n\n::\n\n    Request = namedtuple(\"Request\", [\"uri\", \"data\"])\n    pool.imap(Request(**ujson.loads(line)) for line in fileinput.input()).block_ignore_output()\n\nThis reads lines from a file containing JSON input, and sends the loaded\nentries to the workers. In the real world, you'd probably want to make\nthe workers do the JSON loading. The ``.block_ignore_output()`` will\ncause the entire iterable (input file) to be read, and [by default]\nclose the pool after it's done.\n\nvariations on imap\n------------------\n\nThe input binder\n~~~~~~~~~~~~~~~~\n\nThe first Variation on Imap tuples inputs with outputs. So, you have\nsome [lazy] iterable of inputs,\n\n::\n\n    x1, x2, x3, x4, x5, ...\n\nand when you ``vimap`` this with some function ``f``, you get back\ntuples,\n\n::\n\n    (x1, f(x1)), (x2, f(x2)), ...\n\npossibly not in order. Since it's streaming, one shouldn't need to keep\naround inputs for long -- the input binder will keep around O(#\nprocesses) inputs, hence it's safe to iterate through large inputs. In\ncode, this could look like,\n\n::\n\n    for input, output in pool.imap(iterable).zip_in_out():\n        results[input] = output\n    # do some more processing\n\nHandling exceptions\n~~~~~~~~~~~~~~~~~~~\n\nIf you want to gracefully handle exceptions that bubble up to the main\nfunction of your worker processes, you can request that vimap yield back\nto you any exceptions it receives from the workers.\n\n::\n\n    for input, output, typ in pool.imap(iterable).zip_in_out_typ():\n        if typ == 'exception':\n            print('Worker had an exception:')\n            print(output.formatted_traceback)\n        elif typ == 'output':\n            print('I got some actual output from a worker!')\n            print(output)\n\n``output`` will be an ``ExceptionContext`` namedtuple if the return type\nis ``exception``; those contain a ``value`` of the exception raised and\n``formatted_traceback`` string of the traceback that would have been\nprinted to stderr.\n\n.. |Build Status| image:: https://travis-ci.org/gatoatigrado/vimap.png\n   :target: https://travis-ci.org/gatoatigrado/vimap",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/gatoatigrado/vimap",
        "keywords": null,
        "license": "UNKNOWN",
        "maintainer": null,
        "maintainer_email": null,
        "name": "vimap",
        "package_url": "https://pypi.org/project/vimap/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/vimap/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "https://github.com/gatoatigrado/vimap"
        },
        "release_url": "https://pypi.org/project/vimap/0.1.9-alpha-deadlock-quickfix/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "vimap",
        "version": "0.1.9-alpha-deadlock-quickfix"
    },
    "last_serial": 1270635,
    "releases": {
        "0.1.9-alpha-deadlock-quickfix": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "be0481d493d45ab959f2f4dd47cf3e69",
                    "sha256": "4e6176a69b2d382b65e42e4d00987aab60d7ebef9ce374b6e3dda9183f03859d"
                },
                "downloads": -1,
                "filename": "vimap-0.1.9-alpha-deadlock-quickfix.tar.gz",
                "has_sig": false,
                "md5_digest": "be0481d493d45ab959f2f4dd47cf3e69",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 20220,
                "upload_time": "2014-10-14T23:48:16",
                "url": "https://files.pythonhosted.org/packages/e7/98/32083ed01a8bb25563859008ae2a70d4b672a4a2a9aa08e0b2e968301fda/vimap-0.1.9-alpha-deadlock-quickfix.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "be0481d493d45ab959f2f4dd47cf3e69",
                "sha256": "4e6176a69b2d382b65e42e4d00987aab60d7ebef9ce374b6e3dda9183f03859d"
            },
            "downloads": -1,
            "filename": "vimap-0.1.9-alpha-deadlock-quickfix.tar.gz",
            "has_sig": false,
            "md5_digest": "be0481d493d45ab959f2f4dd47cf3e69",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 20220,
            "upload_time": "2014-10-14T23:48:16",
            "url": "https://files.pythonhosted.org/packages/e7/98/32083ed01a8bb25563859008ae2a70d4b672a4a2a9aa08e0b2e968301fda/vimap-0.1.9-alpha-deadlock-quickfix.tar.gz"
        }
    ]
}