{
    "info": {
        "author": "Yeray Diaz Diaz",
        "author_email": "yeraydiazdiaz@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 3 - Alpha",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7",
            "Programming Language :: Python :: Implementation :: CPython",
            "Topic :: Software Development :: Libraries :: Python Modules"
        ],
        "description": "# Futureproof - Bulletproof concurrent.futures\n\n[![Build Status](https://dev.azure.com/yeraydiazdiaz/futureproof/_apis/build/status/yeraydiazdiaz.futureproof?branchName=master)](https://dev.azure.com/yeraydiazdiaz/futureproof/_build/latest?definitionId=1&branchName=master)\n[![Supported Python Versions](https://img.shields.io/pypi/pyversions/futureproof.svg)](https://pypi.org/project/futureproof/)\n[![PyPI](https://img.shields.io/pypi/v/futureproof.svg)](https://pypi.org/project/futureproof/)\n\n[`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) is amazing, but it's got some sharp edges that have bit me many times in the past.\n\nFutureproof is a thin wrapper around it addressing some of these problems and adding some usability features.\n\n## Features:\n\n- **Monitoring**: a summary of completed tasks is logged by default.\n- **Fail fast**: errors cause the main thread to raise an exception and stop by default.\n- **Error policy**: the user can decide whether to raise, log or completely ignore errors on tasks.\n- **Backpressure control**: large collections of tasks are consumed lazily as the executor completes tasks, drastically reducing memory consumption and improving responsiveness in these situations.\n\n## Current status: Alpha\n\nThe API is subject to change, any changes will be documented in the changelog.\n\nFutureproof was designed to wrap ThreadPoolExecutor, however version 0.2+ includes limited support ProcessPoolExecutor but only for Python3.7+.\n\nWhen using ProcessPoolExecutor a \"OSError: handle is closed\" error is known to be printed from the interpreter exit handler, seemingly caused by worker processed being shut down prematurely. This is not affect the execution of the tasks.\n\n## concurrent.futures has problems? What problems?\n\nLet's have a look at the canonical example for ThreadPoolExecutor:\n\n```python\nimport concurrent.futures\nimport urllib.request\n\nURLS = ['http://www.foxnews.com/',\n        'http://www.cnn.com/',\n        'http://europe.wsj.com/',\n        'http://www.bbc.co.uk/',\n        'http://some-made-up-domain-that-definitely-does-not-exist.com/']\n\n# Retrieve a single page and report the URL and contents\ndef load_url(url, timeout):\n    with urllib.request.urlopen(url, timeout=timeout) as conn:\n        return conn.read()\n\n# We can use a with statement to ensure threads are cleaned up promptly\nwith concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:\n    # Start the load operations and mark each future with its URL\n    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}\n    for future in concurrent.futures.as_completed(future_to_url):\n        url = future_to_url[future]\n        try:\n            data = future.result()\n        except Exception as exc:\n            print('%r generated an exception: %s' % (url, exc))\n        else:\n            print('%r page is %d bytes' % (url, len(data)))\n```\n\nJust to reiterate, this is amazing, the fact that the barrier of entry for multithreading is this small is really a testament to the great work done by Brian Quinlan and the core Python developers.\n\nHowever, I see two problems with this:\n\n1. The boilerplate. We need to enter a context manager, call `submit` manually keeping track of the futures and its arguments, call `as_completed` which actually returns an iterator, call `result` on the future remembering to handle the exception.\n2. It's surprising. Why do we need to get the result in order to raise? What if we don't expect it to raise? We probably want to know as soon as possible.\n\nIf you run this code you get the following output (at the time of this writing):\n\n```\n'http://some-made-up-domain-that-definitely-does-not-exist.com/' generated an exception: <urlopen error [Errno 8] nodename nor servname provided, or not known>\n'http://www.foxnews.com/' page is 248838 bytes\n'http://www.bbc.co.uk/' page is 338658 bytes\n'http://www.cnn.com/' page is 991167 bytes\n'http://europe.wsj.com/' page is 970346 bytes\n```\n\nWhich is perfect. How does futureproof compare?\n\n```python\nexecutor = futureproof.FutureProofExecutor(max_workers=5)\nwith futureproof.TaskManager(executor) as tm:\n    for url in URLS:\n        tm.submit(load_url, url, 60)\n    for task in tm.as_completed():\n        print(\"%r page is %d bytes\" % (task.args[0], len(task.result)))\n```\n\nThat looks quite similar, there's an executor and a *task manager*. `submit` and `as_completed` are methods on it and there's no `try..except`. If we run it we get:\n\n```\n'http://www.foxnews.com/' page is 248838 bytes\nTraceback (most recent call last):\n  File \"/Users/yeray/.pyenv/versions/3.7.3/lib/python3.7/urllib/request.py\", line 1317, in do_open\n    encode_chunked=req.has_header('Transfer-encoding'))\n  ... omitted traceback output ...\nsocket.gaierror: [Errno 8] nodename nor servname provided, or not known\n```\n\nNotice that `futureproof` raised the exception that ocurred immediately and everything stopped, as you would've expected in normal non-threaded Python, no surprises.\n\nIf we prefer `futureproof` gives you the option to log or even ignore exceptions using error policies. Say we want to log the exceptions:\n\n```python\nlogging.basicConfig(\n    level=logging.INFO,\n    format=\"[%(asctime)s %(thread)s] %(message)s\",\n    datefmt=\"%H:%M:%S\",\n)\n\nexecutor = futureproof.FutureProofExecutor(max_workers=5)\nwith futureproof.TaskManager(executor, error_policy=\"log\") as tm:\n    for url in URLS:\n        tm.submit(load_url, url, 60)\n        for task in tm.as_completed():\n            if not isinstance(task.result, Exception):\n                print(\"%r page is %d bytes\" % (task.args[0], len(task.result)))\n```\n\nNote we've added a check to only print the result in case it's not an exception, this outputs:\n\n```\n'http://www.foxnews.com/' page is 251088 bytes\n[12:09:15 4350641600] Task Task(fn=<function load_url at 0x1029ef1e0>, args=('http://some-made-up-domain-that-definitely-does-not-exist.com/', 60), kwargs={}, result=URLError(gaierror(8, 'nodename nor servname provided, or not known')),\n complete=True) raised an exception\nTraceback (most recent call last):\n  File \"/Users/yeray/.pyenv/versions/3.7.3/lib/python3.7/urllib/request.py\", line 1317, in do_open\n    encode_chunked=req.has_header('Transfer-encoding'))\n    ... omitted long traceback ...\n  File \"/Users/yeray/.pyenv/versions/3.7.3/lib/python3.7/urllib/request.py\", line 1319, in do_open\n    raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>\n'http://some-made-up-domain-that-definitely-does-not-exist.com/' generated an exception: <urlopen error [Errno 8] nodename nor servname provided, or not known>\n'http://www.bbc.co.uk/' page is 339087 bytes\n'http://www.cnn.com/' page is 991167 bytes\n[12:09:16 123145404444672] 5 task completed in the last 1.18 second(s)\n'http://europe.wsj.com/' page is 970880 bytes\n```\n\nNote we only had to configure logging and pass the appropriate error policy, everything else was taken care for us. You can also choose to ignore exceptions completely and manage them yourself accessing `result`, which is the workflow when using `concurrent.futures`.\n\n### `as_completed`?\n\nIf you think about it, why do we need `as_completed`?\n\nThe answer is for monitoring and error handling.\n\nIf we had loads of URLs, you don't want to wait until all URLs are back to show output, it could take ages. But really it just adds complexity to the code. What does the example look like if you don't use `as_completed`?\n\n```python\nwith concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:\n    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}\n\nfor future, url in future_to_url.items():\n    try:\n        data = future.result()\n    except Exception as exc:\n        print(\"%r generated an exception: %s\" % (url, exc))\n    else:\n        print(\"%r page is %d bytes\" % (url, len(data)))\n```\n\nWhich is arguably more readable, however, it has a subtle difference: there's no output until all the futures are complete. If you imagine tasks taking longer you're left wondering if things are even working at all.\n\nLet's compare to the `futureproof` version:\n\n```python\nexecutor = futureproof.FutureProofExecutor(max_workers=5)\nwith futureproof.TaskManager(executor, error_policy=\"ignore\") as tm:\n    for url in URLS:\n        tm.submit(load_url, url, 60)\n\nfor task in tm.completed_tasks:\n    if isinstance(task.result, Exception):\n        print(\"%r generated an exception: %s\" % (task.args[0], task.result))\n    else:\n        print(\"%r page is %d bytes\" % (task.args[0], len(task.result)))\n```\n\n```\n[12:40:28 123145393414144] Starting executor monitor\n[12:40:29 123145393414144] 5 task completed in the last 1.01 second(s)\n[12:40:29 123145393414144] Shutting down monitor...\n'http://www.foxnews.com/' page is 252016 bytes\n'http://some-made-up-domain-that-definitely-does-not-exist.com/' generated an exception: <urlopen error [Errno 8] nodename nor servname provided, or not known>\n'http://www.cnn.com/' page is 992648 bytes\n'http://www.bbc.co.uk/' page is 338987 bytes\n'http://europe.wsj.com/' page is 969285 bytes\n```\n\n`futureproof` defaults to logging monitoring information on the tasks so you always know if things are working. Note how the task manager exposes `completed_tasks` allowing easy access to the results without having to manually keep track of futures. Finally, as mentioned previously, you're also in total control over exception handling so you don't need to add code for that either.\n\nThese are fairly minor problems that we can work around manually using `concurrent.futures` but when starting to deal higher number of tasks other problems arise, check out the [examples directory](https://github.com/yeraydiazdiaz/futureproof/tree/master/examples/) for a hands-on comparison between `futureproof` and `concurrent.futures` in other more serious scenarios.\n\n## Alternatives\n\nI am by no means the first person to address these problems. Here a few similar, more stable and feature full, albeit restrictively licensed alternatives:\n\n- [Pebble](https://pebble.readthedocs.io/en/latest/), LGPL 3.0\n- [more-executors](https://github.com/rohanpm/more-executors), GPL 3.0\n\n`futureproof` is licensed MIT.\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/yeraydiazdiaz/futureproof",
        "keywords": "concurrent futures multithreading",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "futureproof",
        "package_url": "https://pypi.org/project/futureproof/",
        "platform": "",
        "project_url": "https://pypi.org/project/futureproof/",
        "project_urls": {
            "Homepage": "https://github.com/yeraydiazdiaz/futureproof"
        },
        "release_url": "https://pypi.org/project/futureproof/0.2.0/",
        "requires_dist": [
            "attrs",
            "pytest-mock ; extra == 'tests'",
            "pytest-timeout ; extra == 'tests'",
            "coverage ; extra == 'tests'"
        ],
        "requires_python": "",
        "summary": "Bulletproof concurrent.futures",
        "version": "0.2.0"
    },
    "last_serial": 5490915,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "616a5e18f9bded5e4414b84a4c70abdb",
                    "sha256": "5639bf3c0309413cbf117607dbc315232df0de5909e9e50129c31d8d85839098"
                },
                "downloads": -1,
                "filename": "futureproof-0.1.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "616a5e18f9bded5e4414b84a4c70abdb",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 6111,
                "upload_time": "2019-06-09T14:51:57",
                "url": "https://files.pythonhosted.org/packages/4a/93/860c46f3157b4b5dd66da9f14e731186efd1d86241235c5a26231b988449/futureproof-0.1.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "6ece029e6fbbc287834a9937d32769a9",
                    "sha256": "f90993a87026211aa5cc6bb4b6e8632c6e32ef38b56a2b5b39ad6266a2b155a8"
                },
                "downloads": -1,
                "filename": "futureproof-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "6ece029e6fbbc287834a9937d32769a9",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6129,
                "upload_time": "2019-06-09T14:51:59",
                "url": "https://files.pythonhosted.org/packages/26/9d/9f937d368255700dbc94c92110ef3291aa9faa4df2afce8a38942f95d130/futureproof-0.1.0.tar.gz"
            }
        ],
        "0.1.0.dev0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ef4cf6ef119cc3e7bb33b8992d411780",
                    "sha256": "2f0c7f866a2190ba2e534ab60fab67f9757794352dec0bf5ff94c010b3e4210a"
                },
                "downloads": -1,
                "filename": "futureproof-0.1.0.dev0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ef4cf6ef119cc3e7bb33b8992d411780",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 6051,
                "upload_time": "2019-06-08T13:54:00",
                "url": "https://files.pythonhosted.org/packages/ea/dc/fd98d83105204f0f1244334a2abbcff535342ba35f62746e192ea503115d/futureproof-0.1.0.dev0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "0d227cd64f609d9389b4ee792add0d04",
                    "sha256": "ba56a6b2b3e6228045820758170df8af923a6990770df1a4da2cd32121c52714"
                },
                "downloads": -1,
                "filename": "futureproof-0.1.0.dev0.tar.gz",
                "has_sig": false,
                "md5_digest": "0d227cd64f609d9389b4ee792add0d04",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 4662,
                "upload_time": "2019-06-08T13:54:02",
                "url": "https://files.pythonhosted.org/packages/db/28/81d48a7658d336b34922d5ddf0a1d6a6a4afb4fcbdef9049db56ff3c3270/futureproof-0.1.0.dev0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d776a09a0fbc8cfd0bc3ae1152d6df56",
                    "sha256": "84293f5e68b0d596612715ffd4a0cb53233269303d6669d9c24d704874e5048d"
                },
                "downloads": -1,
                "filename": "futureproof-0.1.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d776a09a0fbc8cfd0bc3ae1152d6df56",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 6152,
                "upload_time": "2019-06-09T16:18:24",
                "url": "https://files.pythonhosted.org/packages/b7/11/8d942caecd1f2fc43102638188c6bca47f7f15eefff27399444f9baa82b9/futureproof-0.1.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "42b85cfc766ef8ee09215d7c54c4b9aa",
                    "sha256": "bde7f7dabfa37d30fb3022393535edde95ee8123a7e27e5b5a70ec844ebe428b"
                },
                "downloads": -1,
                "filename": "futureproof-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "42b85cfc766ef8ee09215d7c54c4b9aa",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6261,
                "upload_time": "2019-06-09T16:18:26",
                "url": "https://files.pythonhosted.org/packages/74/b7/778ca0d98e077aaab32e295fc638381fa3a76622485cf73c9f0b379e68e0/futureproof-0.1.1.tar.gz"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7b54d222659041d1623f83c109c51346",
                    "sha256": "945cc5495c73a50fcb6cd2bb861964d8818e3abcaf43b7b2b71ff5e049741c6e"
                },
                "downloads": -1,
                "filename": "futureproof-0.1.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "7b54d222659041d1623f83c109c51346",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 6277,
                "upload_time": "2019-06-29T16:09:20",
                "url": "https://files.pythonhosted.org/packages/be/c8/66f0fc371c4af9e16faee3808755e8d4e74bfbe3e2d4bf3fb6eb12214ef1/futureproof-0.1.2-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "dd35dee4874919b049e2904f48cb183a",
                    "sha256": "19be32e8bae40c341c06999c755e1c54f72ecd993b2abbfff6fc46bc598d6146"
                },
                "downloads": -1,
                "filename": "futureproof-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "dd35dee4874919b049e2904f48cb183a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6538,
                "upload_time": "2019-06-29T16:09:22",
                "url": "https://files.pythonhosted.org/packages/0c/39/c4fd0a53b8fb5f0b23adbf76925e5405ffc23cb4809976904ff454c852ea/futureproof-0.1.2.tar.gz"
            }
        ],
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b2619d96e270528c0a380202f00e7345",
                    "sha256": "fe996bf3bfa4dc3b5ca3690657f87592946c9b343ae1ff6fc371d8378a581f96"
                },
                "downloads": -1,
                "filename": "futureproof-0.2.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "b2619d96e270528c0a380202f00e7345",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 9384,
                "upload_time": "2019-07-05T11:17:59",
                "url": "https://files.pythonhosted.org/packages/ed/12/074ae7ce99fd5f722856d34f4f12b9d42e43901507d0a5ef5169fd81dfba/futureproof-0.2.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "9914712f68df63fdd21a7b6401b30cfd",
                    "sha256": "df5be7aff672feab5438630059d4bdf4ca093ec67f470a047bfb801373a6dc8d"
                },
                "downloads": -1,
                "filename": "futureproof-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "9914712f68df63fdd21a7b6401b30cfd",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10921,
                "upload_time": "2019-07-05T11:18:00",
                "url": "https://files.pythonhosted.org/packages/f7/0e/0ffe29970f0821755e7b14c42606b65387811bec87f849dc9c54aea2fbdc/futureproof-0.2.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "b2619d96e270528c0a380202f00e7345",
                "sha256": "fe996bf3bfa4dc3b5ca3690657f87592946c9b343ae1ff6fc371d8378a581f96"
            },
            "downloads": -1,
            "filename": "futureproof-0.2.0-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "b2619d96e270528c0a380202f00e7345",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 9384,
            "upload_time": "2019-07-05T11:17:59",
            "url": "https://files.pythonhosted.org/packages/ed/12/074ae7ce99fd5f722856d34f4f12b9d42e43901507d0a5ef5169fd81dfba/futureproof-0.2.0-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "9914712f68df63fdd21a7b6401b30cfd",
                "sha256": "df5be7aff672feab5438630059d4bdf4ca093ec67f470a047bfb801373a6dc8d"
            },
            "downloads": -1,
            "filename": "futureproof-0.2.0.tar.gz",
            "has_sig": false,
            "md5_digest": "9914712f68df63fdd21a7b6401b30cfd",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 10921,
            "upload_time": "2019-07-05T11:18:00",
            "url": "https://files.pythonhosted.org/packages/f7/0e/0ffe29970f0821755e7b14c42606b65387811bec87f849dc9c54aea2fbdc/futureproof-0.2.0.tar.gz"
        }
    ]
}