{
    "info": {
        "author": "Sergi Blanch-Torn'e",
        "author_email": "srgblnchtrn@protonmail.ch",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# Yet Another Multiprocessing Pool (yamp)\n\n![license GPLv3+](https://img.shields.io/badge/license-GPLv3+-green.svg) ![about](https://img.shields.io/badge/Subject-multiprocessing-orange.svg?style=social) ![2 - Pre-Alpha](https://img.shields.io/badge/Development_Status-2_--_pre--alpha-orange.svg)\n\nPython module to manage multiprocessing of a queue. Quite similar to _multiprocessing.Pool_ but more to my liking.\n\nI've needed and reproduced a similar functionality twice in a while and a third was coming the way. That's _enough_ to encapsulate it in a module.\n\nPerhaps the name could be better but it is a coincidence with [young tramp](http://www.urbandictionary.com/define.php?term=yamp) or young vagabond. No need to mention all respect to homeless.\n\n\n## TODO:\n\n- [x] Use python Logging.\n- [ ] Extend the ending condition.\n- [ ] Improve the input: no need to be build before, but can be generated while needed.\n- [x] Control memory usage (psutil).\n- [x] Control machine load.\n  - [ ] Default warning when last minute reach the number of cores\n  - [ ] Default limit when the three values reach the number of cores\n- [x] Hooks. At least, after the worker target execution allow to execute something, but with in a Lock because is a place to report results in a file (for example).\n- [ ] Python 3.5 together with the current 2.7 support.\n- [ ] Cythonize.\n- [ ] Look on *pkg_resources* to improve version numbering.\n- [x] When enter in _pause_ mode, use _psutil_ to suspend the workers until conditions recovers and the work can be resumed (then resume the workers).\n  - [ ] This many enter in a loop of _suspend-resume_. Raise the bell to reduce the number of parallel workers.\n  - [ ] Trigger the _pause_ when the limit is reached, but _resume_ when warning is clean.\n  - [ ] In the warning sections of memory use and machine load, those workers can be _reniced_ to reduce their priority.\n\n\n## Usage:\n\nGet the sources as usual.\n\n```\n$ git clone git@github.com:srgblnch/yamp.git\n```\n\nBuild them using the setup tools.\n\n```\n$ python setup.py build\n$ python setup.py install --prefix ~/usr\n```\n\nUse in a python interpreter\n\n```python\n>>> import yamp\n>>> yamp.version()\n    '0.0.5-0'\n>>> arginLst = range(20)\n>>> checkPeriod = 2\n>>> from random import randint\n>>> from time import sleep\n>>> def tester(argin):\n...:     argout = argin**2\n...:     sleep(randint(checkPeriod/2, checkPeriod*2))\n...:     return argout\n>>> pool = yamp.Pool(tester, arginLst)\n>>> pool.checkPeriod = checkPeriod\n>>> pool.start()\n```\n\nThe control of the *main thread* will be returned and it can be checked the progress for the procedure:\n\n```python\n>>> pool.progress\n0.407407\n```\n\n## Known Issues\n\n- It has been saw that, when there are more than 1 _Worker_ in the _Pool_, the '_start event_' is propagated to the last of them almost immediately, but the others will receive it around 58 o 59 seconds later.\n\nThe script '_testing/pool.py_' with the parameters '_--samples=10_' and '_--processors=3_' will produce an stdout similar to:\n\n    (...)\n    2016-12-06 19:19:43.311158 -    DEBUG - yamp - MainProcess - MainThread - input: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] (10)\n    2016-12-06 19:19:43.323326 -    DEBUG - yamp - MainProcess - MainThread - 3 workers ready\n    2016-12-06 19:19:44.325203 -    DEBUG - yamp - MainProcess - MainThread - START has been requested to the Pool\n    2016-12-06 19:19:44.325444 -    DEBUG - yamp - MainProcess - MainThread - START event emitted\n    2016-12-06 19:19:44.325639 -     INFO - yamp - MainProcess - Monitor2 - Start to work: creating the fork\n    2016-12-06 19:19:44.327255 -    DEBUG - yamp - MainProcess - Monitor2 - Start monitoring\n    2016-12-06 19:19:44.331702 -    DEBUG - yamp - Process2 - Worker2 - Fork starts 0:00:00.006179 after the event trigger\n    2016-12-06 19:19:44.331925 -    DEBUG - yamp - Process2 - Worker2 - argin: 0\n    (...)\n    2016-12-06 19:20:22.369294 -     INFO - yamp - Process2 - Worker2 - argout: 0\n    2016-12-06 19:20:22.370414 -    DEBUG - yamp - Process2 - Worker2 - STEP event emitted\n    2016-12-06 19:20:22.370479 -    DEBUG - yamp - MainProcess - PoolMonitor - STEP event catch\n    2016-12-06 19:20:22.370571 -    DEBUG - yamp - Process2 - Worker2 - argin: 1\n    2016-12-06 19:20:22.371047 -    DEBUG - yamp - MainProcess - PoolMonitor - STEP event received\n    2016-12-06 19:20:22.371420 -    DEBUG - yamp - MainProcess - PoolMonitor - collect 1 outputs\n    2016-12-06 19:20:22.371743 -  WARNING - yamp - MainProcess - PoolMonitor - Worker 0 hasn't start when it should have (event: True, worker: False)\n    2016-12-06 19:20:22.371993 -  WARNING - yamp - MainProcess - PoolMonitor - Worker 1 hasn't start when it should have (event: True, worker: False)\n    (...)\n    2016-12-06 19:20:43.318191 -    DEBUG - yamp - MainProcess - Monitor0 - Waiting to start\n    2016-12-06 19:20:43.318308 -     INFO - yamp - MainProcess - Monitor0 - Start to work: creating the fork\n    2016-12-06 19:20:43.319511 -    DEBUG - yamp - MainProcess - Monitor0 - Start monitoring\n    2016-12-06 19:20:43.320754 -    DEBUG - yamp - MainProcess - Monitor1 - Waiting to start\n    2016-12-06 19:20:43.320849 -     INFO - yamp - MainProcess - Monitor1 - Start to work: creating the fork\n    2016-12-06 19:20:43.322135 -    DEBUG - yamp - MainProcess - Monitor1 - Start monitoring\n    2016-12-06 19:20:43.323962 -    DEBUG - yamp - Process0 - Worker0 - Fork starts 0:00:58.998431 after the event trigger\n    2016-12-06 19:20:43.324197 -    DEBUG - yamp - Process0 - Worker0 - argin: 2\n    2016-12-06 19:20:43.326832 -    DEBUG - yamp - Process1 - Worker1 - Fork starts 0:00:59.001310 after the event trigger\n    2016-12-06 19:20:43.327090 -    DEBUG - yamp - Process1 - Worker1 - argin: 3\n    (...)\n\n* Try to use _threading.Event_ instead of _multiprocessing.Event_ for the _start_ (only used by the _MainProcess_) didn't change this behaviour of delayed start.\n",
        "description_content_type": null,
        "docs_url": null,
        "download_url": null,
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/srgblnch/yamp",
        "keywords": null,
        "license": "GPLv3+",
        "maintainer": null,
        "maintainer_email": null,
        "name": "yamp",
        "package_url": "https://pypi.org/project/yamp/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/yamp/",
        "project_urls": {
            "Homepage": "https://github.com/srgblnch/yamp"
        },
        "release_url": "https://pypi.org/project/yamp/0.0.5-6/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "UNKNOWN",
        "version": "0.0.5-6"
    },
    "last_serial": 2564020,
    "releases": {
        "0.0.5-6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8cd33e10e0d6478f3d30bf50d97f4a58",
                    "sha256": "6ecc01415b322a8ec2ca768d518cb3ddc43b1729cc7a20a75db1ea935b5758c1"
                },
                "downloads": -1,
                "filename": "yamp-0.0.5-6.tar.gz",
                "has_sig": false,
                "md5_digest": "8cd33e10e0d6478f3d30bf50d97f4a58",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 96722,
                "upload_time": "2017-01-10T08:35:43",
                "url": "https://files.pythonhosted.org/packages/f2/b2/35b3dca35daf978c88159385f2d290a966386f1d2bec0b676b977eee9c88/yamp-0.0.5-6.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "8cd33e10e0d6478f3d30bf50d97f4a58",
                "sha256": "6ecc01415b322a8ec2ca768d518cb3ddc43b1729cc7a20a75db1ea935b5758c1"
            },
            "downloads": -1,
            "filename": "yamp-0.0.5-6.tar.gz",
            "has_sig": false,
            "md5_digest": "8cd33e10e0d6478f3d30bf50d97f4a58",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 96722,
            "upload_time": "2017-01-10T08:35:43",
            "url": "https://files.pythonhosted.org/packages/f2/b2/35b3dca35daf978c88159385f2d290a966386f1d2bec0b676b977eee9c88/yamp-0.0.5-6.tar.gz"
        }
    ]
}