{
    "info": {
        "author": "Max Horn",
        "author_email": "maexlich@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3 :: Only"
        ],
        "description": "simple_gpu_scheduler\n--------------------\n\nA simple scheduler to run your commands on individual GPUs.  Following the\n[KISS principle](https://en.wikipedia.org/wiki/KISS_principle), this script\nsimply accepts commands via `stdin` and executes them on a specific GPU by\nsetting the `CUDA_VISIBLE_DEVICES` variable.\n\nThe commands read are executed using the login shell, thus redirections `>`\npipes `|` and all other kinds of bash magic can be used.\n\nInstallation\n------------\n\nThe package can simply be installed from\n[pypi](https://pypi.org/project/simple-gpu-scheduler/)\n```bash\n$ pip install simple_gpu_scheduler\n```\n\nExample\n-------\n\nTo show how this generally works, we will create jobs that simply outputs\na job id and the value of `CUDA_VISIBLE_DEVICES`:\n\n```bash\nfor i in {0..10}; do echo \"echo job_id=$i device=\\$CUDA_VISIBLE_DEVICES && sleep 3\"; done | simple_gpu_scheduler --gpus 0,1,2\n```\n\nwhich results in the following output:\n\n```\nProcessing `command echo job_id=0 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2\nProcessing `command echo job_id=1 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 1\nProcessing `command echo job_id=2 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0\njob_id=0 device=2\njob_id=1 device=1\njob_id=2 device=0\n--- 3 seconds no output ---\nProcessing command `echo job_id=3 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2\nProcessing command `echo job_id=4 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 1\nProcessing command `echo job_id=5 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0\njob_id=3 device=2\njob_id=4 device=1\njob_id=5 device=0\n--- 3 seconds no output ---\nProcessing command `echo job_id=6 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2\nProcessing command `echo job_id=7 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 1\nProcessing command `echo job_id=8 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0\njob_id=6 device=2\njob_id=7 device=1\njob_id=8 device=0\n--- 3 seconds no output ---\nProcessing command `echo job_id=9 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2\nProcessing command `echo job_id=10 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0\njob_id=9 device=2\njob_id=10 device=0\n```\n\nThis is equivalent to creating a file `commands.txt` with the following content:\n\n```bash\necho job_id=0 device=$CUDA_VISIBLE_DEVICES && sleep 3\necho job_id=1 device=$CUDA_VISIBLE_DEVICES && sleep 3\necho job_id=2 device=$CUDA_VISIBLE_DEVICES && sleep 3\necho job_id=3 device=$CUDA_VISIBLE_DEVICES && sleep 3\necho job_id=4 device=$CUDA_VISIBLE_DEVICES && sleep 3\necho job_id=5 device=$CUDA_VISIBLE_DEVICES && sleep 3\necho job_id=6 device=$CUDA_VISIBLE_DEVICES && sleep 3\necho job_id=7 device=$CUDA_VISIBLE_DEVICES && sleep 3\necho job_id=8 device=$CUDA_VISIBLE_DEVICES && sleep 3\necho job_id=9 device=$CUDA_VISIBLE_DEVICES && sleep 3\necho job_id=10 device=$CUDA_VISIBLE_DEVICES && sleep 3\n```\n\nand running\n```bash\nsimple_gpu_scheduler --gpus 0,1,2 < commands.txt\n```\n\nSimple scheduler for jobs\n-------------------------\n\nCombined with some basic command line tools, one can set up a very basic\nscheduler which waits for new jobs to be \"submitted\" and executes them in order\nof submission.\n\nSetup and start scheduler in background or in a separate permanent session\n(using for example `tmux`):\n```bash\ntouch gpu.queue\ntail -f -n 0 gpu.queue | simple_gpu_scheduler --gpus 0,1,2\n```\nthe command `tail -f -n 0` follows the end of the gpu.queue file. Thus if there\nwas anything written into `gpu.queue` prior to the execution of the command it\nwill not be passed to `simple_gpu_scheduler`.\n\nThen submitting commands boils down to appending text to the `gpu.queue` file:\n\n```bash\necho \"my_command_with | and stuff > logfile\" >> gpu.queue\n```\n\nHyperparameter search\n---------------------\n\nIn order to allow user friendly utilization of the scheduler in the common\nscenario of hyperparameter search, a convenience script `simple_hypersearch` is\nincluded in the package.\n\n```bash\nsimple_hypersearch -h\n```\n\n```bash\nusage: simple_hypersearch [-h] [--sampling-mode {shuffled_grid,grid}]\n                          [--n-samples N_SAMPLES] [--seed SEED]\n                          [-p NAME [VALUES ...]]\n                          command_pattern\n\nConvenience tool to generate hyperparameter search commands from a command pattern and parameter ranges.\n\npositional arguments:\n  command_pattern       Command pattern where placeholders with {parameter_name} should be replaced.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --sampling-mode {shuffled_grid,grid}\n                        Determine how to sample commands. Either in the grid order [grid]\n                        or in a shuffled order [shuffled_grid, default].\n  --n-samples N_SAMPLES\n                        Number of samples to draw. If not provided use all possible combinations.\n  --seed SEED           Random seed to ensure reproducability when using randomized order of the grid.\n  -p NAME [VALUES ...], --parameter NAME [VALUES ...]\n                        Name of parameter followed by values that should be considered for hyperparameter search.\n                        Example: `-p lr 0.01 0.001 0.0001`\n\nUsage example:\n    simple_hypersearch \"my_program --param1 {param1} --param2 {param2}\" -p param1 0 1 -p param2 2 3\n    will generate the output:\n    my_program --param1 0 --param2 2\n    my_program --param1 0 --param2 3\n    my_program --param1 1 --param2 2\n    my_program --param1 1 --param2 3\n```\n\nThis allows to easily perform hyperparameter searches over a grid of values or\nuniform samples of the grid (dependent on the setting of `sampling-mode`).\nThe output can directly be piped into `simple_gpu_scheduler` or appended to the\n\"queue file\" (see [Simple scheduler for jobs](#simple-scheduler-for-jobs)).\n\nHere some more concrete examples:\n\n**Grid of all possible parameter configurations in random order:**\n```bash\nsimple_hypersearch \"python3 train_dnn.py --lr {lr} --batch_size {bs}\" -p lr 0.001 0.0005 0.0001 -p bs 32 64 128 | simple_gpu_scheduler --gpus 0,1,2\n```\n\n**5 uniformly sampled parameter configurations:**\n```bash\nsimple_hypersearch \"python3 train_dnn.py --lr {lr} --batch_size {bs}\" --n-samples 5 -p lr 0.001 0.0005 0.0001 -p bs 32 64 128 | simple_gpu_scheduler --gpus 0,1,2\n```\n\nTODO\n----\n\n - Multi line jobs (evtl. we would then need a submission script after all)\n - Stop, but let commands finish when receiving a defined signal\n - Tests would be nice, until now the project is still __very small__ but if it\n   grows tests should be added\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/ExpectationMax/simple_gpu_scheduler",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "simple-gpu-scheduler",
        "package_url": "https://pypi.org/project/simple-gpu-scheduler/",
        "platform": "",
        "project_url": "https://pypi.org/project/simple-gpu-scheduler/",
        "project_urls": {
            "Homepage": "https://github.com/ExpectationMax/simple_gpu_scheduler"
        },
        "release_url": "https://pypi.org/project/simple-gpu-scheduler/0.1.4/",
        "requires_dist": null,
        "requires_python": ">3.6",
        "summary": "A simple scheduler for running commands on multiple GPUs.",
        "version": "0.1.4"
    },
    "last_serial": 5983236,
    "releases": {
        "0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6b28b5c091db50ff7caf85f3db7ed6d8",
                    "sha256": "93ec8bb64e96371da3e63250210cc4baa26f2f9b244135bdb204e631ddd9ef20"
                },
                "downloads": -1,
                "filename": "simple_gpu_scheduler-0.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "6b28b5c091db50ff7caf85f3db7ed6d8",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">3.6",
                "size": 15458,
                "upload_time": "2019-10-11T07:30:58",
                "url": "https://files.pythonhosted.org/packages/2f/d6/592f6a8ec1391886643557524d72ddd30c23ffaba08b37bbe41f71cb23db/simple_gpu_scheduler-0.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "789d2e4e1d04acbfe972ebf57792a102",
                    "sha256": "2b6fec2a0baab6f2aaf8ec0aaed97d856bbbaed73df214e9461272a81e139ac7"
                },
                "downloads": -1,
                "filename": "simple_gpu_scheduler-0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "789d2e4e1d04acbfe972ebf57792a102",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">3.6",
                "size": 15488,
                "upload_time": "2019-10-11T07:31:02",
                "url": "https://files.pythonhosted.org/packages/bf/f8/1f3870466628e04728f4d526e9d72a77a24abb01af6ceae9307f885eca3d/simple_gpu_scheduler-0.1.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "805fbe97b66bcf129defa3ae2d725e7e",
                    "sha256": "ad6aa3aa7c2ddf6eebe82efa9981b27e30be8043bb1bc4d0637e9d7ddee3a94e"
                },
                "downloads": -1,
                "filename": "simple_gpu_scheduler-0.1.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "805fbe97b66bcf129defa3ae2d725e7e",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">3.6",
                "size": 16594,
                "upload_time": "2019-10-11T07:38:34",
                "url": "https://files.pythonhosted.org/packages/19/ee/13580b122aabb071e8654b3837884b49c9fc9a6f2f2b5c11ece531093b36/simple_gpu_scheduler-0.1.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "73e9b39e4ba5b20a515c69c143aee766",
                    "sha256": "fd5201751a3374d0b129e07194baa21c0e5c07ed8aa785c781d6eecab465ccb7"
                },
                "downloads": -1,
                "filename": "simple_gpu_scheduler-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "73e9b39e4ba5b20a515c69c143aee766",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">3.6",
                "size": 15948,
                "upload_time": "2019-10-11T07:38:36",
                "url": "https://files.pythonhosted.org/packages/2f/11/8b4fc11793f13a72f1a032834e2886c0df3f454eb7d26373fcfbc39fb577/simple_gpu_scheduler-0.1.1.tar.gz"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4fa58b811edb77ffaf73b680652e0fca",
                    "sha256": "6fac7b895b951fd2c65e3d3c946bc0001ac1485f5eecf362abdfe4e79950f9f5"
                },
                "downloads": -1,
                "filename": "simple_gpu_scheduler-0.1.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "4fa58b811edb77ffaf73b680652e0fca",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">3.6",
                "size": 16617,
                "upload_time": "2019-10-11T07:49:31",
                "url": "https://files.pythonhosted.org/packages/ae/a1/b8da588d1b69b6911f438437d2ad912f87444ebc29ce7591ef476e7dbc1e/simple_gpu_scheduler-0.1.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "8bd17e908f49da4f60120e869634039f",
                    "sha256": "cb6d5f1da83504f6a3b3f688cee24eda00cf5e053b0efe185553688bdf5df8aa"
                },
                "downloads": -1,
                "filename": "simple_gpu_scheduler-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "8bd17e908f49da4f60120e869634039f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">3.6",
                "size": 15988,
                "upload_time": "2019-10-11T07:49:33",
                "url": "https://files.pythonhosted.org/packages/aa/da/3f8edc6bc639cc544654f0b5fdfed6fda333520a2aeb0c05a73ea45b9671/simple_gpu_scheduler-0.1.2.tar.gz"
            }
        ],
        "0.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "93f4ec8f18f9c14edba99c02b8db72a9",
                    "sha256": "a1bc227294cbb0c0f3505fe3d133e8410b056eb9e44385a9d82e592847074dcf"
                },
                "downloads": -1,
                "filename": "simple_gpu_scheduler-0.1.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "93f4ec8f18f9c14edba99c02b8db72a9",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">3.6",
                "size": 20511,
                "upload_time": "2019-10-16T12:39:09",
                "url": "https://files.pythonhosted.org/packages/47/2f/eebf9d526c6c4c27abd2dd628d96ff48ca4c0b79174ddbe0ade622ec8d7c/simple_gpu_scheduler-0.1.3-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e3f26e7887ae49b0436313baf6189490",
                    "sha256": "6c4f514e0f3d9cb0a0dd0ee15f7c058d3dac079b2350ba3c31a4c785b0d79c8d"
                },
                "downloads": -1,
                "filename": "simple_gpu_scheduler-0.1.3.tar.gz",
                "has_sig": false,
                "md5_digest": "e3f26e7887ae49b0436313baf6189490",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">3.6",
                "size": 18137,
                "upload_time": "2019-10-16T12:39:11",
                "url": "https://files.pythonhosted.org/packages/0c/67/1c5f200c5c30642371e0c4c875388900b6b6e216a2a9a09e23b4800d82d9/simple_gpu_scheduler-0.1.3.tar.gz"
            }
        ],
        "0.1.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ec3bf3b5203f9d89271605f7f873f7cf",
                    "sha256": "bc3a8d7268eac63891e051807f61eb2d4155701f3aa821f50052ae128617ebda"
                },
                "downloads": -1,
                "filename": "simple_gpu_scheduler-0.1.4-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ec3bf3b5203f9d89271605f7f873f7cf",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">3.6",
                "size": 20513,
                "upload_time": "2019-10-16T12:58:58",
                "url": "https://files.pythonhosted.org/packages/5f/e8/f88a719d4c6c1e58446a126d7a78f17d29df41a23fa15f30542e8c02f245/simple_gpu_scheduler-0.1.4-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "5def38730b75192393d587d16951f789",
                    "sha256": "9b976e5ecbda2a4b9ba1e7651995232192572a72acff6176444be9f051b6e13f"
                },
                "downloads": -1,
                "filename": "simple_gpu_scheduler-0.1.4.tar.gz",
                "has_sig": false,
                "md5_digest": "5def38730b75192393d587d16951f789",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">3.6",
                "size": 18142,
                "upload_time": "2019-10-16T12:59:00",
                "url": "https://files.pythonhosted.org/packages/7a/65/0e26c47691274fcb14616f5cd6075c4b67534fadaedee95588c961339114/simple_gpu_scheduler-0.1.4.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "ec3bf3b5203f9d89271605f7f873f7cf",
                "sha256": "bc3a8d7268eac63891e051807f61eb2d4155701f3aa821f50052ae128617ebda"
            },
            "downloads": -1,
            "filename": "simple_gpu_scheduler-0.1.4-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "ec3bf3b5203f9d89271605f7f873f7cf",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">3.6",
            "size": 20513,
            "upload_time": "2019-10-16T12:58:58",
            "url": "https://files.pythonhosted.org/packages/5f/e8/f88a719d4c6c1e58446a126d7a78f17d29df41a23fa15f30542e8c02f245/simple_gpu_scheduler-0.1.4-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "5def38730b75192393d587d16951f789",
                "sha256": "9b976e5ecbda2a4b9ba1e7651995232192572a72acff6176444be9f051b6e13f"
            },
            "downloads": -1,
            "filename": "simple_gpu_scheduler-0.1.4.tar.gz",
            "has_sig": false,
            "md5_digest": "5def38730b75192393d587d16951f789",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">3.6",
            "size": 18142,
            "upload_time": "2019-10-16T12:59:00",
            "url": "https://files.pythonhosted.org/packages/7a/65/0e26c47691274fcb14616f5cd6075c4b67534fadaedee95588c961339114/simple_gpu_scheduler-0.1.4.tar.gz"
        }
    ]
}