{ "info": { "author": "Suriyadeepan Ramamoorthy", "author_email": "suriyadeepan.r@gmail.com", "bugtrack_url": null, "classifiers": [ "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Programming Language :: Python :: 3" ], "description": "

\n\n

\n\n---\n\nA sweet tool for Remote Execution.\n\n\nWhat is the chillest way one can train models in remote machines? \n\n- Do not worry about environment setup (dependencies)\n- Don't bother choosing an instance to run on\n- No more bash scripts to copy files back and forth\n\n## Principles\n\n**re** provides a suite of features that empowers the user to focus on the experiments without having to worry about boring details listed above.\n\n- Almost zero conf\n- Abstract away boring repetitive details\n- Ease of execution\n\n## Conventions\n\nYou do need to follow a couple of conventions.\n\n- Data goes into `data/`\n- Any non-python file that is necessary for remote execution should be added to `.recompute/include`\n- Any python file that shouldn't be pushed to remote machine should be added to `.recompute/exclude`\n\n## Setup\n\n```bash\n# install sshpass\nbrew install http://git.io/sshpass.rb\npip install --user recompute\n```\n\n## Configuration\n\nThe configuration file is super-short.\n\n```ini\n[general]\ninstance = 0\nremote_home = projects/\n\n[instance 0]\nusername = grenouille\nhost = grasse.local\npassword = hen0s3datru1h\n\n```\n\nYou can add credentials for remote machines directly into the configuration file or add them sequentially via command-line `re sshadd --instance='user@remotehost'`.\n\n## Workflow\n\nMy machine learning workflow follows these steps:\n\n- Copy code to remote machine `rsync`\n- Setup dependencies `pip install`\n- Download dataset and place them in `data/`\n- Execute code in remote machine\n- Get execution log\n- Copy binaries generated `bin/`\n\nwiht **re**, the tasks listed above can be accomplished with 4 commands, as below:\n\n```bash\n# re sshadd --instance='\nre init # initalize [rsync, install]\nre async \"python3 x.py\" # start execution in remote\n# (or) re sync \"python3 x.py\" # blocking run (wait for completion)\nre log # after a while\nre pull \"bin/ ./bin/\" . # pull generated binaries\n```\n\n- `init` creates local configuration files, setting up the environment for remote execution\n - Makes a list of local dependencies (python files)\n - Populates `requirements.txt` with required pypi packages\n - Installs pypi packages in remote machine\n - Copies local dependencies to remote machine using `rsync`\n - A copy of local folder is created in the remote machine, under `~/projects/`\n- We could start execution in remote machine and wait for it to complete by using `sync` mode or just start remote execution and move on, using `async` mode\n - The command to be executed in remote machine, should be given as a string next to `sync` or `async` mode\n- `re log` fetches log from remote machine\n- `re pull` pulls any file from remote machine\n - Files are addressed by their relative paths\n\n## Logging\n\n**re** redirects the `stdout` and `stderr` of remote execution into `.log`, which could be pulled to local machine by running `re log`. More often than not, it takes a while for execution to complete. So we start the execution in remote machine and check the log once in a while using `re log`. Or you could put this \"once in a while\" as a command-line argument and **re** pulls the log and shows you every \"once in a while\". It is recommended to use `logging` module to print information onto stdout, instead of `print` statements.\n\n```bash\n# fetch log from remote machine\nre log\n# . start execution in remote machine\n# .. fetch log\nre async \"python3 nn.py\"\nre log\n# . start execution \n# .. pull log every 20 seconds\nre async \"python3 nn.py\"\nre log --loop=20\n```\n\n## rsync\n\nFiles (local dependencies) can be synchronized by using `rsync` command. `rsync` is run in the background which copies files listed in `.recompute/rsync.db` to remote machine. `--force` switch forces **re** to figure out the local dependencies and update `rsync.db`.\n\n```bash\nre rsync # --force updates .recompute/rsync.db\n```\n\n## Dependencies\n\n`requirements.txt` is populated with python packages necessary for execution (uses `pipreqs` behind the scenes). `re install` reads `requirements.txt` and installs the packages in remote system.\n\n```bash\n# install dependencies\nre install # --force updates requirements.txt\n# manual install\nre install \"torch tqdm\"\n```\n\n## Manages Processes\n\n**re** keeps track of all the remote processes it has spawned. We could list them out using `list` command and selectively kill processes using `kill` command.\n\n```bash\n# list live processes\nre list\n# +-------+--------------+-------+\n# | Index | Name | PID |\n# +-------+--------------+-------+\n# | 0 | all | * |\n# | 1 | zombie/spawn | 30601 |\n# | 2 | runner | 31036 |\n# +-------+--------------+-------+\n# kill process [1]\nre kill --idx=1\n# kill them all\nre purge\n# or kill interactively with just `re kill`\n```\n\n## Upload/Download\n\nYou might wanna download or upload a file just once without having to include it in rsync database. We have `push` and `pull` commands. And there is a special command named `data` which downloads from space separated urls from command-line, into remote machine's `data/` directory.\n\n```bash\n# . upload from local machine to remote\n# .. copy [current_dir/x/localfile] to [remote_home/projects/mynn1/x/]\nre push \"x/localfile x/\"\n# . download from remote machine to local\n# .. copy [remote_home/projects/mynn1/y/remotefile] to [current_dir/y/remotefile]\nre pull \"y/remotefile y/\"\n# download IRIS dataset to remote machine's [data/]\nre data https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data # more urls can be added, separated by spaces\n```\n\n## Notebook\n\nSometimes you wanna run code snippets in a notebook. `re notebook` starts a remote jupyter notebook server and hooks it to a local port. The remote server is tracked (`re list`) and could be killed whenever necessary.\n\n```bash\n# . start notebook server in remote machine\n# .. hook to local port\nre notebook # Cntl-c to quit\n```\n\n## Probe\n\n`probe` command probes remote machines and provides us with a table of available machines with info on available resources.\n\n```bash\nre probe\n# +--------------------------------+--------+----------+-----------+\n# | Machine | Status | GPU (MB) | Disk (MB) |\n# +--------------------------------+--------+----------+-----------+\n# | grenouille@grasse.local | active | 10432 | 4238 |\n# | slartibartfast@magrathea.local | active | 8642 | 12012 |\n# +--------------------------------+--------+----------+-----------+\n```\n\n## Manual\n\n`re man` gives you a detailed manual.\n\n| Mode | Description | Options | Example |\n|----------|-----------------------------------------------------|-----------------------|----------------------------------|\n| init | Setup current directory for remote execution | --instance-idx | re init |\n| | | | re init --instance-idx=1 |\n| rsync | Use rsync to synchronize local files with remote | --force | re rsync |\n| sshadd | Add a new instance to config | --instance | re sshadd --instance=\"usr@host\" |\n| install | Install pypi packages in requirements.txt in remote | cmd, --force | re install |\n| | | | re install \"pytorch tqdm\" |\n| sync | Synchronous execution of \"args.cmd\" in remote | cmd, --force, --rsync | re sync \"python3 x.py\" |\n| async | Asynchronous execution of \"args.cmd\" in remote | cmd, --force, --rsync | re async \"python3 x.py\" |\n| log | Fetch log from remote machine | --loop, --filter | re log |\n| | | | re log --loop=2 |\n| | | | re log --filter=\"pattern\" |\n| list | List out processes alive in remote machine | --force | re list |\n| kill | Kill a process by index | --idx | re kill |\n| | | | re kill --idx=1 |\n| purge | Kill all remote process that are alive | None | re purge |\n| ssh | Create an ssh session in remote machine | None | re ssh |\n| notebook | Create jupyter notebook in remote machine | --run-async | re notebook |\n| push | Upload file to remote machine | cmd | re push \"x.py y/\" |\n| pull | Download file from remote machine | cmd | re pull \"y/z.py .\" |\n| data | Download data from web into data/ folder of remote | cmd | re data \"url1 url2 url3\" |\n| man | Show this man page | None | re man \n\n## Contribution\n\nAll kinds of contribution are welcome.\n\n- Somethin went wrong?\n- What feature is missing?\n- What could be done better?\n\nRaise an [issue](https://github.com/suriyadeepan/recompute.py/issues).\nAdd a pull request.\n\n## License\n\nCopyright (c) 2019 Suriyadeepan Ramamoorthy. All rights reserved.\n\nThis work is licensed under the terms of the MIT license. \nFor a copy, see .\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/suriyadeepan/recompute.py", "keywords": "", "license": "", "maintainer": "", "maintainer_email": "", "name": "recompute", "package_url": "https://pypi.org/project/recompute/", "platform": "", "project_url": "https://pypi.org/project/recompute/", "project_urls": { "Homepage": "https://github.com/suriyadeepan/recompute.py" }, "release_url": "https://pypi.org/project/recompute/0.9.14/", "requires_dist": [ "prettytable", "pipreqs", "pytest" ], "requires_python": "", "summary": "Remote Execution Framework", "version": "0.9.14" }, "last_serial": 4985931, "releases": { "0.9.10": [ { "comment_text": "", "digests": { "md5": "1ad1f2f2951821b8e73c6dc8dba31da7", "sha256": "de3ab921a579c7ee835925eec6134abd87ea80e437bb843e60053107bbb3f504" }, "downloads": -1, "filename": "recompute-0.9.10.tar.gz", "has_sig": false, "md5_digest": "1ad1f2f2951821b8e73c6dc8dba31da7", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24432, "upload_time": "2019-03-25T14:02:57", "url": "https://files.pythonhosted.org/packages/fc/cb/ae8b18fa0c448b70b9f89868d5d2318c7a1bad460e4b3690cd714f091817/recompute-0.9.10.tar.gz" } ], "0.9.11": [ { "comment_text": "", "digests": { "md5": "e82d805b476783c92f314d3bf165c44f", "sha256": "c148f005da42aa04e6b8ff231401b4c6576ea97d2b1fd2c72f0fc178c4b13fb7" }, "downloads": -1, "filename": "recompute-0.9.11-py3.6.egg", "has_sig": false, "md5_digest": "e82d805b476783c92f314d3bf165c44f", "packagetype": "bdist_egg", "python_version": "3.6", "requires_python": null, "size": 50279, "upload_time": "2019-03-25T14:07:21", "url": "https://files.pythonhosted.org/packages/d9/11/582576e9dacde0e509f2fa1f24168c8799ea8544b84e8e87f5dd78d093e0/recompute-0.9.11-py3.6.egg" }, { "comment_text": "", "digests": { "md5": "f359f7e81e0d7f9671252f079cb986ce", "sha256": "74409a2fe1552fcc7eba513e6bc5136dfd6a255866c41cd5600aa55ca2b31624" }, "downloads": -1, "filename": "recompute-0.9.11-py3-none-any.whl", "has_sig": false, "md5_digest": "f359f7e81e0d7f9671252f079cb986ce", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 25739, "upload_time": "2019-03-25T14:07:19", "url": "https://files.pythonhosted.org/packages/d1/7c/7f13aaf151c7e5359dafcb37b6066b2ed140a0ee61254b1c4d9cf8bb32c8/recompute-0.9.11-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c35b771653fff3e018cf46a7e5d1c3ab", "sha256": "2b3e78dd8c44fa2592537ca5dfeca24476b53b99ccc2ba3fde2eeab00d38365b" }, "downloads": -1, "filename": "recompute-0.9.11.tar.gz", "has_sig": false, "md5_digest": "c35b771653fff3e018cf46a7e5d1c3ab", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24426, "upload_time": "2019-03-25T14:07:23", "url": "https://files.pythonhosted.org/packages/7e/ed/bb7345fb0e2f30d8cbdfda16ffc0e384032cb16559994df6395d81e42b83/recompute-0.9.11.tar.gz" } ], "0.9.12": [ { "comment_text": "", "digests": { "md5": "5b36725a2eb43db71a5b255dfa529ecf", "sha256": "2bfeb0717db42e43f792514baeb20d1d14122a6accfd4d68f1a867c45d58abc1" }, "downloads": -1, "filename": "recompute-0.9.12.tar.gz", "has_sig": false, "md5_digest": "5b36725a2eb43db71a5b255dfa529ecf", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24472, "upload_time": "2019-03-25T14:14:37", "url": "https://files.pythonhosted.org/packages/7c/ac/128accf993a05eb86cd1ae2ce643d7f7b28d62b349715acc2e6f5a2f24ca/recompute-0.9.12.tar.gz" } ], "0.9.14": [ { "comment_text": "", "digests": { "md5": "8a1f1dea4029b4d8cdb71e1d93f3245c", "sha256": "4862be30ddb1cbda0ed90fe2a1cf7ada16fb73d29e31fad0a6806eb0bbc0904d" }, "downloads": -1, "filename": "recompute-0.9.14-py3.6.egg", "has_sig": false, "md5_digest": "8a1f1dea4029b4d8cdb71e1d93f3245c", "packagetype": "bdist_egg", "python_version": "3.6", "requires_python": null, "size": 50568, "upload_time": "2019-03-26T04:47:40", "url": "https://files.pythonhosted.org/packages/2b/42/59e6a2562cc54591170a92f87be1db87aaa31ec6140c30334c9100d06811/recompute-0.9.14-py3.6.egg" }, { "comment_text": "", "digests": { "md5": "a08dd90a91f50bf843a9b2510e05bdb6", "sha256": "d1a3dd341794ccdb4fdf3669609decd5d7a58227501ba315270a1398b3f0c3d1" }, "downloads": -1, "filename": "recompute-0.9.14-py3-none-any.whl", "has_sig": false, "md5_digest": "a08dd90a91f50bf843a9b2510e05bdb6", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 25843, "upload_time": "2019-03-26T04:47:37", "url": "https://files.pythonhosted.org/packages/66/e7/cc3511fbaf8fb0843e5f864c8891fbc4aa0b94b946800383b556cedc3e87/recompute-0.9.14-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "8124086c08209d16daacafdf4e681aa4", "sha256": "958c0b67d6c833a46b06ef38b73a5c631389491d1941e96a52da23ab7fffaeb4" }, "downloads": -1, "filename": "recompute-0.9.14.tar.gz", "has_sig": false, "md5_digest": "8124086c08209d16daacafdf4e681aa4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24588, "upload_time": "2019-03-26T04:47:42", "url": "https://files.pythonhosted.org/packages/6c/89/1e5d4efc3830923d4090a8c20d3e0e4dfc973591e73fee61d3e61e24cd9f/recompute-0.9.14.tar.gz" } ], "0.9.2": [ { "comment_text": "", "digests": { "md5": "b0336d36dc981fa69c3b5e77f717a864", "sha256": "f870125df4e3bc477b713b7ee592c632c8717951d86fd13ca479dd70fcc92b53" }, "downloads": -1, "filename": "recompute-0.9.2.tar.gz", "has_sig": false, "md5_digest": "b0336d36dc981fa69c3b5e77f717a864", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24366, "upload_time": "2019-03-25T13:28:55", "url": "https://files.pythonhosted.org/packages/f2/48/0fad6cf9c09ae913d6d129136b64112b25506a72ba3d806a7fa7b725979c/recompute-0.9.2.tar.gz" } ], "0.9.3": [ { "comment_text": "", "digests": { "md5": "e5d0ca8aa068bb5c3e66a69322053baf", "sha256": "1897b58c96bfe0fd82494eb5468bbe1c435df97eb890f5b33170544a1404bb9b" }, "downloads": -1, "filename": "recompute-0.9.3.tar.gz", "has_sig": false, "md5_digest": "e5d0ca8aa068bb5c3e66a69322053baf", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24357, "upload_time": "2019-03-25T13:31:25", "url": "https://files.pythonhosted.org/packages/15/48/507a97255f681df4cb260a801e7f2552a6b50012c6cb6ef03c08abfeb03f/recompute-0.9.3.tar.gz" } ], "0.9.4": [ { "comment_text": "", "digests": { "md5": "f614d09fe6c310f0bcac9dfaf0e08e14", "sha256": "b69fa6ebd5569fcd9053a1fd591995b582d353ebf4d8cce764c4cb87eea4ff37" }, "downloads": -1, "filename": "recompute-0.9.4.tar.gz", "has_sig": false, "md5_digest": "f614d09fe6c310f0bcac9dfaf0e08e14", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24433, "upload_time": "2019-03-25T13:32:37", "url": "https://files.pythonhosted.org/packages/b8/75/856737a7ffda8fdf3546e1693547715730e5257e74a52da159ad06c371e4/recompute-0.9.4.tar.gz" } ], "0.9.5": [ { "comment_text": "", "digests": { "md5": "fcb1ed0dbdffb2d397eca279c24e9037", "sha256": "c70926829880a77b4bfb5b94b81d235de3daee4e758c3178cf60413a16198cce" }, "downloads": -1, "filename": "recompute-0.9.5.tar.gz", "has_sig": false, "md5_digest": "fcb1ed0dbdffb2d397eca279c24e9037", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24497, "upload_time": "2019-03-25T13:43:54", "url": "https://files.pythonhosted.org/packages/87/d9/ecbe5c4845d61326178fe5f7a8096eb1754840e3217cc5ce6c56332844ec/recompute-0.9.5.tar.gz" } ], "0.9.6": [ { "comment_text": "", "digests": { "md5": "24d73da4ba253aa392f22196d68ca1f4", "sha256": "0cf325be82948f92c03881bb45d5dfacb13bb3f017750ba042130fb8da241590" }, "downloads": -1, "filename": "recompute-0.9.6.tar.gz", "has_sig": false, "md5_digest": "24d73da4ba253aa392f22196d68ca1f4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24520, "upload_time": "2019-03-25T13:52:39", "url": "https://files.pythonhosted.org/packages/68/15/2be34e679f207bb33ab67fa2e9eccf146deb261a029861c67646c79b637b/recompute-0.9.6.tar.gz" } ], "0.9.7": [ { "comment_text": "", "digests": { "md5": "30a6981d4d4f57f953dbc1a12be90a9d", "sha256": "a005f2de5608335e564d15ad1ff4ec6d0a7be8c53bc04f7183431770bf8c6de9" }, "downloads": -1, "filename": "recompute-0.9.7.tar.gz", "has_sig": false, "md5_digest": "30a6981d4d4f57f953dbc1a12be90a9d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24525, "upload_time": "2019-03-25T13:56:40", "url": "https://files.pythonhosted.org/packages/5b/86/3b53c3e2969c1c8fab54542257948f063e7e4cb6e22b023c35317d3b9a6d/recompute-0.9.7.tar.gz" } ], "0.9.8": [ { "comment_text": "", "digests": { "md5": "95a9ce9e1deec14c1b8d86d57ce5ab25", "sha256": "9d88f77512e44292528b82e565d6a1ef6d06ad5db51e0756d72c1dd874d10a9d" }, "downloads": -1, "filename": "recompute-0.9.8.tar.gz", "has_sig": false, "md5_digest": "95a9ce9e1deec14c1b8d86d57ce5ab25", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24458, "upload_time": "2019-03-25T13:59:10", "url": "https://files.pythonhosted.org/packages/4e/7d/809916b85b8862b4fedba34a4b91dd7e44b733aa70c3b8ffede4eb13d7d8/recompute-0.9.8.tar.gz" } ], "0.9.9": [ { "comment_text": "", "digests": { "md5": "cfdb0614f7be7414e86c1750e5e3e1cf", "sha256": "c3d8a44262e804e0cca0e8408c4900371b6a7b87f2581651d853346c5e2bda00" }, "downloads": -1, "filename": "recompute-0.9.9.tar.gz", "has_sig": false, "md5_digest": "cfdb0614f7be7414e86c1750e5e3e1cf", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24476, "upload_time": "2019-03-25T14:00:58", "url": "https://files.pythonhosted.org/packages/bb/bb/686432ebc2619ef9e2db907645608ebe063a68a55a01b87728cde570b2e5/recompute-0.9.9.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "8a1f1dea4029b4d8cdb71e1d93f3245c", "sha256": "4862be30ddb1cbda0ed90fe2a1cf7ada16fb73d29e31fad0a6806eb0bbc0904d" }, "downloads": -1, "filename": "recompute-0.9.14-py3.6.egg", "has_sig": false, "md5_digest": "8a1f1dea4029b4d8cdb71e1d93f3245c", "packagetype": "bdist_egg", "python_version": "3.6", "requires_python": null, "size": 50568, "upload_time": "2019-03-26T04:47:40", "url": "https://files.pythonhosted.org/packages/2b/42/59e6a2562cc54591170a92f87be1db87aaa31ec6140c30334c9100d06811/recompute-0.9.14-py3.6.egg" }, { "comment_text": "", "digests": { "md5": "a08dd90a91f50bf843a9b2510e05bdb6", "sha256": "d1a3dd341794ccdb4fdf3669609decd5d7a58227501ba315270a1398b3f0c3d1" }, "downloads": -1, "filename": "recompute-0.9.14-py3-none-any.whl", "has_sig": false, "md5_digest": "a08dd90a91f50bf843a9b2510e05bdb6", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 25843, "upload_time": "2019-03-26T04:47:37", "url": "https://files.pythonhosted.org/packages/66/e7/cc3511fbaf8fb0843e5f864c8891fbc4aa0b94b946800383b556cedc3e87/recompute-0.9.14-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "8124086c08209d16daacafdf4e681aa4", "sha256": "958c0b67d6c833a46b06ef38b73a5c631389491d1941e96a52da23ab7fffaeb4" }, "downloads": -1, "filename": "recompute-0.9.14.tar.gz", "has_sig": false, "md5_digest": "8124086c08209d16daacafdf4e681aa4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24588, "upload_time": "2019-03-26T04:47:42", "url": "https://files.pythonhosted.org/packages/6c/89/1e5d4efc3830923d4090a8c20d3e0e4dfc973591e73fee61d3e61e24cd9f/recompute-0.9.14.tar.gz" } ] }