{ "info": { "author": "Oscar Martinez-Rubi", "author_email": "o.rubi@esciencecenter.nl", "bugtrack_url": null, "classifiers": [], "description": "**This repository is currently not maintained. We welcome people to fork this repository for further development and maintenance.** \n\n\n# pycoeman\n\n[![Build Status](https://travis-ci.org/NLeSC/pycoeman.svg?branch=master)](https://travis-ci.org/NLeSC/pycoeman)\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/41716b834df24626a8cce742ac068fce)](https://www.codacy.com/app/omrubi/pycoeman?utm_source=github.com&utm_medium=referral&utm_content=NLeSC/pycoeman&utm_campaign=Badge_Grade)\n[![DOI](https://zenodo.org/badge/22553/NLeSC/pycoeman.svg)](https://zenodo.org/badge/latestdoi/22553/NLeSC/pycoeman)\n\nPython Commands Execution Manager\n\npycoeman is a Python toolkit for executing command-line commands. It allows the execution of:\n\n- Sequential commands: this is a chain of command-line commands which will be executed one after the other. In other words, this is a set of commands that you would traditionally execute in a Bash script. Normally there are IO dependencies between commands (one command requires the output from one or previous ones).\n\n- Parallel commands: this is a set of command-line commands which are executed in parallel. In other words, this is a set of commands that you would traditionally execute in a Bash script with all the commands as background jobs (with the & at the end). There cannot be IO dependencies between commands. This is can be useful for pleasingly parallel solutions, i.e. tools which are single-core at programming level but can be parallelized at data level and usually require some final merging process.\n\npycoeman adds CPU/MEM/disk monitoring during the execution of the commands and it allows to create clean execution environments for easier management of your executions (the commands will be executed in different folders separated from where the input data is). pycoeman has tools to run both sequential and parallel commands locally (in the local computer), and also to run parallel commands in a set of remote hosts accessible via SSH as well as in SGE clusters (computer clusters with Sun Grid Engine batch-queuing system). pycoeman is configured using XML files.\n\npycoeman is used by pymicmac (https://github.com/ImproPhoto/pymicmac)\n\n## Installation\n\nFor now pycoeman only works in Linux systems. It requires Python 3.5 and it is installed using pip.\nIf Python 3.5 is not the default Python on user's system, the user should use **pip3** to install *pycoeman*.\nTo avoid issues with dependencies and versions, and indirectly permissions, the use should use [**virtualenv**](https://virtualenv.pypa.io/en/stable/).\n\n* Installing system package dependencies\n```\nsudo apt-get install freetype libssl-dev libffi-dev\n```\n\n* Installing *pycoeman* from sources:\n```\ngit clone https://github.com/NLeSC/pycoeman\ncd pycoeman\n\n#If virtualenv is not installed:\nsudo apt-get install virtualenv\n\nvirtualenv pycoeman_env\n. pycoeman_env/bin/activate\n\npip3 -r requirements.txt install .\n```\n\n* Installing *pycoeman* package:\n```\n#If virtualenv is not installed:\nsudo apt-get install virtualenv\n\nvirtualenv pycoeman_env\n. pycoeman_env/bin/activate\n\npip3 -r requirements.txt install pycoeman\n```\n\nThe installation makes the following command-line tools available: `coeman-seq-local`, `coeman-par-local`, `coeman-par-ssh` and `coeman-par-sge`\n\n## Deployment\n\nWith *Pycoeman* user's commands are executed either in *sequential mode* or *parallel mode*. The commands, their arguments and the required files/directories\nare listed in a XML configuration file. Independent from the execution mode, the commands are either executed at the user's computer (commands with *-local*\nsuffix) or distributed among a set of machines (commands with either *-ssh* or *-sge* suffix). The ones with *-ssh* suffix use **ssh** to run commands at the\nremote hosts. The ones with *-sge* suffix submit the commands as jobs to a Sun Grid Engine queuing system.\n\n### Sequential mode\n\nThe sequential commands XML configuration file must contain a root tag `` and for each command a XML element `` is added with two\nmandatory nested XML elements, `` and ``. Dependencies on files/directories should be specified with `` or `` tags. (Soft)\nlinks are created in the execution folder for the specified files/folders. Using `` is recommended for small number of required files/folders, and\nthey are specified comma-separated. Using `` is recommended when the number of required files/folders is large. In this case they are specified in\na separate ASCII file, one file/folder per line. Both `` and `` can be simultaneously used. \n\nOnce the sequential mode XML configuration file is defined the user should use the tools `coeman-seq-[local | ssh | sge]` to execute them. For\nlocal executions, it is important to notice that since all the commands are executed in the same execution folder file/folders already linked\nfor one of the commands (using `` or ``) they don't need to be linked again. In the following example, *inputfile1* and\n*inputfile2* are linked for *Executable1* and re-used for *Executable2*.\n```\n\n \n Executable1 \n listimages.txt\n inputfile1 inputfile2\n Executable1 *jpg inputfile1 inputfile2 -o outputfile1 \n \n \n Executable2 \n Executable2 inputfile1 outputfile1 \n \n\n```\n\n### Parallel mode\n\nThe parallel commands XML configuration file must contain a root tag `` and for each command a XML element `` is added with three\nmandatory nested XML elements, ``, `` and ``. The latter determines which files and directories should be collected as output. Dependencies\non files/directories should be specified with `` or `` tags. (Soft) links are created in the execution folder for the specified\nfiles/folders. Using `` is recommended for small number of required files/folders, and they are specified comma-separated. Using ``\nis recommended when the number of required files/folders is large. In this case they are specified in a separate ASCII file, one file/folder per line.\nBoth `` and `` can be simultaneously used. \n\nOnce the parallel mode XML configuration file is defined the user should use the tools `coeman-par-[local | ssh | sge]` to execute them. When running in\nparallel mode, commands are executed in a different execution folder and possibly in a more than one computer. For each command, the required data is\ncopied/linked from the location where the pycoeman tool is launched to the remote execution folders. After a successful execution of the commands, the\nfiles listed under the tag `` are copied back to the location where the pycoeman tool was launched. \n\nFor data availability the execution model is share-nothing, i.e., the data indicated by `` and `` is not shared between different\ncommands execution. Hence, in each command `` and `` must indicate ALL the required data. This is different than in the sequential\ncommands execution where the required data from a *command A* can be shared with the other commands since they are all executed in the same execution folder.\nIn the following example, *listimage.list* needs to be specified as ** for both *Executable1* and *Executable2* command.\n```\n\n \n Executable1\n listimages.list\n ParallelConfig/0_Config.xml\n Executable1 *jpg 0_Config.xml -o output1\n output1\n \n \n Executable1\n listimages.list\n ParallelConfig/1_Config.xml\n Executable1 *jpg 1_Config.xml -o output2\n output2\n \n\n```\n\n### Remote execution\n\n\nThe tools either with the `-ssh` or `-sge` suffix are used to run commands at remote hosts. In addition to the XML configuration file, the user needs\nto specify the hosts XML file. The host XML file has a `` XML element per each host and five nested XML elements: the `` to define host\nname, `` user name at the remote host, `` to set environment before each command execution at the remote host, `` to\ndefine the number of commands executed in simultaneously, and `` specifies the root directory for each command's execution directory\n(each command will then use `/` as execution directory). An example of the hosts XML file follows:\n\n```\n\n \n localhost \n oscarr \n /home/oscarr/.bashrc \n 4 \n /home/oscarr/test_remote_ssh \n \n\n```\n\n**IMPORTANT:**\n* All the required data is sent to the remote nodes using SCP, therefore, the host name must be a valid ssh-reachable host name. It is assumed that\npassword-less ssh connections are possible with all the involved hosts. So, before running `coeman-par-ssh` make sure this is the case. To set password-less\nconnections with remote hosts use SSH keys: generate a key locally with `ssh-keygen` and add the local public key to the remote machine with `ssh-copy-id `.\n* It is assumed that pycoeman and all software required by each commands is installed on all remote hosts.\n* The file specified by `` is used to load the environment at the remote hosts. Hence, the user should make sure all the dependencies, including\n*pycoeman* are loaded through ``. The same holds for environment variables.\n\n#### SGE clusters\n\nThe tools with the suffix `-sge` run commands in clusters with Sun Grid Engine (SGE) queuing system. SGE clusters should have all a shared folder accessable on all the nodes. Having all nodes simultaneously reading and writing to the shared storage is discouraged. Hence, for performance reasons the user should\ncopy the input data from the shared storage to the node's local storage and use a local directory to store command's outputs.\n\nThe *setenv* file and the XML configuration file should be placed at the shared folder. The output of the tools with `-sge` suffix is a submission script.\nThe user should run it to submit the different jobs to a SGE queuing system. It is assumed that the software locations are shared between all the nodes\nand that the setenv file will set the environment properly in all the nodes.\n\n### Monitoring\n\nFor both sequential and parallel commands, during the execution of each command of the specified in the XML configuration file, the CPU and memory and usage of the involved processes are monitored (child processes from any of the executed commands are also considered). There is also monitoring of the disk usage (free vs. used file system storage of the data directory). Monitoring files are created in the execution folder. Concretely a .mon file, a .mon.disk and a .log file. The first one contains CPU/MEM usage monitoring, the second one contains disk usage monitoring and the third one is the log produced by the executed command. To get statistics of CPU and memory usage use the tool `coeman-mon-stats`, to plot the CPU and memory usage use the tool`coeman-mon-plot-cpu-mem`, and to plot the disk usage use the tool `coeman-mon-plot-disk`.\n", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/NLeSC/pycoeman", "keywords": "", "license": "Apache License 2.0", "maintainer": "", "maintainer_email": "", "name": "pycoeman", "package_url": "https://pypi.org/project/pycoeman/", "platform": "", "project_url": "https://pypi.org/project/pycoeman/", "project_urls": { "Homepage": "https://github.com/NLeSC/pycoeman" }, "release_url": "https://pypi.org/project/pycoeman/1.1.1/", "requires_dist": null, "requires_python": "", "summary": "", "version": "1.1.1" }, "last_serial": 3331778, "releases": { "1.1.0": [ { "comment_text": "", "digests": { "md5": "aa2c0eea25151a006d6451968790322c", "sha256": "c9a63f0a46ce6e3484df715e71cf93671f0ac4f4d5a3c7b1e8d46abad23416cc" }, "downloads": -1, "filename": "pycoeman-1.1.0.tar.gz", "has_sig": false, "md5_digest": "aa2c0eea25151a006d6451968790322c", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14743, "upload_time": "2017-11-14T13:38:03", "url": "https://files.pythonhosted.org/packages/06/19/e22339ede0a27aa5b9940a194e159ca8d5a94b34c8535938b2fe1932bbd4/pycoeman-1.1.0.tar.gz" } ], "1.1.1": [ { "comment_text": "", "digests": { "md5": "c8a6ecd1a5989bc7f9aedde5bd6a1b37", "sha256": "18bfcb4a0f98547b4ddbf774bbe5cd371d41b70c52f387176e8e2f98a4e5977c" }, "downloads": -1, "filename": "pycoeman-1.1.1.tar.gz", "has_sig": false, "md5_digest": "c8a6ecd1a5989bc7f9aedde5bd6a1b37", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18495, "upload_time": "2017-11-14T14:19:48", "url": "https://files.pythonhosted.org/packages/5d/d8/392aa7cf522503cd9e51e3b447f52952c5fdc49e69b8c50b47506f69099d/pycoeman-1.1.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "c8a6ecd1a5989bc7f9aedde5bd6a1b37", "sha256": "18bfcb4a0f98547b4ddbf774bbe5cd371d41b70c52f387176e8e2f98a4e5977c" }, "downloads": -1, "filename": "pycoeman-1.1.1.tar.gz", "has_sig": false, "md5_digest": "c8a6ecd1a5989bc7f9aedde5bd6a1b37", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18495, "upload_time": "2017-11-14T14:19:48", "url": "https://files.pythonhosted.org/packages/5d/d8/392aa7cf522503cd9e51e3b447f52952c5fdc49e69b8c50b47506f69099d/pycoeman-1.1.1.tar.gz" } ] }