{ "info": { "author": "Jingcheng Yang", "author_email": "yjcyxky@163.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Environment :: Console", "Environment :: Web Environment", "Framework :: Flask", "Intended Audience :: Developers", "Intended Audience :: Healthcare Industry", "Intended Audience :: Science/Research", "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)", "Operating System :: OS Independent", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3 :: Only", "Topic :: Scientific/Engineering :: Bio-Informatics" ], "description": "> Author: Jingcheng Yang\n>\n> Email: yjcyxky@163.com\n>\n> Date: 2018-12-13\n\n# Choppy for Reproducible Omics Pipeline\n\n[\u4e2d\u6587\u6587\u6863](http://docs.3steps.cn)\n\n## Introduction\n\nChoppy is a command-line & web tool for executing WDL workflows on Cromwell servers. (Based on widdler, but more.)\nFeatures include:\n\n- Workflow execution: Execute a workflow on a specified Cromwell server.\n- Workflow restart: Restart a previously executed workflow.\n- Workflow queries: Get the status, metadata, or logs for a specific workflow.\n- Workflow result explanation: Get more detailed information on fails at the command line.\n- Workflow monitoring: Monitor a specific workflow or set of user-specific workflows to completion.\n- Workflow abortion: Abort a running workflow.\n- JSON validation: Validate a JSON input file against the WDL file intended for use.\n\n## Dependencies\n\nChoppy requires Python 3+ and Java-1.8 to be loaded in your environment in order for full functionality to work.\n\n## Installation\n\n```\nvirtualenv .env\nsource .env/bin/activate\npip install choppy-0.2.0.tar.gz\n\n# Activate bash auto-complete\nactivate-global-python-argcomplete\neval \"$(register-python-argcomplete choppy)\"\n```\n\n## Usage\n\nBelow is choppy's basic help text. Choppy expects one of three usage modes to\nbe indicated as it's first argument: run, query, or abort.\n\n```\nusage: choppy []\n\nDescription: A tool for executing and monitoring WDLs to Cromwell instances.\n\npositional arguments:\n {restart,explain,log,abort,monitor,query,run,validate,label,email,upload,batch,testapp,install,apps}\n\noptional arguments:\n -h, --help show this help message and exit\n```\n\n### choppy submit\n\nBelow is choppy's submit help text. It expects the user to provide a wdl file,\njson file, and to indicate one of the available servers for execution. The validate option\nvalidates both the WDL and the JSON file submitted and is on by default.\n\n```\nusage: choppy submit []\n\nSubmit a WDL & JSON for execution on a Cromwell VM.\n\npositional arguments:\n wdl Path to the WDL to be executed.\n json Path the json inputs file.\n\noptional arguments:\n -h, --help show this help message and exit\n -v, --validate Validate WDL inputs in json file. (default: False)\n -l LABEL, --label LABEL\n A key:value pair to assign. May be used multiple\n times. (default: None)\n -m, --monitor Monitor the workflow and receive an e-mail\n notification when it terminates. (default: False)\n -i INTERVAL, --interval INTERVAL\n If --monitor is selected, the amount of time in\n seconds to elapse between status checks. (default: 30)\n -o EXTRA_OPTIONS, --extra_options EXTRA_OPTIONS\n Additional workflow options to pass to Cromwell.\n Specify as k:v pairs. May be specified multipletimes\n for multiple options. See\n https://github.com/broadinstitute/cromwell#workflow-\n optionsfor available options. (default: None)\n -V, --verbose If selected, choppy will write the current status to\n STDOUT until completion while monitoring. (default:\n False)\n -n, --no_notify When selected, disable choppy e-mail notification of\n workflow completion. (default: False)\n -d DEPENDENCIES, --dependencies DEPENDENCIES\n A zip file containing one or more WDL files that the\n main WDL imports. (default: None)\n -D, --disable_caching\n Don\\'t used cached data. (default: False)\n -S {localhost,remote}, --server {localhost,remote}\n Choose a cromwell server from ['localhost','remote'] (default: None)\n```\n\nFor example:\n\n`choppy submit myworkflow.wdl myinput.json -S remote`\n\nThis will return a workflow ID and status if successfully submitted, for example:\n\n`{'id': '2f8bb5c6-8254-4d38-b010-620913dd325e', 'status': 'Submitted'}`\n\nThis will execute a workflow that uses subworkflows:\n\n`choppy submit myworkflow.wdl myinput.json -S remote -d mydependencies.zip`\n\nUsers may also invoke Choppy's monitoring capabilities when initiating a workflow. See below for an\nexplanation of monitoring options.\n\n### choppy restart\n\nIf a workflow has been previously executed to a Cromwell server, it is possible to restart the workflow after it has\ncompleted and run it again with the same inputs simply by providing the workflow ID and server of the original run.\nThe usage for performing this action is as follows:\n\n```\nusage: choppy restart \n\nRestart a submitted workflow.\n\npositional arguments:\n workflow_id workflow id of workflow to restart.\n\noptional arguments:\n -h, --help show this help message and exit\n -S {localhost,remote}, --server {localhost,remote}\n Choose a cromwell server from ['localhost','remote']\n (default: None)\n```\n\nFor example:\n\n```\nchoppy restart b931c639-e73d-4b59-9333-be5ede4ae2cb -S remote\n```\n\nWill restart workflow b931xxx and return the new workflow id like so:\n\n```\nWorkflow restarted successfully; new workflow-id: 164678b8-2a52-40f3-976c-417c777c78ef\n```\n\nFinally, any restarted workflows will inherit the labels of it's originating workflow.\n\n### choppy query\n\nBelow is choppy's query help text. Aside from the workflow ID it expects one or more optional\narguments to request basic status, metadata, and/or logs.\n\n```\nusage: choppy query []\n\nQuery cromwell for information on the submitted workflow.\n\npositional arguments:\n workflow_id workflow id for workflow execution of interest.\n (default: None)\n\noptional arguments:\n -h, --help show this help message and exit\n -s, --status Print status for workflow to stdout (default: False)\n -m, --metadata Print metadata for workflow to stdout (default: False)\n -l, --logs Print logs for workflow to stdout (default: False)\n -u USERNAME, --username USERNAME\n Owner of workflows to monitor. (default: amr)\n -L LABEL, --label LABEL\n Query status of all workflows with specific label(s).\n (default: None)\n -d DAYS, --days DAYS Last n days to query. (default: 7)\n -S {localhost,remote}, --server {localhost,remote}\n Choose a cromwell server from ['localhost','remote'] (default: None)\n -f {Running,Submitted,QueuedInCromwell,Failed,Aborted,Succeeded}, --filter {Running,Submitted,QueuedInCromwell,Failed,Aborted,Succeeded}\n Filter by a workflow status from those listed above.\n May be specified more than once. (default: None)\n -a, --all Query for all users. (default: False)\n```\n\nFor example:\n`choppy 2f8bb5c6-8254-4d38-b010-620913dd325e query -s -S remote`\n\nwill return something like this:\n\n`[{'id': '2f8bb5c6-8254-4d38-b010-620913dd325e', 'status': 'Running'}]`\n\nand:\n\n`choppy query 2f8bb5c6-8254-4d38-b010-620913dd325e -m -s -S remote`\n\nwill return a ton of information like so (truncated for viewability):\n\n```\n{'status': 'Running', 'submittedFiles': {'workflow': '# GATK WDL\\r\\n# import \"hc_scatter.wdl\" as sub\\r\\n\\r\\ntask VersionCheck {\\r\\n String gatk\\r\\n command {\\r\\n source\n/broad/software/scripts/useuse\\r\\n use Java-1.8\\r\\n use Python-2.7\\r\\n... 'ref': '/cil/shed/sandboxes/amr/dev/gatk_pipeline/output/pfal_5/Plasmodium_falciparum_3D7.fasta'}}]}, 'submi\nssion': '2017-07-14T11:26:05.931-04:00', 'workflowName': 'gatk', 'outputs': {}, 'id': '2f8bb5c6-8254-4d38-b010-620913dd325e'}]\n```\n\nand:\n\n`choppy query 2f8bb5c6-8254-4d38-b010-620913dd325e -l -s -S remote`\n\n```\n[{'id': '2f8bb5c6-8254-4d38-b010-620913dd325e', 'calls': {'gatk.MakeSampleDir': [{'shardIndex': 0, 'attempt': 1, 'stderr': '/cil/shed/apps/internal/cromwell_new/cromwell-executions/ga\n tk/2f8bb5c6-8254-4d38-b010-620913dd325e/call-MakeSampleDir/shard-0/execution/stderr', 'stdout': '/cil/shed/apps/internal/cromwell_new/cromwell-executions/gatk/2f8bb5c6-8254-4d38-b010-\n 620913dd325e/call-MakeSampleDir/shard-0/execution/stdout'}\n```\n\n### choppy abort\n\nBelow is choppy's abort usage. Simply provide the\n\n```\nusage: choppy abort \n\nAbort a submitted workflow.\n\npositional arguments:\n workflow_id workflow id of workflow to abort.\n\noptional arguments:\n -h, --help show this help message and exit\n -S {localhost,remote}, --server {localhost,remote}\n Choose a cromwell server from ['localhost','remote']\n (default: None)\n```\n\nThis example:\n`choppy abort 2f8bb5c6-8254-4d38-b010-620913dd325e -S remote`\n\nwill return:\n\n```\n{'status': 'Aborted', 'id': '2f8bb5c6-8254-4d38-b010-620913dd325e'}\n```\n\n### choppy explain\n\nRunning choppy explain will provide information at command line similar to the monitor e-mail, including workflow\nstatus, root directory, stdout and stderr information, and useful links. Usage is as follows:\n\n```\nusage: choppy explain \n\nExplain the status of a workflow.\n\npositional arguments:\n workflow_id workflow id of workflow to abort.\n\noptional arguments:\n -h, --help show this help message and exit\n -S {localhost,remote}, --server {localhost,remote}\n Choose a cromwell server from ['localhost','remote']\n (default: None)\n```\n\nThis example:\n\n```\nchoppy explain b931c639-e73d-4b59-9333-be5ede4ae2cb -S remote\n```\n\nwill return:\n\n```\n-------------Workflow Status-------------\n{'id': 'b931c639-e73d-4b59-9333-be5ede4ae2cb',\n 'status': 'Failed',\n 'workflowRoot': '/cil/shed/apps/internal/cromwell_gaag/cromwell-executions/gatk/b931c639-e73d-4b59-9333-be5ede4ae2cb'}\n-------------Failed Stdout-------------\n/cil/shed/apps/internal/cromwell_gaag/cromwell-executions/gatk/b931c639-e73d-4b59-9333-be5ede4ae2cb/call-ApplySnpRecalibration/execution/stdout:\n[Errno 2] No such file or directory: u'/cil/shed/apps/internal/cromwell_gaag/cromwell-executions/gatk/b931c639-e73d-4b59-9333-be5ede4ae2cb/call-ApplySnpRecalibration/execution/stdout'\n-------------Failed Stderr-------------\n/cil/shed/apps/internal/cromwell_gaag/cromwell-executions/gatk/b931c639-e73d-4b59-9333-be5ede4ae2cb/call-ApplySnpRecalibration/execution/stderr:\n[Errno 2] No such file or directory: u'/cil/shed/apps/internal/cromwell_gaag/cromwell-executions/gatk/b931c639-e73d-4b59-9333-be5ede4ae2cb/call-ApplySnpRecalibration/execution/stderr'\n-------------Cromwell Links-------------\nhttp://ale:9000/api/workflows/v1/b931c639-e73d-4b59-9333-be5ede4ae2cb/metadata\nhttp://ale:9000/api/workflows/v1/b931c639-e73d-4b59-9333-be5ede4ae2cb/timing\n```\n\nNote that in this case, there were no stdout or stderr for the step that failed in the workflow.\n\n## Validation\n\n(Requires Java-1.8, so make sure to 'use Java-1.8' before trying validation)\n\nChoppy validation attempts to validate the inputs in the user's supplied json file against the WDL\narguments in the supplied WDL file. Validation is OFF by default and so users must specify it using\nthe -v flag if using choppy submit. Validaton can also be performed using choppy validate if you\nwish to validate inputs without executing the workflow.\n\nIt will validate the following:\n\n- That the value of a parameter in the json matches the same type of value the WDL expects. For example\n if the WDL expects an integer and the parameter supplies a float, this will be flagged as an error.\n- That if the parameter is of type File, that the file exists on the file system.\n- If a parameter specified in the json is not expected by the WDL.\n- If a parameter contains the string 'samples_file' it's value will be interpreted as an input TSV file in which\n the last column of every row indicates a sample file. In this case, an existence check will be made on each\n sample file.\n\nIt will NOT validate the following:\n\n- The contents of arrays. It can't tell the difference between an array of strings and an array of integers, but\n it can tell they are arrays, and if a parameter expects an array but is provided something else this will\n be logged as an error.\n\nA note on validating WDL files with dependencies: due to the limitations of the current implementation\nof depedency validation, WDL file dependencies must be present in the same directory as the main WDL file\nand must be unzipped. Otherwise validation may not work.\n\nValidation may also be run as a stand-alone operation using choppy validate. Usage is as follows:\n\n```\nusage: choppy validate \n\nValidate (but do not run) a json for a specific WDL file.\n\npositional arguments:\n wdl Path to the WDL associated with the json file.\n json Path the json inputs file to validate.\n\noptional arguments:\n -h, --help show this help message and exit\n```\n\nFor example:\n\n`choppy mywdl.wdl myjson.json`\n\nIf the json file has errors, a list of errors will be reported in the same way that the runtime validation reports.\nFor example:\n\n```\nbad.json input file contains the following errors:\ngatk.ts_filter_snp: 99 is not a valid Float.\ngatk.tcir: False is not a valid Boolean. Note that JSON boolean values must not be quoted.\ngatk.ploidy: 2.0 is not a valid Int.\nRequired parameter gatk.snp_annotation is missing from input json.\nRequired parameter gatk.ref_file is missing from input json.\n```\n\n### choppy log\n\nRunning 'choppy log' will print to screen the commands used by each task of a workflow. For example, running:\n\n```\nchoppy log becb307f-4718-4d8b-836f-5780d64c4a82 -S remote\n```\n\nResults in the following:\n\n```\n{u'hello.helloWorld': [{u'attempt': 1, u'shardIndex': -1, u'stderr': u'/btl/store/cromwell_executions/hello/becb307f-4718-4d8b-836f-5780d64c4a82/call-helloWorld/execution/stderr', u'stdout': u'/btl/store/cromwell_executions/hello/becb307f-4718-4d8b-836f-5780d64c4a82/call-helloWorld/execution/stdout'}]}\nhello.helloWorld:\n\n#!/bin/bash\ntmpDir=$(mktemp -d /cil/shed/apps/internal/cromwell_new/cromwell-executions/hello/d90bf4f3-d9fb-4f07-92d9-0d46c40355f1/call-helloWorld/execution/tmp.XXXXXX)\nchmod 777 $tmpDir\nexport _JAVA_OPTIONS=-Djava.io.tmpdir=$tmpDir\nexport TMPDIR=$tmpDir\n\n(\ncd /cil/shed/apps/internal/cromwell_new/cromwell-executions/hello/d90bf4f3-d9fb-4f07-92d9-0d46c40355f1/call-helloWorld/execution\necho Hello, amr\n)\necho $? > /cil/shed/apps/internal/cromwell_new/cromwell-executions/hello/d90bf4f3-d9fb-4f07-92d9-0d46c40355f1/call-helloWorld/execution/rc.tmp\n(\ncd /cil/shed/apps/internal/cromwell_new/cromwell-executions/hello/d90bf4f3-d9fb-4f07-92d9-0d46c40355f1/call-helloWorld/execution\n\n)\nsync\nmv /cil/shed/apps/internal/cromwell_new/cromwell-executions/hello/d90bf4f3-d9fb-4f07-92d9-0d46c40355f1/call-helloWorld/execution/rc.tmp /cil/shed/apps/internal/cromwell_new/cromwell-executions/hello/d90bf4f3-d9fb-4f07-92d9-0d46c40355f1/call-helloWorld/execution/rc\n```\n\n### choppy monitor\n\nChoppy allows the monitoring of workflow(s). Unlike the query options, monitoring persists until a workflow reaches\na terminal state (any state besides 'Running' or 'Submitted'). While monitoring, it can optionally print the status of\na workflow to the screen, and when a terminal state is reached, it can optionally e-mail the user (users are assumed\nto be of the broadinstitute.org domain) when the workflow is finished.\n\nMonitoring usage is as follows:\n\n```\nusage: choppy monitor []\n\nMonitor a particular workflow and notify user via e-mail upon completion. If\naworkflow ID is not provided, user-level monitoring is assumed.\n\npositional arguments:\n workflow_id workflow id for workflow to monitor. Do not specify if\n user-level monitoring is desired. (default: None)\n\noptional arguments:\n -h, --help show this help message and exit\n -u USERNAME, --username USERNAME\n Owner of workflows to monitor. (default: )\n -i INTERVAL, --interval INTERVAL\n Amount of time in seconds to elapse between status\n checks. (default: 30)\n -V, --verbose When selected, choppy will write the current status\n to STDOUT until completion. (default: False)\n -n, --no_notify When selected, disable choppy e-mail notification of\n workflow completion. (default: False)\n -S {localhost,remote}, --server {localhost,remote}\n Choose a cromwell server from ['localhost','remote']\n (default: None)\n```\n\n#### Single Workflow Monitoring\n\nAside from monitoring of a single workflow with choppy's run command, you can also execute a monitor as in the\nfollowing example:\n\n```\nchoppy monitor 7ff17cb3-12f1-4bf0-8754-e3a0d39178ea -S remote\n```\n\nIn this case, choppy will continue to silently monitor this workflow until it detects a terminal status. An\ne-mail will be sent to @broadinstitute.org when a terminal status is detected, which will include\nthe metadata of the workflow.\n\nIf --verbose were selected, the user would have seen a STDOUT message indicating the workflows status at intervals\ndefined by the --interval parameter, which has a default of 30 seconds.\n\nIf --no_notify were selected, an e-mail would not be sent.\n\n#### User Workflow Monitoring\n\n(Note this feature is still under active development and is currently quite primitive)\n\nUser's may also monitor all workflows for a given user name by omitting the workflow_id parameter and specifying the\n--user parameter like so:\n\n```\nchoppy monitor -u amr -n -S remote\n```\n\nHere, the user 'amr' is monitoring all workflows ever executed by him using choppy. Any workflows not executed by\nchoppy will not be monitored. Workflows in a terminal state prior to execution will have an e-mail sent immediately\nregarding their status, and any running workflows will result in an e-mail once they terminate. Using the --verbose\noption here would result in STDOUT output for each workflow that is monitored at intervals specified by --interval.\n\n## Logging\n\nChoppy logs information in the application's logs directory in a file called choppy.log.\nThis can be useful to find information on choppy executions including workflow id and query\nresults and can help users locate workflow IDs if they've been lost. Each execution in the log\nis presented like so, with the user's username indicated in the start/stop separators for\nconvenient identification.\n\n```\n-------------New Choppy Execution by amr-------------\n2017-07-14 12:10:44,746 - choppy - INFO - Parameters chosen: {'logs': False, 'func': , 'status': True, 'workflow_id': '7ff17cb3-12f1-4bf0-8754-e3a0d39178ea', 'server': 'btl-cromwell', 'metadata': False}\n2017-07-14 12:10:44,746 - choppy.cromwell.Cromwell - INFO - URL:http://btl-cromwell:9000/api/workflows/v1\n2017-07-14 12:10:44,746 - choppy.cromwell.Cromwell - INFO - Querying status for workflow 7ff17cb3-12f1-4bf0-8754-e3a0d39178ea\n2017-07-14 12:10:44,747 - choppy.cromwell.Cromwell - INFO - GET REQUEST:http://btl-cromwell:9000/api/workflows/v1/7ff17cb3-12f1-4bf0-8754-e3a0d39178ea/status\n2017-07-14 12:10:44,812 - choppy - INFO - Result: [{'id': '7ff17cb3-12f1-4bf0-8754-e3a0d39178ea', 'status': 'Running'}]\n2017-07-14 12:10:44,813 - choppy - INFO -\n-------------End Choppy Execution by amr-------------\n```\n\n## Roadmap\n\n### 2018-12-12\n\n- Add batch mode for batch submmitting WDL workflow.\n- Add app repo. The apps are used repeatedly on some similar projects", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://choppy.3steps.cn", "keywords": "choppy,data platform,command-line tool", "license": "", "maintainer": "Choppy team", "maintainer_email": "pgx@fudan.edu.cn", "name": "choppy-pipe", "package_url": "https://pypi.org/project/choppy-pipe/", "platform": "any", "project_url": "https://pypi.org/project/choppy-pipe/", "project_urls": { "Homepage": "http://choppy.3steps.cn" }, "release_url": "https://pypi.org/project/choppy-pipe/0.3.8.dev0/", "requires_dist": null, "requires_python": "", "summary": "An integrated analysis platform for efficiently achieving computational reproducibility", "version": "0.3.8.dev0" }, "last_serial": 5288989, "releases": { "0.3.8.dev0": [ { "comment_text": "", "digests": { "md5": "17eca09550bcddf4a00a47fe05ad5565", "sha256": "fa4b160169d0f6ef619212f7cad6f4374196d5054622560a0bb314bd32efaad3" }, "downloads": -1, "filename": "choppy-pipe-0.3.8.dev1.tar.gz", "has_sig": false, "md5_digest": "17eca09550bcddf4a00a47fe05ad5565", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8349261, "upload_time": "2019-05-19T17:50:02", "url": "https://files.pythonhosted.org/packages/d2/dd/c751af2dc8974481f9bdd49c014cd54fee2cf40f085d540e661d17b11e8a/choppy-pipe-0.3.8.dev1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "17eca09550bcddf4a00a47fe05ad5565", "sha256": "fa4b160169d0f6ef619212f7cad6f4374196d5054622560a0bb314bd32efaad3" }, "downloads": -1, "filename": "choppy-pipe-0.3.8.dev1.tar.gz", "has_sig": false, "md5_digest": "17eca09550bcddf4a00a47fe05ad5565", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8349261, "upload_time": "2019-05-19T17:50:02", "url": "https://files.pythonhosted.org/packages/d2/dd/c751af2dc8974481f9bdd49c014cd54fee2cf40f085d540e661d17b11e8a/choppy-pipe-0.3.8.dev1.tar.gz" } ] }