{
    "info": {
        "author": "",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [
            "Intended Audience :: Developers",
            "License :: OSI Approved :: Apache Software License",
            "Natural Language :: English",
            "Operating System :: POSIX",
            "Programming Language :: Python :: 3"
        ],
        "description": "# SevenBridges CWL Draft2 Upgrader (BETA)\n\n[![Travis](https://travis-ci.org/sbg/sevenbridges-cwl-draft2-upgrader.svg?branch=master)](https://travis-ci.org/sbg/sevenbridges-cwl-draft2-upgrader)\n[![PyPI version](https://badge.fury.io/py/sevenbridges-cwl-draft2-upgrader.svg)](https://pypi.python.org/pypi/sevenbridges-cwl-draft2-upgrader)\n\n\nSevenBridges CWL Draft2 Upgrader is a Python3 package that comes with three command line tools:\n- `sbg_cwl_upgrader`\n- `sbg_cwl_decomposer`\n- `sbg_validate_js_cwl_v1`\n\nCommand line tool `sbg_cwl_upgrader` allows you to upgrade CWL \ntools and workflows written in sbg:draft-2 to CWL v1.0 or v1.1. The source CWL can be on your local machine or on the Seven Bridges platform and the\nconverted CWL can be output to your local machine or onto the Seven Bridges platform. For more details see [this section](#recommended-draft2-to-cwl10-upgrade-flow).  \nNote that the conversion process is semi-automated, manual intervention may be required. For more details see the [known limitations section](#known-limitations).  \nManual part of the conversion process may be tedious and require changes to the CWL code. Therefore, strong understanding of the CWL syntax is desirable.\n\nCommand line tool `sbg_cwl_decomposer` installs individual components of a workflow and re-links them back in the workflow. For more details see the [usage section](#decompose-a-platform-workflow).\n\nCommand line tool `sbg_validate_js_cwl_v1` checks a CWL document for non-strict syntax in JavaScript expressions and produces warnings. For more details see the [usage section](#checking-javascript-version-compatibility-in-expressions).\n\n## Installation\n\nThis tool requires Python3. In general, it is helpful to install user space\nPython applications in [virtual environments][venv].\n\n### Install using pip \n\n[venv]: https://packaging.python.org/guides/installing-using-pip-and-virtualenv/\n\nThe package can be installed with `pip`, using\n\n```\npython3 -m pip install sevenbridges-cwl-draft2-upgrader\n```\n\n### Install from source code\n\nInstall from source by cloning the repo, using\n```\ngit clone https://github.com/sbg/sevenbridges-cwl-draft2-upgrader.git\npython3 -m pip install ./sevenbridges-cwl-draft2-upgrader\n```\n\n## Usage\n\nPackage commands use Platform authentication by reading default config from \nthe `~/.sevenbridges/credentials` file.\nFor additional ways of providing authentication, please consult `sbg_cwl_upgrader --help`.\n\n### Save upgraded CWL to a local file\nAs an example, we'll take a publicly available workflow on the Seven Bridges platform, the\nGATK4 WES workflow, convert it to CWL v1.0 and save it locally:\n```\nsbg_cwl_upgrader -i admin/sbg-public-data/whole-exome-sequencing-bwa-gatk-4-0 -o wes.cwl\n```\n\nThe arguments to `-i` and `-o` are interpreted as local CWL files if they end\nin `.cwl`, `.yml`, `.yaml`, or `.json` and Apps on the Seven Bridges platform otherwise.\n\n### Save upgraded CWL to a Platform project\nWe can also directly upload the converted CWL1.0 workflow to the Platform:\n```\nsbg_cwl_upgrader -i admin/sbg-public-data/whole-exome-sequencing-bwa-gatk-4-0 -o username/usernames-demo-project/wes\n```\nYou will additionally be asked if you want to decompose the workflow after installation. \n\n### Save upgraded CWL in version v1.1\nThe CWL version of the upgraded app can be controlled with `-c` parameter:\n```\nsbg_cwl_upgrader -i admin/sbg-public-data/whole-exome-sequencing-bwa-gatk-4-0 -c v1.1 -o username/usernames-demo-project/wes\n```\n\n### Decompose a Platform workflow\nSometimes, you want all workflow components to be available in the same project as the workflow. This can be done using the `sbg_cwl_decomposer` tool.  \nThis tool will:\n- Find all component tools of a workflow\n- Install them in the project where the workflow is installed\n- Insert them in the workflow so they are editable in that project\n\nExample command:\n```\nsbg_cwl_decomposer -a username/usernames-demo-project/wes\n```\n\n### Decompose a local workflow\nSometimes, you want all workflow components to be available under the `steps` folder, next to the main workflow. This can be done using the `sbg_cwl_decomposer` tool.  \nThis tool will:\n- Find all component tools of a workflow\n- Install them in the `steps` folder alongside the main workflow\n- Insert them in the workflow as strings (relative paths to the main workflow)\n\nExample command:\n```\nsbg_cwl_decomposer -a workflow.cwl\n```\n\n## Recommended draft2 to CWL1.0 upgrade flow\n\n### Convert a Platform draft2 workflow to CWL1.0 and run it on the Seven Bridges Platform\n\n- Prepare a development project and copy the workflow there.\n- Run a task in the development project with the draft2 workflow.\n- Run the upgrader script with the same app ID for `-i` and `-o` parameters to create a new revision over the draft2 wf:  \n`sbg_cwl_upgrader -i <app_id> -o <app_id> -d -u`.\n- This will upgrade the workflow and the tools/subworkflows, and install them in the project as separate apps.\n- Rerun the task that used the draft2 workflow with the same inputs.\n- Debug failures. Consult the [known limitations section](#known-limitations) and do a manual intervention\n- If a tool in the workflow requires modifications, find the tool in the project, apply a fix there, and update the tool in the workflow. Good practice is to test the modified tool individually before updating in the workflow.\n- Once the CWL1.0 task completes, check if all the expected outputs are produced properly.\n\n### Convert a Platform draft2 workflow to CWL1.0 and run it using cwltool\n\n- Run the upgrader script with the `-o` parameter pointing to a local file:  \n`sbg_cwl_upgrader -i <app_id> -o /path/to/app.cwl -d -v`.\n- This will upgrade the workflow and the tools/subworkflows, and install them in the `steps` folder alongside the main workflow.\n- Validate JS expressions in the workflow - [check this section](#checking-javascript-version-compatibility-in-expressions).\n- Validate the workflow with `cwltool --validate /path/to/app.cwl` and check for potential errors - see the [portability notes](#portability-notes) section.\n- Prepare workflow inputs for local execution and describe them in `<name>_inputs.json` file (can be either JSON or YAML).\n- Run with cwltool using `cwltool /path/to/app.cwl <name>_inputs.json`.\n- Debug failures. Consult the [known limitations section](#known-limitations) and do a manual intervention.\n- If a tool in the workflow requires modifications, find the tool in the `steps` folder, apply a fix there, and rerun the workflow. Good practice is to test the modified tool individually before updating in the workflow.\n- Once the CWL1.0 task completes, check if all the expected outputs are produced properly.\n\n\n## Known limitations\n\n### Conversion notes\nThe conversion process is semi-automated, meaning some differences between CWL versions can't be automatically resolved and require manual intervention.\nManual intervention may be required when:\n- Tool parses the `job.json` file, which is a Seven Bridges Platform specific log file. This should be avoided in tools.\n- File in `InitialWorkdirRequirement` is created in a subfolder, e.g. `entryname: foo/bar`. Files need to be created directly in the working directory.\n- Glob is catching files outside the working directory, e.g. `glob: $(inputs.input.path)`. In CWL, every job has its own working directory in order to keep things clean. Glob should only capture files inside the working directory (meaning tools should only write outputs in the working directory).\n- JS expression is missing a `return` statement. Every expression should return something (at least `null`), and must use `return`.\n- JS expression is editing the `inputs` variable and then accessing it. The tool converts `$job.inputs` to `inputs`, so in CWL1.0 JS expressions should no longer redefine the `inputs` variable.\n- The converter is not fully tested on upgrading mixed-version workflows. \n\n### Portability notes\nThe CWL standard specifies that glob patterns must follow POSIX(3) pattern matching and JS expressions should follow ES5.1 strict syntax. The Seven Bridges platform allows some additional features for glob and JS, which may not work on executors that strictly follow the CWL standard. Common examples include:  \n- Glob pattern with brace expand syntax [not capturing outputs in cwltool](https://github.com/common-workflow-language/cwltool/issues/870), e.g. `glob: {*.txt,*.pdf}`. Using brace expand pattern (curly brackets) is a bash extension to the glob specification. This pattern will work on the Seven Bridges platform, but not with cwltool.  \nTo get this glob working with all executors, break down the single string pattern into a list of string patterns, i.e. convert `glob: \"{*.txt,*.pdf}\"` to `glob: [\"*.txt\",\"*.pdf\"]`\n- Some CWL executors require strict type matching between connected ports, e.g. a `File` object can't connect to a `File[]` input and vice versa.  \nThese connection mismatches can be handled by adding a step input `valueFrom` expression to transform a `File` to `File[]` with `valueFrom: $([self])`, or transform `File[]` to `File` with `valueFrom: $(self[0])`.\n- Some CWL executors require Javascript expressions to be written in ES5.1 strict mode (more info [here](#note-on-javascript-versions-in-expressions)).\n\n### Note on Javascript versions in expressions\n\nWhile the Seven Bridges platform executor correctly evaluates both [JS ES6 and JS ES5.1][js-ver]\ncode in expressions, some CWL executors only evaluate JS ES5.1 in strict mode. If a converted \nworkflow contains JS ES6 or JS with non-strict syntax, the workflow may not be executable on all executors.  \n\n[js-ver]: https://www.w3schools.com/js/js_versions.asp  \n\n### Checking Javascript version compatibility in expressions\n\nPreliminary validation of JS expressions in an app can be done using the `sbg_validate_js_cwl_v1` tool. Running the command:  \n```\nsbg_validate_js_cwl_v1  -i app.cwl\n```\nwill write the expression evaluation result to the stderr stream.  \nTo get a list of undeclared variables without duplicates, run:\n```\nsbg_validate_js_cwl_v1 -i app.cwl 2>&1 >/dev/null | awk 'NR % 3 == 0' | uniq\n```\nand add `var` to the first instances of all undeclared variables.",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/sbg/sevenbridges-cwl-draft2-upgrader",
        "keywords": "sevenbridges,sbg,cwl",
        "license": "Apache Software License 2.0",
        "maintainer": "Seven Bridges Genomics Inc.",
        "maintainer_email": "bogdan.gavrilovic@sbgenomics.com",
        "name": "sevenbridges-cwl-draft2-upgrader",
        "package_url": "https://pypi.org/project/sevenbridges-cwl-draft2-upgrader/",
        "platform": "POSIX",
        "project_url": "https://pypi.org/project/sevenbridges-cwl-draft2-upgrader/",
        "project_urls": {
            "Homepage": "https://github.com/sbg/sevenbridges-cwl-draft2-upgrader"
        },
        "release_url": "https://pypi.org/project/sevenbridges-cwl-draft2-upgrader/0.4.0/",
        "requires_dist": null,
        "requires_python": ">=3.6.0",
        "summary": "Seven Bridges CWL sbg:draft2 to v1.0 upgrader.",
        "version": "0.4.0",
        "yanked": false,
        "yanked_reason": null
    },
    "last_serial": 8897538,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6fff304da8263dc545f4e54b24340b41",
                    "sha256": "e342b7b3c038052475c8759267b57937d0c330556e3289c7851aceac6b5e832a"
                },
                "downloads": -1,
                "filename": "sevenbridges-cwl-draft2-upgrader-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "6fff304da8263dc545f4e54b24340b41",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6.0",
                "size": 34841,
                "upload_time": "2019-09-03T12:33:32",
                "upload_time_iso_8601": "2019-09-03T12:33:32.831248Z",
                "url": "https://files.pythonhosted.org/packages/f1/1d/5ff1be0af8dacbcf185ae0ff459fc39bd7004e880f5576c9aec4490c06f3/sevenbridges-cwl-draft2-upgrader-0.1.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "926ea55dd38353d75bbf5c0c5f92b83f",
                    "sha256": "cb423cbf5dc705fa507cfac821ca18e38b28069ffe2beb186f3cecb065205492"
                },
                "downloads": -1,
                "filename": "sevenbridges-cwl-draft2-upgrader-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "926ea55dd38353d75bbf5c0c5f92b83f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6.0",
                "size": 34891,
                "upload_time": "2019-10-25T12:09:07",
                "upload_time_iso_8601": "2019-10-25T12:09:07.698386Z",
                "url": "https://files.pythonhosted.org/packages/ec/e2/a2212de145c58297170283f1c44dbd0c879422ea137e100459bfe608b03e/sevenbridges-cwl-draft2-upgrader-0.1.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2ee050d36742cb8221a205f9825fb6a6",
                    "sha256": "a505d8f1fdf43bcbfefeaa61852e1b65d5110f1bcedac7e6b4b5d37397c88780"
                },
                "downloads": -1,
                "filename": "sevenbridges-cwl-draft2-upgrader-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "2ee050d36742cb8221a205f9825fb6a6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6.0",
                "size": 37952,
                "upload_time": "2020-02-11T15:44:07",
                "upload_time_iso_8601": "2020-02-11T15:44:07.912433Z",
                "url": "https://files.pythonhosted.org/packages/af/e1/35a76c1c21eafab2dd7dcb3b21b44b56f084d384a66c2fe24243d342b381/sevenbridges-cwl-draft2-upgrader-0.2.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2d477f5d0569e01c79aef05472afba9a",
                    "sha256": "997efaaaa3fd8e10ba5dbaae3b472b26dda269e582ffb733239cc45382e54b5c"
                },
                "downloads": -1,
                "filename": "sevenbridges-cwl-draft2-upgrader-0.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "2d477f5d0569e01c79aef05472afba9a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6.0",
                "size": 34696,
                "upload_time": "2020-02-20T16:35:32",
                "upload_time_iso_8601": "2020-02-20T16:35:32.134974Z",
                "url": "https://files.pythonhosted.org/packages/7c/5c/a588e221fc8f403efcb43c5202e16838359e28bfb2e4446411ad06dede5b/sevenbridges-cwl-draft2-upgrader-0.2.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.2.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8dc445c09341522d7a71eff9b95b43ad",
                    "sha256": "1af5f2e10e12eea078ab0a473921d455cdf553c55001a4076cc9f83cadfdebd5"
                },
                "downloads": -1,
                "filename": "sevenbridges-cwl-draft2-upgrader-0.2.2.tar.gz",
                "has_sig": false,
                "md5_digest": "8dc445c09341522d7a71eff9b95b43ad",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6.0",
                "size": 38301,
                "upload_time": "2020-03-10T13:29:57",
                "upload_time_iso_8601": "2020-03-10T13:29:57.403806Z",
                "url": "https://files.pythonhosted.org/packages/43/52/0af553ffb64617b8c98858e7a48d859b0573c1e7a246fc3dc6d1f7270a6d/sevenbridges-cwl-draft2-upgrader-0.2.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d0a6d25e6682783514e8e6c7bfe834df",
                    "sha256": "ed712b40362562933e66506c28c3b350175fee026658db4c1921195d1a654be8"
                },
                "downloads": -1,
                "filename": "sevenbridges-cwl-draft2-upgrader-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "d0a6d25e6682783514e8e6c7bfe834df",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6.0",
                "size": 35301,
                "upload_time": "2020-08-05T15:38:50",
                "upload_time_iso_8601": "2020-08-05T15:38:50.626787Z",
                "url": "https://files.pythonhosted.org/packages/19/62/6bfd21a90504cf8f428aaebb40b5e93d02ac1ffde211d661c4f7e629b0b6/sevenbridges-cwl-draft2-upgrader-0.3.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "96922a7977b55e69a2a11ee9d4eb6fbf",
                    "sha256": "6c8c98e134054632d865d265304012e42133f203f058a6e39a5467a1d40bdcb7"
                },
                "downloads": -1,
                "filename": "sevenbridges-cwl-draft2-upgrader-0.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "96922a7977b55e69a2a11ee9d4eb6fbf",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6.0",
                "size": 38789,
                "upload_time": "2020-12-14T18:03:42",
                "upload_time_iso_8601": "2020-12-14T18:03:42.786569Z",
                "url": "https://files.pythonhosted.org/packages/f5/80/7ef1db7a69236051a58a8dad43eec4a1ca05761c5a091267831d446314d6/sevenbridges-cwl-draft2-upgrader-0.4.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "96922a7977b55e69a2a11ee9d4eb6fbf",
                "sha256": "6c8c98e134054632d865d265304012e42133f203f058a6e39a5467a1d40bdcb7"
            },
            "downloads": -1,
            "filename": "sevenbridges-cwl-draft2-upgrader-0.4.0.tar.gz",
            "has_sig": false,
            "md5_digest": "96922a7977b55e69a2a11ee9d4eb6fbf",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6.0",
            "size": 38789,
            "upload_time": "2020-12-14T18:03:42",
            "upload_time_iso_8601": "2020-12-14T18:03:42.786569Z",
            "url": "https://files.pythonhosted.org/packages/f5/80/7ef1db7a69236051a58a8dad43eec4a1ca05761c5a091267831d446314d6/sevenbridges-cwl-draft2-upgrader-0.4.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "vulnerabilities": []
}