{ "info": { "author": "Jochem Bijlard", "author_email": "jochem@thehyve.nl", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Intended Audience :: Developers", "License :: OSI Approved :: GNU General Public License (GPL)", "Natural Language :: English", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Topic :: Scientific/Engineering :: Bio-Informatics" ], "description": "transmart-packer\n================\n\n|Build status| |codecov| |pypi| |downloads|\n\n.. |Build status| image:: https://travis-ci.org/thehyve/transmart-packer.svg?branch=master\n :alt: Build status\n :target: https://travis-ci.org/thehyve/transmart-packer/branches\n.. |codecov| image:: https://codecov.io/gh/thehyve/transmart-packer/branch/master/graph/badge.svg\n :alt: codecov\n :target: https://codecov.io/gh/thehyve/transmart-packer\n.. |pypi| image:: https://img.shields.io/pypi/v/transmart-packer.svg\n :alt: PyPI\n :target: https://pypi.org/project/transmart-packer/\n.. |downloads| image:: https://img.shields.io/pypi/dm/transmart-packer.svg\n :alt: PyPI - Downloads\n :target: https://pypi.org/project/transmart-packer/\n\nRun data transformation jobs for TranSMART_.\n\n.. _TranSMART: https://github.com/thehyve/transmart-core\n\n\nInstall\n-------\n\nFirst make virtual environment to install dependencies using `Python3.6+`\n\n.. code-block:: bash\n\n pip install transmart-packer\n\nOr from source:\n\n.. code-block:: bash\n\n git clone https://github.com/thehyve/transmart-packer.git\n cd transmart-packer\n pip install .\n\n\nDependencies\n------------\n\n* a Redis server running on localhost (or update ``packer/config.py``)\n\n\nRunning\n-------\n\nFrom root dir run:\n\n.. code-block:: bash\n\n redis-server\n\n celery -A packer.tasks worker --loglevel=info\n\n transmart-packer\n\n\n*Environment variables:*\n\n============================== =================\nVariable Description\n============================== =================\n``TRANSMART_URL`` The URL of the TranSMART API server\n``KEYCLOAK_SERVER_URL`` Keycloak server URL, e.g., ``https://keycloak-dwh-test.thehyve.net``\n``KEYCLOAK_REALM`` The Keycloak realm, e.g., ``transmart-dev``\n``KEYCLOAK_CLIENT_ID`` The Keycloak client ID (default: ``transmart-client``)\n``REDIS_URL`` Redis server URL (default: ``redis://localhost:6379``)\n``DATA_DIR`` Directory to write export data (default: ``/tmp/packer/``)\n``LOG_CFG`` Logging configuration (default: ``packer/logging.yaml``)\n``CLIENT_ORIGIN_URL`` URLs to restrict cross-origin requests to (CORS) (default: ``*``)\n============================== =================\n\nAn optional variable `VERIFY_CERT` can be used to specify the path of a certificate collection file (`.pem`)\nused to verify HTTP requests.\n\n\nAlternatively, you could build and run the stack from code using ``docker-compose``.\n\n.. code-block:: bash\n\n # Downloads redis image and creates image with project dependencies.\n docker-compose build\n\n # After build is complete, start via:\n docker-compose up\n\nOn code change the webserver will automatically restart, but the Celery workers will not.\nAfter updating the Celery task logic, you will need to restart the Docker container.\n\n\nUsage\n-----\n\nAvailable handlers:\n\n============================== =================\nPath Description\n============================== =================\n``GET /jobs`` List all known jobs for this user.\n``POST /jobs/create`` Create a new job by providing `job_type` and `job_parameters`, creates the job and returns a `task_id`.\n``GET /jobs/status/`` Get status details for a specific task.\n``GET /jobs/cancel/`` Cancel scheduled or abort a running task.\n``GET /jobs/data/`` Download the data that this task produced.\n``WS /jobs/subscribe`` Open websocket connection to get live updates on job progress.\n============================== =================\n\nTo start the toy job \"add\" on the localhost machine\nmake call to ``http://localhost:8999/jobs/create?job_type=add&job_parameters={%22x%22:500,%22y%22:1501}``.\n\n\nDevelopment\n-----------\n\nComponents\n++++++++++\n\n.. figure:: images/transmart-packer.svg\n :alt: Overview of the components of transmart-packer and the interaction with external components.\n\n Overview of the components of transmart-packer and the interaction with external components.\n\n\nTesting\n+++++++\n\nTo run the test suite, we have to start redis-server and celery workers with the commands above.\nThen you can run:\n\n.. code-block:: bash\n\n python setup.py test\n\nExtending\n+++++++++\n\nNew jobs can be added by adding a new Celery function to the jobs folder and adding\nthe function to the jobs registry. See the `packer/jobs/example.py`_ to learn how.\n\n.. _packer/jobs/example.py: https://github.com/thehyve/transmart-packer/blob/master/packer/jobs/example.py\n\n\nExisting jobs\n-------------\n\nBasic export job\n++++++++++++++++\n\nExport transmart api client observation dataframe to tsv file\n\n.. code-block:: json\n\n {\n \"job_type\":\"basic_export\",\n \"job_parameters\": {\n \"constraint\": {\n \"type\":\"study_name\",\n \"studyId\":\"CSR\"\n\n },\n \"custom_name\":\"name of the export\"\n }\n }\n\n\nCSR export\n++++++++++\n\n`The Central Subject Registry (CSR) data model`_ specific export.\nThe model contains individual, diagnosis, biosource and biomaterial entities,\nfollowing the hierarchy: patient > diagnosis > biosource > biomaterial.\nThe entities IDs are first 4 columns of the export file. The rest of the columns are concepts.\nHigher level concepts (e.g Age that is specific to Patient level)\nget distributed to all rows specific to lower levels (e.g. Diagnosis)\n\nSee the CSR_ test study as an example.\n\n.. _CSR: https://github.com/thehyve/transmart-core/tree/dev/transmart-data/test_studies/CSR\n.. _The Central Subject Registry (CSR) data model: https://github.com/thehyve/python_csr2transmart/blob/master/csr/csr.py\n\n.. code-block:: json\n\n {\n \"job_type\":\"csr_export\",\n \"job_parameters\": {\n \"constraint\": {\n \"type\":\"study_name\",\n \"studyId\":\"CSR\"\n\n },\n \"custom_name\":\"name of the export\",\n \"row_filter\": {\n \"type\":\"patient_set\",\n \"subjectIds\": [\"P2\", \"P6\"]\n }\n }\n }\n\n\nwhere:\n\n- ``job_parameters.constraint`` - any `transmart v2 api constraint`_\n or composition of them that used to get data from transmart.\n- ``job_parameters.custom_name`` (optional) - name of the export job and the output ``tsv`` file.\n- ``job_parameters.row_filter`` (optional) - any `transmart v2 api constraint`_\n or composition of them to fetch keys (``[[[[patient], diagnosis], biosource], biomaterial]``) that will make it to the end result.\n e.g. Given the `CSR` study and query above only rows specific to `P2` and `P6` patients will end up to the result table such as `P2`, `D2`, `BS2`, `BM2`, ... row.\n Please note that keys do not have to be equals in length. A row gets selected if only part of keys matches. e.g. `P1` vs `P1`, `D1`\n\n.. _`transmart v2 api constraint`: https://github.com/thehyve/transmart-core/blob/dev/open-api/swagger.yaml\n\n\nAdding new entity to CSR data model:\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen the CSR data model is extended with new entities, the export transformation code\nhas to be changed as well in order to include a column with the ID of the new entity as one of the identifying columns.\n\nIn order to do this, ``_ file has to be modified.\nRequired changes:\n\n1) extend the ``ID_COLUMNS`` list,\n\n2) modify ``from_obs_df_to_csr_df(obs: DataFrame)`` function in order to properly rename columns.\n\n\nLicense\n-------\n\nCopyright \u00a9 2019 The Hyve B.V.\n\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU Affero General Public License as\npublished by the Free Software Foundation, either version 3 of the\nLicense, or (at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU Affero General Public License for more details.\n\nYou should have received a copy of the `GNU Affero General Public License`_\nalong with this program. If not, see https://www.gnu.org/licenses/.\n\n.. _`GNU Affero General Public License`: LICENSE\n\n\n\n\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/thehyve/transmart-packer", "keywords": "transmart-packer,transmart", "license": "GNU General Public License v3 or later", "maintainer": "", "maintainer_email": "", "name": "transmart-packer", "package_url": "https://pypi.org/project/transmart-packer/", "platform": "", "project_url": "https://pypi.org/project/transmart-packer/", "project_urls": { "Homepage": "https://github.com/thehyve/transmart-packer" }, "release_url": "https://pypi.org/project/transmart-packer/0.3.0/", "requires_dist": [ "PyYAML (<5.2,>=5.1)", "transmart[full] (==0.2.6)", "redis (==3.2.1)", "celery (==4.3.0)", "tornado (==6.0.2)", "aioredis (==1.2.0)", "cryptography (==2.6.1)", "pyjwt (==1.7.1)", "requests (==2.22.0)", "pandas (<0.26.0,>=0.25.1)", "numpy (==1.17.1)", "prospector[with_pyroma] ; extra == 'dev'", "pygments ; extra == 'dev'", "yapf ; extra == 'dev'", "isort ; extra == 'dev'" ], "requires_python": ">=3.6.0", "summary": "Data transformation jobs for TranSMART", "version": "0.3.0" }, "last_serial": 5994515, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "37b0645f800cf34607c8239f7f16fec1", "sha256": "59f1d5c312a9b11a799c4c3f62b5a6a73f9b6562e7f4634fc62a5ba58b888dfa" }, "downloads": -1, "filename": "transmart_packer-0.1.0-py3-none-any.whl", "has_sig": false, "md5_digest": "37b0645f800cf34607c8239f7f16fec1", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 31385, "upload_time": "2019-05-14T16:21:14", "url": "https://files.pythonhosted.org/packages/a6/d6/4c0cbcad43f82a62526cc20b98a5158a4da5c7cd2517e61b3a31a92da72d/transmart_packer-0.1.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ae20a03ae587ca0fda3c34c8fe833520", "sha256": "eb8f1a22e59f6e0be83d9cec7acaf5e2169604874f725ad89626b4fadfa138b0" }, "downloads": -1, "filename": "transmart-packer-0.1.0.tar.gz", "has_sig": false, "md5_digest": "ae20a03ae587ca0fda3c34c8fe833520", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 17523, "upload_time": "2019-05-14T16:21:16", "url": "https://files.pythonhosted.org/packages/e5/f2/3fd2572302b1dc23b4adaa18a8ee5b8ebf40d36a43ffb33c9260641b8341/transmart-packer-0.1.0.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "0fe52f20927931ef80a3e879c16d9d10", "sha256": "150ed04fe497ca9cdf6789c14f6b1d746256aa49fd1547c055c77938c7dcbe52" }, "downloads": -1, "filename": "transmart_packer-0.1.1-py3-none-any.whl", "has_sig": false, "md5_digest": "0fe52f20927931ef80a3e879c16d9d10", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 31383, "upload_time": "2019-05-14T16:37:17", "url": "https://files.pythonhosted.org/packages/d7/c0/78444ac35c49df92a8c81c81cd62f4cf707df603e6d0596e18c5e97f0e54/transmart_packer-0.1.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "e7366000d0d4f2c62ef8e5c3b2aad8e4", "sha256": "3add787a572e3d196cf5b3668b1bb904e8586744e76648510ce7d2be4060c8a1" }, "downloads": -1, "filename": "transmart-packer-0.1.1.tar.gz", "has_sig": false, "md5_digest": "e7366000d0d4f2c62ef8e5c3b2aad8e4", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 17522, "upload_time": "2019-05-14T16:37:18", "url": "https://files.pythonhosted.org/packages/cc/81/3cc7f1b6d4937d6733db8a21741427ab043016f910140518d1310a91d65b/transmart-packer-0.1.1.tar.gz" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "4d73f67a0d2bdf16631a2908a79b4df9", "sha256": "6ca30af60fb3d5c3313ccbdc552ba54b9cc0f3e54c33986d4d97291192850219" }, "downloads": -1, "filename": "transmart_packer-0.1.2-py3-none-any.whl", "has_sig": false, "md5_digest": "4d73f67a0d2bdf16631a2908a79b4df9", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 31385, "upload_time": "2019-05-14T19:42:54", "url": "https://files.pythonhosted.org/packages/22/ce/e4ff4263620b6575d8abaaeb2688906d59ca23dfc92d96c91701b6364cf4/transmart_packer-0.1.2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "04b0d900b3b3ce5d0ac0a26fe70b4aa9", "sha256": "03b12ec3076786b8472804b07084bb876ab8a05e1a8f782a616f17659194bee8" }, "downloads": -1, "filename": "transmart-packer-0.1.2.tar.gz", "has_sig": false, "md5_digest": "04b0d900b3b3ce5d0ac0a26fe70b4aa9", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 17517, "upload_time": "2019-05-14T19:42:55", "url": "https://files.pythonhosted.org/packages/e6/f2/7bad485bcd67bcc86127134066d1a388ffcf4368ac90cd659b099b36766b/transmart-packer-0.1.2.tar.gz" } ], "0.1.3": [ { "comment_text": "", "digests": { "md5": "6ab11c2bb1d673595c7c6b5b111629cd", "sha256": "4b6673205ba898d64714e7d7a50229e9dc9ac6c7b700f1fd29e870447014c4b1" }, "downloads": -1, "filename": "transmart_packer-0.1.3-py3-none-any.whl", "has_sig": false, "md5_digest": "6ab11c2bb1d673595c7c6b5b111629cd", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 31385, "upload_time": "2019-05-14T20:06:34", "url": "https://files.pythonhosted.org/packages/6c/a4/fa72c99b72fad35cbdadada8a7381d6de13b1b67e98f9210b518a2f86fe9/transmart_packer-0.1.3-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "3979fb3aede1820db9d2b242455fe98c", "sha256": "aa2d7f61285ad8827ca583146ce2be2db2068245055172d15cc790896ab708ff" }, "downloads": -1, "filename": "transmart-packer-0.1.3.tar.gz", "has_sig": false, "md5_digest": "3979fb3aede1820db9d2b242455fe98c", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 17516, "upload_time": "2019-05-14T20:06:35", "url": "https://files.pythonhosted.org/packages/a7/2f/bbb7a6b3b5319b6b1f2794f27230028a795ac595fd803869027d6776b9a7/transmart-packer-0.1.3.tar.gz" } ], "0.1.4": [ { "comment_text": "", "digests": { "md5": "d92477f9ce92d93be56547a14ca04264", "sha256": "bb6ca692789e01045d1a7e495470ed2c27e1dce62e062bdc939a5215d8b8cb6d" }, "downloads": -1, "filename": "transmart_packer-0.1.4-py3-none-any.whl", "has_sig": false, "md5_digest": "d92477f9ce92d93be56547a14ca04264", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 31973, "upload_time": "2019-08-27T16:27:13", "url": "https://files.pythonhosted.org/packages/6b/9d/bc902e3c8802cf875d6cdd8d7bde79c76f69920eee54d6b496ad84dfeae5/transmart_packer-0.1.4-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "6d66eda0facfa8bbc2471129976a7f18", "sha256": "b12e9d063fc17db66b6b19df24fd5b68f986da5e34c027b2a96997ba83104d75" }, "downloads": -1, "filename": "transmart-packer-0.1.4.tar.gz", "has_sig": false, "md5_digest": "6d66eda0facfa8bbc2471129976a7f18", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 18487, "upload_time": "2019-08-27T16:27:15", "url": "https://files.pythonhosted.org/packages/81/63/20537a314299564e1df3c205a90c46a280a711d5934220020d7628fa65be/transmart-packer-0.1.4.tar.gz" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "d6ea6e5da7657ef23da6f248f53d03c5", "sha256": "8c762bb1e08c035de1693cfac1ecf93e026804f42c97951e98987bc0737f7937" }, "downloads": -1, "filename": "transmart_packer-0.2.0-py3-none-any.whl", "has_sig": false, "md5_digest": "d6ea6e5da7657ef23da6f248f53d03c5", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 32033, "upload_time": "2019-09-23T08:16:57", "url": "https://files.pythonhosted.org/packages/7a/80/14ecc6b254cb4dfe7871effd65e04bf191c1e8e71b554a6a77b5527a151e/transmart_packer-0.2.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "7b3eaa2796cb4f09b6ceae5c2e73682e", "sha256": "ab9552ad6d1fc2a2111d829eb053c7a8bb2a77a99c26bf17ec57e855c588d8bb" }, "downloads": -1, "filename": "transmart-packer-0.2.0.tar.gz", "has_sig": false, "md5_digest": "7b3eaa2796cb4f09b6ceae5c2e73682e", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 18525, "upload_time": "2019-09-23T08:16:58", "url": "https://files.pythonhosted.org/packages/71/e1/b546a9bb2c32b098131094e7e8a16095840a9d5f02fdaaa82b068096fe4b/transmart-packer-0.2.0.tar.gz" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "441dc15d6f34bb33e810e5987aecd013", "sha256": "a9b87ab71424fec659c9d8cd467ddf733eb6d7901310eac5d3ad420403e20203" }, "downloads": -1, "filename": "transmart_packer-0.2.1-py3-none-any.whl", "has_sig": false, "md5_digest": "441dc15d6f34bb33e810e5987aecd013", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 32248, "upload_time": "2019-09-23T22:22:15", "url": "https://files.pythonhosted.org/packages/0e/f8/24c7d05835cfcebc42dff68dca8969f495c83d1f3639964091b0ec32f7ee/transmart_packer-0.2.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "3819c13dd186fd3d1a1baf949bf40d99", "sha256": "d5c1b26b842fb804e837e4ab6752e6c09a1ceb759ef938f15af24ff3b64c422a" }, "downloads": -1, "filename": "transmart-packer-0.2.1.tar.gz", "has_sig": false, "md5_digest": "3819c13dd186fd3d1a1baf949bf40d99", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 18747, "upload_time": "2019-09-23T22:22:17", "url": "https://files.pythonhosted.org/packages/46/70/fa21cb4b061f6a6486907b1186dee63dcb3f2c3b8024021410478de001f9/transmart-packer-0.2.1.tar.gz" } ], "0.3.0": [ { "comment_text": "", "digests": { "md5": "fd11cc4a0c22f86d9b4ad38dd6e84f74", "sha256": "2c45618dc3faababe07ce86f13a13218bde00cc48fc2726074f4b9bbb02e310d" }, "downloads": -1, "filename": "transmart_packer-0.3.0-py3-none-any.whl", "has_sig": false, "md5_digest": "fd11cc4a0c22f86d9b4ad38dd6e84f74", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 32344, "upload_time": "2019-10-18T09:00:53", "url": "https://files.pythonhosted.org/packages/fd/32/7a2e3dd685fd92843d1567c307cda6b243645abdff2d840a7b607aff2bf2/transmart_packer-0.3.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "bd7e8b2d785c6e1a06d1100610c6a659", "sha256": "7103813d5c38b327cfe9e22066d7cc6f9ead3a2ef537fc501271c8906393123e" }, "downloads": -1, "filename": "transmart-packer-0.3.0.tar.gz", "has_sig": false, "md5_digest": "bd7e8b2d785c6e1a06d1100610c6a659", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 19353, "upload_time": "2019-10-18T09:00:56", "url": "https://files.pythonhosted.org/packages/2c/e0/b34ca9563999b47abeaecffb8da6008c609685bcf559f8c16a390d457094/transmart-packer-0.3.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "fd11cc4a0c22f86d9b4ad38dd6e84f74", "sha256": "2c45618dc3faababe07ce86f13a13218bde00cc48fc2726074f4b9bbb02e310d" }, "downloads": -1, "filename": "transmart_packer-0.3.0-py3-none-any.whl", "has_sig": false, "md5_digest": "fd11cc4a0c22f86d9b4ad38dd6e84f74", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 32344, "upload_time": "2019-10-18T09:00:53", "url": "https://files.pythonhosted.org/packages/fd/32/7a2e3dd685fd92843d1567c307cda6b243645abdff2d840a7b607aff2bf2/transmart_packer-0.3.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "bd7e8b2d785c6e1a06d1100610c6a659", "sha256": "7103813d5c38b327cfe9e22066d7cc6f9ead3a2ef537fc501271c8906393123e" }, "downloads": -1, "filename": "transmart-packer-0.3.0.tar.gz", "has_sig": false, "md5_digest": "bd7e8b2d785c6e1a06d1100610c6a659", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 19353, "upload_time": "2019-10-18T09:00:56", "url": "https://files.pythonhosted.org/packages/2c/e0/b34ca9563999b47abeaecffb8da6008c609685bcf559f8c16a390d457094/transmart-packer-0.3.0.tar.gz" } ] }