{
"info": {
"author": "Nick Watts",
"author_email": "nick@nawatts.com",
"bugtrack_url": null,
"classifiers": [
"Development Status :: 4 - Beta",
"Environment :: Console",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: GNU General Public License v2 (GPLv2)",
"Natural Language :: English",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Topic :: Scientific/Engineering"
],
"description": "What is the Big Data Smart Socket (BDSS)?\n=========================================\n\nThe increasing size of datasets used in scientific computing has made it difficult or impossible for a\nresearcher to store all their data at the compute site they are using to process it. This has necessitated\nthat a data transfer step become a key consideration in experimental design. Accordingly, scientific data\nrepositories such as NCBI have begun to offer services such as dedicated data transfer machines and advanced\ntransfer clients. Despite this, many researchers continue familiar but suboptimal practices: using slow\ntransfer clients like a web browser or ``scp``, transferring data over wireless networks, etc.\n\nBDSS aims to alleviate this problem by shifting the burden of learning about alternative file mirrors,\ntransfer clients, tuning parameters, etc. from the end user researcher to a group of \"data curators\". It\nconsists of three parts:\n\nComponents\n----------\n\n- Metadata repository\n\n - Central database managed by data curators\n - Matches patterns of data file URLs and maps them to alternate sources\n - Includes information about the transfer tool to use to retrieve the data\n\n- BDSS transfer client\n\n - Consumes information from metadata repository\n - Invokes transfer tools\n - Reports analytics to metadata repository\n\n- Integration as a `Galaxy `__ data transfer tool\n\nGet Started\n-----------\n\n- Moving data with the BDSS client:\n\n - `From the command line `__\n `(Video) `__\n - `From Galaxy `__\n `(Video) `__\n\n- `Managing an existing metadata repository\n `__\n\n- Setting up a new metadata repository\n\n - `For development or testing\n `__\n - `For production use\n `__\n\nExamples\n--------\n\nAll examples here require a metadata repository configured to support them. The default metadata repository\nat https://bdss.bioinfo.wsu.edu/ supports these examples and the necessary configuration is also listed here.\n\n- `NCBI SRA archive `__\n- `JGI Genome Portal `__\n\nNCBI SRA archive\n~~~~~~~~~~~~~~~~\n\nNCBI makes files available for `transfer using Aspera Connect `__,\na tool with \"improved data transfer characteristics\" vs FTP or HTTP. If ``ascp`` is installed on your machine,\nBDSS can handle building the appropriate command.\n\nWithout BDSS:\n\n.. code:: shell\n\n ascp -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh -T anonftp@ftp.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR039/SRR039885/SRR039885.sra ./\n\nWith BDSS:\n\n.. code:: shell\n\n bdss transfer -u 'ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR039/SRR039885/SRR039885.sra'\n\nMetadata repository configuration:\n\n.. code:: json\n\n {\n \"data_sources\": [\n {\n \"description\": \"\",\n \"label\": \"NCBI Sequence Read Archive with FTP\",\n \"test_files\": [],\n \"transfer_mechanism\": {\n \"options\": {},\n \"type\": \"curl\"\n },\n \"transforms\": [\n {\n \"for_destinations\": [],\n \"options\": {\n \"new_scheme\": \"aspera\"\n },\n \"target\": \"NCBI Sequence Read Archive with Aspera\",\n \"type\": \"change_scheme\"\n }\n ],\n \"url_matchers\": [\n {\n \"options\": {\n \"pattern\": \"^ftp://ftp\\\\.ncbi\\\\.nlm\\\\.nih\\\\.gov/sra\"\n },\n \"type\": \"regular_expression\"\n }\n ]\n },\n {\n \"description\": \"\",\n \"label\": \"NCBI Sequence Read Archive with Aspera\",\n \"test_files\": [],\n \"transfer_mechanism\": {\n \"options\": {\n \"disable_encryption\": true,\n \"username\": \"anonftp\"\n },\n \"type\": \"aspera\"\n },\n \"transforms\": [],\n \"url_matchers\": [\n {\n \"options\": {\n \"pattern\": \"^aspera://ftp\\\\.ncbi\\\\.nlm\\\\.nih\\\\.gov/sra\"\n },\n \"type\": \"regular_expression\"\n }\n ]\n }\n ],\n \"destinations\": []\n }\n\nJGI Genome Portal\n~~~~~~~~~~~~~~~~~\n\nTo download files from the `JGI Genome Portal `__, you must first\n`authenticate `__. BDSS can prompt for credentials and handle\nstoring your session cookies.\n\nWithout BDSS:\n\n.. code:: shell\n\n curl 'https://signon.jgi.doe.gov/signon/create' --data-urlencode 'login=USER_NAME' --data-urlencode 'password=USER_PASSWORD' -c cookies > /dev/null\n curl 'http://genome.jgi.doe.gov/ext-api/downloads/get-directory?organism=PhytozomeV10' -b cookies > get-directory\n\nWith BDSS:\n\n.. code:: shell\n\n bdss transfer -u 'http://genome.jgi.doe.gov/ext-api/downloads/get-directory?organism=PhytozomeV10'\n JGI Genome Portal username?USER_NAME\n JGI Genome Portal password?USER_PASSWORD\n\nMetadata repository configuration:\n\n.. code:: json\n\n {\n \"data_sources\": [\n {\n \"description\": \"\",\n \"label\": \"JGI Genome Portal\",\n \"test_files\": [],\n \"transfer_mechanism\": {\n \"options\": {\n \"auth_url\": \"https://signon.jgi.doe.gov/signon/create\",\n \"password_field\": \"password\",\n \"password_prompt\": \"JGI Genome Portal password?\",\n \"username_field\": \"login\",\n \"username_prompt\": \"JGI Genome Portal username?\"\n },\n \"type\": \"session_authenticated_curl\"\n },\n \"transforms\": [],\n \"url_matchers\": [\n {\n \"options\": {\n \"pattern\": \"http:\\\\/\\\\/genome\\\\.jgi\\\\.doe\\\\.gov\\\\/ext-api\"\n },\n \"type\": \"regular_expression\"\n }\n ]\n }\n ],\n \"destinations\": []\n }\n\n\n",
"description_content_type": null,
"docs_url": null,
"download_url": "",
"downloads": {
"last_day": -1,
"last_month": -1,
"last_week": -1
},
"home_page": "https://github.com/feltus/BDSS",
"keywords": "",
"license": "GPLv2",
"maintainer": "",
"maintainer_email": "",
"name": "bdss-client",
"package_url": "https://pypi.org/project/bdss-client/",
"platform": "",
"project_url": "https://pypi.org/project/bdss-client/",
"project_urls": {
"Homepage": "https://github.com/feltus/BDSS"
},
"release_url": "https://pypi.org/project/bdss-client/1.0.1b4/",
"requires_dist": [
"colorlog (==2.7.0)",
"requests (==2.8.1)",
"six (==1.10.0)",
"voluptuous (==0.8.11)"
],
"requires_python": "",
"summary": "Big Data Smart Socket client",
"version": "1.0.1b4"
},
"last_serial": 3252591,
"releases": {
"1.0.0b1": [
{
"comment_text": "",
"digests": {
"md5": "b3ce25a3d16a193e9d5159340602058a",
"sha256": "dc3b29673850e9a19e9b401b1ac74648212437943c8e938c0a887d04b72cb425"
},
"downloads": -1,
"filename": "bdss_client-1.0.0b1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b3ce25a3d16a193e9d5159340602058a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 46075,
"upload_time": "2016-08-29T18:17:50",
"url": "https://files.pythonhosted.org/packages/1b/65/65f0b4ac9ceebe3437760808ea0e1d326b69b435a36393d864ccbcdd11c7/bdss_client-1.0.0b1-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "baca88a3b631af326b8c4ba16af7fb7a",
"sha256": "31372ea73e18faebe9b1620d3c53c8080adc2d1bf0b68670d6f44da45429fb9a"
},
"downloads": -1,
"filename": "bdss_client-1.0.0b1.tar.gz",
"has_sig": false,
"md5_digest": "baca88a3b631af326b8c4ba16af7fb7a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 16210,
"upload_time": "2016-08-29T18:17:52",
"url": "https://files.pythonhosted.org/packages/b5/40/e4ae4cc98be98e847992cda773e0cf4bd23633984ef6e458d2f82d78b10a/bdss_client-1.0.0b1.tar.gz"
}
],
"1.0.1b1": [
{
"comment_text": "",
"digests": {
"md5": "84660a56806f235b03b62473fc2cd8c0",
"sha256": "6ade15e6ab89760702a8d86b8fbf18180e7aa7fe9b520c34d4201ed9e2f71bd6"
},
"downloads": -1,
"filename": "bdss_client-1.0.1b1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "84660a56806f235b03b62473fc2cd8c0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 50082,
"upload_time": "2016-08-29T18:33:49",
"url": "https://files.pythonhosted.org/packages/b2/8e/8e06b855e61e54e17feaafaab960c4eecd48d2061da7f431870314e44384/bdss_client-1.0.1b1-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "94d15b2de8fcb785001c1b1cd9796ff4",
"sha256": "7a3f40194a85d51fd2de2b5d22e814fc6857c589efc438af692f249eeb08d58a"
},
"downloads": -1,
"filename": "bdss_client-1.0.1b1.tar.gz",
"has_sig": false,
"md5_digest": "94d15b2de8fcb785001c1b1cd9796ff4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 20204,
"upload_time": "2016-08-29T18:33:52",
"url": "https://files.pythonhosted.org/packages/d2/9f/81389f9d96f819b7fc51250270217979f2b3cf139659d03ad416bc51e367/bdss_client-1.0.1b1.tar.gz"
}
],
"1.0.1b2": [
{
"comment_text": "",
"digests": {
"md5": "5225ae9ca3fb2fbc3fcd759fa29c71df",
"sha256": "2bb547fc5082d3559b3bfd7a8921eb398b8aa1a11bd426f2b98d1ce73ff056df"
},
"downloads": -1,
"filename": "bdss_client-1.0.1b2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5225ae9ca3fb2fbc3fcd759fa29c71df",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 50093,
"upload_time": "2016-11-20T00:39:08",
"url": "https://files.pythonhosted.org/packages/46/06/cc562cea05e87b9dd8ad3469c02472292582bea31ad99bd0cc647a35b976/bdss_client-1.0.1b2-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "8a6dc24ee3b8a4bc8de7d0279dae072d",
"sha256": "f942d80762abdbc27bcd7961fc00310b6e23cdb9df1dbed2191df53702c542c4"
},
"downloads": -1,
"filename": "bdss_client-1.0.1b2.tar.gz",
"has_sig": false,
"md5_digest": "8a6dc24ee3b8a4bc8de7d0279dae072d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 20164,
"upload_time": "2016-11-20T00:39:10",
"url": "https://files.pythonhosted.org/packages/30/2d/1947511783742b5c889e3e4bf705c1fea6b67ba89a707ba7f0e81f8a1142/bdss_client-1.0.1b2.tar.gz"
}
],
"1.0.1b4": [
{
"comment_text": "",
"digests": {
"md5": "46dbd99857110b839395f30224f99918",
"sha256": "724bdbd8c2adc0461f0f7bc7f9ba055e858aea725ffe94a1ffdc0abddbab6656"
},
"downloads": -1,
"filename": "bdss_client-1.0.1b4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "46dbd99857110b839395f30224f99918",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 51951,
"upload_time": "2017-10-16T03:14:07",
"url": "https://files.pythonhosted.org/packages/1e/c6/23af8c8115ae2ee62bc9831f3fc8a30f132ee7760bab390d8052367151c4/bdss_client-1.0.1b4-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "d59d515c9a083c46403763871d7fea19",
"sha256": "97e39aa649bd0454fb73e7f3c52db5805f0d5935dea7298533d520ac780b6e00"
},
"downloads": -1,
"filename": "bdss_client-1.0.1b4.tar.gz",
"has_sig": false,
"md5_digest": "d59d515c9a083c46403763871d7fea19",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 18140,
"upload_time": "2017-10-16T03:14:08",
"url": "https://files.pythonhosted.org/packages/55/16/408c57feefda772b6b07f5b71557f7fe35d5bffb5f16cb7bbf2613127d8f/bdss_client-1.0.1b4.tar.gz"
}
]
},
"urls": [
{
"comment_text": "",
"digests": {
"md5": "46dbd99857110b839395f30224f99918",
"sha256": "724bdbd8c2adc0461f0f7bc7f9ba055e858aea725ffe94a1ffdc0abddbab6656"
},
"downloads": -1,
"filename": "bdss_client-1.0.1b4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "46dbd99857110b839395f30224f99918",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 51951,
"upload_time": "2017-10-16T03:14:07",
"url": "https://files.pythonhosted.org/packages/1e/c6/23af8c8115ae2ee62bc9831f3fc8a30f132ee7760bab390d8052367151c4/bdss_client-1.0.1b4-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "d59d515c9a083c46403763871d7fea19",
"sha256": "97e39aa649bd0454fb73e7f3c52db5805f0d5935dea7298533d520ac780b6e00"
},
"downloads": -1,
"filename": "bdss_client-1.0.1b4.tar.gz",
"has_sig": false,
"md5_digest": "d59d515c9a083c46403763871d7fea19",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 18140,
"upload_time": "2017-10-16T03:14:08",
"url": "https://files.pythonhosted.org/packages/55/16/408c57feefda772b6b07f5b71557f7fe35d5bffb5f16cb7bbf2613127d8f/bdss_client-1.0.1b4.tar.gz"
}
]
}