{ "info": { "author": "Scott Gigante", "author_email": "scottgigante@gmail.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Intended Audience :: Science/Research", "License :: OSI Approved :: GNU General Public License (GPL)", "Topic :: Scientific/Engineering :: Bio-Informatics" ], "description": "Picopore v1.2.0\n===============\n\nA tool for reducing the size of Oxford Nanopore Technologies' datasets without losing information.\n\nIf you find Picopore useful, please cite it at http://dx.doi.org/10.12688/f1000research.11022.1\n\nOptions:\n\n* Raw compression: reduces footprint by removing event detection and basecall data, leaving only raw signal, configuration data and a choice of FASTQ data, basecall summary, both or neither;\n* Lossless compression: reduces footprint without reducing the ability to use other nanopore tools by using HDF5's inbuilt gzip functionality; *(NOTE: as of May 2017, Oxford Nanopore Technologies implemented all compression strategies used in Picopore's lossless compression. Recently basecalled files will therefore not benefit from this compression.)*\n* Deep lossless compression: reduces footprint without removing any data by indexing basecalled dataset to the event detection dataset. *(NOTE: deep lossless compression will have the greatest impact on 2D datasets. Further work to implement 1D^2 compression is in progress.)*\n\nAuthor: Scott Gigante, Walter & Eliza Hall Institute of Medical\nResearch. Contact: `Email `_, `Twitter `_\n\nInstallation\n------------\n\nInstall via pypi\n~~~~~~~~~~~~~~~~\n\nThe latest stable version of Picopore is available on PyPi. Install it using the following command:\n\n::\n\n pip install picopore\n\nInstall via conda\n~~~~~~~~~~~~~~~~~\n\nPicopore and dependencies could also be installed using conda.\n\n::\n\n conda install picopore -c bioconda -c conda-forge\n\nInstall from source\n~~~~~~~~~~~~~~~~~~~\n\nFor the bleeding edge, clone and install from GitHub.\n\n::\n\n git clone https://www.github.com/scottgigante/picopore\n cd picopore\n python setup.py install\n\nCurrently, ``h5py`` is only available on Windows via ``conda``.\n\nRequirements\n~~~~~~~~~~~~\n\nPicopore runs on Python 2.7, 3.4, 3.5, 3.6 or 3.7 with development headers (``python-dev`` or similar).\n\nPicopore requires ``h5repack`` from ``hdf5-tools``, which can be\ndownloaded from https://support.hdfgroup.org/downloads/index.html or\nusing ``sudo apt-get install hdf5-tools`` or similar.\n\nPicopore requires the following Python packages:\n\n* ``h5py``\n* ``future``\n* ``watchdog`` (for real-time compression)\n\nIn addition, ``h5py`` requires HDF5 1.8.4 or later (``libhdf5-dev`` or similar). Difficulties resolving dependencies of ``h5py`` can be resolved by installing from your package manager, using ``sudo apt-get install python-h5py`` or similar.\n\nUsage\n-----\n\n::\n\n commands: picopore\n picopore-realtime monitors a directory for new reads and compresses them in real time\n picopore-test compresses to temporary files and checks that all datasets and attributes are equal (lossless modes only)\n picopore-rename renames groups and datasets within FAST5 files\n\n::\n\n usage: picopore [-h] --mode {lossless,deep-lossless,raw} [--revert] [--fastq]\n [--summary] [--manual STR] [-v] [-y] [-t INT] [--prefix STR]\n [--skip-root] [--print-every INT]\n [input [input ...]]\n\n::\n\n positional arguments:\n input list of directories or fast5 files to shrink\n\n optional arguments:\n -h, --help show this help message and exit\n --mode {lossless,deep-lossless,raw}\n choose compression mode\n --revert reverts files to original size (lossless modes only)\n --fastq, --no-fastq retain FASTQ data (raw mode only) (Default: --fastq)\n --summary, --no-summary\n retain summary data (raw mode only) (Default: --no-\n summary)\n --manual STR manually remove only groups whose paths contain STR\n (raw mode only, regular expressions permitted,\n overrides defaults)\n -v, --version show version number and exit\n -y skip confirm step\n -t INT, --threads INT\n number of threads (Default: 1)\n --prefix STR add prefix to output files to prevent overwrite\n --skip-root, --no-skip-root\n ignore files in root input directories for albacore\n realtime compression (Default: --no-skip-root)\n --print-every INT print a dot every approximately INT files, or -1 to\n silence (Default: 100)\n\nIt is necessary to choose one compression mode out of ``lossless``,\n``deep-lossless``, and ``raw``.\n\nNote that only ``lossless`` and ``deep-lossless`` are options for ``--revert``.\n\nFor ``--manual`` raw compression, the entire group path is used for matching. For example,\nyou could use the command ``picopore --mode raw --manual 1D.*Events [...]`` to remove the\ngroups ``/Analyses/Basecall_1D_000/BaseCalled_template/Events`` and\n``/Analyses/Basecall_1D_000/BaseCalled_complement/Events``.\n\nCompression Modes\n-----------------\n\nPicopore compression allows most nanopore tools to operate unimpeded. We\nprovide a list of software tools which can operate on compressed files\nunimpeded, and the process required to recover the necessary data if\nthis is not possible.\n\n====================== ============= ======================= ============================= =============================\nFunctionality Lossless Deep Lossless Raw Raw ``--no-fastq``\n====================== ============= ======================= ============================= =============================\nMetrichor yes ``picopore --revert`` yes yes\nnanonetcall yes ``picopore --revert`` yes yes\nporetools fastq yes ``picopore --revert`` yes ``nanonetcall / Metrichor``\npoRe printfastq yes ``picopore --revert`` yes ``nanonetcall / Metrichor``\nnanopolish consensus yes ``picopore --revert`` ``nanonetcall / Metrichor`` ``nanonetcall / Metrichor``\n====================== ============= ======================= ============================= =============================\n\nFAQs\n----\n\nWhy would I want to shrink my fast5 files?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNanopore runs are big. Really big. Over a long period of time, the\nstorage footprint of a Nanopore lab will increase to unsustainable\nlevels.\n\nA large proportion of the data stored in ONT's fast5 files is\nunnecessary for the average end-user; during the basecalling process, a\nlarge amount of intermediary data is generated, and for most users who\nsimply need the FASTQ, this data is useless.\n\nPicopore solves this problem. Without removing the raw signal or\nconfiguration data used for basecalling, Picopore removes the\nintermediary datasets to reduce the size of your Nanopore dataset.\n\nDo I lose functionality when using Picopore?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nLossless\n^^^^^^^^\n\nLossless compression uses HDF5's builtin compression, so all existing\nfast5 tools will work seamlessly.\n\n- Use case: power users who wish to reduce server storage footprint\n\nDeep Lossless\n^^^^^^^^^^^^^\n\nDeep lossless compression modifies the structure of your fast5 file: any\ndata extraction tools will not work until you run\n``python picopore.py --revert --mode deep-lossless [input]``.\n\n- Use case: power users who wish to reduce the size of their files during data transfer, or for long-term storage\n\nRaw\n^^^\n\nRaw compression removes the \"squiggle-space\" data. For most users, this\ndata is not critical; the only tools we know of which use the\nsquiggle-space data are ``nanopolish``, ``nanoraw`` and\n``nanonettrain``. If you do not intend on using these tools, your tools\nwill work as before. If you do intend to use these tools, the raw signal\nis retained, and you can resubmit the files for basecalling to generate\nnew squiggle-space data.\n\n- Use case: end users who are only interested in using the FASTQ data\n- Use case: power users running local basecalling with limited local disk space, who wish to use FASTQ immediately and will submit reads to Metrichor at a later date\n\nRaw ``--no-fastq``\n^^^^^^^^^^^^^^^^^^\n\nMinimal compression removes all data not required to rerun basecalling\non the fast5 files. This is only recommended for long-term storage, and\nrequires files to be re-basecalled for any data to be retrieved.\n\n- Use case: users storing historical runs for archive purposes, with no short-term plans to use these reads\n\nDo I lose any data when using Picopore?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nTechnically yes, but nothing that cannot be recovered. In the case where\nyou need to access the data which has been removed, you can regenerate\nit using either picopore (on lossless compression) or using any\nbasecaller provided by ONT (for other methods.)\n\nNote that, since ONT's base calling is continuously improving, the\nbasecalls generated when re-basecalling your data may not be the same,\nbut in fact higher quality than before. If it is important that you\nretain the squiggle-space of the original called sequence, it is\nrecommended that you use a lossless compression method.", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/scottgigante/picopore", "keywords": "nanopore,compression,hdf5,fast5,oxford,minion", "license": "", "maintainer": "", "maintainer_email": "", "name": "picopore", "package_url": "https://pypi.org/project/picopore/", "platform": "", "project_url": "https://pypi.org/project/picopore/", "project_urls": { "Homepage": "https://github.com/scottgigante/picopore" }, "release_url": "https://pypi.org/project/picopore/1.2.0/", "requires_dist": null, "requires_python": "", "summary": "A tool for reducing the size of Oxford Nanopore Technologies' datasets without losing information.", "version": "1.2.0" }, "last_serial": 3136678, "releases": { "0.1.1": [], "0.1.10": [ { "comment_text": "", "digests": { "md5": "8a4c02e7640d8ed735cecc59de244b3e", "sha256": "fbe7226d295383cb0b6cab79e5cc935d85878c7f5e3b04c1a6fd03497fbc1d15" }, "downloads": -1, "filename": "picopore-0.1.10.zip", "has_sig": false, "md5_digest": "8a4c02e7640d8ed735cecc59de244b3e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 16583, "upload_time": "2017-02-11T13:46:22", "url": "https://files.pythonhosted.org/packages/35/9c/f330c03d6eebdd13a9895f102075f4f07ea3d74a8585ce1d6e9e56d00014/picopore-0.1.10.zip" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "40ec8df49f7a64d0d91079db299fd45d", "sha256": "5ab3fe34995a36bb6f8c3dff448a6a4adbb3bd2f9eaf54602bbf89b229d5d398" }, "downloads": -1, "filename": "picopore-0.1.2.zip", "has_sig": false, "md5_digest": "40ec8df49f7a64d0d91079db299fd45d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13204, "upload_time": "2017-02-11T12:02:36", "url": "https://files.pythonhosted.org/packages/78/64/a6f0532004e6be2cae80d4df7077f677e8c207789efa6ead044f0b746c2b/picopore-0.1.2.zip" } ], "0.1.3": [ { "comment_text": "", "digests": { "md5": "17cd59afb92a121b4cbe878eea013179", "sha256": "8eec639ac825f0de2bc65218be5348b46aca00e309116e065f8241c653fc84b2" }, "downloads": -1, "filename": "picopore-0.1.3.zip", "has_sig": false, "md5_digest": "17cd59afb92a121b4cbe878eea013179", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14010, "upload_time": "2017-02-11T12:29:08", "url": "https://files.pythonhosted.org/packages/16/75/542927007264eb01e46c109eba9c37086ebfd82f0ff0285920aa3c6d7a4a/picopore-0.1.3.zip" } ], "0.1.4": [ { "comment_text": "", "digests": { "md5": "9e1f687d4c4bb53c980122b0810ecd9b", "sha256": "fc3e2721a708074032620e04c44facd176d6b4a971b56db7249e5f6d787035f5" }, "downloads": -1, "filename": "picopore-0.1.4.zip", "has_sig": false, "md5_digest": "9e1f687d4c4bb53c980122b0810ecd9b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13973, "upload_time": "2017-02-11T12:32:11", "url": "https://files.pythonhosted.org/packages/f7/0a/41862e75362fd4ad327c4fbd0987c1f0ec879bc78bc687a97392747f74dd/picopore-0.1.4.zip" } ], "0.1.5": [ { "comment_text": "", "digests": { "md5": "65ae85601c1db5c56ef811eaad66cfa3", "sha256": "9a341affe0c77b8d354f7162448f0db37c0b757bdb846cd3f90485bf08f613b8" }, "downloads": -1, "filename": "picopore-0.1.5.zip", "has_sig": false, "md5_digest": "65ae85601c1db5c56ef811eaad66cfa3", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14289, "upload_time": "2017-02-11T12:34:02", "url": "https://files.pythonhosted.org/packages/0f/ec/37ebe3bb471a10c2eaecc2e09f965a481f03d35d30d111e4a9728ea16050/picopore-0.1.5.zip" } ], "0.1.6": [ { "comment_text": "", "digests": { "md5": "56313c0e5e75e7f14c056a5792adc6d2", "sha256": "8aab9a18e31b60d8816c21f7edd9877fa9176f29df555769ed0c8a049d100e95" }, "downloads": -1, "filename": "picopore-0.1.6.zip", "has_sig": false, "md5_digest": "56313c0e5e75e7f14c056a5792adc6d2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14302, "upload_time": "2017-02-11T12:36:27", "url": "https://files.pythonhosted.org/packages/62/8c/dc86c5db0107e7d7e73079dbc28eafed0a7b99d85b8d36ae5cd6acdd3dc2/picopore-0.1.6.zip" } ], "0.1.7": [ { "comment_text": "", "digests": { "md5": "76087f5dcad2a92fbe5bb2a864100529", "sha256": "672978577b7bbcd1ef2535ddf5bd90cef4734d531445ad4ffd6e22cf0824e0b1" }, "downloads": -1, "filename": "picopore-0.1.7.zip", "has_sig": false, "md5_digest": "76087f5dcad2a92fbe5bb2a864100529", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14210, "upload_time": "2017-02-11T12:42:05", "url": "https://files.pythonhosted.org/packages/ec/11/ad549e7e04e1f6afacad429e917cd38848762a82627a0a4853f50e3e7baf/picopore-0.1.7.zip" } ], "0.1.8": [ { "comment_text": "", "digests": { "md5": "66cace846b180e074280ee002e89ecf5", "sha256": "0af0a6235b4c25a571acdf2d6197112fa262abb4555baf1d3516ea5f9e09bf72" }, "downloads": -1, "filename": "picopore-0.1.8.zip", "has_sig": false, "md5_digest": "66cace846b180e074280ee002e89ecf5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14219, "upload_time": "2017-02-11T12:48:04", "url": "https://files.pythonhosted.org/packages/cb/74/a4871869a2d8bd06bd65a776d93c126fef3ac3176b6771deb7351d3db55f/picopore-0.1.8.zip" } ], "0.1.9": [ { "comment_text": "", "digests": { "md5": "b29e2963f9835488044c82975f5728f2", "sha256": "e4024544667873ef2ed717e82c3064c9c058fa997f2f80cd611d1c280bdb1ab7" }, "downloads": -1, "filename": "picopore-0.1.9.zip", "has_sig": false, "md5_digest": "b29e2963f9835488044c82975f5728f2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 16399, "upload_time": "2017-02-11T13:38:00", "url": "https://files.pythonhosted.org/packages/5e/7a/9a472c5e4803ee0315afffa0cfff57ed6dfa663468e2e3037f2414f034f7/picopore-0.1.9.zip" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "cd9ec649615f16dd66a4a6f7bf0b3df6", "sha256": "a72927f394a14d17375b5fa68ab59e9bceeced5f10b741016b3f71588913f7f8" }, "downloads": -1, "filename": "picopore-0.2.0.zip", "has_sig": false, "md5_digest": "cd9ec649615f16dd66a4a6f7bf0b3df6", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17396, "upload_time": "2017-02-12T03:56:26", "url": "https://files.pythonhosted.org/packages/eb/ee/adcd5305663ceb4b4938d5ae78861bca3c4fe9743b011bba40552137935b/picopore-0.2.0.zip" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "bd5c6dff41f10a823c161489225d422e", "sha256": "6e55af5d19bb1b3a25e74f126f9c7d2e1a36b14adc3bc6cfcaec623e17a8c4b2" }, "downloads": -1, "filename": "picopore-0.2.1.zip", "has_sig": false, "md5_digest": "bd5c6dff41f10a823c161489225d422e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 21889, "upload_time": "2017-02-12T11:17:09", "url": "https://files.pythonhosted.org/packages/22/40/479f74aa3f6c9334436d7bc25a79fd287e113617acb174bd577668f37bb5/picopore-0.2.1.zip" } ], "0.2.2": [ { "comment_text": "", "digests": { "md5": "3069121d61c7b7eabcd99497c037d280", "sha256": "c1748bba238d7b7f38c73f4439df1d743feef2ff8c8ebaa9827136de5a3303d9" }, "downloads": -1, "filename": "picopore-0.2.2.zip", "has_sig": false, "md5_digest": "3069121d61c7b7eabcd99497c037d280", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24414, "upload_time": "2017-02-20T23:45:31", "url": "https://files.pythonhosted.org/packages/8a/e0/6e5493522cefe8c314adb33c9318691658c5e87cd0b44243b96bf0381d6d/picopore-0.2.2.zip" } ], "1.0.0": [ { "comment_text": "", "digests": { "md5": "1364d2306b7627ddc846335fd9048526", "sha256": "16efb0adfa369f53422dd53538f7b8b2914db3a3c3912a94a99066c783249795" }, "downloads": -1, "filename": "picopore-1.0.0.zip", "has_sig": false, "md5_digest": "1364d2306b7627ddc846335fd9048526", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24418, "upload_time": "2017-02-24T05:44:05", "url": "https://files.pythonhosted.org/packages/15/3e/665087074c9c68550ea7e9091048e29360f5be007958cdc68d71a0bb2d02/picopore-1.0.0.zip" } ], "1.1.1": [ { "comment_text": "", "digests": { "md5": "f8724715da0fce857046965caf165c21", "sha256": "d0a74a097b97ed6231aba75dbe6f840c4aa536271954a8e1848b9c4a78e4d6d2" }, "downloads": -1, "filename": "picopore-1.1.1.zip", "has_sig": false, "md5_digest": "f8724715da0fce857046965caf165c21", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24880, "upload_time": "2017-03-04T10:49:58", "url": "https://files.pythonhosted.org/packages/2a/40/d3adc24813ff490104e0a4f5db8b4ca92d20c37c79fd1c3b4967a46fa45b/picopore-1.1.1.zip" } ], "1.1.1a": [ { "comment_text": "", "digests": { "md5": "e0d5ef4242b88cc4650159147d70af75", "sha256": "eed43027050fbcfac9581ef1b334e39b89be95dbcfd8c331e0428a6720e84f07" }, "downloads": -1, "filename": "picopore-1.1.1a.tar.gz", "has_sig": false, "md5_digest": "e0d5ef4242b88cc4650159147d70af75", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13021, "upload_time": "2017-03-22T13:43:24", "url": "https://files.pythonhosted.org/packages/e2/db/5fba0165e7765c2b08165d3bacbbf009ab28cbd75f10282ca933f363d07a/picopore-1.1.1a.tar.gz" } ], "1.1.2": [ { "comment_text": "", "digests": { "md5": "bd16455fd1065bb6dbb200ae5994982b", "sha256": "94fb078779c9104aeeafbb8aa8b05d4e7fbe445d7f7d084d769443af25c20a9e" }, "downloads": -1, "filename": "picopore-1.1.2.tar.gz", "has_sig": false, "md5_digest": "bd16455fd1065bb6dbb200ae5994982b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13574, "upload_time": "2017-03-28T04:59:53", "url": "https://files.pythonhosted.org/packages/0c/a3/06b6630153eb8d9a868c9377a7c489ace77f75925f461cf8ff92991138f0/picopore-1.1.2.tar.gz" } ], "1.1.3": [ { "comment_text": "", "digests": { "md5": "610d1218e8e0c25ff390ab71ebf02552", "sha256": "8fc3c6fc846ee93c59eb5967a77450cfb8fa433d4abde657b76dbe54dacbed4d" }, "downloads": -1, "filename": "picopore-1.1.3.zip", "has_sig": false, "md5_digest": "610d1218e8e0c25ff390ab71ebf02552", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 27655, "upload_time": "2017-05-01T10:37:16", "url": "https://files.pythonhosted.org/packages/b6/98/d2b71d87ccfd8f33a13dcd8c7c94f510aaaf0a565d40978dfb4baa99beb0/picopore-1.1.3.zip" } ], "1.1.4": [ { "comment_text": "", "digests": { "md5": "6703677292ff144304ae29af60a81961", "sha256": "be5414a55359bcc4d5a480b37d7b968cacd9621b3067c833a1ec2ea049ae9659" }, "downloads": -1, "filename": "picopore-1.1.4.zip", "has_sig": false, "md5_digest": "6703677292ff144304ae29af60a81961", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 27828, "upload_time": "2017-05-01T11:09:50", "url": "https://files.pythonhosted.org/packages/bb/7e/87cc180009254d68bd70ce3467d1cf13ddae2f1dcdf4edd49133108bce70/picopore-1.1.4.zip" } ], "1.1.5": [ { "comment_text": "", "digests": { "md5": "651e3f405b138bb702513f0d4c9d309f", "sha256": "f487e276d076107dbacab21c34d202dfd8ba85706e85599e3e187f067d24fe58" }, "downloads": -1, "filename": "picopore-1.1.5.zip", "has_sig": false, "md5_digest": "651e3f405b138bb702513f0d4c9d309f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 29128, "upload_time": "2017-06-25T09:17:54", "url": "https://files.pythonhosted.org/packages/a9/e3/3fd0cb7ea34331d6a6caaea5e3a0883596e484fab3e442a07a1d68012f7d/picopore-1.1.5.zip" } ], "1.2.0": [ { "comment_text": "", "digests": { "md5": "5022bda29808de873c676b47727d1d6d", "sha256": "38a3e4239f0467d598bd62e5f0682baa724abe091628bcf5f68ef108127d1560" }, "downloads": -1, "filename": "picopore-1.2.0.zip", "has_sig": false, "md5_digest": "5022bda29808de873c676b47727d1d6d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 31988, "upload_time": "2017-08-31T04:56:28", "url": "https://files.pythonhosted.org/packages/98/e6/2431eebb5370c84e4071475206be805f7059d83ead8c53ba509ecf6e25d6/picopore-1.2.0.zip" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "5022bda29808de873c676b47727d1d6d", "sha256": "38a3e4239f0467d598bd62e5f0682baa724abe091628bcf5f68ef108127d1560" }, "downloads": -1, "filename": "picopore-1.2.0.zip", "has_sig": false, "md5_digest": "5022bda29808de873c676b47727d1d6d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 31988, "upload_time": "2017-08-31T04:56:28", "url": "https://files.pythonhosted.org/packages/98/e6/2431eebb5370c84e4071475206be805f7059d83ead8c53ba509ecf6e25d6/picopore-1.2.0.zip" } ] }