{
    "info": {
        "author": "gneiss development team",
        "author_email": "jamietmorton@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "License :: OSI Approved :: BSD License",
            "Operating System :: MacOS :: MacOS X",
            "Operating System :: POSIX",
            "Operating System :: Unix",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3 :: Only",
            "Topic :: Scientific/Engineering",
            "Topic :: Scientific/Engineering :: Bio-Informatics",
            "Topic :: Software Development :: Libraries"
        ],
        "description": "[![Build Status](https://travis-ci.org/biocore/rhapsody.svg?branch=master)](https://travis-ci.org/biocore/rhapsody)\n\n# rhapsody\nNeural networks for estimating microbe-metabolite interactions through their co-occurence probabilities.\n\n# Installation\n\nRhapsody can be installed via pypi as follows\n\n```\npip install rhapsody\n```\n\nIf you are planning on using GPUs, be sure to `pip install tensorflow-gpu`.\n\nRhapsody can also be installed via conda as follows\n\n```\nconda install rhapsody -c conda-forge\n```\n\nNote that this option may not work in cluster environments, it maybe workwhile to pip install within a virtual environment.  It is possible to pip install rhapsody within a conda environment, including qiime2 conda environments.  However, pip and conda are known to have compatibility issues, so proceed with caution.\n\n# Getting started\n\nTo get started you can run a quick example as follows.  This will learn microbe-metabolite vectors (mmvec)\nwhich can be used to estimate microbe-metabolite conditional probabilities that are accurate up to rank.\n\n```\nrhapsody mmvec \\\n\t--otu-file data/otus.biom \\\n\t--metabolite-file data/ms.biom \\\n\t--summary-dir summary\n```\n\nWhile this is running, you can open up another session and run `tensorboard --logdir .` for diagnosis, see FAQs below for more details.\n\nIf you investigate the summary folder, you will notice that there are a number of files deposited.\n\nSee the following url for a more complete tutorial with real datasets.\n\nhttps://github.com/knightlab-analyses/multiomic-cooccurences\n\nMore information can found under `rhapsody --help`\n\n# Qiime2 plugin\n\nIf you want to make this qiime2 compatible, install this in your\nqiime2 conda environment (see qiime2 installation instructions [here](https://qiime2.org/)) and run the following\n\n```\npip install git+https://github.com/biocore/rhapsody.git\nqiime dev refresh-cache\n```\n\nThis should allow your q2 environment to recognize rhapsody. Before we test\nthe qiime2 plugin, run the following commands to import an example dataset\n\n```\nqiime tools import \\\n\t--input-path data/otus_nt.biom \\\n\t--output-path otus_nt.qza \\\n\t--type FeatureTable[Frequency]\n\nqiime tools import \\\n\t--input-path data/lcms_nt.biom \\\n\t--output-path lcms_nt.qza \\\n\t--type FeatureTable[Frequency]\n```\n\nThen you can run mmvec\n```\nqiime rhapsody mmvec \\\n\t--i-microbes otus_nt.qza \\\n\t--i-metabolites lcms_nt.qza \\\n\t--o-conditionals ranks.qza \\\n\t--o-conditional-biplot biplot.qza\n```\nIn the results, there are two files, namely `results/conditional_biplot.qza` and `results/conditionals.qza`. The conditional biplot is a biplot representation the\nconditional probability matrix so that you can visualize these microbe-metabolite interactions in an exploratory manner.  This can be directly visualized in\nEmperor as shown below.  We also have the estimated conditional probability matrix given in `results/conditionals.qza`,\nwhich an be unzip to yield a tab-delimited table via `unzip results/conditionals`. Each row can be ranked,\nso the top most occurring metabolites for a given microbe can be obtained by identifying the highest co-occurrence probabilities for each microbe.\n\nIt is worth your time to investigate the logs (labeled under `logdir**`) that are deposited using Tensorboard.\nThe actual logfiles within this directory are labeled `events.out.tfevents.*` : more discussion on this later.\n\n\nTensorboard can be run via\n```\ntensorboard --logdir .\n```\n\nYou may need to tinker with the parameters to get readable tensorflow results, namely `--p-summary-interval`,\n`--epochs` and `--batch-size`.\n\nA description of these two graphs is outlined in the FAQs below.\n\n\nThen you can run the following to generate a emperor biplot.\n\n```\nqiime emperor biplot \\\n\t--i-biplot conditional_biplot.qza \\\n\t--m-sample-metadata-file data/metabolite-metadata.txt \\\n\t--m-feature-metadata-file data/microbe-metadata.txt \\\n\t--o-visualization emperor.qzv\n\n```\n\nThe resulting biplot should look like something as follows\n\n![biplot](https://github.com/biocore/rhapsody/raw/master/img/biplot.png \"Biplot\")\n\nHere, the metabolite represent points and the arrows represent microbes.  The points close together are indicative of metabolites that\nfrequently co-occur with each other.  Furthermore, arrows that have a small angle between them are indicative of microbes that co-occur with each other.\nArrows that point in the same direction as the metabolites are indicative of microbe-metabolite co-occurrences.  In the biplot above, the red arrows\ncorrespond to Pseudomonas aeruginosa, and the red points correspond to Rhamnolipids that are likely produced by Pseudomonas aeruginosa.\n\nMore information behind the parameters can found under `qiime rhapsody --help`\n\n# FAQs\n\n**Q**: Looks like there are two different commands, a standalone script and a qiime2 interface.  Which one should I use?!?\n\n**A**:  It'll depend on how deep in the weeds you'll want to get.  For most intents and purposes, the qiime2 interface will more practical for most analyses.  There are 3 major reasons why the standalone scripts are more preferable to the qiime2 interface, namely\n\n1. Customized acceleration : If you want to bring down your runtime from a few days to a few hours, you may need to compile Tensorflow to handle hardware specific instructions (i.e. GPUs / SIMD instructions).  It probably is possible to enable GPU compatiability within a conda environment with some effort, but since conda packages binaries, SIMD instructions will not work out of the box.\n\n2. Checkpoints : If you are not sure how long your analysis should run, the standalone script can allow you record checkpoints, which can allow you to recover your model parameters.  This enables you to investigate your model while the model is training.\n\n3. More model parameters : The standalone script will return the bias parameters learned for each dataset (i.e. microbe and metabolite abundances).  These are stored under the summary directory (specified by `--summary`) under the names `embeddings.csv`. This file will hold the coordinates for the microbes and metabolites, along with biases.  There are 4 columns in this file, namely `feature_id`, `axis`, `embed_type` and `values`.  `feature_id` is the name of the feature, whether it be a microbe name or a metabolite feature id.  `axis` corresponds to the name of the axis, which either corresponds to a PC axis or bias.  `embed_type` denotes if the coordinate corresponds to a microbe or metabolite.  `values` is the coordinate value for the given `axis`, `embed_type` and `feature_id`.  This can be useful for accessing the raw parameters and building custom biplots / ranks visualizations - this also has the advantage of requiring much less memory to manipulate.\n\n**Q** : You mentioned that you can use GPUs.  How can you do that??\n\n**A** : This can be done by running `pip install tensorflow-gpu` in your environment.  See details [here](https://www.tensorflow.org/install/gpu).\n\nAt the moment, these capabilities are only available for the standalone CLI due to complications of installation.  See the `--arm-the-gpu` option in the standalone interface.\n\n**Q** : Neural networks scare me - don't they overfit the crap out of your data?\n\n**A** : Here, we are using shallow neural networks (so only two layers).  This falls under the same regime as PCA and SVD.  But just as you can overfit PCA/SVD, you can also overfit mmvec.  Which is why we have Tensorboard enabled for diagnostics. You can visualize the `cv_rmse` to gauge if there is overfitting -- if your run is strictly decreasing, then that is a sign that you are probably not overfitting.  But this is not necessarily indicative that you have reach the optimal -- you want to check to see if `logloss` has reached a plateau as shown above.\n\n**Q** : I'm confused, what is Tensorboard?\n\n**A** : Tensorboard is a diagnostic tool that runs in a web browser. To open tensorboard, make sure you\u2019re in the rhapsody environment and cd into the folder you are running the script above from. Then run:\n\n```\ntensorboard --logdir .\n```\n\nReturning line will look something like:\n\n```\nTensorBoard 1.9.0 at http://Lisas-MacBook-Pro-2.local:6006 (Press CTRL+C to quit)\n```\nOpen the website (highlighted in red) in a browser. (Hint; if that doesn\u2019t work try putting only the port number (here it is 6006), adding localhost, localhost:6006). Leave this tab alone. Now any rhapsody output directories that you add to the folder that tensorflow is running in will be added to the webpage.\n\n\nIf working properly, it will look something like this\n![tensorboard](https://github.com/biocore/rhapsody/raw/master/img/tensorboard.png \"Tensorboard\")\n\nFIRST graph in Tensorflow; 'Prediction accuracy'. Labelled `cv_rmse`\n\nThis is a graph of the prediction accuracy of the model; the model will try to guess the metabolite intensitiy values for the testing samples that were set aside in the script above, using only the microbe counts in the testing samples. Then it looks at the real values and sees how close it was.\n\nThe second graph is the `likelihood` - if your `likelihood` values are plateaued, that is a sign that you have converged and reached at a local minima.\n\nThe x-axis is the number of iterations (meaning times the model is training across the entire dataset). Every time you iterate across the training samples, you also run the test samples and the averaged results are being plotted on the y-axis.\n\n\nThe y-axis is the average number of counts off for each feature. The model is predicting the sequence counts for each feature in the samples that were set aside for testing. So in the graph above it means that, on average, the model is off by ~0.75 intensity units, which is low. However, this is ABSOLUTE error not relative error (unfortunately we don't know how to compute relative errors because of the sparsity in these datasets).\n\nYou can also compare multiple runs with different parameters to see which run performed the best.  If you are doing this, be sure to look at the `training-column` example make the testing samples consistent across runs.\n\n**Q** : What's up with the `--training-column` argument?\n\n**A** : That is used for cross-validation if you have a specific reproducibility question that you are interested in answering. It can also make it easier to compare cross validation results across runs. If this is specified, only samples labeled \"Train\" under this column will be used for building the model and samples labeled \"Test\" will be used for cross validation. In other words the model will attempt to predict the microbe abundances for the \"Test\" samples. The resulting prediction accuracy is used to evaluate the generalizability of the model in order to determine if the model is overfitting or not. If this argument is not specified, then 10 random samples will be chosen for the test dataset. If you want to specify more random samples to allocate for cross-validation, the `num-random-test-examples` argument can be specified.\n\n\n**Q** : What sort of parameters should I focus on when picking a good model?\n\n**A** : There are 3 different parameters to focus on, `input-prior`, `output-prior` and `latent-dim`\n\nThe `--input-prior`  and `--output-prior` options specifies the width of the prior distribution of the coefficients, where the `--input-prior` is typically specific to microbes and the `--output-prior` is specific to metabolites.\nFor a prior of 1, this means 99% of entries in the embeddings (typically given in the `U.txt` and `V.txt` files are within -3 and +3 (log fold change). The higher differential-prior is, the more parameters can have bigger changes, so you want to keep this relatively small. If you see overfitting (accuracy and fit increasing over iterations in tensorboard) you may consider reducing the `--input-prior` and `--output-prior` in order to reduce the parameter space.\n\nAnother parameter worth thinking about is `--latent-dim`, which controls the number of dimensions used to approximate the conditional probability matrix.  This also specifies the dimensions of the microbe/metabolite embeddings `U.txt` and `V.txt`.  The more dimensions this has, the more accurate the embeddings can be -- but the higher the chance of overfitting there is.  The rule of thumb to follow is in order to fit these models, you need at least 10 times as many samples as there are latent dimensions (this is following a similar rule of thumb for fitting straight lines).  So if you have 100 samples, you should definitely not have a latent dimension of more than 10.  Furthermore, you can still overfit certain microbes and metabolites.  For example, you are fitting a model with those 100 samples and just 1 latent dimension, you can still easily overfit microbes and metabolites that appear in less than 10 samples -- so even fitting models with just 1 latent dimension will require some microbes and metabolites that appear in less than 10 samples to be filtered out.\n\n\n**Q** : What does a good model fit look like??\n\n**A** : Again the numbers vary greatly by dataset. But you want to see the both the `logloss` and `cv_rmse` curves decaying, and plateau as close to zero as possible.\n\n**Q** : How long should I expect this program to run?\n\n**A** : Both `epochs` and `batch-size` contribute to determining how long the algorithm will run, namely\n\n**Number of iterations = `epoch #` multiplied by the ( Total # of microbial reads / `batch-size` parameter)**\n\nThis also depends on if your program will converge. The `learning-rate` specifies the resolution (smaller step size = smaller resolution, but may take longer to converge). You will need to consult with Tensorboard to make sure that your model fit is sane. See this paper for more details on gradient descent: https://arxiv.org/abs/1609.04747\n\nIf you are running this on a CPU, 16 cores, a run that reaches convergence should take about 1 day.\nIf you have a GPU - you maybe able to get this down to a few hours.  However, some finetuning of the `batch-size` parameter maybe required -- instead of having a small `batch-size` < 100, you'll want to bump up the `batch-size` to between 1000 and 10000 to fully leverage the speedups available on the GPU.\n\nCredits to Lisa Marotz ([@lisa55asil](https://github.com/lisa55asil)),  Yoshiki Vazquez-Baeza ([@ElDeveloper](https://github.com/ElDeveloper)) and Julia Gauglitz ([@jgauglitz](https://github.com/jgauglitz)) for their README contributions.",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "",
        "keywords": "",
        "license": "BSD-3-Clause",
        "maintainer": "gneiss development team",
        "maintainer_email": "jamietmorton@gmail.com",
        "name": "rhapsody",
        "package_url": "https://pypi.org/project/rhapsody/",
        "platform": "",
        "project_url": "https://pypi.org/project/rhapsody/",
        "project_urls": null,
        "release_url": "https://pypi.org/project/rhapsody/0.4.0/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Microbe-metabolite interactions through neural networks",
        "version": "0.4.0"
    },
    "last_serial": 5569765,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9e860a278c5a1ab8611a1b64a2c58333",
                    "sha256": "5f0be550bacd9880a3371e42c3e5761edd5c59b63131a24dccbaec810f451746"
                },
                "downloads": -1,
                "filename": "rhapsody-0.1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "9e860a278c5a1ab8611a1b64a2c58333",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 12279,
                "upload_time": "2019-03-01T19:18:24",
                "url": "https://files.pythonhosted.org/packages/18/4f/3e5e3dcbc947324cc6977abd1e4eff4f3510306fc51bf52d286493987384/rhapsody-0.1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ad0b9d8c014d19f2183e79d467e958c3",
                    "sha256": "913204023e8cf4933b1e2cd1cb03a3dcb76dfb7b3c0c5b38e43057e472f2d1eb"
                },
                "downloads": -1,
                "filename": "rhapsody-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "ad0b9d8c014d19f2183e79d467e958c3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10510,
                "upload_time": "2019-03-01T19:18:26",
                "url": "https://files.pythonhosted.org/packages/3c/2b/d703040e0a26489f9d87b09d1b968249fa38e18165a0b654d618f5f96c1d/rhapsody-0.1.0.tar.gz"
            }
        ],
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e03ffaf2279f3386ccd4e4644fd8508f",
                    "sha256": "982d2d65a28c1c85284ad6635ad0036211a120a25a728d417167ad665c11886a"
                },
                "downloads": -1,
                "filename": "rhapsody-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "e03ffaf2279f3386ccd4e4644fd8508f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17847,
                "upload_time": "2019-04-22T16:38:33",
                "url": "https://files.pythonhosted.org/packages/a0/aa/87489110b92952b10afe67351da41466d7b8791c3f359d950650f4236fd5/rhapsody-0.2.0.tar.gz"
            }
        ],
        "0.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bc7a92190bbed43f0dfab7549628a1c4",
                    "sha256": "d73e0af9f57fdb7e48bd7bab4f946fbeb0240f6d44cb47e0257a277df7051beb"
                },
                "downloads": -1,
                "filename": "rhapsody-0.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "bc7a92190bbed43f0dfab7549628a1c4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17024,
                "upload_time": "2019-04-22T18:25:32",
                "url": "https://files.pythonhosted.org/packages/8b/e9/b63934b1988f52e7b568456823ea3de9c40bde0270a90e2af27e237aaa61/rhapsody-0.2.1.tar.gz"
            }
        ],
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "041a08c22ba557f0b882449f15323357",
                    "sha256": "225fcaabcde9cdba5e08f7484f24d6c8a76789808a33b7eacb1e96146a4f2887"
                },
                "downloads": -1,
                "filename": "rhapsody-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "041a08c22ba557f0b882449f15323357",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 24595,
                "upload_time": "2019-06-21T03:49:04",
                "url": "https://files.pythonhosted.org/packages/66/29/08a4745dbd7714b8ee8706a85c2d48b58c604a869cb2894e6103d7a62347/rhapsody-0.3.0.tar.gz"
            }
        ],
        "0.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4f3a0ff44823627ba9f4acdefc673302",
                    "sha256": "159cad70b5c34e9b36f24f8975933de06e541240151bd977be8c79ee609045b8"
                },
                "downloads": -1,
                "filename": "rhapsody-0.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "4f3a0ff44823627ba9f4acdefc673302",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 18570,
                "upload_time": "2019-07-23T00:20:59",
                "url": "https://files.pythonhosted.org/packages/21/5b/4dc0b411d571f5bb14c17d267de49631e6a2aaba63e3a9fbebddd094b0c8/rhapsody-0.4.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "4f3a0ff44823627ba9f4acdefc673302",
                "sha256": "159cad70b5c34e9b36f24f8975933de06e541240151bd977be8c79ee609045b8"
            },
            "downloads": -1,
            "filename": "rhapsody-0.4.0.tar.gz",
            "has_sig": false,
            "md5_digest": "4f3a0ff44823627ba9f4acdefc673302",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 18570,
            "upload_time": "2019-07-23T00:20:59",
            "url": "https://files.pythonhosted.org/packages/21/5b/4dc0b411d571f5bb14c17d267de49631e6a2aaba63e3a9fbebddd094b0c8/rhapsody-0.4.0.tar.gz"
        }
    ]
}