{
    "info": {
        "author": "MIT Data To AI Lab",
        "author_email": "dailabmit@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 2 - Pre-Alpha",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Natural Language :: English",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7"
        ],
        "description": "\n<p align=\"left\">\n<img width=15% src=\"https://dai.lids.mit.edu/wp-content/uploads/2018/06/Logo_DAI_highres.png\" alt=\u201cDAI-Lab\u201d />\n<i>An open source project from Data to AI Lab at MIT.</i>\n</p>\n\n\n[![PyPI Shield](https://img.shields.io/pypi/v/piex.svg)](https://pypi.python.org/pypi/piex)\n[![Travis CI Shield](https://travis-ci.org/HDI-Project/piex.svg?branch=master)](https://travis-ci.org/HDI-Project/piex)\n\n# Pipeline Explorer\n\nClasses and functions to explore and reproduce the performance obtained by\nthousands of MLBlocks pipelines and templates across hundreds of datasets.\n\n- Free software: MIT license\n- Documentation: https://HDI-Project.github.io/piex\n- Homepage: https://github.com/HDI-Project/piex\n\n\n# Overview\n\nThis repository contains a collection of classes and functions which allows a user to easily\nexplore the results of a series of experiments run by team MIT using MLBlocks pipelines over\na large collection of Datasets.\n\nAlong with this library we are releasing a number of fitted pipelines, their performance on\ncross validation, test data and metrics. The results of these experiments were stored in a\nDatabase and later on uploaded to Amazon S3, from where they can be downloaded and analyzed\nusing the Pipeline Explorer.\n\nWe will continuously add more pipelines, templates and datasets to our experiments and make\nthem publicly available to the community.\n\nThese can be used for the following purposes:\n\n* Find what is the best score we found so far for a given dataset and task type (given the\n  search space we defined and our tuners)\n* Use information about pipeline performance to do meta learning\n\nCurrent summary of our experiments is:\n\n<div>\n<table class=\"dataframe\">\n  <thead>\n    <tr>\n      <th># of</th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>datasets</th>\n      <td>453</td>\n    </tr>\n    <tr>\n      <th>pipelines</th>\n      <td>2115907</td>\n    </tr>\n    <tr>\n      <th>templates</th>\n      <td>63</td>\n    </tr>\n    <tr>\n      <th>tests</th>\n      <td>2152</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n## Concepts\n\nBefore diving into the software usage, we briefly explain some concepts and terminology.\n\n### Primitives\n\nWe call the smallest computational blocks used in a Machine Learning process\n**primitives**, which:\n\n* Can be either classes or functions.\n* Have some initialization arguments, which MLBlocks calls `init_params`.\n* Have some tunable hyperparameters, which have types and a list or range of valid values.\n\n### Templates\n\nPrimitives can be combined to form what we call **Templates**, which:\n\n* Have a list of primitives.\n* Have some initialization arguments, which correspond to the initialization arguments\n  of their primitives.\n* Have some tunable hyperparameters, which correspond to the tunable hyperparameters\n  of their primitives.\n\n### Pipelines\n\nTemplates can be used to build **Pipelines** by taking and fixing a set of valid\nhyperparameters for a Template. Hence, Pipelines:\n\n* Have a list of primitives, which corresponds to the list of primitives of their template.\n* Have some initialization arguments, which correspond to the initialization arguments\n  of their template.\n* Have some hyperparameter values, which fall within the ranges of valid tunable\n  hyperparameters of their template.\n\nA pipeline can be fitted and evaluated using the MLPipeline API in MLBlocks.\n\n### Datasets\n\nA collection of ~450 datasets was used covering 6 different data modalities and 17 task types.\n\nEach dataset was split using a holdout method in two parts, training and testing, which were\nused respectively to find and fit the optimal pipeline for each dataset, and to later on\nevaluate the goodness-of-fit of each pipeline against a specific metric for each dataset.\n\nThis collection of datasets is stored in an Amazon S3 Bucket in the [D3M format](https://github.com/mitll/d3m-schema),\nincluding the training and testing partitioning, and can be downloaded both using piex or a web browser following this link: https://d3m-data-dai.s3.amazonaws.com/index.html\n\n### What is an experiment/test?\n\nThroughout our description we will refer to a search process as an **experiment** or a **test**.\nAn experiment/test is defined as follows:\n\n* It is given a dataset and a task\n* It is given a template\n* It then searches using a Bayesian tuning algorithm (using a tuner from our BTB library). Tuning\n  algorithm tests multiple pipelines derived from the template and tries to find the best set of\n  hyperparameters possible for that template on each dataset.\n* During the search process, a collection of information is stored in the database and is\n  available through piex. They are:\n    * Cross Validation score obtained over the training partition by each pipeline fitted during\n      the search process.\n    * In parallel, at some points in time the best pipeline already found was validated against\n      the testing data, and the   obtained score was also stored in the database.\n\nEach experiment was given one or more of the following configuration values:\n\n* Timeout: Maximum time that the search process is allowed to run.\n* Budget: Maximum number of tuning iterations that the search process is allowed to perform.\n* Checkpoints: List of points in time, in seconds, where the best pipeline so far was\n  scored against the testing data.\n* Pipeline: The name of the template to use to build the pipelines.\n* Tuner Type: The type of tuner to use, `gp` or `uniform`.\n\n\n# Getting Started\n\n## Installation\n\nThe simplest and recommended way to install the Pipeline Explorer is using pip:\n\n```bash\npip install piex\n```\n\nAlternatively, you can also clone the repository and install it from sources\n\n```bash\ngit clone git@github.com:HDI-Project/piex.git\ncd piex\npip install -e .\n```\n\n# Usage\n\n## The S3PipelineExplorer\n\nThe **S3PipelineExplorer** class provides methods to download the results from previous\ntests executions from S3, see which pipelines obtained the best scores and load them\nas a dictionary, ready to be used by an MLPipeline.\n\nTo start working with it, it needs to be given the name of the S3 Bucket from which\nthe data will be downloaded.\n\nFor this examples, we will be using the `ml-pipelines-2018` bucket, where the results\nof the experiments run for the Machine Learning Bazaar paper can be found.\n\n\n```python\nfrom piex.explorer import S3PipelineExplorer\n\npiex = S3PipelineExplorer('ml-pipelines-2018')\n```\n\n### The Datasets\n\nThe `get_datasets` method returns a `pandas.DataFrame` with information about the\navailable datasets, their data modalities, task types and task subtypes.\n\n\n```python\ndatasets = piex.get_datasets()\ndatasets.shape\n```\n\n\n\n\n    (453, 4)\n\n\n\n\n```python\ndatasets.head()\n```\n\n\n\n\n<div>\n<table class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>dataset</th>\n      <th>data_modality</th>\n      <th>task_type</th>\n      <th>task_subtype</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>314</th>\n      <td>124_120_mnist</td>\n      <td>image</td>\n      <td>classification</td>\n      <td>multi_class</td>\n    </tr>\n    <tr>\n      <th>315</th>\n      <td>124_138_cifar100</td>\n      <td>image</td>\n      <td>classification</td>\n      <td>multi_class</td>\n    </tr>\n    <tr>\n      <th>316</th>\n      <td>124_153_svhn_cropped</td>\n      <td>image</td>\n      <td>classification</td>\n      <td>multi_class</td>\n    </tr>\n    <tr>\n      <th>317</th>\n      <td>124_174_cifar10</td>\n      <td>image</td>\n      <td>classification</td>\n      <td>multi_class</td>\n    </tr>\n    <tr>\n      <th>318</th>\n      <td>124_178_coil100</td>\n      <td>image</td>\n      <td>classification</td>\n      <td>multi_class</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\ndatasets = piex.get_datasets(data_modality='multi_table', task_type='regression')\ndatasets.head()\n```\n\n\n\n\n<div>\n<table class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>dataset</th>\n      <th>data_modality</th>\n      <th>task_type</th>\n      <th>task_subtype</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>311</th>\n      <td>uu2_gp_hyperparameter_estimation</td>\n      <td>multi_table</td>\n      <td>regression</td>\n      <td>multivariate</td>\n    </tr>\n    <tr>\n      <th>312</th>\n      <td>uu3_world_development_indicators</td>\n      <td>multi_table</td>\n      <td>regression</td>\n      <td>univariate</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n### The Experiments\n\nThe list of tests that have been executed can be obtained with the method `get_tests`.\n\nThis method returns a `pandas.DataFrame` that contains a row for each experiment that has been run on each dataset.\nThis dataset includes information about the dataset, the configuration used for the experiment, such as the\ntemplate, the checkpoints or the budget, and information about the execution, such as the timestamp, the exact\nsoftware version, the host that executed the test and whether there was an error or not.\n\nJust like the `get_datasets`, any keyword arguments will be used to filter the results.\n\n\n```python\nimport pandas as pd\n\ntests = piex.get_tests()\ntests.head().T\n```\n\n\n\n\n<div>\n<table class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>0</th>\n      <th>1</th>\n      <th>2</th>\n      <th>3</th>\n      <th>4</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>budget</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>checkpoints</th>\n      <td>[900, 1800, 3600, 7200]</td>\n      <td>[900, 1800, 3600, 7200]</td>\n      <td>[900, 1800, 3600, 7200]</td>\n      <td>[900, 1800, 3600, 7200]</td>\n      <td>[900, 1800, 3600, 7200]</td>\n    </tr>\n    <tr>\n      <th>commit</th>\n      <td>4c7c29f</td>\n      <td>4c7c29f</td>\n      <td>4c7c29f</td>\n      <td>4c7c29f</td>\n      <td>4c7c29f</td>\n    </tr>\n    <tr>\n      <th>dataset</th>\n      <td>196_autoMpg</td>\n      <td>26_radon_seed</td>\n      <td>LL0_1027_esl</td>\n      <td>LL0_1028_swd</td>\n      <td>LL0_1030_era</td>\n    </tr>\n    <tr>\n      <th>docker</th>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>error</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>hostname</th>\n      <td>ec2-52-14-97-167.us-east-2.compute.amazonaws.com</td>\n      <td>ec2-18-223-109-53.us-east-2.compute.amazonaws.com</td>\n      <td>ec2-18-217-79-23.us-east-2.compute.amazonaws.com</td>\n      <td>ec2-18-217-239-54.us-east-2.compute.amazonaws.com</td>\n      <td>ec2-18-225-32-252.us-east-2.compute.amazonaws.com</td>\n    </tr>\n    <tr>\n      <th>image</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>insert_ts</th>\n      <td>2018-10-24 20:05:01.872</td>\n      <td>2018-10-24 20:05:02.778</td>\n      <td>2018-10-24 20:05:02.879</td>\n      <td>2018-10-24 20:05:02.980</td>\n      <td>2018-10-24 20:05:03.081</td>\n    </tr>\n    <tr>\n      <th>pipeline</th>\n      <td>categorical_encoder/imputer/standard_scaler/xg...</td>\n      <td>categorical_encoder/imputer/standard_scaler/xg...</td>\n      <td>categorical_encoder/imputer/standard_scaler/xg...</td>\n      <td>categorical_encoder/imputer/standard_scaler/xg...</td>\n      <td>categorical_encoder/imputer/standard_scaler/xg...</td>\n    </tr>\n    <tr>\n      <th>status</th>\n      <td>done</td>\n      <td>done</td>\n      <td>done</td>\n      <td>done</td>\n      <td>done</td>\n    </tr>\n    <tr>\n      <th>test_id</th>\n      <td>20181024200501872083</td>\n      <td>20181024200501872083</td>\n      <td>20181024200501872083</td>\n      <td>20181024200501872083</td>\n      <td>20181024200501872083</td>\n    </tr>\n    <tr>\n      <th>timeout</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>traceback</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>tuner_type</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>update_ts</th>\n      <td>2018-10-24 22:05:55.386</td>\n      <td>2018-10-24 22:05:57.508</td>\n      <td>2018-10-24 22:05:56.337</td>\n      <td>2018-10-24 22:05:56.112</td>\n      <td>2018-10-24 22:05:56.164</td>\n    </tr>\n    <tr>\n      <th>data_modality</th>\n      <td>single_table</td>\n      <td>single_table</td>\n      <td>single_table</td>\n      <td>single_table</td>\n      <td>single_table</td>\n    </tr>\n    <tr>\n      <th>task_type</th>\n      <td>regression</td>\n      <td>regression</td>\n      <td>regression</td>\n      <td>regression</td>\n      <td>regression</td>\n    </tr>\n    <tr>\n      <th>task_subtype</th>\n      <td>univariate</td>\n      <td>univariate</td>\n      <td>univariate</td>\n      <td>univariate</td>\n      <td>univariate</td>\n    </tr>\n    <tr>\n      <th>metric</th>\n      <td>meanSquaredError</td>\n      <td>rootMeanSquaredError</td>\n      <td>meanSquaredError</td>\n      <td>meanSquaredError</td>\n      <td>meanSquaredError</td>\n    </tr>\n    <tr>\n      <th>dataset_id</th>\n      <td>196_autoMpg_dataset_TRAIN</td>\n      <td>26_radon_seed_dataset_TRAIN</td>\n      <td>LL0_1027_esl_dataset_TRAIN</td>\n      <td>LL0_1028_swd_dataset_TRAIN</td>\n      <td>LL0_1030_era_dataset_TRAIN</td>\n    </tr>\n    <tr>\n      <th>problem_id</th>\n      <td>196_autoMpg_problem_TRAIN</td>\n      <td>26_radon_seed_problem_TRAIN</td>\n      <td>LL0_1027_esl_problem_TRAIN</td>\n      <td>LL0_1028_swd_problem_TRAIN</td>\n      <td>LL0_1030_era_problem_TRAIN</td>\n    </tr>\n    <tr>\n      <th>target</th>\n      <td>class</td>\n      <td>log_radon</td>\n      <td>out1</td>\n      <td>Out1</td>\n      <td>out1</td>\n    </tr>\n    <tr>\n      <th>size</th>\n      <td>24</td>\n      <td>160</td>\n      <td>16</td>\n      <td>52</td>\n      <td>32</td>\n    </tr>\n    <tr>\n      <th>size_human</th>\n      <td>24K</td>\n      <td>160K</td>\n      <td>16K</td>\n      <td>52K</td>\n      <td>32K</td>\n    </tr>\n    <tr>\n      <th>test_features</th>\n      <td>7</td>\n      <td>28</td>\n      <td>4</td>\n      <td>10</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>test_samples</th>\n      <td>100</td>\n      <td>183</td>\n      <td>100</td>\n      <td>199</td>\n      <td>199</td>\n    </tr>\n    <tr>\n      <th>train_features</th>\n      <td>7</td>\n      <td>28</td>\n      <td>4</td>\n      <td>10</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>train_samples</th>\n      <td>298</td>\n      <td>736</td>\n      <td>388</td>\n      <td>801</td>\n      <td>801</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\npd.DataFrame(tests.groupby(['data_modality', 'task_type']).size(), columns=['count'])\n```\n\n\n\n\n<div>\n<table class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th></th>\n      <th>count</th>\n    </tr>\n    <tr>\n      <th>data_modality</th>\n      <th>task_type</th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th rowspan=\"4\" valign=\"top\">graph</th>\n      <th>community_detection</th>\n      <td>5</td>\n    </tr>\n    <tr>\n      <th>graph_matching</th>\n      <td>18</td>\n    </tr>\n    <tr>\n      <th>link_prediction</th>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>vertex_nomination</th>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th rowspan=\"2\" valign=\"top\">image</th>\n      <th>classification</th>\n      <td>57</td>\n    </tr>\n    <tr>\n      <th>regression</th>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th rowspan=\"2\" valign=\"top\">multi_table</th>\n      <th>classification</th>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>regression</th>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th rowspan=\"4\" valign=\"top\">single_table</th>\n      <th>classification</th>\n      <td>1405</td>\n    </tr>\n    <tr>\n      <th>collaborative_filtering</th>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>regression</th>\n      <td>430</td>\n    </tr>\n    <tr>\n      <th>time_series_forecasting</th>\n      <td>175</td>\n    </tr>\n    <tr>\n      <th>text</th>\n      <th>classification</th>\n      <td>17</td>\n    </tr>\n    <tr>\n      <th>timeseries</th>\n      <th>classification</th>\n      <td>37</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\ntests = piex.get_tests(data_modality='graph', task_type='link_prediction')\ntests[['dataset', 'pipeline', 'checkpoints', 'test_id']]\n```\n\n\n\n\n<div>\n<table class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>dataset</th>\n      <th>pipeline</th>\n      <th>checkpoints</th>\n      <th>test_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1716</th>\n      <td>59_umls</td>\n      <td>NaN</td>\n      <td>[900, 1800, 3600, 7200]</td>\n      <td>20181031040541366347</td>\n    </tr>\n    <tr>\n      <th>2141</th>\n      <td>59_umls</td>\n      <td>graph/link_prediction/random_forest_classifier</td>\n      <td>[900, 1800, 3600, 7200]</td>\n      <td>20181031182305995728</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n### The Experiment Results\n\nThe results of the experiments can be seen using the `get_experiment_results` method.\n\nThese results include both the cross validation score obtained by the pipeline during\nthe tuning, as well as the score obtained by this pipeline once it has been fitted\nusing the training data and then used to make predictions over the test data.\n\nJust like the `get_datasets`, any keyword arguments will be used to filter the results,\nincluding the `test_id`.\n\n\n```python\nresults = piex.get_test_results(test_id='20181031182305995728')\nresults[['test_id', 'pipeline', 'score', 'cv_score', 'elapsed', 'iterations']]\n```\n\n\n\n\n<div>\n<table class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>test_id</th>\n      <th>pipeline</th>\n      <th>score</th>\n      <th>cv_score</th>\n      <th>elapsed</th>\n      <th>iterations</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>7464</th>\n      <td>20181031182305995728</td>\n      <td>graph/link_prediction/random_forest_classifier</td>\n      <td>0.499853</td>\n      <td>0.843175</td>\n      <td>900.255511</td>\n      <td>435.0</td>\n    </tr>\n    <tr>\n      <th>7465</th>\n      <td>20181031182305995728</td>\n      <td>graph/link_prediction/random_forest_classifier</td>\n      <td>0.499853</td>\n      <td>0.854603</td>\n      <td>1800.885417</td>\n      <td>805.0</td>\n    </tr>\n    <tr>\n      <th>7466</th>\n      <td>20181031182305995728</td>\n      <td>graph/link_prediction/random_forest_classifier</td>\n      <td>0.499853</td>\n      <td>0.854603</td>\n      <td>3600.005072</td>\n      <td>1432.0</td>\n    </tr>\n    <tr>\n      <th>7467</th>\n      <td>20181031182305995728</td>\n      <td>graph/link_prediction/random_forest_classifier</td>\n      <td>0.785568</td>\n      <td>0.860000</td>\n      <td>7200.225256</td>\n      <td>2366.0</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n### The Best Pipeline\n\nInformation about the best pipeline for a dataset can be obtained using the `get_best_pipeline` method.\n\nThis method returns a `pandas.Series` object with information about the pipeline that obtained the\nbest cross validation score during the tuning, as well as the template that was used to build it.\n\n**Note**: This call will download some data in the background the first time that it is run,\nso it might take a while to return.\n\n\n```python\npiex.get_best_pipeline('185_baseball')\n```\n\n\n\n\n    id                            17385666-31da-4b6e-ab7f-8ac7080a4d55\n    dataset                                 185_baseball_dataset_TRAIN\n    metric                                                     f1Macro\n    name             categorical_encoder/imputer/standard_scaler/xg...\n    rank                                                      0.307887\n    score                                                     0.692113\n    template                                  5bd0ce5249e71569e8bf8003\n    test_id                                       20181024234726559170\n    pipeline         categorical_encoder/imputer/standard_scaler/xg...\n    data_modality                                         single_table\n    task_type                                           classification\n    Name: 1149699, dtype: object\n\n\n\nApart from obtaining this information, we can use the `load_best_pipeline` method\nto load its JSON specification, ready to be using in an `mlblocks.MLPipeline` object.\n\n\n```python\npipeline = piex.load_best_pipeline('185_baseball')\npipeline['primitives']\n```\n\n\n\n\n    ['mlprimitives.feature_extraction.CategoricalEncoder',\n     'sklearn.preprocessing.Imputer',\n     'sklearn.preprocessing.StandardScaler',\n     'mlprimitives.preprocessing.ClassEncoder',\n     'xgboost.XGBClassifier',\n     'mlprimitives.preprocessing.ClassDecoder']\n\n\n\n### The Best Template\n\nJust like the best pipeline, the best template for a given dataset can be obtained using\nthe `get_best_template` method.\n\nThis returns just the name of the template that was used to build the best pipeline.\n\n\n```python\ntemplate_name = piex.get_best_template('185_baseball')\ntemplate_name\n```\n\n\n\n\n    'categorical_encoder/imputer/standard_scaler/xgbclassifier'\n\n\n\nThis can be later on used to explore the template, obtaining its default hyperparameters:\n\n\n```python\ndefaults = piex.get_default_hyperparameters(template_name)\ndefaults\n```\n\n\n\n\n    {'mlprimitives.feature_extraction.CategoricalEncoder#1': {'copy': True,\n      'features': 'auto',\n      'max_labels': 0},\n     'sklearn.preprocessing.Imputer#1': {'missing_values': 'NaN',\n      'axis': 0,\n      'copy': True,\n      'strategy': 'mean'},\n     'sklearn.preprocessing.StandardScaler#1': {'with_mean': True,\n      'with_std': True},\n     'mlprimitives.preprocessing.ClassEncoder#1': {},\n     'xgboost.XGBClassifier#1': {'n_jobs': -1,\n      'n_estimators': 100,\n      'max_depth': 3,\n      'learning_rate': 0.1,\n      'gamma': 0,\n      'min_child_weight': 1},\n     'mlprimitives.preprocessing.ClassDecoder#1': {}}\n\n\n\nOr obtaining the corresponding tunable ranges, ready to be used with a tuner:\n\n\n```python\ntunable = piex.get_tunable_hyperparameters(template_name)\ntunable\n```\n\n\n\n\n    {'mlprimitives.feature_extraction.CategoricalEncoder#1': {'max_labels': {'type': 'int',\n       'default': 0,\n       'range': [0, 100]}},\n     'sklearn.preprocessing.Imputer#1': {'strategy': {'type': 'str',\n       'default': 'mean',\n       'values': ['mean', 'median', 'most_frequent']}},\n     'sklearn.preprocessing.StandardScaler#1': {'with_mean': {'type': 'bool',\n       'default': True},\n      'with_std': {'type': 'bool', 'default': True}},\n     'mlprimitives.preprocessing.ClassEncoder#1': {},\n     'xgboost.XGBClassifier#1': {'n_estimators': {'type': 'int',\n       'default': 100,\n       'range': [10, 1000]},\n      'max_depth': {'type': 'int', 'default': 3, 'range': [3, 10]},\n      'learning_rate': {'type': 'float', 'default': 0.1, 'range': [0, 1]},\n      'gamma': {'type': 'float', 'default': 0, 'range': [0, 1]},\n      'min_child_weight': {'type': 'int', 'default': 1, 'range': [1, 10]}},\n     'mlprimitives.preprocessing.ClassDecoder#1': {}}\n\n\n\n# Scoring Templates and Pipelines\n\nThe **S3PipelineExplorer** class also allows cross validating templates and pipelines\nover any of the datasets.\n\n## Scoring a Pipeline\n\nThe simplest use case is cross validating a pipeline over a dataset.\nFor this, we must pass the ID of the pipeline and the name of the dataset to the method `score_pipeline`.\n\nThe dataset can be the one that was used during the experiments or a different one.\n\n\n```python\npiex.score_pipeline(pipeline['id'], '185_baseball')\n```\n\n\n\n\n    (0.6921128080904511, 0.09950216269594728)\n\n\n\n\n```python\npiex.score_pipeline(pipeline['id'], 'uu4_SPECT')\n```\n\n\n\n\n    (0.8897656842904123, 0.037662864373452655)\n\n\n\nOptionally, the cross validation configuration can be changed\n\n\n```python\npiex.score_pipeline(pipeline['id'], 'uu4_SPECT', n_splits=3, random_state=43)\n```\n\n\n\n\n    (0.8869488536155202, 0.019475563687443638)\n\n\n\n## Scoring a Template\n\nA Template can also be tested over any dataset by passing its name, the dataset and, optionally,\nthe cross validation specification. You have to make sure to choose template that is relevant for\nthe task/data modality for which you want to use it.\n\nIf no hyperparameters are passed, the default ones will be used:\n\n\n```python\npiex.score_template(template_name, 'uu4_SPECT', n_splits=3, random_state=43)\n```\n\n\n\n\n    (0.8555346666968675, 0.028343173498423108)\n\n\n\nYou can get the default hyperparameters, and update the hyperparameters by setting values\nin the dictionary:\n\n**With this anyone can tune the templates that we have for different task/data modality\ntypes using their own AutoML routine. If you choose to do so, let us know the score you\nare getting and the pipeline and we will add to our database.**\n\n\n```python\nhyperparameters = piex.get_default_hyperparameters(template_name)\nhyperparameters['xgboost.XGBClassifier#1']['learning_rate'] = 1\n\npiex.score_template(template_name, 'uu4_SPECT', hyperparameters, n_splits=3, random_state=43)\n```\n\n\n\n\n    (0.8754554700753094, 0.019151608028236813)\n\n\n# History\n\n## 0.2.0\n\n* Pin dependency versions to ensure reproducibility\n* Allow S3PipelineExplorer usage without AWS credentials\n* Fix minor bug in S3PipelineExplorer\n\n## 0.1.1\n\n* Improved documentation\n* Minor bugfixes and improvements to the explorer API\n\n## 0.1.0\n\n* First release on PyPI\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/HDI-Project/piex",
        "keywords": "piex",
        "license": "MIT license",
        "maintainer": "",
        "maintainer_email": "",
        "name": "piex",
        "package_url": "https://pypi.org/project/piex/",
        "platform": "",
        "project_url": "https://pypi.org/project/piex/",
        "project_urls": {
            "Homepage": "https://github.com/HDI-Project/piex"
        },
        "release_url": "https://pypi.org/project/piex/0.2.0/",
        "requires_dist": [
            "boto3 (==1.9.27)",
            "numpy (==1.15.2)",
            "mit-d3m (==0.1.1)",
            "mlblocks (==0.2.3)",
            "mlprimitives (==0.1.3)",
            "pandas (==0.23.4)",
            "pymongo (==3.7.2)",
            "scikit-learn (==0.20.0)",
            "scipy (==1.1.0)",
            "jupyter (==1.0.0) ; extra == 'demo'",
            "jupyter (==1.0.0) ; extra == 'dev'",
            "bumpversion (>=0.5.3) ; extra == 'dev'",
            "pip (>=9.0.1) ; extra == 'dev'",
            "watchdog (>=0.8.3) ; extra == 'dev'",
            "m2r (>=0.2.0) ; extra == 'dev'",
            "Sphinx (>=1.7.1) ; extra == 'dev'",
            "sphinx-rtd-theme (>=0.2.4) ; extra == 'dev'",
            "flake8 (>=3.5.0) ; extra == 'dev'",
            "isort (>=4.3.4) ; extra == 'dev'",
            "autoflake (>=1.1) ; extra == 'dev'",
            "autopep8 (>=1.3.5) ; extra == 'dev'",
            "twine (>=1.10.0) ; extra == 'dev'",
            "wheel (>=0.30.0) ; extra == 'dev'",
            "coverage (>=4.5.1) ; extra == 'dev'",
            "pytest (>=3.4.2) ; extra == 'dev'",
            "tox (>=2.9.1) ; extra == 'dev'",
            "coverage (>=4.5.1) ; extra == 'test'",
            "pytest (>=3.4.2) ; extra == 'test'",
            "tox (>=2.9.1) ; extra == 'test'"
        ],
        "requires_python": ">=3.4",
        "summary": "Pipeline Explorer",
        "version": "0.2.0"
    },
    "last_serial": 5300109,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a25519e88f80a25b1c4d090f36e74ce2",
                    "sha256": "c16f53b400de362ae550a61634e31ae007da210b108edf508cae57ba9d0652e2"
                },
                "downloads": -1,
                "filename": "piex-0.1.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a25519e88f80a25b1c4d090f36e74ce2",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.4",
                "size": 9883,
                "upload_time": "2018-11-16T17:32:56",
                "url": "https://files.pythonhosted.org/packages/a6/a1/3f7592e5d5a4c7ded6c70432359652863d6fddc0ecf1d32e4878d9374c02/piex-0.1.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "f166bb8f85bf5e52dd0c7e64d49d5906",
                    "sha256": "26ae6a8a70886f5402122d682d64f4bff14e1a5e597f819c421b9fe40136f662"
                },
                "downloads": -1,
                "filename": "piex-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "f166bb8f85bf5e52dd0c7e64d49d5906",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.4",
                "size": 21160,
                "upload_time": "2018-11-16T17:32:59",
                "url": "https://files.pythonhosted.org/packages/ab/b8/5ab2080f6fe0bd7b111d1e2d68c5c82f57f309f0a1c6dd0f3bf41d12be68/piex-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bc11518733ea200c88be48802e40f11b",
                    "sha256": "098b2a1d048d0d7b04cf513f305db77c52d251fd907f4be2c4eb8ef1cd01c481"
                },
                "downloads": -1,
                "filename": "piex-0.1.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "bc11518733ea200c88be48802e40f11b",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.4",
                "size": 12675,
                "upload_time": "2018-11-28T20:05:04",
                "url": "https://files.pythonhosted.org/packages/79/c6/264f13eb46cc624a801e3b359fae41a29af2cdf1a121888f95b9f224bbdc/piex-0.1.1-py2.py3-none-any.whl"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6b109a00ef5f896a6944686577c0ed2f",
                    "sha256": "94a8cb4c9855b0e0ad24db54429eb4b47f83717d1c300b41c76af33af78dde5b"
                },
                "downloads": -1,
                "filename": "piex-0.1.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "6b109a00ef5f896a6944686577c0ed2f",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.4",
                "size": 12966,
                "upload_time": "2019-05-21T22:35:12",
                "url": "https://files.pythonhosted.org/packages/1f/90/1c29037268b0cec0211e8c80e64876a53f1b2745210bf58e06c19c2599e6/piex-0.1.2-py2.py3-none-any.whl"
            }
        ],
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "81fa32e115353bd8f2deb483a84ddd20",
                    "sha256": "1d3a45d9ce7479d6b30823783a205356eb3ce01414f73977124d93d810ec3a76"
                },
                "downloads": -1,
                "filename": "piex-0.2.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "81fa32e115353bd8f2deb483a84ddd20",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.4",
                "size": 12967,
                "upload_time": "2019-05-21T22:38:24",
                "url": "https://files.pythonhosted.org/packages/86/a8/8d43acb88a5b5e247879e93776fe9f8c99bb31cabbedb42099a251cedb2b/piex-0.2.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "9513ca888ab4e30624d813db1a8836dd",
                    "sha256": "c6ebb98ac1ea7d07dcb63b14f674158c66e670ce1fbad9b1fc6da3d6a903fbff"
                },
                "downloads": -1,
                "filename": "piex-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "9513ca888ab4e30624d813db1a8836dd",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.4",
                "size": 30551,
                "upload_time": "2019-05-21T22:38:26",
                "url": "https://files.pythonhosted.org/packages/b3/ac/a1ae4c66b0c91d51715e2d3bb402d92eea03e8fd4500021a7219fa36521c/piex-0.2.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "81fa32e115353bd8f2deb483a84ddd20",
                "sha256": "1d3a45d9ce7479d6b30823783a205356eb3ce01414f73977124d93d810ec3a76"
            },
            "downloads": -1,
            "filename": "piex-0.2.0-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "81fa32e115353bd8f2deb483a84ddd20",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": ">=3.4",
            "size": 12967,
            "upload_time": "2019-05-21T22:38:24",
            "url": "https://files.pythonhosted.org/packages/86/a8/8d43acb88a5b5e247879e93776fe9f8c99bb31cabbedb42099a251cedb2b/piex-0.2.0-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "9513ca888ab4e30624d813db1a8836dd",
                "sha256": "c6ebb98ac1ea7d07dcb63b14f674158c66e670ce1fbad9b1fc6da3d6a903fbff"
            },
            "downloads": -1,
            "filename": "piex-0.2.0.tar.gz",
            "has_sig": false,
            "md5_digest": "9513ca888ab4e30624d813db1a8836dd",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.4",
            "size": 30551,
            "upload_time": "2019-05-21T22:38:26",
            "url": "https://files.pythonhosted.org/packages/b3/ac/a1ae4c66b0c91d51715e2d3bb402d92eea03e8fd4500021a7219fa36521c/piex-0.2.0.tar.gz"
        }
    ]
}