{ "info": { "author": "Simon-Martin Schroeder", "author_email": "martin.schroeder@nerdluecht.de", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Intended Audience :: Science/Research", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7" ], "description": "# experitur\n\n[![](https://img.shields.io/pypi/v/experitur.svg?style=flat)](https://pypi.org/project/experitur/) [![Build Status](https://travis-ci.org/moi90/experitur.svg?branch=master)](https://travis-ci.org/moi90/experitur) [![codecov](https://codecov.io/gh/moi90/experitur/branch/master/graph/badge.svg)](https://codecov.io/gh/moi90/experitur) ![](https://img.shields.io/pypi/pyversions/experitur.svg?style=flat) [![Documentation Status](https://readthedocs.org/projects/experitur/badge/?version=latest)](https://experitur.readthedocs.io/en/latest/?badge=latest)\n\n**experitur** automates machine learning and other computer experiments. Includes grid search and resuming aborted experiments. No lock-in, all your data is easily accessible in the text-based, machine-readable YAML format.\n\n![example](https://raw.githubusercontent.com/moi90/experitur/master/examples/simple.gif)\n\n## Experiment description\n\nEvery experiment is described in a regular python file. The `@experiment` decorator is used to mark experiment entry-points.\n\n```python\nfrom experitur import experiment\n\n@experiment(\n parameter_grid={\n \"parameter_1\": [1,2,3],\n \"parameter_2\": [\"a\", \"b\", \"c\"],\n })\ndef example(trial):\n \"\"\"This is an example experiment.\"\"\"\n ...\n```\n\n### Parameter grid\n\nThe core of an experiment is its *parameter grid*. It works like [`sklearn.model_selection.ParameterGrid`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ParameterGrid.html). Each parameter has a list of values that it can take. A number of *trials* is generated from the cross product of the values of each parameter.\n\n### Entry point\n\nAn experiment is a regular function that is decorated with `@experiment` (unless it is *abstract* or *derived*). Upon execution, the function gets called with the current trial. It may return a result dictionary.\n\n**Signature:** `(trial) -> dict`\n\n```python\nfrom experitur import experiment\n\n@experiment(\n parameter_grid={\n \"parameter_1\": [1,2,3],\n \"parameter_2\": [\"a\", \"b\", \"c\"],\n })\ndef example(trial):\n \"\"\"This is an example experiment.\"\"\"\n print(\"parameters:\", pformat(parameters))\n return {}\n```\n\nNow, you can run the experiment:\n\n```\n$ experitur run example.py\n...\n```\n\nAs you can see, `run` was called four times with every combination of [1,2] x [a,b].\n\n### Multiple experiments\n\nThe Python file can contain multiple experiments:\n\n```python\nfrom experitur import experiment\n\n@experiment(...)\ndef example1(trial):\n ...\n\n@experiment(...)\ndef example2(trial):\n ...\n```\n\n### Experiment inheritance\n\nOne experiment may inherit the settings of another, using the `parent` parameter.\n\n```python\nfrom experitur import experiment\n\n@experiment(...)\ndef example1(trial):\n ...\n\n# Derived with own entry point:\n@experiment(parent=example1)\ndef example2(trial):\n ...\n\n# Derived with inherited entry point:\nexample3 = experiment(\"example3\", parent=example2)\n```\n\n### Parameter substitution\n\n`experitur` includes a recursive parameter substitution engine. Each value string is treated as a *recursive format string* and is resolved using the whole parameter set of a trial.\n\n```python\n@experiment(\n parameter_grid={\n \"a1\": [1],\n \"a2\": [2],\n \"b\": [1, 2],\n \"a\": [\"{a_{b}}\"],\n })\ndef example(trial):\n ...\n```\n\n```\n$ experitur run parsub\n...\n```\n\nThis way, you can easily run complicated setups with settings that depend on other settings.\n\nRecursive format strings work like `string.Formatter` with two exceptions:\n\n1. **Recursive field names:** The field name itself may be a format string:\n\n ```\n format(\"{foo_{bar}}\", bar=\"baz\", foo_baz=\"foo\") -> \"foo\"\n ```\n\n2. **Literal output:** If the format string consist solely of a replacement field and does not contain a format specification, no to-string conversion is performed:\n\n ```\n format(\"{}\", 1) -> 1\n ```\n\n This allows the use of format strings for non-string values.\n\n#### Application\n\nThis feature is especially useful if you want to run your experiments for different datasets but need slightly different settings for each dataset.\n\nLet's assume we have two datasets, \"bees\" and \"flowers\".\n\n```python\n@experiment(\n parameter_grid={\n \"dataset\": [\"bees\", \"flowers\"],\n \"dataset_fn\": [\"/data/{dataset}/index.csv\"],\n \"bees-crop\": [10],\n \"flowers-crop\": [0],\n \"crop\": [\"{{dataset}-crop}\"]\n }\n)\ndef example(trial):\n ...\n```\n\nThe experiment will be executed once for each dataset, with `trial[\"crop\"]==10` for the \"bees\" dataset and `trial[\"crop\"]==0` for the \"flowers\" dataset.\n\n## The `trial` object\n\nEvery experiment receives a `trial` object that allows access to the parameters and meta-data of the trial.\n\nParameters are accessed with the `[]` operator (e.g. `trial[\"a\"]`), meta-data is accessed with the `.` operator (e.g. `trial.wdir`).\n\n### Access of parent data\n\n...\n\n## Files\n\nWhen `experitur` executes a script, it creates the following file structure in the directory where the DOX file is located:\n\n```\n/\n+- script.py\n+- script/\n| +- experiment_id/\n| | +- trial_id/\n| | | +- experitur.yaml\n| | ...\n| ...\n```\n\n`