\n \n \n \n
Experiment design, deployment, and optimization
\n\n\n[](http://www.apache.org/licenses/LICENSE-2.0.html)\n\n\nEXP is a python experiment management toolset created to simplify two simple use cases: design and deploy experiments in the form of python modules/files.\n\nAn experiment is a series of runs of a given configurable module for a specified set of parameters. This tool covers one of the most prevalent experiment deployment scenarios: testing a set of parameters in parallel in a local machine or homogeneous cluster. EXP also supports [global optimization](https://www.cs.ox.ac.uk/people/nando.defreitas/publications/BayesOptLoop.pdf) using **gaussian processes** or other surrogate models such as **random forests**. This can be used for instance as a tool for **hyperoparameter tuning** for machine learning models.\n\n## Features\n* **parameter space design** based on configuration files ([TOML](https://github.com/toml-lang/toml) format);\n* **parallel experiment deployment** using ``multiprocessing`` processes;\n* **CUDA gpu workers** one parallel process per available GPUs: uses the variable [CUDA_VISIBLE_DEVICES](https://devblogs.nvidia.com/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices);\n* **global optimization** from parameter spaces (e.g. for hyperparameter tunning) using [scikit-optimize](https://scikit-optimize.github.io/).\n\n## Installation\n``pip install exp`` \n\n``pipenv install exp`` with [pipenv](https://pipenv.readthedocs.io/en/latest/install/#pragmatic-installation-of-pipenv)\n\n## Available CLI tools\nEXP provides two CLI modules:\n* exp.run: ``python -m exp.run -p basic.conf -m runnable.py --workers 10``\n* exp.gopt:``python -m exp.gopt -p basic.conf -m runnable.py --workers 4 -n 100 --plot``\n\nfor more information check each commands help:\n\n``python -m exp.run -h``\n\n## Getting Started: Optimization\n\n### 1. Runnable Module\nThe first step is to create a module to use in our experiments. A basic configurable module ``runnable.py`` looks like this:\n\n```python\ndef run(x=1, **kwargs):\n return x ** 2\n```\n\nThe module simply computes the square of a parameter ``x``. Note that ``kwargs`` is included in case the experiment runner dispatcher sends other parameters (that might not be used by your module). Since run receives a dictionary, you could also define it as follows.\n\n```python\ndef run(**kwargs):\n x = kwargs.get('x',1)\n return x ** 2\n```\n\n### 2. Parameter Space Definition\nNext, we need a configuration file ``basic.conf`` were the parameters are specified:\n```markdown\n[x]\ntype = \"range\"\nbounds = [-10,10]\n```\nThis defines a parameter space with a single parameter ``x`` with values in the range ``[-10,10]``. For how to specify parameter spaces, see the Parameter Space Specification.\n\n### 3. Module Optimization\nOur simple module returns the ``x**2``, the optimizer tries to find the minimum value of this function based on the parameter space given by the configuration file. In this case, the optimizer will look at values of ``x`` between ``[-10,10]`` and try to find the minimum value.\n\n```bash\npython -m exp.gopt --params basic.conf --module runnable.py --n 20 --workers 4\n```\n\n\n ![](\"https://raw.githubusercontent.com/davidenunes/exp/master/extras/getting_started.gif\")
\n
\n ![](\"https://raw.githubusercontent.com/davidenunes/exp/master/extras/convergence.png\")
\n