{
    "info": {
        "author": "Bradley Arsenault (Electric Brain)",
        "author_email": "brad@electricbrain.io",
        "bugtrack_url": null,
        "classifiers": [
            "Environment :: Console",
            "Intended Audience :: Developers",
            "Intended Audience :: Education",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: BSD License",
            "Operating System :: MacOS :: MacOS X",
            "Operating System :: POSIX",
            "Operating System :: Unix",
            "Programming Language :: Python",
            "Programming Language :: Python :: 3",
            "Topic :: Scientific/Engineering",
            "Topic :: Software Development"
        ],
        "description": "# Introduction\n\nHypermax is a power tool for optimizing algorithms. It builds on the powerful TPE algorithm with additional features\nmeant to help you get to your optimal hyper parameters faster and easier. We call our algorithm Adaptive-TPE, and it is\nfast and accurate optimizer that trades off between explore-style and exploit-style strategies in an intelligent manner\nbased on your results. It depends upon pretrained machine learning models that have been taught how to optimize\nyour machine learning model as fast as possible. Read the research behind ATPE in [Optimizing Optimization](https://www.electricbrain.io/blog/optimizing-optimization) and [Learning to Optimize](https://www.electricbrain.io/blog/learning-to-optimize), and use it for yourself by downloading Hypermax.\n\nIn addition, Hypermax automatically gives you a variety of charts and graphs based on your hyperparameter results.\nHypermax can be restarted easily in-case of a crash. Hypermax can monitor the CPU and RAM usage of your algorithms - \nautomatically killing your process if it takes too long to execute or uses too much RAM. Hypermax even has a UI.\nHypermax makes it easier and faster to get to those high performing hyper-parameters that you crave so much.\n\nStart optimizing today!\n\n![Screenshot of Hypermax](https://raw.githubusercontent.com/electricbrainio/hypermax/master/docs/main_screen.png \"Screenshot of Hypermax\")\n\n# Installation\n\nInstall using pip:\n\n```bash\npip3 install hypermax -g\n```\n\nPython3 is required.\n\n# Getting Started (Using Python Library)\n\nIn Hypermax, you define your hyper-parameter search, including the variables, method of searching, and \nloss functions, using a JSON object as you configuration file.\n\n\n# Getting Started (Using CLI)\n\nHere is an example. Lets say you have the following file, model.py:\n\n```python\nimport sklearn.datasets\nimport sklearn.ensemble\nimport sklearn.metrics\nimport datetime\n\ndef trainModel(params):\n    inputs, outputs = sklearn.datasets.make_hastie_10_2()\n\n    startTime = datetime.now()\n\n    model = sklearn.ensemble.RandomForestClassifier(n_estimators=int(params['n_estimators']))\n    model.fit(inputs, outputs)\n    predicted = model.predict(inputs)\n\n    finishTime = datetime.now()\n\n    auc = sklearn.metrics.auc(outputs, predicted)\n\n    return {\"loss\": auc, \"time\": (finishTime - startTime).total_seconds()}\n```\n\nYou configure your hyper parameter search space by defining a JSON-schema object with the needed values:\n\n```json\n{\n    \"hyperparameters\": {\n        \"type\": \"object\",\n        \"properties\": {\n            \"n_estimators\": {\n                \"type\": \"number\",\n                \"min\": 1,\n                \"max\": 1000,\n                \"scaling\": \"logarithmic\"\n            }\n        }\n    }\n}\n```\n\nNext, define how you want to execute your optimization function:\n\n```json\n{\n    \"function\": {\n        \"type\": \"python_function\",\n        \"module\": \"model.py\",\n        \"name\": \"trainModel\",\n        \"parallel\": 1\n    }\n}\n```\n\nNext, you need to define your hyper parameter search:\n\n```json\n{\n    \"search\": {\n        \"method\": \"atpe\",\n        \"iterations\": 1000\n    }\n}\n```\n\nNext, setup where you wants results stored and if you want\ngraphs generated:\n\n\n```json\n{\n    \"results\": {\n        \"directory\": \"results\",\n        \"graphs\": true\n    }\n}\n```\n\n\nLastly, you need to provide indication if you want to use the UI:\n\n```json\n{\n    \"ui\": {\n        \"enabled\": true\n    }\n}\n```\n\n**NOTE:** At the moment the console UI is not supported in Windows environments, so you will need to specify `false` in\nthe `enabled` property. We use the `urwid.raw_display` module which relies on `fcntl`. For more information, [see here](https://github.com/urwid/urwid/issues/152).\n\nPulling it all together, you create a file like this `search.json`, defining your hyper-parameter search:\n\n```json\n{\n    \"hyperparameters\": {\n        \"type\": \"object\",\n        \"properties\": {\n            \"n_estimators\": {\n                \"type\": \"number\",\n                \"min\": 1,\n                \"max\": 1000,\n                \"scaling\": \"logarithmic\"\n            }\n        }\n    },\n    \"function\": {\n        \"type\": \"python_function\",\n        \"module\": \"model\",\n        \"name\": \"trainModel\",\n        \"parallel\": 1\n    },\n    \"search\": {\n        \"method\": \"atpe\",\n        \"iterations\": 1000\n    },\n    \"results\": {\n        \"directory\": \"results\",\n        \"graphs\": true\n    },\n    \"ui\": {\n        \"enabled\": true\n    }\n}\n```\n\nAnd now you can run your hyper-parameter search\n\n```bash\n$ hypermax search.json\n```\n\nHypermax will automatically begin searching your hyperparameter space. If your computer dies and you need to restart\nyour hyperparameter search, its as easy as providing it the existing results directory as a second parameter. Hypermax\nwill automatically pick up where it left off.\n\n```bash\n$ hypermax search.json results_0/\n```\n\n# Optimization Algorithms\n\nHypermax supports 4 different optimization algorithms:\n\n - \"random\" - Does a fully random search\n - \"tpe\" - The classic TPE algorithm, with its default configuration\n - \"atpe\" - Our Adaptive-TPE algorithm, a good general purpose optimizer \n - \"abh\" - Adaptive Bayesian Hyperband - this is an optimizer that its able to learn from partially trained algorithms in order to optimizer your fully trained algorithm\n\nThe first three optimizers - random, tpe, and atpe, can all be used with no configuration. \n\n## Adaptive Bayesian Hyperband\n\nThe only optimizer in our toolkit that requires additional configuration is the Adaptive Bayesian Hyperband algorithm.\n\nABH works by training your network with different amounts of resources. Your \"Resource\" can be any parameter that significantly effects\nthe execution time of your model. Typically, training-time, size of dataset, or # of epochs are used as the resource. The amount of resource\nis referred to as the \"budget\" of a particular execution.\n\nBy using partially trained networks, ABH is able to explore more widely over more combinations of hyperparameters, and then triage\nthe knowledge it gains up to the fully trained model. It does this by using a tournament of sorts, promoting the best performing \nparameters on smaller budget runs to be trained at larger budgets. See the following chart as an example:\n\n![Hyperband CPU Allocation](https://raw.githubusercontent.com/electricbrainio/hypermax/master/docs/abh_cpu_allocation.png \"Single Parameter Loss Chart\")\n\nThere are many ways you could configure such a system. Hyperband is just a mathematically and theoretically sound way of choosing\nhow many brackets to run and with what budgets. ABH is a method that combines Hyperband with ATPE, in the same way that BOHB combines\nHyperband with conventional TPE.\n\nABH requires you to select three additional parameters:\n- min_budget - Sets the minimum amount of resource that must be allocated to a single run\n- max_budget - Sets the maximum amount of resource that can be allocated to a single run\n- eta - Defines how much the budget is reduced for each bracket. The theoretically optimum value is technically E or 2.71, but values of 3 or 4 are more typical and work fine in practice.\n\nYou define these parameters like so:\n\n```json\n{\n  \"search\": {\n      \"method\": \"abh\",\n      \"iterations\": 1000,\n      \"min_budget\": 1,\n      \"max_budget\": 30,\n      \"eta\": 3\n  }\n}\n```\n\nThis configuration will result in Hyperband testing 4 different brackets: 30 epochs, 10 epochs, 3.333 epochs (rounded down), and 1.1111 epochs (rounded down)\n\nThe budget for each run is provided as a hyperparameter to your function, along side your other hyperparameters. The budget will be given as \nthe \"$budget\" key in the Python dictionary that is passed to your model function.\n\nTips:\n - ABH only works well when the parameters for a run with a small budget correlates strongly with the parameters for a run with a high budget.\n - Try to eliminate any parameters whose behaviour and effectiveness might change depending on the budget. E.g. a parameter for % of budget in mode 1, % of budget in mode 2 will not\n   work well with ABH. \n - Don't test too wide of a range of budgets. As a general rule of thumb, never set min_budget lower then max_budget/eta^4\n - Never test a min_budget thats so low that your model doesn't train at all. The minimum is there for a reason\n - If you find that ABH is getting stuck in a local minima, choosing parameters that work well on few epochs but work poorly on many epochs, your \n   better off using vanilla ATPE and just training networks fully on each run.\n\n\n# Results\n\nHypermax automatically generates a wide variety of different types of results for you to analyze.\n\n## Hyperparameter Correlations\n\nThe hyperparameter correlations can be viewed from within the user-interface or in \"correlations.csv\" within\nyour results directory. The correlations can help you tell which hyper-parameter combinations are moving the\nneedle the most. Remember that a large value either in the negative or positive indicates a strong correlation\nbetween those two hyper-parameters. Values close to 0 indicate that there is little correlation between those\nhyper-parameters. The diagonal access will give you the single-parameter correlations.\n\nIt should also be noted that these numbers get rescaled to fall roughly between -10 and +10 (preserving the original sign),\nand thus are not the mathematically defined covariances. This is done to make it easier to see the important relationships.\n\n## Single Parameter Loss Charts\n\n![Single Parameter Loss Chart](https://raw.githubusercontent.com/electricbrainio/hypermax/master/docs/single_parameter_scatter.png \"Single Parameter Loss Chart\")\n\nThe single parameter loss charts create a Scatter diagram between the parameter and the loss. These are the most useful charts and are usually the go-to for attempting\nto interpret the results. Hypermax is going to generate several different versions of this chart. The original version will have every tested value. The \"bucketed\"\nversion will attempt to combine hyper-parameter values into \"buckets\" and give you the minimum value for each bucket - useful for continuous valued hyper-parameters\nthat you have a lot of results for. The \"top_10_percent\" version is just showing you the scatter for only the top-10% of results - useful when you want to home in on\nthose top-performing values.\n\nYou will also get a version of this chart for the time that your model takes to execute. This can be useful if trading off between accuracy and time taken is important\nto you.\n\n## Two Parameter Loss Matrix\n\nThe two parameter loss matrixes are a color-coded diagram that helps you to determine the optimal value between two hyper-parameters.\n\n![Two Parameter Loss Matrix](https://raw.githubusercontent.com/electricbrainio/hypermax/master/docs/loss_matrix.png \"Loss Matrix\")\n\nThe graph is color coded in the following way:\n\n    Red: 90th percentile of squares\n    Yellow: 40th percentile of squares\n    Green: 20th percentile of squares\n    Blue: 5th percentile of squares\n\nWe use quadric interpolation which both gives nice smooth rounded corners but does not excessively blur key areas. Thee chart is generated by\ndividing your hyperparameter values into 10 buckets each, resulting in a 10x10 grid of squares. We compute a value for each square in two ways:\n\nOne version of the graph computes the value for each square by taking the Min of all values in that grid square. This is usually the most useful\nchart. The other version computes the value by taking the Mean. This second version can be susceptible to outlier results, but can show interesting\npatterns sometimes.\n\nYou also get versions of this graph done which only use the top 10% of your results, helping you to further focus in on the top performing area of\nyour hyper parameter space. In addition, you get a version of this matrix done for the execution time of your model - in caes that is important.\n\n## Two Parameter Response Matrix\n\nThe response matrixes are very similar to the Loss matrices. In fact - it displays all of the same data. They are just color-coded differently\nto highlight different things. The Loss matrix defines its colors based on global statistics. The Response Matrix defines its colors\nbased only one the values within each row. This often highlights important patterns - such as that the optimal value for one hyperparameter\nis always the same, regardless of the other hyper parameter (like this chart below, where optimal boosting_rounds appears to be around 40, no \nmatter what the max_depth is.)\n\n![Two Parameter Response Matrix](https://raw.githubusercontent.com/electricbrainio/hypermax/master/docs/response_matrix.png \"Response Matrix\")\n\nThe graph is color coded in the following way:\n\n    Red: 90th percentile of row\n    Yellow: 60th percentile of row\n    Green: 30th percentile of row\n    Blue: 10th percentile of row\n\n## Two Parameter Scatter\n\nThe two parameter scatters go along-side the two-parameter loss matrices. If you are concerned that the Loss Matrixes may be trying to extrapolate too\nmuch from very few data-points, you can check the scatter in order to check if you actually have a decent sample of results that fall within that area.\n\n![Two Parameter Scatter](https://raw.githubusercontent.com/electricbrainio/hypermax/master/docs/two_parameter_scatter.png \"Two Parameter Scatter\")\n\nThe color coding is the same as it is for the Loss Matrix, but percentiles are calculated over all results and not over the 10x10 grid of squares.\n\n    Red: 90th percentile of values\n    Yellow: 40th percentile of values\n    Green: 20th percentile of values\n    Blue: 5th percentile of values\n\nThe size of the markers will also vary - larger and bluer is more accurate. Smaller and redder is less accurate.\n\n## CSV Files\n\nFor all of the above mentioned charts, you will automatically get CSV files containing all of the raw data used to generate that chart.\n\n# Detailed Configuration\n\n## Hyper Parameter Space\n\nYou define your hyper-parameter space within the \"hyperparameters\" section of the configuration file. The format is reminiscent of JSON-schema, however, only\na limited set of options are supported.\n\n### Number hyper-parameters\n\nMost of the hyper-parameters that you are going to be tuning are expected to be numbers. The configuration of the number hyper-parameter looks like so:\n\n```json\n{\n    \"parameter_name\": {\n        \"type\": \"number\",\n        \"mode\": \"uniform\",\n        \"scaling\": \"logarithmic\",\n        \"min\": 1,\n        \"max\": 1000,\n        \"rounding\": 1\n    }\n}\n```\n\nThere are 3 required parameters - type, min and max. Type should be set to 'number', and the min and max should represent the minimum and maximum values of\nyour range.\n\nThere are also three optional parameters. `mode` can be either `uniform` or `normal` (defaults to `uniform`). The `scaling` parameter can be either `linear`\nor `logarithmic` (default to `linear`). And you can additionally set `rounding` if you want values to be rounded to some fixed interval. A rounding set to 1\nwill make your parameter an integer.\n\n\n### Enumerations\n\nWhen you have several different possible values that are categorically distinct, you can use an enumeration to specify the possible options:\n\n```json\n{\n    \"activation_function\": {\n        \"type\": \"string\",\n        \"enum\": [\"relu\", \"elu\", \"tanh\", \"sigmoid\", \"swish\", \"prelu\", \"selu\"]\n    }\n}\n```\n\n\n### Object hyper-parameters\n\nYour hyper-parameter space can contain JSON objects which contain other hyper parameters. In fact, the bottom layer must be made as an object. Simply\nset the type to `object` and provide it a `properties` field.\n\n```json\n{\n    \"parameter_object\": {\n        \"type\": \"object\",\n        \"properties\": {\n            \"parameter_name\": {\n                \"type\": \"number\",\n                \"mode\": \"uniform\",\n                \"scaling\": \"logarithmic\",\n                \"min\": 1,\n                \"max\": 1000,\n                \"rounding\": 1\n            }\n        }\n    }\n}\n```\n\n### Choices & Decision Points\n\nThe true power of the TPE algorithm comes from its ability to optimize categorical hyper-parameters, including ones which make other hyper-parameters\navailable. To do this, you can provide either a `oneOf` or `anyOf` field. Note that \"oneOf\" and \"anyOf\" behave exactly the same - we allow\nboth in order to match JSON-Schema specifications.\n\n\n```json\n{\n    \"choice_parameter\": {\n        \"anyOf\": [\n            {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"parameter_name\": {\n                        \"type\": \"number\",\n                        \"min\": 1,\n                        \"max\": 1000\n                    }\n                }\n            },\n            {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"other_parameter_name\": {\n                        \"type\": \"number\",\n                        \"min\": 1,\n                        \"max\": 1000\n                    }\n                }\n            }\n        ]\n    }\n}\n```\n\nHypermax will add in an additional \"parameter_name.$index\" field into the parameters it sends to your algorithm, so that you\ncan tell which side of the branch you are on.\n\nImportant! When using oneOf or anyOf, each of the options MUST be \"object\" type hyperparameters, as shown above.\n\n\n### Constants\n\nWhen using decision points, you may find it convenient to add in a constant value to tell you which side of the branch you are on.\n\nThis is easy using a \"constant\" parameter. You can have the same parameter name on both sides of the branch.\n\nThis allows you to, for example, test two different neural network optimizers, and the various learning rates\nattached to each, without having to worry that the algorithm is going to learn an \"average\" learning rate that \nworks for both optimizers. Both sides of the branch will be kept separate during optimization, even though\nthey share the same parameter names.\n\nConstant parameters are ignored for optimization purposes, but are still passed into your function, making them mostly\nuseful when you want to lock in a parameter without changing your code, or when using decision points like so:\n\n```json\n{\n    \"optimizer\": {\n        \"oneOf\": [\n            {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"optimizerName\": {\n                        \"type\": \"string\",\n                        \"constant\": \"adam\"\n\n                    },\n                    \"learningRate\": {\n                        \"type\": \"number\",\n                        \"min\": 1e-5,\n                        \"max\": 1e-3\n                    }\n                }\n            },\n            {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"optimizerName\": {\n                        \"type\": \"string\",\n                        \"constant\": \"sgd\"\n\n                    },\n                    \"learningRate\": {\n                        \"type\": \"number\",\n                        \"min\": 1e-5,\n                        \"max\": 1e-3\n                    }\n                }\n            }\n        ]\n    }\n}\n```\n\n\n## Model Execution\n\nThere are several different ways of executing your model.\n\n### Python Functions\n\nThe most straight forward way to execute your model is by defining a Python function. To do this, simply provide the \nname of the module and the name of the function in the \"module\" and \"name\" functions, like so:\n\n```json\n{\n    \"function\": {\n        \"type\": \"python_function\",\n        \"module\": \"model\",\n        \"name\": \"trainModel\"\n    }\n}\n```\n\nRemember that you do not include the extension of the name of your module, there is no \".py\" on it. The module is\nreferenced using Pythons standard system. This means that you can directly reference any files in the current working\ndirectory simply by their file-name. Alternatively, you can reference a system-package or a Python package that is\nsetup elsewhere. As long as this works:\n\n```bash\n    $ python3\n\n    Python 3.6.5 (default, Mar 29 2018, 18:20:46) \n    [GCC 8.0.1 20180317 (Red Hat 8.0.1-0.19)] on linux\n    Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n    >>> import module_name\n    >>> module.foobar()\n```\n\nThen this will to:\n\n```json\n{\n    \"function\": {\n        \"type\": \"python_function\",\n        \"module\": \"module_name\",\n        \"name\": \"foobar\"\n    }\n}\n```\n\n### Format of the result\n\nThe results can be provided in one of two formats. The simplest is to just return the loss directly as a single floating point value\nfrom your cost function, or print it to standard output in your executable. For example:\n\n```python\ndef trainModel(parameters):\n    # Do some fancy stuff\n    loss = 1.0\n    return loss\n```\n\nor as an executable:\n\n```python\n#!/usr/bin/python3\n\n# Do some fany stuff\nloss = 1.0\nprint(loss)\n```\n\nIf you are using multiple losses though, you will have to return each of them as part of a JSON object. For example:\n\n```python\ndef trainModel(parameters):\n    # Do some fancy stuff\n    accuracy = 0.9\n    stddev = 0.1\n    return {\"accuracy\": accuracy, \"stddev\": stddev}\n```\n\nor as an executable:\n\n```python\n#!/usr/bin/python3\n\nimport json\n\n# Do some fancy stuff\naccuracy = 0.9\nstddev = 0.1\nprint(json.dumps({\"accuracy\": accuracy, \"stddev\": stddev}))\n```\n\nIf you want to store additional metadata with your model, you can. Any fields that are unrecognized for any other purpose will be automatically considered as metadata.\n\n```python\ndef trainModel(parameters):\n    # Do some fancy stuff\n    loss = 1.0\n    additional_statistic = 42.0\n    return {\"loss\": loss, \"additional_statistic\": additional_statistic}\n```\n\nThe time your model takes is automatically measured by Hypermax (time can be used for punishing your model for taking too long, see Losses section).\nHowever, you may only care about the execution / run-time of your model, and not about the training time. In these cases, you can return `time` as\nan additional variable.\n\n```python\ndef trainModel(parameters):\n    # Do some fancy stuff\n    model = Model()\n    model.train()\n    start = datetime.now()\n    loss = model.test()\n    end = datetime.now()\n    return {\"loss\": loss, \"time\": (end-start).total_seconds()}\n```\n\nIt should be noted that this time is not the same time used for `auto_kill` purposes. This is the time that will be showed in the UI and considered for optimization\npurposes.\n\n### Automatically killing models due to running time or RAM usage\n\nSometimes, your models may be behaving very poorly in certain parts of your hyper-parameter space. It is thus possible,\nand indeed recommended, to set add limits on how long your model can be running for and how much RAM it can use. This\nprevents your optimization routine from getting hung due to a model that takes too long to train, or crashing entirely\nbecause it uses too much RAM.\n\nTo do this, simply add in a `auto_kill_max_time`, `auto_kill_max_ram`, or `auto_kill_max_system_ram` option, and set a \na `kill_loss` variable to indicate what the loss should be for models which are killed.\n\nauto_kill_max_time is specified in seconds. `auto_kill_max_ram` and `auto_kill_max_system_ram` are both specified in\nmegabytes, the kind which are based by `1024` (not `1000`).\n\n`auto_kill_max_ram` only measures the RAM of the model process. However, if your cost-function has other various\nsub-processes which take up RAM, these will not be counted. Therefore, you can use `auto_kill_max_system_ram` in\nthese cases to prevent total system RAM usage from creeping too high (the assumption being your model is what is\ntaking up the systems RAM). You are able to provide both at the same time (if you want to).\n\n`auto_kill_loss` is just a floating point indicating the total loss that should be given to the optimizer when the model\nis killed. This helps teach the optimizer to avoid hyper-parameters which lead to models being killed.\n\n```json\n{\n    \"function\": {\n        \"type\": \"python_function\",\n        \"module\": \"model\",\n        \"name\": \"trainModel\",\n        \"auto_kill_max_time\": 120.0,\n        \"auto_kill_max_ram\": 512,\n        \"auto_kill_max_system_ram\": 3800,\n        \"auto_kill_loss\": 1.0\n    }\n}\n```\n\n## Loss / Cost Functions (UNDER CONSTRUCTION)\n\nPLEASE NOTE THIS SECTION IS DESCRIBING FUTURE FUNCTIONALITY AND IS NOT YET SUPPORTED. PLEASE IGNORE THIS SECTION.\n\nWe support several different types of cost functions.\n\n## Timing Loss (UNDER CONSTRUCTION)\n\nYou can include the time your model takes to train as one of your loss functions as well.  This makes it convenient \nto teach the algorithm to avoid bad hyper parameters which lead to long training times. Many algorithms have poor\ncombinations of parameters which can lead to long execution time with no improvement in performance.\n\nIf the algorithm takes less then the `target_time`, then no penalty is incurred. As the time taken goes between\n`target_time` and `max_time`, the penalty is introduced quadratically. At `max_time`, the penalty is exactly\n`penalty_at_max`.\n\nThis usually results in the algorithm choosing a value between `target_time` and `max_time`, but closer\nto `target_time`. For example, with the following:\n\n```json\n{\n    \"metrics\": {\n        \"time\": {\n            \"type\": \"time\",\n            \"target_time\": 5,\n            \"max_time\": 10,\n            \"penalty_at_max\": 0.1\n        }\n    }\n}\n```\n\nIf the algorithm takes 5.0 seconds, no penalty is introduced. At 6.0 seconds, the penalty is:\n\n    = ((6 - 5) ^ 2 / (10 - 5)^2)*0.1\n    = 0.0025\n\nAt 9 seconds, the penalty is:\n\n    = ((9 - 5) ^ 2 / (10 - 5) ^ 2)*0.1\n    = 0.064\n\nAt 10 seconds, the penalty is:\n\n    = ((10 - 5) ^ 2 / (10 - 5) ^ 2)*0.1\n    = 0.1\n\nLonger times will have even larger penalties.\n\n# Details on Adaptive-TPE\n\nSee here:\n\n- [Optimizing Optimization](https://www.electricbrain.io/blog/optimizing-optimization)\n- [Learning to Optimize](https://www.electricbrain.io/blog/learning-to-optimize)\n\n# Todo & Wishlist\n\nFeel free to contribute! Reach out to Brad at brad@electricbrain.io or here on Github. We welcome additional contributors.\n\nThis is the grand-todo list and was created on August 4, 2018. Some items may have been completed.\n\n- Results\n    - Able to configure save-location.\n    - Automatic uploading of results to Google Drive\n- Model Execution\n    - Autokill models that take too much GPU RAM\n    - Autokill models that take too much Disk / Network (not sure about this one)\n    - Fix bug related to using too many file handlers.\n    - Execute model as an executable\n    - Execute model remotely, through ssh\n    - Execute model by sending a message through message-buffer like RabbitMQ or Kafka (receive results same way)\n    - Rsync a folder prior to remote execution\n    - Can attach additional arbitrary metadata to your model results\n    - Able to have \"speed-tests\" on models, where your hyper-parameters are tested on a reduced dataset in order to measure the speed. Useful to eliminate bad hyper-parameters without executing the full model.\n    - Similarly, able to run \"memory tests\" on models, ensuring your hyper-parameters don't lead to excessive ram usage\n    - Able to automatically run additional cross-folds on your best-models, to ensure they aren't statistical flukes\n    - Hyper-parameters with rounding set to 1 should be automatically converted to integer\n- Configuration:\n    - JSON-schema for the configuration files\n    - validation of json-schemas\n    - Ability to accept yaml as well as JSON\n    - Able to have fixed values inside of the hyper-parameter schemas.\n    - Able to have \"unbounded\" hyperparameters (only when using iterative optimization, since TPE doesn't inherently do this)\n    - Ability to have hyper-parameters within arrays, such as a list of layers\n- Reliability:\n    - Hypermax saves results / reloads from where it left off by default\n    - Try to lock in package versions related to Hyperopt so people don't have problems on installation\n    - Need a better way of handling exceptions that happen in UI code\n    - Execution threads should only communicate through queues, eliminate shared variables (and put locks in the ones we can't eliminate)\n    - Control random seed and ensure that runs are reproducible\n- General User Interface:\n    - Change User-Interface code to use proper organized classes and not ad-hoc style like it is currently\n    - View recently trained models\n    - Able to view Training Loss VS. Testing Loss on UI\n    - View models which had errors\n    - Fix UI issues related to data-tables (such as in hyperparameter correlations)\n    - Able to adjust min,max,smoothing, and domain of the loss chart\n    - Predict model execution time based on hyper-parameters and prior data\n    - Progress-bar on model training\n    - Can change the file-name when exporting the hyper-parameter correlations\n    - Can instantly open files which are exported, using xdg-open & equivalents\n    - Widget which allows easily viewing/scrolling a large block of text.\n    - View the raw hyper-parameter search configuration, as JSON\n    - Exporting the hyper-parameters should save them in the exact format they are fed into the model, not in a flattened structure\n    - View the hyper-parameter space while model is running\n    - Can view arbitrary metadata that was attached to models\n    - Make the UI responsive to different console sizes (like a grid system)\n    - Fix the bug where the UI doesn't automatically resize when terminal resizes\n    - Access the UI through a web-browser\n    - Password-protection on web-browser UI\n    - Able to monitor the RAM (and GPU RAM) usage of currently executing models\n    - Able to monitor the disk usage of currently executing models\n    - Able to monitor the network usage of currently executing models\n    - Able to monitor general system stats, such as CPU, network, disk, and ram\n- Losses:\n    - Able to have multiple weighted loss functions\n    - Automatically pass loss-function through math-function based on min,max,target to add in asymptotes at target values\n    - Write documentation related to how your cost-function is manipulated to improve results\n    - Convenient way to add Time as an additional loss on your model\n    - Time computed automatically, but can be overridden if provided in results\n    - Can view all the losses for a given model in the UI, not just final loss\n    - Convenient way to add in peak or median RAM (and GPU RAM) as an additional loss on your model\n    - Convenient way to add in disk / network usage as a additional loss on your model\n- Tools for Hyperparameter Tuning:\n    - Improve feature-vector design for hyper-parameter correlations\n    - Edit / change the hyper-parameter space while the model is running\n    - Estimate the cardinality of the search-space\n    - Estimate number of runs per parameter (or something like this)\n    - Ability to fit a hyper-model and do a simulated extension of your hyper-parameter search, trying to predict if there are better values that you haven't searched\n    - Ability to use the hyper-model to hold certain hyper-parameters constant, and determine the optimal values for remaining hyper-parameters\n    - Staged tuning - able to have multiple tuning \"stages\", which tune only certain hyper-parameters at a time or with different configurations\n    - Can have a 'default' value for each of the hyperparameters, e.g. your current best model.\n    - Incremental tuning - basically only tunes a handful of hyper-parameters at a time. Can be random or specified\n    - Ability to change the hyper parameters for TPE\n    - Research some automatic way to guess good TPE hyper-parameters\n    - Integrate Bayesian hyper-parameter optimization as an alternative to TPE\n    - Integrate grid-search as an alternative to TPE\n    - Integrate genetic-algo as an alternative to TPE\n- Command-line interface\n    - Execute model without CUI\n    - Sample next hyper-parameters to test\n    - Export all of the existing types of result-analysis (correlations, hotspot-grids, param vs loss, etc..)\n    - Launch web browser UI (without CUI)\n    - Write a template configuration file to the current directory\n- Library interface:\n    - Able to activate hypermax by calling a library function\n    - Able to provide our cost-function directly as a python function, rather then just as a JSON description\n- Testing\n    - Write unit tests for hyper-parameter configuration\n    - Write unit tests for model execution\n    - Write unit tests for loss functions\n    - Write unit tests for the optimizer\n    - Write unit tests for the results generation\n    - Write unit tests related to reliability (starting / stopping models)\n    - Write unit tests for command line interface\n    - Write unit tests for web-UI module (just ensure it loads)\n    - Write UI tests for the CUI\n    - Write end-to-end optimization tests for a few different real datasets / algos\n- Template hyperparameter spaces:\n    - Create a template hyper-parameter search for lightgbm\n    - Create a template hyper-parameter search for xgbboost\n    - Create template hyper-parameter searches for various scikit-learn estimators\n    - Ability to reference template hyper-parameter searches in your own JSON schemas ($ref)\n- Release / Launch:\n    - Test hypermax in following environments, and fix any issues with installation configuration\n        - Fresh virtual environment\n        - Fresh fedora installation\n        - Fresh ubuntu installation\n        - macOS\n    - Write all the documentation for the README file for Hypermax\n    - Create Github wiki, duplicate README documentation in Github Wiki\n    - Create Hypermax web-page as sub-domain of Electric Brain, duplicate information from README\n    - Write a blog post discussing hypermax and hyper-optimization in general\n    - Create optimization examples for:\n        - Python\n        - Java\n        - NodeJS\n        - Ruby\n        - R\n    - Create system packages for:\n        - Fedora / Redhat / CentOS\n        - Debian / Ubuntu\n        - macOS\n        - Windows\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/electricbrainio/hypermax",
        "keywords": "",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "hypermax",
        "package_url": "https://pypi.org/project/hypermax/",
        "platform": "Linux",
        "project_url": "https://pypi.org/project/hypermax/",
        "project_urls": {
            "Homepage": "https://github.com/electricbrainio/hypermax"
        },
        "release_url": "https://pypi.org/project/hypermax/0.5.1/",
        "requires_dist": [
            "hyperopt",
            "scikit-learn",
            "numpy",
            "scipy",
            "jsonschema",
            "pyyaml",
            "urwid",
            "panwid (==0.2.5)",
            "lightgbm",
            "psutil",
            "matplotlib",
            "colors.py",
            "pymongo"
        ],
        "requires_python": ">=3",
        "summary": "Better, faster hyperparameter optimization by mixing the best of humans and machines.",
        "version": "0.5.1",
        "yanked": false,
        "yanked_reason": null
    },
    "last_serial": 6019207,
    "releases": {
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b49cc12d3c6f1db1c64e3f975424fb8b",
                    "sha256": "8b05d8095813cecd040a788122632155f00061c619e181b102c13ce9a7917e42"
                },
                "downloads": -1,
                "filename": "hypermax-0.1.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "b49cc12d3c6f1db1c64e3f975424fb8b",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3",
                "size": 27937,
                "upload_time": "2018-08-06T21:07:30",
                "upload_time_iso_8601": "2018-08-06T21:07:30.907137Z",
                "url": "https://files.pythonhosted.org/packages/ea/a3/19f7a838b57ff512cb0e3b39650dcb801fd9f3333642b9367cda68e9a4e4/hypermax-0.1.1-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "9334fbfe7333aad6dcb22a140c679c06",
                    "sha256": "cb39b27746a7b85fc3621eb612ff0315de900bd7384f720dd38d59989400dadd"
                },
                "downloads": -1,
                "filename": "hypermax-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "9334fbfe7333aad6dcb22a140c679c06",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3",
                "size": 26774,
                "upload_time": "2018-08-06T21:07:32",
                "upload_time_iso_8601": "2018-08-06T21:07:32.343347Z",
                "url": "https://files.pythonhosted.org/packages/b6/2c/47e658fbc01420412727b7a29a2b194a1e2298159ac1557f3562292d1964/hypermax-0.1.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a5d2e7f524ae48af7d80ff11c2d0f35c",
                    "sha256": "d01bb1d825b9cbf8dbb57698e13dac5887f2cc3605ff77d179f12e1aa5e32527"
                },
                "downloads": -1,
                "filename": "hypermax-0.1.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a5d2e7f524ae48af7d80ff11c2d0f35c",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3",
                "size": 29897,
                "upload_time": "2018-08-07T05:06:41",
                "upload_time_iso_8601": "2018-08-07T05:06:41.155862Z",
                "url": "https://files.pythonhosted.org/packages/be/86/e7fab4f4f610326682d3b5526fdbfa65cbbbd1cb5a01a7df4170df67b724/hypermax-0.1.2-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f2c6883effca505d0e3c29868130a0ff",
                    "sha256": "3d7b7af2f75647d66aa20b548da150b0139918fc3950879ce9259dad72cd589d"
                },
                "downloads": -1,
                "filename": "hypermax-0.1.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "f2c6883effca505d0e3c29868130a0ff",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3",
                "size": 32620,
                "upload_time": "2018-08-07T15:03:24",
                "upload_time_iso_8601": "2018-08-07T15:03:24.581114Z",
                "url": "https://files.pythonhosted.org/packages/59/93/30218cd487f6f8dbb51141c5fa7417c98093021e48f897a7fd535e75d65d/hypermax-0.1.3-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "aed9f0f023515e447ff5b4b25665854e",
                    "sha256": "7b80a394216561bdaf4a180e03fdb549a74fffa37863e7751a19ba5dab45fea7"
                },
                "downloads": -1,
                "filename": "hypermax-0.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "aed9f0f023515e447ff5b4b25665854e",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3",
                "size": 36785,
                "upload_time": "2018-08-17T01:38:37",
                "upload_time_iso_8601": "2018-08-17T01:38:37.471984Z",
                "url": "https://files.pythonhosted.org/packages/29/31/9c22018cf24a17468e02caaa5ad4e8afc1781603ec844d99e48d1fca2909/hypermax-0.2-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f1dc945a1fef32d7f7de82403cae0064",
                    "sha256": "88544e10aaea2864f2a0aafb82812be523e2b62856e1f41dddd632c02bdb739b"
                },
                "downloads": -1,
                "filename": "hypermax-0.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "f1dc945a1fef32d7f7de82403cae0064",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3",
                "size": 864200,
                "upload_time": "2018-08-27T20:26:35",
                "upload_time_iso_8601": "2018-08-27T20:26:35.865132Z",
                "url": "https://files.pythonhosted.org/packages/d2/a7/07c941a81e2acff01ab750a99f6c5411a533bad0fa474a9b56d4611ce8aa/hypermax-0.3-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.3.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b9ce5cc874b096f22fa47aed1e4d20d8",
                    "sha256": "0a5ca6441e5110b734ef7658164f3089b7a98e8c75eea9aad99652ad8e18f3c1"
                },
                "downloads": -1,
                "filename": "hypermax-0.3.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "b9ce5cc874b096f22fa47aed1e4d20d8",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3",
                "size": 864216,
                "upload_time": "2018-08-27T21:19:51",
                "upload_time_iso_8601": "2018-08-27T21:19:51.032988Z",
                "url": "https://files.pythonhosted.org/packages/83/92/e0f4fced0eb6814617ea8fa949a5405bd34a071ef7edfba614142ff0677a/hypermax-0.3.1-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.3.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fc9f83eb9ba28f4f5a0cc749ed8c8da7",
                    "sha256": "afb8936dd821fe23d7bb09a0aac707eda51e98f453d81b1df51af6b48a4222e3"
                },
                "downloads": -1,
                "filename": "hypermax-0.3.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "fc9f83eb9ba28f4f5a0cc749ed8c8da7",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3",
                "size": 858270,
                "upload_time": "2018-10-13T17:14:58",
                "upload_time_iso_8601": "2018-10-13T17:14:58.388947Z",
                "url": "https://files.pythonhosted.org/packages/b2/d4/a76605a01193e134e7b9680f3d47f0527347e08fd48759eff6d5b52d76c9/hypermax-0.3.2-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "04a721114378419d6e72d709cded7877",
                    "sha256": "3028463eba498b2e17c032290aad4a71e9f8d551ed897f69e359afc6f0bd9d17"
                },
                "downloads": -1,
                "filename": "hypermax-0.4.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "04a721114378419d6e72d709cded7877",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3",
                "size": 859788,
                "upload_time": "2018-10-27T14:08:29",
                "upload_time_iso_8601": "2018-10-27T14:08:29.035332Z",
                "url": "https://files.pythonhosted.org/packages/26/9b/4e1fdfd8e7fd46713392d2d70f92f568eded059283763fa6ffda79e988ef/hypermax-0.4.0-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.4.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "482e710281e13709cbaaf7f88d9086a2",
                    "sha256": "552ad0735cbe2595843947f5cabac5ff4170be0550988a2b3b68cd05963544e2"
                },
                "downloads": -1,
                "filename": "hypermax-0.4.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "482e710281e13709cbaaf7f88d9086a2",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3",
                "size": 871102,
                "upload_time": "2018-10-27T22:05:00",
                "upload_time_iso_8601": "2018-10-27T22:05:00.786223Z",
                "url": "https://files.pythonhosted.org/packages/c8/2e/74984451fdb5ff2a4a52de0faebe8c4ff8beea32bf29086a003ac15bb1f5/hypermax-0.4.1-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.5.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3d8e14e05e6f94cc2da37d7959df5a4b",
                    "sha256": "d8c7e388a7ee09bf90328f95b09ac3f49ffed2d04ce71f9e504c698c5cb3ff32"
                },
                "downloads": -1,
                "filename": "hypermax-0.5.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "3d8e14e05e6f94cc2da37d7959df5a4b",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3",
                "size": 875887,
                "upload_time": "2019-01-18T03:43:39",
                "upload_time_iso_8601": "2019-01-18T03:43:39.342935Z",
                "url": "https://files.pythonhosted.org/packages/a1/30/3d6c6c4aa6fa9d7b9d538c5221895c8ff5fc62efba11be5ff7ed44aa0267/hypermax-0.5.0-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.5.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c0569c5735350bdd38363a0843e2494a",
                    "sha256": "5af8e9391199ad8362388d39897c2b98caa266278cb3ff41babebd81ccdd05a3"
                },
                "downloads": -1,
                "filename": "hypermax-0.5.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "c0569c5735350bdd38363a0843e2494a",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3",
                "size": 878384,
                "upload_time": "2019-10-23T15:40:12",
                "upload_time_iso_8601": "2019-10-23T15:40:12.043597Z",
                "url": "https://files.pythonhosted.org/packages/1a/20/637a4de5dabbd810755999da74eea0695696a583dd87095251a3d7c4ecaa/hypermax-0.5.1-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "c0569c5735350bdd38363a0843e2494a",
                "sha256": "5af8e9391199ad8362388d39897c2b98caa266278cb3ff41babebd81ccdd05a3"
            },
            "downloads": -1,
            "filename": "hypermax-0.5.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "c0569c5735350bdd38363a0843e2494a",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3",
            "size": 878384,
            "upload_time": "2019-10-23T15:40:12",
            "upload_time_iso_8601": "2019-10-23T15:40:12.043597Z",
            "url": "https://files.pythonhosted.org/packages/1a/20/637a4de5dabbd810755999da74eea0695696a583dd87095251a3d7c4ecaa/hypermax-0.5.1-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "vulnerabilities": []
}