{
    "info": {
        "author": "Alex Gajewski",
        "author_email": "agajews@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3"
        ],
        "description": "# EvoGrad\n\nEvoGrad is a lightweight tool for differentiating through expectation,\nbuilt on top of PyTorch.\n\nTools that enable fast and flexible experimentation democratize and accelerate machine learning research.\nHowever, one field that so far has not been greatly impacted by automatic differentiation tools is evolutionary computation\nThe reason is that most evolutionary algorithms are gradient-free:\nthey do not follow any explicit mathematical gradient (i.e., the mathematically optimal local direction of improvement), and instead proceed through a generate-and-test heuristic.\nIn other words, they create new variants, test them out, and keep the best.\n\nRecent and exciting research in evolutionary algorithms for deep reinforcement learning, however, has highlighted how a specific class of evolutionary algorithms can benefit from auto-differentiation.\nWork from OpenAI demonstrated that a form of Natural Evolution Strategies (NES) is massively scalable, and competitive with modern deep reinforcement learning algorithms.\n\nEvoGrad enables fast prototyping of NES-like algorithms.\nWe believe there are many interesting algorithms yet to be discovered in this vein, and we hope this library will help to catalyze progress in the machine learning community.\n\n## Examples\n### Natural Evolution Strategies\nAs a first example, we\u2019ll implement the simplified NES algorithm of [Salimans et al. (2017)](https://openai.com/blog/evolution-strategies/) in EvoGrad.\nEvoGrad provides several probability distributions which may be used in the expectation function.\nWe will use a normal distribution because it is the most common choice in practice.\n\nLet\u2019s consider the problem of finding a fitness peak in a simple 1-D search space.\n\nWe can define our population distribution over this search space to be initially centered at 1.0, with a fixed variance of 0.05, with the following Python code:\n\n```python\nmu = torch.tensor([1.0], requires_grad=True)\np = Normal(mu, 0.05)\n```\n\nNext, let\u2019s define a simple fitness function that rewards individuals for approaching the location 5.0 in the search space:\n\n```python\ndef fitness(xs):\n\treturn -(x - 5.0) ** 2\n```\n\nEach generation of evolution in NES takes samples from the population distribution and evaluates the fitness of each of those individual samples. Here we sample and evaluate 100 individuals from the distribution:\n\n```python\nsample = p.sample(n=100)\nfitnesses = fitness(sample)\n```\n\nOptionally, we can apply a [whitening transformation](https://en.wikipedia.org/wiki/Whitening_transformation) to the fitnesses (a form of pre-processing that often increases NES performance), like this:\n\n```python\nfitnesses = (fitnesses - fitnesses.mean()) / fitnesses.std()\n```\n\nNow we can use these calculated fitness values to estimate the mean fitness over our population distribution:\n\n```python\nmean = expectation(fitnesses, sample, p=p)\n```\n\nAlthough we could have estimated the mean value directly with the snippet:\n`mean = fitnesses.mean()`, what we gain by instead using the EvoGrad `expectation` function is the ability to backpropagate through mean.\nWe can then use the resulting auto-differentiated gradients to optimize the center of the 1D Gaussian population distribution (mu) through gradient descent (here, to increase the expected fitness value of the population):\n\n```python\nmean.backward()\nwith torch.no_grad():\n\tmu += alpha * mu.grad\n\tmu.grad.zero_()\n```\n\n### Maximizing Variance\nAs a more sophisticated example, rather than maximizing the mean fitness, we can maximize the variance of behaviors in the population.\nWhile fitness is a measure of quality for a fixed task, in some situations we want to prepare for the unknown, and instead might want our population to contain a diversity of behaviors that can easily be adapted to solve a wide range of possible future tasks.\n\nTo do so, we need a quantification of behavior, which we can call a behavior characterization. Similarly to how you can evaluate an individual parameter vector drawn from the population distribution to establish its fitness (e.g. how far does this controller cause a robot to walk?), you could evaluate such a draw and return some quantification of its behavior (e.g., what position does a robot controlled by this neural network locomote to?).\n\nFor this example, let\u2019s choose a simple but illustrative, 1D behavior characterization, namely, the product of two sine waves (one with much faster frequency than the other):\n\n```python\ndef behavior(x):\n\treturn 5 * torch.sin(0.2 * x) * torch.sin(20 * x)\n```\n\nNow, instead of estimating the mean fitness, we can calculate a statistic that reflects the diversity of sampled behaviors. The variance of a distribution is one metric of diversity, and one variant of evolvability ES measures and optimizes such variance of behaviors sampled from the population distribution:\n\n```python\nsample = p.sample(n=100)\nbehaviors = behavior(sample)\nzscore = (behaviors - behaviors.mean()) / behaviors.std()\nvariance = expectation(zscore ** 2, sample, p=p)\n```\n\n### Maximizing Entropy\nIn the previous example, the gradient would be relatively straightforward to compute by hand.\nHowever, sometimes we may need to maximize objectives whose derivatives would be much more challenging to derive.\nIn particular, this final example will seek to maximize the entropy of the distribution of behaviors (a variant of evolvability ES).\n\nNote that for this example you'll also have to install `scipy` from pip.\n\nTo create a differentiable estimate of entropy, we first compute the pairwise distances between the different behaviors.\nNext, we create a smooth probability distribution by fitting a [kernel density estimate](https://en.wikipedia.org/wiki/Kernel_density_estimation):\n\n```python\ndists = scipy.spatial.distance.squareform(scipy.spatial.distance.pdist(behaviors, \"sqeuclidean\"))\nkernel = torch.tensor(scipy.exp(-dists / k_sigma ** 2), dtype=torch.float32)\np_x = expectation(kernel, sample, p=p, dim=1)\n```\n\nThen, we can use these probabilities to estimate the [entropy of the distribution](https://en.wikipedia.org/wiki/Entropy_(information_theory)), and run gradient descent on it as before:\n\n```python\nentropy = expectation(-torch.log(p_x), sample, p=p)\n```\n\nFull code for these examples can be found in the `demos` directory of\nthis repository.\n\n## Installation\nEither install EvoGrad from pip:\n```\npip install evograd\n```\nOr from the source code in this repository:\n```\ngit clone github.com/uber-research/EvoGrad\ncd EvoGrad\npip install -r requirements.txt\npip install -e .\n```\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/uber-research/EvoGrad",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "evograd",
        "package_url": "https://pypi.org/project/evograd/",
        "platform": "",
        "project_url": "https://pypi.org/project/evograd/",
        "project_urls": {
            "Homepage": "https://github.com/uber-research/EvoGrad"
        },
        "release_url": "https://pypi.org/project/evograd/0.1.2/",
        "requires_dist": [
            "numpy",
            "torch"
        ],
        "requires_python": "",
        "summary": "A lightweight tool for differentiating through expectations",
        "version": "0.1.2"
    },
    "last_serial": 5832775,
    "releases": {
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "33635a52b7a65d99d2bb0e038fb2d556",
                    "sha256": "1d79b6e5b4c2fbb17b378b89a83326f4937bc0e9122bd2355e3ac3ad9dba29a8"
                },
                "downloads": -1,
                "filename": "evograd-0.1.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "33635a52b7a65d99d2bb0e038fb2d556",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 9338,
                "upload_time": "2019-09-15T18:39:50",
                "url": "https://files.pythonhosted.org/packages/8d/8d/853a4cd56fe43e8d32d7406429a3a4c0684e53fc9c9375ce6b37ee6a5ccf/evograd-0.1.1-py3-none-any.whl"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4cbbe9253713c8326e50606e83c6a049",
                    "sha256": "2aba49e465c684bfa68422b2c57f5aaf95bc2f038e986ab0d97bb3117baa3b6b"
                },
                "downloads": -1,
                "filename": "evograd-0.1.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "4cbbe9253713c8326e50606e83c6a049",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 9347,
                "upload_time": "2019-09-15T18:41:12",
                "url": "https://files.pythonhosted.org/packages/7a/c4/9dfd838c3e1e0e263a3f1fd516f88af883d31a65bd117a016c0e858b4ea3/evograd-0.1.2-py3-none-any.whl"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "4cbbe9253713c8326e50606e83c6a049",
                "sha256": "2aba49e465c684bfa68422b2c57f5aaf95bc2f038e986ab0d97bb3117baa3b6b"
            },
            "downloads": -1,
            "filename": "evograd-0.1.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "4cbbe9253713c8326e50606e83c6a049",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 9347,
            "upload_time": "2019-09-15T18:41:12",
            "url": "https://files.pythonhosted.org/packages/7a/c4/9dfd838c3e1e0e263a3f1fd516f88af883d31a65bd117a016c0e858b4ea3/evograd-0.1.2-py3-none-any.whl"
        }
    ]
}