{
    "info": {
        "author": "David Silva",
        "author_email": "davidtvs10@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Intended Audience :: Developers",
            "Intended Audience :: Education",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python",
            "Topic :: Scientific/Engineering :: Artificial Intelligence",
            "Topic :: Software Development"
        ],
        "description": "# PyTorch learning rate finder\n\nA PyTorch implementation of the learning rate range test detailed in [Cyclical Learning Rates for Training Neural Networks](https://arxiv.org/abs/1506.01186) by Leslie N. Smith and the tweaked version used by [fastai](https://github.com/fastai/fastai).\n\nThe learning rate range test is a test that provides valuable information about the optimal learning rate. During a pre-training run, the learning rate is increased linearly or exponentially between two boundaries. The low initial learning rate allows the network to start converging and as the learning rate is increased it will eventually be too large and the network will diverge.\n\nTypically, a good static learning rate can be found half-way on the descending loss curve. In the plot below that would be `lr = 0.002`.\n\nFor cyclical learning rates (also detailed in Leslie Smith's paper) where the learning rate is cycled between two boundaries `(base_lr, max_lr)`, the author advises the point at which the loss starts descending and the point at which the loss stops descending or becomes ragged for `base_lr` and `max_lr` respectively.  In the plot below, `base_lr = 0.0002` and `max_lr=0.2`.\n\n![Learning rate range test](images/lr_finder_cifar10.png)\n\n\n## Installation\n\nPython 2.7 and above:\n``pip install torch-lr-finder``\n\n## Implementation details and usage\n\n### Tweaked version from fastai\n\nIncreases the learning rate in an exponential manner and computes the training loss for each learning rate. `lr_finder.plot()` plots the training loss versus logarithmic learning rate.\n\n```python\nfrom torch_lr_finder import LRFinder\n\nmodel = ...\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(model.parameters(), lr=1e-7, weight_decay=1e-2)\nlr_finder = LRFinder(model, optimizer, criterion, device=\"cuda\")\nlr_finder.range_test(trainloader, end_lr=100, num_iter=100)\nlr_finder.plot()\n```\n\n### Leslie Smith's approach\n\nIncreases the learning rate linearly and computes the evaluation loss for each learning rate. `lr_finder.plot()` plots the evaluation loss versus learning rate.\nThis approach typically produces more precise curves because the evaluation loss is more susceptible to divergence but it takes significantly longer to perform the test, especially if the evaluation dataset is large.\n\n```python\nfrom torch_lr_finder import LRFinder\n\nmodel = ...\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(model.parameters(), lr=0.1, weight_decay=1e-2)\nlr_finder = LRFinder(model, optimizer, criterion, device=\"cuda\")\nlr_finder.range_test(trainloader, end_lr=1, num_iter=100, step_mode=\"linear\")\nlr_finder.plot(log_lr=False)\n```\n\n### Notes\n\n- Examples for CIFAR10 and MNIST can be found in the examples folder.\n- `LRFinder.range_test()` will change the model weights and the optimizer parameters. Both can be restored to their initial state with `LRFinder.reset()`.\n- The learning rate and loss history can be accessed through `lr_finder.history`. This will return a dictionary with `lr` and `loss` keys.\n- When using `step_mode=\"linear\"` the learning rate range should be within the same order of magnitude.\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/davidtvs/pytorch-lr-finder",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "torch-lr-finder",
        "package_url": "https://pypi.org/project/torch-lr-finder/",
        "platform": "",
        "project_url": "https://pypi.org/project/torch-lr-finder/",
        "project_urls": {
            "Homepage": "https://github.com/davidtvs/pytorch-lr-finder"
        },
        "release_url": "https://pypi.org/project/torch-lr-finder/0.0.1/",
        "requires_dist": [
            "matplotlib",
            "numpy",
            "torch (>=0.4.1)",
            "tqdm"
        ],
        "requires_python": ">=2.7",
        "summary": "Pytorch implementation of the learning rate range test",
        "version": "0.0.1"
    },
    "last_serial": 5830188,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "134a574f9dab12ebc005ddff441694ec",
                    "sha256": "7f690a2e093b10c0e1935e013572c4be7225820c5be40af56139a6c28626db9e"
                },
                "downloads": -1,
                "filename": "torch_lr_finder-0.0.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "134a574f9dab12ebc005ddff441694ec",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=2.7",
                "size": 7397,
                "upload_time": "2019-09-14T20:48:59",
                "url": "https://files.pythonhosted.org/packages/62/7c/397078ef7ca83880a35037285fd9d1f0e70ab5414db3a6f81e868c7230e4/torch_lr_finder-0.0.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "f5f590e69484e6fb07314582669ea875",
                    "sha256": "3d1f91f0232f069325b456348b97d7936921f77393cef10e8253b3cb7cc2932d"
                },
                "downloads": -1,
                "filename": "torch-lr-finder-0.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "f5f590e69484e6fb07314582669ea875",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7",
                "size": 5973,
                "upload_time": "2019-09-14T20:49:02",
                "url": "https://files.pythonhosted.org/packages/5d/cd/91f910b0b05cb72ad66db3b8a82b86f8e1ab4241398eef8dc4dfd033a7eb/torch-lr-finder-0.0.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "134a574f9dab12ebc005ddff441694ec",
                "sha256": "7f690a2e093b10c0e1935e013572c4be7225820c5be40af56139a6c28626db9e"
            },
            "downloads": -1,
            "filename": "torch_lr_finder-0.0.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "134a574f9dab12ebc005ddff441694ec",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=2.7",
            "size": 7397,
            "upload_time": "2019-09-14T20:48:59",
            "url": "https://files.pythonhosted.org/packages/62/7c/397078ef7ca83880a35037285fd9d1f0e70ab5414db3a6f81e868c7230e4/torch_lr_finder-0.0.1-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "f5f590e69484e6fb07314582669ea875",
                "sha256": "3d1f91f0232f069325b456348b97d7936921f77393cef10e8253b3cb7cc2932d"
            },
            "downloads": -1,
            "filename": "torch-lr-finder-0.0.1.tar.gz",
            "has_sig": false,
            "md5_digest": "f5f590e69484e6fb07314582669ea875",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=2.7",
            "size": 5973,
            "upload_time": "2019-09-14T20:49:02",
            "url": "https://files.pythonhosted.org/packages/5d/cd/91f910b0b05cb72ad66db3b8a82b86f8e1ab4241398eef8dc4dfd033a7eb/torch-lr-finder-0.0.1.tar.gz"
        }
    ]
}