{
    "info": {
        "author": "Kakao Brain",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 3 - Alpha",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: Apache Software License",
            "Operating System :: POSIX :: Linux",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7",
            "Programming Language :: Python :: 3 :: Only",
            "Topic :: Scientific/Engineering :: Artificial Intelligence"
        ],
        "description": "# torchlars\n\n[![PyPI](https://img.shields.io/pypi/v/torchlars.svg)](https://pypi.org/project/torchlars)\n[![Build Status](https://travis-ci.org/kakaobrain/torchlars.svg?branch=master)](https://travis-ci.org/kakaobrain/torchlars)\n\nA [LARS](https://arxiv.org/abs/1708.03888) implementation in PyTorch.\n\n```python\nfrom torchlars import LARS\noptimizer = LARS(optim.SGD(model.parameters(), lr=0.1))\n```\n\n## What is LARS?\n\nLARS (Layer-wise Adaptive Rate Scaling) is an optimization algorithm designed\nfor large-batch training published by You, Gitman, and Ginsburg, which\ncalculates the local learning rate per layer at each optimization step.\nAccording to the paper, when training ResNet-50 on ImageNet ILSVRC (2016)\nclassification task with LARS, the learning curve and the best top-1 accuracy\nstay similar to that of the baseline (training with batch size 256 without\nLARS) even if the batch size is scaled up to 32K.\n\n[Large Batch Training of Convolutional Networks](https://arxiv.org/abs/1708.03888)\n\nOriginally, LARS is formulated in terms of SGD optimizer and extension to other\noptimizers was not mentioned in the paper. In contrast, `torchlars` implements\nLARS as a wrapper which can take any optimizers including SGD as the base.\n\nAdditionally, LARS of torchlars is designed to more consider operation in the\nCUDA environment compared to existing implementations. Thanks to this, you can\nsee only the little speed loss appears compared to just using SGD, in an\nenvironment where CPU to GPU synchronization does not occur.\n\n## Usage\n\nCurrently, torchlars requires the following environments:\n\n- Linux\n- Python 3.6+\n- PyTorch 1.1+\n- CUDA 10+\n\nTo use torchlars, install it via PyPI:\n\n```bash\n$ pip install torchlars\n```\n\nTo use LARS, simply wrap your base optimizer with `torchlars.LARS`. LARS\ninherits `torch.optim.Optimizer`, so you can simply use LARS as optimizer on\nyour code. After then, when you call step method of LARS, LARS automatically\ncalculates local learning rate before running base optimizer such as SGD or\nAdam\n\nThe example code below shows how to use LARS using SGD as base optimizer.\n\n```python\nfrom torchlars import LARS\n\nbase_optimizer = optim.SGD(model.parameters(), lr=0.1)\noptimizer = LARS(optimizer=base_optimizer, eps=1e-8, trust_coef=0.001)\n\noutput = model(input)\nloss = loss_fn(output, target)\nloss.backward()\n\noptimizer.step()\n```\n\n## Benchmarking\n\n### ResNet-50 on ImageNet classification\n\n| Batch Size | LR policy    | lr   | warm-up | epoch | Best Top-1 accuracy, % |\n| :--------- | :----------- | :--- | :------ | :---- | :--------------------- |\n| 256        | poly(2)      | 0.2  | N/A     | 90    | 73.79                  |\n| 8k         | LARS+poly(2) | 12.8 | 5       | 90    | 73.78                  |\n| 16K        | LARS+poly(2) | 25.0 | 5       | 90    | 73.36                  |\n| 32K        | LARS+poly(2) | 29.0 | 5       | 90    | 72.26                  |\n\n![](img/resnet50_test_learning_curves.jpg)\n\nAbove image and table show the reproduced performance benchmark on ResNet-50,\nas reported in Table 4 and Figure 5 of the paper.\n\nThe cyan line represents the baseline result, which is training result with\nbatch size 256, and others represent training result of 8K, 16K, 32K\nrespectively. As you see, every result shows a similar learning curve and best\ntop-1 accuracy.\n\nMost experimental conditions are similar to used in the paper, but we slightly\nchange some conditions like learning rate to observe comparable results as\nproposed by the LARS paper.\n\n_Note: We refer\n[log file](https://people.eecs.berkeley.edu/~youyang/publications/batch/)\nprovided by paper to obtain above hyper-parameters._\n\n## Authors and Licensing\n\ntorchlars project is developed by [Chunmyong Park] at [Kakao Brain], with\n[Heungsub Lee], [Myungryong Jeong], [Woonhyuk Baek], and [Chiheon Kim]'s help.\nIt is distributed under [Apache License 2.0](LICENSE).\n\n[chiheon kim]: https://github.com/chiheonk\n[chunmyong park]: https://github.com/cmpark0126\n[heungsub lee]: https://subl.ee/\n[kakao brain]: https://kakaobrain.com/\n[woonhyuk baek]: https://github.com/wbaek\n[myungryong jeong]: https://github.com/mrJeong",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "",
        "keywords": "",
        "license": "",
        "maintainer": "Chunmyong Park",
        "maintainer_email": "",
        "name": "torchlars",
        "package_url": "https://pypi.org/project/torchlars/",
        "platform": "",
        "project_url": "https://pypi.org/project/torchlars/",
        "project_urls": null,
        "release_url": "https://pypi.org/project/torchlars/0.1.0/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "A LARS implementation in PyTorch",
        "version": "0.1.0"
    },
    "last_serial": 5938144,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5c575625d470b9616685514804f7559b",
                    "sha256": "99b7b46ac4957e7268e4eda35b52867a0abc4d8ecb15a6445c9c0932d6daed92"
                },
                "downloads": -1,
                "filename": "torchlars-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "5c575625d470b9616685514804f7559b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6219,
                "upload_time": "2019-10-07T10:50:01",
                "url": "https://files.pythonhosted.org/packages/88/34/65e3fbf961665b4a3407c1bd5446b6aa53c53161f19a2dc2d4e84b6b8a17/torchlars-0.1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "5c575625d470b9616685514804f7559b",
                "sha256": "99b7b46ac4957e7268e4eda35b52867a0abc4d8ecb15a6445c9c0932d6daed92"
            },
            "downloads": -1,
            "filename": "torchlars-0.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "5c575625d470b9616685514804f7559b",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 6219,
            "upload_time": "2019-10-07T10:50:01",
            "url": "https://files.pythonhosted.org/packages/88/34/65e3fbf961665b4a3407c1bd5446b6aa53c53161f19a2dc2d4e84b6b8a17/torchlars-0.1.0.tar.gz"
        }
    ]
}