{
    "info": {
        "author": "Nishant Sinha",
        "author_email": "nishant@offnote.co",
        "bugtrack_url": null,
        "classifiers": [
            "Environment :: Console",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: Apache Software License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3.7",
            "Topic :: Scientific/Engineering :: Artificial Intelligence",
            "Topic :: Software Development"
        ],
        "description": "![experimental](https://img.shields.io/badge/stability-experimental-orange.svg)\n\n\n## LightEx\n\n`LightEx` is a lightweight experiment framework to create, monitor and record your machine learning experiments. Targeted towards individual data scientists, researchers, small teams and, in general, resource-constrained experimentation. Compatible with all machine-learning frameworks.\n\n**Project Status:** Alpha\n\nUnlike most experiment frameworks, `LightEx`sports a modular, and highly configurable design:\n\n* **dispatcher**: run experiments using `process`,`docker` containers or `kubernetes` pods. Switch between modes seamlessly by minor changes to config.\n* **mulogger**: log metrics and artifacts to multiple logger backends, using an *unified* API. Supports `mlflow`, `tensorboard` and `trains` \u2014 add new loggers easily as *plugins*.\n* **namedconf**:  python `dataclass` based flexible and *unified* configuration specification for jobs, parameters and model architectures. Config instances are *named* and can be *locally modified*.\n* **qviz**: query, compare and visualize your experiment results.\n\nThe run environment and parameters for your experiments are specified using a config file `lxconfig.py` in your project directory. Modify, inherit, and create new *named* config instances, on-the-fly, as you perform a series of experiments. \n\nLearn more about the **anatomy** of a ML experimentation framework [here](docs/anatomy.md).\n\n#### Benefits\n\n Start with a basic `train` or `eval` project. In a few minutes,\n\n* introduce systematic logging (multiple loggers) and visualization to your project\n* go from running a single experiment to multiple parameterized experiments, e.g.,\n  * multiple training runs over a set of hyper-parameters.\n  * multiple `efficient-net` or `bert` train or eval runs.\n  * a neural architecture search over multiple architectures in parallel.\n\n\n### Installation\n\n>  pip install -U lightex\n\n### Quick Start\n\nAssume we have an existing `train` project: run trainer with \n\n> `train.py --data-dir ./data \u2014-lr 0.1 -\u2014hidden_dim 512` \n\nIn the main project directory, initialize `lightex` \u2014 this creates files `lxconfig.py` and `run_expts.py`.\n\n> `lx\u2014init`                               \n\nThe file `lxconfig.py` contains pre-defined `dataclass`es for specifying *named* experiment configs.\n\n* The main (controller) class `Config`, contains three fields: `er` , `hp` and `run` (see below). \n* `Config` also includes a `get_experiments` function, which generates a list of experiment configs to be executed by the dispatcher. See [config.md](docs/config.md) for full description of the defined dataclasses.\n\n```python\n\n@dataclass\nclass Config:\n    er: Resources \t\t\t\t\t#(Logger, Storage resources)\n    hp: HP \t\t\t\t\t\t\t#(Hyper-parameters of model, training)\n    run: Run \t\t\t\t\t\t#(Run-time config)\n\n    def get_experiments(self): #required: generate a list of experiments to run\n        expts = [Experiment(er=self.er, hp=self.hp, run=self.run)]\n        return expts\n```\n\nInstantiate class `HP` with actual parameters, and class `Run` to mimic the command with placeholders.\n\n```python\ncmd=\"python train.py --data-dir {{run.data_dir}} --lr {{hp.lr}} --hidden_dim {{hp.hidden_dim}}\" #placeholders refer to fields of Experiment instance\nRu1 = Run(cmd=cmd, experiment_name=\"find_hp\")\nH2 = HP(lr=1e-2, hidden_dim=512)\n\nC1 = Config(er=R1, hp=H1, run=Ru1) #er defined elsewhere\n```\n\nOnce config named `C1` is defined, run your experiments as follows:\n\n> python run_expts.py -c C1\n\n**That's it!** Now, your experiments, logs, metrics and models are organized and recorded systematically.\n\n#### Modify Experiment Parameters, Experiment Groups\n\nModify configs from previous experiments quickly using `replace` and run new experiments. \n\nExample: Create a new `HP` instance and replace it in `C1` to create a new `Config`. Recursive replace also supported.\n\n```python\nH2 = HP(lr=1e-3, hidden_dim=1024)\nC2 = replace(C1, hp=H2) #inherit er=R1 and run=Ru1\n```\n\n> python run_expts.py -c C2\n\nTo specify and run **experiment groups**, specify a set of `HP`s in a `HPGroup` (see [scripts/lxconfig.py](scripts/lxconfig.py)).\n\n**Note**: Although LightEx pre-defines the dataclass hierarchy, it allows the developer plenty of flexibility in defining the individual fields of classes, in particular, the fields of the `HP` class. \n\n#### Adding Logging to your Code\n\nUse the unified `MultiLogger` [API](lightex/mulogger) to log metrics and artifacts to multiple logger backends.\n\n```python\nfrom lightex.mulogger import MLFlowLogger, MultiLogger, PytorchTBLogger\nlogger = MultiLogger(['mlflow', 'trains'])\nlogger.start_run()\n# log to trains only\nlogger.log('trains', ltype='hpdict', value={'alpha': alpha, 'l1_ratio': l1_ratio})\n# log to mlflow only\nlogger.log('mlflow', ltype='scalardict', value={'mae': mae, 'rmse': rmse, 'r2': r2}, step=1)\n# log to all\nlogger.log('*', ltype='scalardict', value={'mae': mae, 'rmse': rmse, 'r2': r2}, step=3)\n# log scalars and tensors, if supported by the logger backend\nlogger.log('trains', ltype='1d', name='W1', value=Tensor(..), histogram=True, step=4)\nlogger.end_run()\n```\n\nOr, use one of the existing loggers' API directly.\n\n```python\nlogger = MLFlowLogger()\nmlflow = logger.mlflow\n# call mlflow API\n\nlogger = PytorchTBLogger()\nwriter = logger.writer\n#call tensorboard's API\n```\n\n**Note**: Except for changes in logging, no changes are required to your existing code!\n\n\n#### Switch to Docker\n\nSetting up the `lxconfig` instances pays off here! \n\nNow, add a `Dockerfile` to your project which builds the runtime environment with all the project dependencies. Update the `Build` instance inside `Resources` config. See [examples/sklearn](examples/sklearn), for example.\n\n> python run_expts.py -c C2 -e docker\n\nBoth your code and data are mounted on the container (no copying involved) \u2014 minimal disruption in your dev cycle.\n\n#### Advanced Features\n\nMore advanced features are in development stage.\n\n**Modifying, Adding Loggers**\n\n```python\nLm = MLFlowConfig(client_in_cluster=False, port=5000)\nL = LoggerConfig(mlflow=Lm)\nfrom lightex.mulogger.trains_logger import TrainsConfig\nL.register_logger('trains', TrainsConfig())\n\nR1 = Resources(build=..., storage=..., ctr=..., loggers=L)\n```\n\nMore loggers and a better plugin system being developed.\n\n**Running Experiments on multiple nodes / servers**\n\nIf you've setup a docker `swarm` or `kubernetes` cluster, few changes to the existing config instance allow changing the underlying experiment dispatcher.\n\nWe need to virtualize code (by adding to Dockerfile) and storage.\n\nCreate a shared NFS on your nodes. Switch storage config to the NFS partition. Setup scripts will be added.\n\n**Setup Summary**\n\nIn summary, `LightEx` involves the following **one-time setup**:\n\n- config values in `lxconfig.py`\n- Setup backend logger servers (only the ones required). Instructions [here](backend/). (Optional)\n- Update logging calls in your code to call `mulogger` API. (Optional)\n- Dockerfile for your project, if you want to use containers for dispatch. (Optional)\n\nWhile `LightEx` is quick to start with, it is advisable to spend some time understanding the [config schema](llightex/config_base.py).\n\n\n\n### Dependencies\n\nPython > 3.6 (require `dataclasses`, included during install). \n\n\n\n### Design Challenges\n\n- A significant portion of experiment manager design is about setting up and propagating a giant web of configuration variables. \n  - No optimal choice here: `json`,`yaml`,`jsonnet`\u2014 all formats have issues. \n  - Using `dataclass`es, we can write complex config specs, with built-in inheritance and ability to do *local updates*. Tiny bit of a learning curve here, bound to python, but we gain a lot of flexibility.\n- A unified `mulogger` API to abstract away the API of multiple logging backends.\n- Designing multiple dispatchers with similar API, enabling containers and varying storage options.\n- Read more on challenges [here](docs/anatomy.md).\n\n### References\n\n- ML Experiment Frameworks: [kubeflow](https://github.com/kubeflow/kubeflow), [mlflow](https://www.mlflow.org/docs/latest/index.html), [polyaxon](https://polyaxon.com/), ...\n- Loggers: [sacred](https://sacred.readthedocs.io/en/latest/index.html), [trains](https://github.com/allegroai/trains), [Trixi](https://github.com/MIC-DKFZ/trixi), [ml_logger](https://github.com/episodeyang/ml_logger)\n- Motivating Dataclasses [intro](https://blog.jetbrains.com/pycharm/2018/04/python-37-introducing-data-class/), [how-different](https://stackoverflow.com/questions/47955263/what-are-data-classes-and-how-are-they-different-from-common-classes)\n- Flexible configuration\n  - in modeling: allennlp, gin, jiant.\n  - in orchestration: [ksonnet](https://github.com/ksonnet), kubernetes-operator \n- On the pains of ML experimentation\n  - an article from [wandb](https://www.wandb.com/articles/iteratively-fine-tuning-neural-networks-with-weights-biases) \n\nMost current (July 2019 end) tools focus on the *logger* component and provide selective `qviz` components. `kubeflow` and `polyaxon` are tied to the (k8s) *dispatcher*. Every tool has its own version of config management \u2014 mostly *yaml* based, where config types are absent or have a non-nested config class. Config-specific languages have been also proposed (ksonnet, sonnet, gin).\n\n\n\n### Yet another experiment framework?\n\nSystematic experimentation tools are essential for a data scientist. Unfortunately, many existing tools (`kubeflow`, `mlflow`, `polyaxon`) are too monolithic, kubernetes-first, cloud-first, target very diverse audiences and hence spread too thin, and yet lack important dev-friendly features. `sacred` 's design is' tightly coupled and requires several `sacred`-specific changes to your main code (plan to add `sacred`logger as backend). Other tools cater only to a specific task , e.g., `tensorboard` only handles log recording and visualization. Also, contrasting different experiment frameworks is hard: there is no standardized expt-management architecture for machine learning and most open-source frameworks are undergoing a process of adhoc requirements discovery. \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/ofnote/lightex",
        "keywords": "",
        "license": "Apache 2.0",
        "maintainer": "",
        "maintainer_email": "",
        "name": "lightex",
        "package_url": "https://pypi.org/project/lightex/",
        "platform": "POSIX",
        "project_url": "https://pypi.org/project/lightex/",
        "project_urls": {
            "Homepage": "https://github.com/ofnote/lightex"
        },
        "release_url": "https://pypi.org/project/lightex/0.0.4/",
        "requires_dist": [
            "easydict",
            "dacite",
            "kubernetes",
            "docker",
            "dataclasses; python_version < \"3.7\""
        ],
        "requires_python": ">3.6.0",
        "summary": "LightEx: A Light Experiment Manager",
        "version": "0.0.4"
    },
    "last_serial": 5617120,
    "releases": {
        "0.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "30d14c6023ce4bb6dd9a9ab72af5c699",
                    "sha256": "2ce9ecd261d688bf3e279844e92b5d3e0e386f01b2ecdeb7532c6009b54356f1"
                },
                "downloads": -1,
                "filename": "lightex-0.0.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "30d14c6023ce4bb6dd9a9ab72af5c699",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 13379,
                "upload_time": "2019-06-21T20:07:23",
                "url": "https://files.pythonhosted.org/packages/d8/45/595c95af5e6acef5d02da1471cfd3bbb5947ad68ee2eb4cb076c71601657/lightex-0.0.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "472f7e941bd62e96f1fdc49729c760f3",
                    "sha256": "28208a0b208b8ef1d1e87298b85f47b839850728f5ff948a1e6111b48014c300"
                },
                "downloads": -1,
                "filename": "lightex-0.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "472f7e941bd62e96f1fdc49729c760f3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7837,
                "upload_time": "2019-06-21T20:07:26",
                "url": "https://files.pythonhosted.org/packages/a3/35/cdf54aeed9ffb6798fb1db715c001ccd31dabd49f532993ecd5cff895b15/lightex-0.0.2.tar.gz"
            }
        ],
        "0.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c4eb847cafd05fb24bcb29cf43619b5f",
                    "sha256": "8ac6c33a9ba6d8f5efc1fcd9b3c932106467092f69a25cff250e65c0d04f244f"
                },
                "downloads": -1,
                "filename": "lightex-0.0.4-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "c4eb847cafd05fb24bcb29cf43619b5f",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">3.6.0",
                "size": 28652,
                "upload_time": "2019-08-01T06:50:52",
                "url": "https://files.pythonhosted.org/packages/6d/76/18cf72c6265ea7aef0915a99caaaf7497bbd66135f4eb2f9ae391a2a00a8/lightex-0.0.4-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d916583a6c64c75ec6712b67a70feb3a",
                    "sha256": "d429b6e2d3e38c5f1eb917ba476fdd5bda1c4998bfdd94d518f1d930fee02c2a"
                },
                "downloads": -1,
                "filename": "lightex-0.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "d916583a6c64c75ec6712b67a70feb3a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">3.6.0",
                "size": 23544,
                "upload_time": "2019-08-01T06:50:54",
                "url": "https://files.pythonhosted.org/packages/ca/37/70486a3e97a9a121e2b4d15cbda13d2679e7d240510f45570c5dcd2b8f86/lightex-0.0.4.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "c4eb847cafd05fb24bcb29cf43619b5f",
                "sha256": "8ac6c33a9ba6d8f5efc1fcd9b3c932106467092f69a25cff250e65c0d04f244f"
            },
            "downloads": -1,
            "filename": "lightex-0.0.4-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "c4eb847cafd05fb24bcb29cf43619b5f",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">3.6.0",
            "size": 28652,
            "upload_time": "2019-08-01T06:50:52",
            "url": "https://files.pythonhosted.org/packages/6d/76/18cf72c6265ea7aef0915a99caaaf7497bbd66135f4eb2f9ae391a2a00a8/lightex-0.0.4-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "d916583a6c64c75ec6712b67a70feb3a",
                "sha256": "d429b6e2d3e38c5f1eb917ba476fdd5bda1c4998bfdd94d518f1d930fee02c2a"
            },
            "downloads": -1,
            "filename": "lightex-0.0.4.tar.gz",
            "has_sig": false,
            "md5_digest": "d916583a6c64c75ec6712b67a70feb3a",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">3.6.0",
            "size": 23544,
            "upload_time": "2019-08-01T06:50:54",
            "url": "https://files.pythonhosted.org/packages/ca/37/70486a3e97a9a121e2b4d15cbda13d2679e7d240510f45570c5dcd2b8f86/lightex-0.0.4.tar.gz"
        }
    ]
}