{
    "info": {
        "author": "Vitaliy Grabovets",
        "author_email": "v.grabovets@gmail.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# Benchmarkit\n\n[![PyPI version](https://badge.fury.io/py/benchmarkit.svg)](https://badge.fury.io/py/benchmarkit)\n[![Build Status](https://travis-ci.org/vgrabovets/benchmarkit.svg?branch=master)](https://travis-ci.org/vgrabovets/benchmarkit)\n[![codecov](https://codecov.io/gh/vgrabovets/benchmarkit/branch/master/graph/badge.svg)](https://codecov.io/gh/vgrabovets/benchmarkit)\n[![CodeFactor](https://www.codefactor.io/repository/github/vgrabovets/benchmarkit/badge)](https://www.codefactor.io/repository/github/vgrabovets/benchmarkit)\n[![Dependabot Status](https://api.dependabot.com/badges/status?host=github&repo=vgrabovets/benchmarkit)](https://dependabot.com)\n\nBenchmark and analyze functions' time execution and results over the course of development. \n\n## Features\n\n- No boilerplate code\n- Saves history and additional info\n- Saves function output and parameters to benchmark data science tasks\n- Easy to analyze results\n- Disables garbage collector during benchmarking\n\n## Motivation\n\n- I need to benchmark execution time of my function\n- I don't want to memorize and write boilerplate code\n- I want to compare results with previous runs before some changes were introduced\n- I don't want to manually write down results somewhere\n- I want to know exact commits of my previous runs months ago\n- I want to benchmark accuracy, precision, recall of my models and keep track of hyperparameters \n\n## Installation\n\n```text\npip install benchmarkit\n```\n\n## Usage\n### Benchmark execution times\n\nPut `@benchmark` decorator over function with piece of code that should be timed\n\n```python\nfrom benchmarkit import benchmark, benchmark_run\n\nN = 10000\nseq_list = list(range(N))\nseq_set = set(range(N))\n\nSAVE_PATH = '/tmp/benchmark_time.jsonl'\n\n\n@benchmark(num_iters=100, save_params=True, save_output=False)\ndef search_in_list(num_items=N):\n    return num_items - 1 in seq_list\n\n\n@benchmark(num_iters=100, save_params=True, save_output=False)\ndef search_in_set(num_items=N):\n    return num_items - 1 in seq_set\n```\n\n- __num_iters__ - how many times to repeat benchmarked function. Default _1_\n- __save_params__ - save parameters passed to the benchmarked function in the file with benchmark results. In the example above `num_items` will be saved. Default _False_\n- __save_output__ - save benchmarked function output. Should return dict `{'name': value}`. Default _False_. See example how to benchmark model results.\n\nRun benchmark:\n\n```python\nbenchmark_results = benchmark_run(\n    [search_in_list, search_in_set],\n    SAVE_PATH,\n    comment='initial benchmark search',\n    rows_limit=10,\n    extra_fields=['num_items'],\n    metric='mean_time',\n    bigger_is_better=False,\n)  \n```\n\n- __functions__ - function or list of functions with `benchmark` decorator\n- __save_file__ - path to file where to save results\n- __comment__ - comment to save alongside the results\n- __rows_limit__ - limit table rows in console output. Default _10_\n- __extra_fields__ - extra fields to include in console output\n- __metric__ - metric which is used for comparison. Default `mean_time`\n- __bigger_is_better__ - whether bigger value of metric indicates that result is better. For time benchmarks should be `False`, for model accuracy should be `True`. Default _False_\n\nPrints to terminal and returns list of dictionaries with data for the last run.\n\n![Benchmark time output1](https://raw.githubusercontent.com/vgrabovets/benchmarkit/master/img/benchmark_time1.jpg)\n\nChange `N=1000000` and rerun\n\n![Benchmark time output2](https://raw.githubusercontent.com/vgrabovets/benchmarkit/master/img/benchmark_time2.jpg)\n\nThe same can be run from command line:\n```text\nbenchmark_run test_data/time/benchmark_functions.py --save_dir /tmp/ --comment \"million items\" --extra_fields num_items\n```\n\n### Benchmark model results\n\n```python\nfrom benchmarkit import benchmark, benchmark_run\nfrom sklearn.datasets import load_iris\nfrom sklearn.linear_model import LogisticRegression\n\nMODEL_BENCHMARK_SAVE_FILE = '/tmp/benchmark_model.jsonl'\n\nx, y = load_iris(return_X_y=True)\n\n@benchmark(save_params=True, save_output=True)\ndef log_regression(C=1.0, fit_intercept=True):\n    clf = LogisticRegression(\n        random_state=0, \n        solver='lbfgs', \n        multi_class='multinomial', \n        C=C,\n        fit_intercept=fit_intercept,\n    )\n    clf.fit(x, y)\n    score = clf.score(x, y)\n    return {'score': score}\n\nmodel_benchmark_results = benchmark_run(\n    log_regression,\n    MODEL_BENCHMARK_SAVE_FILE,\n    comment='baseline model',\n    extra_fields=['C', 'fit_intercept'],\n    metric='score',\n    bigger_is_better=True,\n)\n```\n\n![Benchmark model1](https://raw.githubusercontent.com/vgrabovets/benchmarkit/master/img/benchmark_model1.jpg)\n\nChange hyperparameter `C=0.5` and rerun. Output:\n\n![Benchmark model2](https://raw.githubusercontent.com/vgrabovets/benchmarkit/master/img/benchmark_model2.jpg)\n\nThe same can be run from command line:\n```text\nbenchmark_run file_with_benchmark.py --save_dir /tmp/ --comment \"stronger regularization\" --extra_fields C fit_intercept --metric score --bigger_is_better\n```\n\n### Analyze results from the file\n\n```python\nfrom benchmarkit import benchmark_analyze\n\nSAVE_PATH = '/tmp/benchmark_time.jsonl'\n\nbenchmark_df = benchmark_analyze(\n    SAVE_PATH,\n    func_name=None, \n    rows_limit=10,\n    metric='mean_time',\n    bigger_is_better=False,\n    extra_fields=['num_items'],\n)\n```\n\n- __input_path__ - path to `.jsonl` file or directory with `.jsonl` files with benchmark results \n- __func_name__ - display statistics for particular function. If `None` then all functions, stored in file, are displayed. Default _None_\n- __rows_limit__ - limit table rows in console output. Default _10_\n- __metric__ - metric which is used for comparison. Default `mean_time`\n- __bigger_is_better__ - whether bigger value of metric indicates that result is better. For time benchmarks should be `False`, for model accuracy should be `True`. Default _False_\n- __extra_fields__ - extra fields to include in console output\n\nPrints to terminal and returns pandas `DataFrame`.\n\n![Benchmark analyze](https://raw.githubusercontent.com/vgrabovets/benchmarkit/master/img/benchmark_analyze.jpg)\n\nThe same can be run from command line:\n```text\nbenchmark_analyze /tmp/benchmark_time.jsonl --extra_fields num_items\n```\n\n[Other examples](https://nbviewer.jupyter.org/github/vgrabovets/benchmarkit/blob/master/notebooks/benchmark_examples.ipynb)\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/vgrabovets/benchmarkit",
        "keywords": "benchmark,timeit,time",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "benchmarkit",
        "package_url": "https://pypi.org/project/benchmarkit/",
        "platform": "",
        "project_url": "https://pypi.org/project/benchmarkit/",
        "project_urls": {
            "Homepage": "https://github.com/vgrabovets/benchmarkit"
        },
        "release_url": "https://pypi.org/project/benchmarkit/0.0.3/",
        "requires_dist": [
            "colorama (==0.4.1)",
            "GitPython (==2.1.11)",
            "pandas (==0.24.2)",
            "path.py (==12.0.1)"
        ],
        "requires_python": ">=3.6.0",
        "summary": "Benchmark and analyze functions' time execution and results over the course of development",
        "version": "0.0.3"
    },
    "last_serial": 5333723,
    "releases": {
        "0.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "84dd4a97db59c14c06df31d02ab7f2e5",
                    "sha256": "33f27ec2b926d788b3fd7fb8dab35e8d687b14251b3cefc195680fbec94597d7"
                },
                "downloads": -1,
                "filename": "benchmarkit-0.0.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "84dd4a97db59c14c06df31d02ab7f2e5",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6.0",
                "size": 9229,
                "upload_time": "2019-05-26T10:14:17",
                "url": "https://files.pythonhosted.org/packages/5d/fe/0bbde1c725e8cc822d650d265aee2400f08b959dc490e43e68657ef8ea53/benchmarkit-0.0.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "456a8ef5ef8a5dd510d824c5972f9910",
                    "sha256": "fc588cb7582babcbec4d670c35c475dae844696f9f0231bad740b5561cc4e7b9"
                },
                "downloads": -1,
                "filename": "benchmarkit-0.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "456a8ef5ef8a5dd510d824c5972f9910",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6.0",
                "size": 7412,
                "upload_time": "2019-05-26T10:14:19",
                "url": "https://files.pythonhosted.org/packages/6a/1c/11402b289ef81d54f8824b8bd6e503d4eb8dca746c0f34f05ee3597a14fb/benchmarkit-0.0.2.tar.gz"
            }
        ],
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d43d0ab97e19aac9bf50f69350ff3037",
                    "sha256": "c56048a178cee300fc3be90a39bb3cbd4654936ccac30f4fd056add34097ef69"
                },
                "downloads": -1,
                "filename": "benchmarkit-0.0.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d43d0ab97e19aac9bf50f69350ff3037",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6.0",
                "size": 9802,
                "upload_time": "2019-05-29T18:17:25",
                "url": "https://files.pythonhosted.org/packages/a2/df/1b5ad6a2606b5131ae5cd07ce29625d5b34ed02cdcf11caede664e430454/benchmarkit-0.0.3-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "f717147888011b29a12171e8babef648",
                    "sha256": "dc769c51869efb923cc35a96cbca16fa6cc7aaababee9112494f49f8f663332d"
                },
                "downloads": -1,
                "filename": "benchmarkit-0.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "f717147888011b29a12171e8babef648",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6.0",
                "size": 7613,
                "upload_time": "2019-05-29T18:17:27",
                "url": "https://files.pythonhosted.org/packages/27/04/e2c0c1a51a2d71a217a347056d7c68d56027a8b98b70a1857f6cd8abb5a7/benchmarkit-0.0.3.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "d43d0ab97e19aac9bf50f69350ff3037",
                "sha256": "c56048a178cee300fc3be90a39bb3cbd4654936ccac30f4fd056add34097ef69"
            },
            "downloads": -1,
            "filename": "benchmarkit-0.0.3-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "d43d0ab97e19aac9bf50f69350ff3037",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.6.0",
            "size": 9802,
            "upload_time": "2019-05-29T18:17:25",
            "url": "https://files.pythonhosted.org/packages/a2/df/1b5ad6a2606b5131ae5cd07ce29625d5b34ed02cdcf11caede664e430454/benchmarkit-0.0.3-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "f717147888011b29a12171e8babef648",
                "sha256": "dc769c51869efb923cc35a96cbca16fa6cc7aaababee9112494f49f8f663332d"
            },
            "downloads": -1,
            "filename": "benchmarkit-0.0.3.tar.gz",
            "has_sig": false,
            "md5_digest": "f717147888011b29a12171e8babef648",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6.0",
            "size": 7613,
            "upload_time": "2019-05-29T18:17:27",
            "url": "https://files.pythonhosted.org/packages/27/04/e2c0c1a51a2d71a217a347056d7c68d56027a8b98b70a1857f6cd8abb5a7/benchmarkit-0.0.3.tar.gz"
        }
    ]
}