{
    "info": {
        "author": "Adrien Ball",
        "author_email": "adrien.ball@snips.ai",
        "bugtrack_url": null,
        "classifiers": [
            "Programming Language :: Python :: 2",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7"
        ],
        "description": "Snips NLU Metrics\n=================\n\n.. image:: https://travis-ci.org/snipsco/snips-nlu-metrics.svg?branch=master\n    :target: https://travis-ci.org/snipsco/snips-nlu-metrics\n\n.. image:: https://img.shields.io/pypi/v/snips-nlu-metrics.svg?branch=master\n    :target: https://pypi.python.org/pypi/snips-nlu-metrics\n\n.. image:: https://img.shields.io/pypi/pyversions/snips-nlu-metrics.svg?branch=master\n    :target: https://pypi.python.org/pypi/snips-nlu-metrics\n\n\nThis tools is a python library for computing `cross-validation`_ and\n`train/test`_ metrics on an NLU parsing pipeline such as the `Snips NLU`_ one.\n\nIts purpose is to help evaluating and iterating on the tested intent parsing\npipeline.\n\nInstall\n-------\n\n.. code-block:: console\n\n    $ pip install snips_nlu_metrics\n\n\nNLU Metrics API\n---------------\n\nSnips NLU metrics API consists in the following functions:\n\n* ``compute_train_test_metrics`` to compute `train/test`_ metrics\n* ``compute_cross_val_metrics`` to compute `cross-validation`_ metrics\n\nThe metrics output (json) provides detailed information about:\n\n* `confusion matrix`_\n* `precision, recall and f1 scores`_ of intent classification\n* precision, recall and f1 scores of entity extraction\n* parsing errors\n\nData\n----\n\nSome sample datasets, that can be used to compute metrics, are available\n`here <samples/>`_. Alternatively, you can create your own dataset either by\nusing ``snips-nlu``'s `dataset generation tool`_ or by going on the\n`Snips console`_.\n\nExamples\n--------\n\nThe Snips NLU metrics library can be used with any NLU pipeline which satisfies\nthe ``Engine`` API:\n\n.. code-block:: python\n\n    from builtins import object\n\n    class Engine(object):\n        def fit(self, dataset):\n            # Perform training ...\n            return self\n\n        def parse(self, text):\n            # extract intent and slots ...\n            return {\n                \"input\": text,\n                \"intent\": {\n                    \"intentName\": intent_name,\n                    \"probability\": probability\n                },\n                \"slots\": slots\n            }\n\n\n----------------\nSnips NLU Engine\n----------------\n\nThis library can be used to benchmark NLU solutions such as `Snips NLU`_. To\ninstall the ``snips-nlu`` python library, and fetch the language resources for\nenglish, run the following commands:\n\n.. code-block:: bash\n\n    $ pip install snips-nlu\n    $ snips-nlu download en\n\n\nThen, you can compute metrics for the ``snips-nlu`` pipeline using the metrics\nAPI as follows:\n\n.. code-block:: python\n\n    from snips_nlu import SnipsNLUEngine\n    from snips_nlu_metrics import compute_train_test_metrics, compute_cross_val_metrics\n\n    tt_metrics = compute_train_test_metrics(train_dataset=\"samples/train_dataset.json\",\n                                            test_dataset=\"samples/test_dataset.json\",\n                                            engine_class=SnipsNLUEngine)\n\n    cv_metrics = compute_cross_val_metrics(dataset=\"samples/cross_val_dataset.json\",\n                                           engine_class=SnipsNLUEngine,\n                                           nb_folds=5)\n\n-----------------\nCustom NLU Engine\n-----------------\n\nYou can also compute metrics on a custom NLU engine, here is a simple example:\n\n.. code-block:: python\n\n    import random\n\n    from snips_nlu_metrics import compute_train_test_metrics\n\n    class MyNLUEngine(object):\n        def fit(self, dataset):\n            self.intent_list = list(dataset[\"intents\"])\n            return self\n\n        def parse(self, text):\n            return {\n                \"input\": text,\n                \"intent\": {\n                    \"intentName\": random.choice(self.intent_list),\n                    \"probability\": 0.5\n                },\n                \"slots\": []\n            }\n\n    compute_train_test_metrics(train_dataset=\"samples/train_dataset.json\",\n                               test_dataset=\"samples/test_dataset.json\",\n                               engine_class=MyNLUEngine)\n\nLinks\n-----\n* `Changelog <CHANGELOG.md>`__\n* `Bug tracker <https://github.com/snipsco/snips-nlu-metrics/issues>`__\n* `Snips NLU <https://github.com/snipsco/snips-nlu>`__\n* `Snips NLU Rust <https://github.com/snipsco/snips-nlu-rs>`__: Rust inference pipeline implementation and bindings (C, Swift, Kotlin, Python)\n* `Snips <https://snips.ai/>`__\n\nContributing\n------------\nPlease see the `Contribution Guidelines <CONTRIBUTING.rst>`_.\n\nCopyright\n---------\nThis library is provided by `Snips <https://www.snips.ai>`_ as Open Source software. See `LICENSE <LICENSE>`_ for more information.\n\n.. _cross-validation: https://en.wikipedia.org/wiki/Cross-validation_(statistics)\n.. _train/test: https://en.wikipedia.org/wiki/Training,_test,_and_validation_sets\n.. _Snips NLU: https://github.com/snipsco/snips-nlu\n.. _precision, recall and f1 scores: https://en.wikipedia.org/wiki/Precision_and_recall\n.. _confusion matrix: https://en.wikipedia.org/wiki/Confusion_matrix\n.. _dataset generation tool: http://snips-nlu.readthedocs.io/en/latest/tutorial.html#snips-dataset-format\n.. _Snips console: https://console.snips.ai\n\n",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/snipsco/snips-nlu-metrics",
        "keywords": "metrics nlu nlp intent slots entity parsing",
        "license": "Apache 2.0",
        "maintainer": "",
        "maintainer_email": "",
        "name": "snips-nlu-metrics",
        "package_url": "https://pypi.org/project/snips-nlu-metrics/",
        "platform": "",
        "project_url": "https://pypi.org/project/snips-nlu-metrics/",
        "project_urls": {
            "Homepage": "https://github.com/snipsco/snips-nlu-metrics"
        },
        "release_url": "https://pypi.org/project/snips-nlu-metrics/0.14.5/",
        "requires_dist": [
            "future",
            "numpy (<2.0,>=1.7)",
            "scipy (<2.0,>=1.0)",
            "joblib (<0.14,>=0.13)",
            "scikit-learn (<0.21,>=0.19) ; python_version < \"3.5\"",
            "scikit-learn (<0.22,>=0.21.0) ; python_version >= \"3.5\"",
            "mock (<3.0,>=2.0) ; extra == 'test'"
        ],
        "requires_python": "",
        "summary": "Python package to compute NLU metrics",
        "version": "0.14.5"
    },
    "last_serial": 5704274,
    "releases": {
        "0.11.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "109ba182406b37d1e20ff318707c1b66",
                    "sha256": "60b9e5fcecf317806cfe153a69f02c2074ef834676e1a09d15f642e52151922b"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.11.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "109ba182406b37d1e20ff318707c1b66",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 11941,
                "upload_time": "2018-02-21T14:21:40",
                "url": "https://files.pythonhosted.org/packages/32/32/08b263cd892f33e5a43e59f8f1e9241c7508fe3fb14c1a2e8b92acab6899/snips_nlu_metrics-0.11.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "157de19c5e8f2c2831379d0242fc7bd1",
                    "sha256": "e2f116b8428709abda3c955fc0cb6cd1f49593649a1b79aa2b4f43ca2b33abed"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.11.1.tar.gz",
                "has_sig": false,
                "md5_digest": "157de19c5e8f2c2831379d0242fc7bd1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8899,
                "upload_time": "2018-02-21T14:21:44",
                "url": "https://files.pythonhosted.org/packages/e6/ce/2a6cd839eba6f73e2a52ead2f99b680c4aa556f9104af518652494aeef9f/snips_nlu_metrics-0.11.1.tar.gz"
            }
        ],
        "0.12.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8ac8344baaa033e0ecf852ca451c9433",
                    "sha256": "494081ed91dcc2fcf4af8f2e8719e329753acac44aaac438270737f45eae9ee9"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.12.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "8ac8344baaa033e0ecf852ca451c9433",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 14667,
                "upload_time": "2018-03-29T17:18:31",
                "url": "https://files.pythonhosted.org/packages/cc/11/f1d2bf05270df4e8e35e0ef541ea9bc55947b7fdec5e4b127af992b14f90/snips_nlu_metrics-0.12.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "cdb6fb84e749a5990f5e9d45ecd31414",
                    "sha256": "7ab8d4a648c460f86a9ae12e6151b1d175f8427e130835d16c37005596d5037c"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.12.0.tar.gz",
                "has_sig": false,
                "md5_digest": "cdb6fb84e749a5990f5e9d45ecd31414",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15004,
                "upload_time": "2018-03-29T17:27:21",
                "url": "https://files.pythonhosted.org/packages/e4/54/157b8fac7e1dda41c090a2a8c1a73580c1e153df36d1f3a461932650a1c0/snips_nlu_metrics-0.12.0.tar.gz"
            }
        ],
        "0.13.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "65f21f2ef2872c1b15071e869fe08a62",
                    "sha256": "812b14c15067edff36e68e0e7c899f97489e7fef5691bc291d738808fd778f76"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.13.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "65f21f2ef2872c1b15071e869fe08a62",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 11089,
                "upload_time": "2018-07-25T09:21:04",
                "url": "https://files.pythonhosted.org/packages/45/28/b86365954df8a6e6a95d9a5efaf8df417211014b08a3ebdc803baee2c859/snips_nlu_metrics-0.13.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "11e907ebc6bd15fea85f2443333ca0ba",
                    "sha256": "1ec3b6cd5f2fd1b1eb8868c62777a5d50d1f54310417dc06e930793b1ae0ccd7"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.13.0.tar.gz",
                "has_sig": false,
                "md5_digest": "11e907ebc6bd15fea85f2443333ca0ba",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 14426,
                "upload_time": "2018-07-25T09:21:05",
                "url": "https://files.pythonhosted.org/packages/d8/eb/358846cce9970d4779fbb8efff47a75e180f9012ce803bdf01d70682dae6/snips_nlu_metrics-0.13.0.tar.gz"
            }
        ],
        "0.14.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "13f66fa439e104881698a640c3b3dc49",
                    "sha256": "9d260da00831cac3da8e59927f996fcc72f1441e2afe8fb0d112a25f68c174cc"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "13f66fa439e104881698a640c3b3dc49",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 11742,
                "upload_time": "2018-11-13T09:41:48",
                "url": "https://files.pythonhosted.org/packages/d9/cb/a0f932e0ab84ad311238a70673b124e7c11ce49679249f1e9bc065f3d62b/snips_nlu_metrics-0.14.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "4717aef8cd71f4dd7e2faf2b5540f02f",
                    "sha256": "7cb56c87ae70ff09951b902191b034a1215e3462101e05d5aa09e583f623f3f2"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.0.tar.gz",
                "has_sig": false,
                "md5_digest": "4717aef8cd71f4dd7e2faf2b5540f02f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15022,
                "upload_time": "2018-11-13T09:41:50",
                "url": "https://files.pythonhosted.org/packages/bc/c7/dc13b661a0971636b36c2187452e4ed193e393dc08be29aa82deaefb5200/snips_nlu_metrics-0.14.0.tar.gz"
            }
        ],
        "0.14.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "73222cf6dbe792d5e9a370dfd2394a8f",
                    "sha256": "19b10a4660dd9b0f69052e82a866a3073d443e75a9765ac01b66f43526302d27"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "73222cf6dbe792d5e9a370dfd2394a8f",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 11745,
                "upload_time": "2019-01-07T15:48:42",
                "url": "https://files.pythonhosted.org/packages/63/fa/48ab3a2abab9908e96b2b70c32e724315d4fefe74b2d6379ba6e59a20d05/snips_nlu_metrics-0.14.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e508776ff3e3480a627f7cc556b31b1a",
                    "sha256": "0f07fe8a4b1526d47d94438ffaaa3203e7a9f9928c2cd9ccc94f239fa35bb953"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.1.tar.gz",
                "has_sig": false,
                "md5_digest": "e508776ff3e3480a627f7cc556b31b1a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15049,
                "upload_time": "2019-01-07T15:48:44",
                "url": "https://files.pythonhosted.org/packages/71/5d/2ff4b04f8fd878c8367ab0a5642b8bfebb4301a854cdc0cff73deca93a36/snips_nlu_metrics-0.14.1.tar.gz"
            }
        ],
        "0.14.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6f87f8aa3a6539edd5024a7d36ecbe28",
                    "sha256": "09af4986cb780bcdb2a31868e2796826dd8ad9acf030e14ade73784363778910"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "6f87f8aa3a6539edd5024a7d36ecbe28",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 12233,
                "upload_time": "2019-03-21T17:44:54",
                "url": "https://files.pythonhosted.org/packages/ee/7c/2aab4c566ad75a7c767c952bf09656012f4af45d67544668a11a79e05968/snips_nlu_metrics-0.14.2-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "3ae583a2f278a2a83bddcae6136b5352",
                    "sha256": "1de7238bf50019739caee5f3cb5bfd98cf51561068e799fcbbb7fd23934b0ea9"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.2.tar.gz",
                "has_sig": false,
                "md5_digest": "3ae583a2f278a2a83bddcae6136b5352",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15623,
                "upload_time": "2019-03-21T17:44:55",
                "url": "https://files.pythonhosted.org/packages/6d/ce/2e7977d782fe4cf10c14491c12c9cc5a9cb5a549f8ce499e81a00beb73ed/snips_nlu_metrics-0.14.2.tar.gz"
            }
        ],
        "0.14.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c6bd11d7ac9e460c7cb52b9eaf910bf7",
                    "sha256": "631383390166a94c51b56b6824c8ca79d5bab1ab4321fa4b8815a07d15a09ecb"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.3-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "c6bd11d7ac9e460c7cb52b9eaf910bf7",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 17381,
                "upload_time": "2019-05-10T13:16:28",
                "url": "https://files.pythonhosted.org/packages/13/8e/7711c7cb70b98695095d81b399e2874e3f50146fdf86fb247dd7d1bde5bc/snips_nlu_metrics-0.14.3-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c3fd6851e7d7a108161f79583acfa6a7",
                    "sha256": "14b3ebdd571e2000af272f5132412934badedae8b4cfcc4bc05ee18cbed5e1b3"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.3.tar.gz",
                "has_sig": false,
                "md5_digest": "c3fd6851e7d7a108161f79583acfa6a7",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 32390,
                "upload_time": "2019-05-10T13:16:30",
                "url": "https://files.pythonhosted.org/packages/2a/99/a3ebb230fbeb9bf880dd284ba663d6dd1bdbb2d77d1a812deed45c5d5fea/snips_nlu_metrics-0.14.3.tar.gz"
            }
        ],
        "0.14.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "139434a53092248ca8d9b933a9f58af4",
                    "sha256": "628615301857d32fbf8a0243ff82369fef8e624dc0e8aa1c59617fdee2e47019"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.4-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "139434a53092248ca8d9b933a9f58af4",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 17012,
                "upload_time": "2019-06-18T10:31:27",
                "url": "https://files.pythonhosted.org/packages/67/0f/39fac5cbb57eb0fb0a67deaecf3f1fbb10b83e85c811c203a91bbbab5482/snips_nlu_metrics-0.14.4-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "02a299a9e8bbc8f84867330c3fd74657",
                    "sha256": "466e69f40b6c207775100f20e33b1e385f18fcf36252aba06d3ef7c59c46557d"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.4.tar.gz",
                "has_sig": false,
                "md5_digest": "02a299a9e8bbc8f84867330c3fd74657",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 32495,
                "upload_time": "2019-06-18T10:31:30",
                "url": "https://files.pythonhosted.org/packages/70/11/843824d25c58c27a7e0ae1b51c7964f963a05fc7904804f1ea21dbd629bd/snips_nlu_metrics-0.14.4.tar.gz"
            }
        ],
        "0.14.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "114a746734ce44824e81044474b2eb32",
                    "sha256": "9c5fe7263b99df0a9dcbc4f0e8196bda02945a8c61f8968dadebe97b2adb1434"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.5-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "114a746734ce44824e81044474b2eb32",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 17294,
                "upload_time": "2019-08-20T16:15:00",
                "url": "https://files.pythonhosted.org/packages/1b/88/78713d911cd0b4dd6ff8c2e859009edc80e715b9efc496bab0a949eb18c8/snips_nlu_metrics-0.14.5-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "a47607bbba9c3c7f955f8b50ebc5e433",
                    "sha256": "5901b42fd5f897d7c94ce943a72319bf46f8eb710489315955736c468da2ed7e"
                },
                "downloads": -1,
                "filename": "snips_nlu_metrics-0.14.5.tar.gz",
                "has_sig": false,
                "md5_digest": "a47607bbba9c3c7f955f8b50ebc5e433",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 32664,
                "upload_time": "2019-08-20T16:15:01",
                "url": "https://files.pythonhosted.org/packages/f0/ce/f366653b3dee990ab11f0803d7fd898b497f45ea036f9d50d526332ac2ef/snips_nlu_metrics-0.14.5.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "114a746734ce44824e81044474b2eb32",
                "sha256": "9c5fe7263b99df0a9dcbc4f0e8196bda02945a8c61f8968dadebe97b2adb1434"
            },
            "downloads": -1,
            "filename": "snips_nlu_metrics-0.14.5-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "114a746734ce44824e81044474b2eb32",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 17294,
            "upload_time": "2019-08-20T16:15:00",
            "url": "https://files.pythonhosted.org/packages/1b/88/78713d911cd0b4dd6ff8c2e859009edc80e715b9efc496bab0a949eb18c8/snips_nlu_metrics-0.14.5-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "a47607bbba9c3c7f955f8b50ebc5e433",
                "sha256": "5901b42fd5f897d7c94ce943a72319bf46f8eb710489315955736c468da2ed7e"
            },
            "downloads": -1,
            "filename": "snips_nlu_metrics-0.14.5.tar.gz",
            "has_sig": false,
            "md5_digest": "a47607bbba9c3c7f955f8b50ebc5e433",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 32664,
            "upload_time": "2019-08-20T16:15:01",
            "url": "https://files.pythonhosted.org/packages/f0/ce/f366653b3dee990ab11f0803d7fd898b497f45ea036f9d50d526332ac2ef/snips_nlu_metrics-0.14.5.tar.gz"
        }
    ]
}