{
"info": {
"author": "Google Inc.",
"author_email": "opensource@google.com",
"bugtrack_url": null,
"classifiers": [],
"description": "# BIM - Benchmark Interpretability Method\n\nThis repository contains dataset, models, and metrics for benchmarking\ninterpretability methods (BIM) described in paper:\n\n* Title: \"BIM: Towards Quantitative Evaluation of Interpretability Methods\n with Ground Truth\"\n* Authors: Sherry (Mengjiao) Yang, Been Kim\n\nUpon using this library, please cite:\n\n```\n@Article{BIM2019,\n title = {{BIM: Towards Quantitative Evaluation of Interpretability Methods with Ground Truth}},\n author = {Yang, Mengjiao and Kim, Been},\n year = {2019}\n}\n```\n\n## Setup\n\nRun the following from the home directory of this repository to install python\ndependencies, download BIM models, download [MSCOCO](http://cocodataset.org) and\n[MiniPlaces](https://github.com/CSAILVision/miniplaces), and construct BIM\ndataset.\n\n```\npip install bim\nsource scripts/download_models.sh\nsource scripts/download_datasets.sh\npython scripts/construct_bim_dataset.py\n```\n\n## Dataset\n\n![](\"https://raw.githubusercontent.com/google-research-datasets/bim/master/figures/dataset_demo.png\")
\n\nImages in `data/obj` and `data/scene` are the same but have object and scene\nlabels respectively, as shown in the figure above. `val_loc.txt` records the\ntop-left and bottom-right corner of the object and `val_mask` has the binary\nmasks of the object in the validation set. Additional sets and their usage are\ndescribed in the table below.\n\nName | Training | Validation | Usage | Description\n:---------------------------------------------------------------------------------| :------: | :--------: | :----------------------------------- | :----------\n`obj` | 90,000 | 10,000 | Model contrast | Objects and scenes with object labels\n`scene` | 90,000 | 10,000 | Model contrast & Input dependence | Objects and scenes with scene labels\n`scene_only` | 90,000 | 10,000 | Input dependence | Scene-only images with scene labels\n`dog_bedroom` | - | 200 | Relative model contrast | Dog in bedroom labeled as bedroom\n`bamboo_forest` | - | 100 | Input independence | Scene-only images of bamboo forest\n`bamboo_forest_patch` | - | 100 | Input independence | Bamboo forest with functionally insignificant dog patch\n\n## Models\n\nModels in `models/obj`, `models/scene`, and `models/scene_only` are trained on\n`data/obj`, `data/scene`, and `data/scene_only` respectively. Models in\n`models/scenei` for `i` in `{1...10}` are trained on images where dog is added\nto `i` scene classes, and the rest scene classes do not contain any added\nobjects. All models are in TensorFlow's\n[SavedModel](https://www.tensorflow.org/guide/saved_model) format.\n\n## Metrics\n\nBIM metrics compare how interpretability methods perform across models (model\ncontrast), across inputs to the same model (input dependence), and across\nfunctionally equivalent inputs (input independence).\n\n### Model contrast scores\n\nGiven images that contain both objects and scenes, model contrast measures the\ndifference in attributions between the model trained on object labels and the\nmodel trained on scene labels.\n\n![](\"https://raw.githubusercontent.com/google-research-datasets/bim/master/figures/mc_demo.png\")
\n\n### Input dependence rate\n\nGiven a model trained on scene labels, input dependence measures the percentage\nof inputs where the addition of objects results in the region being attributed\nas less important.\n\n![](\"https://raw.githubusercontent.com/google-research-datasets/bim/master/figures/id_demo.png\")
\n\n### Input independence rate\n\nGiven a model trained on scene-only images, input independence measures the\npercentage of inputs where a functionally insignificant patch (e.g., a dog) does\nnot affect explanations significantly.\n\n![](\"https://raw.githubusercontent.com/google-research-datasets/bim/master/figures/ii_demo.png\")
\n\n## Evaluate saliency methods\n\nTo compute model contrast score (MCS) over randomly selected 10 images, you can\nrun\n\n```\npython bim/metrics.py --metrics=MCS --num_imgs=10\n```\n\nTo compute input dependence rate (IDR), change `--metrics` to `IDR`. To compute\ninput independence rate (IIR), you need to first constructs a set of\nfunctionally insignificant patches by running\n\n```\npython scripts/construct_delta_patch.py\n```\n\nand then evaluate IIR by running\n\n```\npython bim/metrics.py --metrics=IIR --num_imgs=10\n```\n\n## Evaluate TCAV\n\n[TCAV](https://github.com/tensorflow/tcav) is a global concept attribution\nmethod whose MCS can be measured by comparing the TCAV scores of a particular\nobject concept for the object model and the scene model. Run the following to\ncompute the TCAV scores of the dog concept for the object model.\n\n```\npython bim/run_tcav.py --model=obj\n```\n\n## Disclaimer\n\nThis is not an officially supported Google product.\n\n\n",
"description_content_type": "text/markdown",
"docs_url": null,
"download_url": "",
"downloads": {
"last_day": -1,
"last_month": -1,
"last_week": -1
},
"home_page": "https://github.com/google-research-datasets/bim",
"keywords": "",
"license": "Apache 2.0",
"maintainer": "",
"maintainer_email": "",
"name": "bim",
"package_url": "https://pypi.org/project/bim/",
"platform": "",
"project_url": "https://pypi.org/project/bim/",
"project_urls": {
"Homepage": "https://github.com/google-research-datasets/bim"
},
"release_url": "https://pypi.org/project/bim/0.4/",
"requires_dist": [
"numpy",
"tensorflow",
"Pillow (>=6.0.0)",
"tcav (>=0.2.1)",
"saliency (>=0.0.2)",
"Cython (>=0.29.12)",
"pycocotools (>=2.0.0)",
"scikit-learn (>=0.20.3)"
],
"requires_python": "",
"summary": "Benchmark interpretability methods.",
"version": "0.4"
},
"last_serial": 5571175,
"releases": {
"0.0": [
{
"comment_text": "",
"digests": {
"md5": "88334c2c07e223f4cfdd3855d9ecf547",
"sha256": "1413a0871a3bbdf3a8f0f3f02b87bad0ec672d5fb2e43c6485ee1b88c2349052"
},
"downloads": -1,
"filename": "bim-0.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "88334c2c07e223f4cfdd3855d9ecf547",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 17047,
"upload_time": "2019-07-23T08:23:31",
"url": "https://files.pythonhosted.org/packages/f6/d5/bf192f8f3953568262bcd9e61e00216f69a554f47fc05a70d4434ca74117/bim-0.0.0-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "b7fee904c37db2c0183feca2fe26f9e8",
"sha256": "c6010b17e790513ac455d58895816feeea2bff076e2f6b0e1e7725ded46d0711"
},
"downloads": -1,
"filename": "bim-0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b7fee904c37db2c0183feca2fe26f9e8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 17101,
"upload_time": "2019-06-20T21:16:45",
"url": "https://files.pythonhosted.org/packages/30/f4/9dd8aa082031736cc26255a1dcc19d4410ae42f5d405092bdfba778b2920/bim-0.0-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "09bc5ee7c358d98d79cae50467938923",
"sha256": "a656e28e6fee7f2afb0111304486b58878e6944ff1f354bc29c1b34e2157a795"
},
"downloads": -1,
"filename": "bim-0.0.tar.gz",
"has_sig": false,
"md5_digest": "09bc5ee7c358d98d79cae50467938923",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11317,
"upload_time": "2019-06-20T21:16:48",
"url": "https://files.pythonhosted.org/packages/f4/e1/3a43ac5883a9f0033a0d69d56fb796a7ab04437de72b231d4c6717a46753/bim-0.0.tar.gz"
}
],
"0.1": [
{
"comment_text": "",
"digests": {
"md5": "199d5c2d33b5b0862aa86e168a65d7a2",
"sha256": "cc8731e1405acdda7fbd4ca8c9724dfe1093857fd982934b51e761cff5fa8e60"
},
"downloads": -1,
"filename": "bim-0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "199d5c2d33b5b0862aa86e168a65d7a2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 17023,
"upload_time": "2019-07-23T08:26:26",
"url": "https://files.pythonhosted.org/packages/59/a4/3bec517bce13ff90e67cdc2d248dcc2f971b95e40e9a628e580d6f9fd625/bim-0.1-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "65316acfe64ce4cc59ccbe4bf28bddf2",
"sha256": "3808b17bd50b222e4f1a54f7a0e358a08526e0042cf6e5137ebdf94cd5ebaae7"
},
"downloads": -1,
"filename": "bim-0.1.tar.gz",
"has_sig": false,
"md5_digest": "65316acfe64ce4cc59ccbe4bf28bddf2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11091,
"upload_time": "2019-07-23T08:26:27",
"url": "https://files.pythonhosted.org/packages/56/4f/c3d1b601691062e50d4d201a67a9e3c9be614982f0caa9cee46148bcecb3/bim-0.1.tar.gz"
}
],
"0.2": [
{
"comment_text": "",
"digests": {
"md5": "1f7b6b5b7e3374ad10121e675646e571",
"sha256": "271c1d28b122863655b6b795d2d97bd31cc0f2bbb7d8993d830d9d2753668660"
},
"downloads": -1,
"filename": "bim-0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1f7b6b5b7e3374ad10121e675646e571",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 17025,
"upload_time": "2019-07-23T08:27:42",
"url": "https://files.pythonhosted.org/packages/72/55/86d7893ec60e22cd546ec1173bf78badc27d15f0afcf2bb6cf48c4ce5362/bim-0.2-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "2dde531dbda0de9538c5a99acf25b140",
"sha256": "45ff0df1996e92f39d4f1dcf780620f62e1cb2d3c5ebf34d8fe766cca2360f77"
},
"downloads": -1,
"filename": "bim-0.2.tar.gz",
"has_sig": false,
"md5_digest": "2dde531dbda0de9538c5a99acf25b140",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11085,
"upload_time": "2019-07-23T08:27:44",
"url": "https://files.pythonhosted.org/packages/91/74/3022c10769dea1082ced54eda83dc891280738b51973d988543e0f02eccd/bim-0.2.tar.gz"
}
],
"0.3": [
{
"comment_text": "",
"digests": {
"md5": "a6c34e0fbd86ed2a4c7a10326fed73aa",
"sha256": "dc39590835098720c10e4deaa7cf19de6ca8fd0ddbc248ddc8116c3a702672eb"
},
"downloads": -1,
"filename": "bim-0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a6c34e0fbd86ed2a4c7a10326fed73aa",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 17024,
"upload_time": "2019-07-23T08:30:02",
"url": "https://files.pythonhosted.org/packages/4a/dd/3454d1fdaa4473817cd6c6621a9988cf858c26e598eb72d84e7d224857ff/bim-0.3-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "ccbf32428fbd32e22daabfaabe790ad0",
"sha256": "4d6b7b87c0c22a8cdaf9e4b5984c6e72f1b2cda0aad474ceac15ccf7cf8f746d"
},
"downloads": -1,
"filename": "bim-0.3.tar.gz",
"has_sig": false,
"md5_digest": "ccbf32428fbd32e22daabfaabe790ad0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11087,
"upload_time": "2019-07-23T08:30:04",
"url": "https://files.pythonhosted.org/packages/9f/75/b6a6278b5a1d501574363cc19fbfe45eb3e56b4107105fe283cc418fc2f1/bim-0.3.tar.gz"
}
],
"0.4": [
{
"comment_text": "",
"digests": {
"md5": "fc599c94d5436f63cdf33cc65b7a755d",
"sha256": "57bf87164a3eeb4810b8b3d1ef879b9211c96a90d44d78c3c45bfbfc59b17079"
},
"downloads": -1,
"filename": "bim-0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fc599c94d5436f63cdf33cc65b7a755d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 17023,
"upload_time": "2019-07-23T08:34:20",
"url": "https://files.pythonhosted.org/packages/4c/42/44067c7e08fcc42b9897efcce27db106740d6c5c20fc9aa29c60e74dbeac/bim-0.4-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "84425e7a6c8a09971f52a84c78c4ea06",
"sha256": "8b13bdc128e92f3eb52f03919aa90e6df0bcedde980906387bd38dfd65a9a327"
},
"downloads": -1,
"filename": "bim-0.4.tar.gz",
"has_sig": false,
"md5_digest": "84425e7a6c8a09971f52a84c78c4ea06",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11091,
"upload_time": "2019-07-23T08:34:22",
"url": "https://files.pythonhosted.org/packages/27/9e/c21cf240a8f36b7690327fdd3ca3a8d37aaa01217a7590c57f2a28c45f73/bim-0.4.tar.gz"
}
]
},
"urls": [
{
"comment_text": "",
"digests": {
"md5": "fc599c94d5436f63cdf33cc65b7a755d",
"sha256": "57bf87164a3eeb4810b8b3d1ef879b9211c96a90d44d78c3c45bfbfc59b17079"
},
"downloads": -1,
"filename": "bim-0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fc599c94d5436f63cdf33cc65b7a755d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 17023,
"upload_time": "2019-07-23T08:34:20",
"url": "https://files.pythonhosted.org/packages/4c/42/44067c7e08fcc42b9897efcce27db106740d6c5c20fc9aa29c60e74dbeac/bim-0.4-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "84425e7a6c8a09971f52a84c78c4ea06",
"sha256": "8b13bdc128e92f3eb52f03919aa90e6df0bcedde980906387bd38dfd65a9a327"
},
"downloads": -1,
"filename": "bim-0.4.tar.gz",
"has_sig": false,
"md5_digest": "84425e7a6c8a09971f52a84c78c4ea06",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11091,
"upload_time": "2019-07-23T08:34:22",
"url": "https://files.pythonhosted.org/packages/27/9e/c21cf240a8f36b7690327fdd3ca3a8d37aaa01217a7590c57f2a28c45f73/bim-0.4.tar.gz"
}
]
}