{
"info": {
"author": "Lorenzo Bertoni",
"author_email": "lorenzo.bertoni@epfl.ch",
"bugtrack_url": null,
"classifiers": [],
"description": "# Monoloco\n\n> We tackle the fundamentally ill-posed problem of 3D human localization from monocular RGB images. \nDriven by the limitation of neural networks outputting point estimates, \nwe address the ambiguity in the task with a new neural network predicting confidence intervals through a \nloss function based on the Laplace distribution. \nOur architecture is a light-weight feed-forward neural network which predicts the 3D coordinates given 2D human pose. \nThe design is particularly well suited for small training data and cross-dataset generalization. \nOur experiments show that (i) we outperform state-of-the art results on KITTI and nuScenes datasets, \n(ii) even outperform stereo for far-away pedestrians, and (iii) estimate meaningful confidence intervals. \nWe further share insights on our model of uncertainty in case of limited observation and out-of-distribution samples.\n\n```\n@article{bertoni2019monoloco,\n title={MonoLoco: Monocular 3D Pedestrian Localization and Uncertainty Estimation},\n author={Bertoni, Lorenzo and Kreiss, Sven and Alahi, Alexandre},\n journal={arXiv preprint arXiv:1906.06059},\n year={2019}\n}\n```\n\n* **Check our video with method description and qualitative results on [YouTube](https://www.youtube.com/watch?v=ii0fqerQrec)**\n\n* **Live demo available! (more info in the webcam section)**\n\n![](\"docs/pull.png\")
\n\n# Setup\n\n### Install\nPython 3 is required. Python 2 is not supported. \nDo not clone this repository and make sure there is no folder named monoloco in your current directory.\n\n```\npip3 install monoloco\n```\n\nFor development of the monoloco source code itself, you need to clone this repository and then:\n```\npip3 install -e '.[test, prep]'\n```\nPython 3.6 or 3.7 is required for nuScenes development kit. \nAll details for Pifpaf pose detector at [openpifpaf](https://github.com/vita-epfl/openpifpaf).\n\n\n\n### Data structure\n\n Data \n \u251c\u2500\u2500 arrays \n \u251c\u2500\u2500 models\n \u251c\u2500\u2500 kitti\n \u251c\u2500\u2500 nuscenes\n \u251c\u2500\u2500 logs\n\n\nRun the following to create the folders:\n```\nmkdir data\ncd data\nmkdir arrays models kitti nuscenes logs\n```\n\n### Pre-trained Models\n* Download a MonoLoco pre-trained model from \n[Google Drive](https://drive.google.com/open?id=1F7UG1HPXGlDD_qL-AN5cv2Eg-mhdQkwv) and save it in `data/models` \n(default) or in any folder and call it through the command line option `--model `\n* Pifpaf pre-trained model will be automatically downloaded at the first run. \nThree standard, pretrained models are available when using the command line option \n`--checkpoint resnet50`, `--checkpoint resnet101` and `--checkpoint resnet152`.\nAlternatively, you can download a Pifpaf pre-trained model from [openpifpaf](https://github.com/vita-epfl/openpifpaf)\n and call it with `--checkpoint `\n\n\n# Interfaces\nAll the commands are run through a main file called `main.py` using subparsers.\nTo check all the commands for the parser and the subparsers (including openpifpaf ones) run:\n\n* `python3 -m monoloco.run --help`\n* `python3 -m monoloco.run predict --help`\n* `python3 -m monoloco.run train --help`\n* `python3 -m monoloco.run eval --help`\n* `python3 -m monoloco.run prep --help`\n\nor check the file `monoloco/run.py`\n\n# Prediction\nThe predict script receives an image (or an entire folder using glob expressions), \ncalls PifPaf for 2d human pose detection over the image\nand runs Monoloco for 3d location of the detected poses.\nThe command `--networks` defines if saving pifpaf outputs, MonoLoco outputs or both.\nYou can check all commands for Pifpaf at [openpifpaf](https://github.com/vita-epfl/openpifpaf).\n\n\nOutput options include json files and/or visualization of the predictions on the image in *frontal mode*, \n*birds-eye-view mode* or *combined mode* and can be specified with `--output_types`\n\n\n### Ground truth matching\n* In case you provide a ground-truth json file to compare the predictions of MonoLoco,\n the script will match every detection using Intersection over Union metric. \n The ground truth file can be generated using the subparser `prep` and called with the command `--path_gt`.\n Check preprocess section for more details or download the file from \n [here](https://drive.google.com/open?id=1F7UG1HPXGlDD_qL-AN5cv2Eg-mhdQkwv).\n\n* In case you don't provide a ground-truth file, the script will look for a predefined path. \nIf it does not find the file, it will generate images\nwith all the predictions without ground-truth matching.\n\nBelow an example with and without ground-truth matching. They have been created (adding or removing `--path_gt`) with:\n`python3 -m monoloco.run predict --glob docs/002282.png --output_types combined --scale 2 \n--model data/models/monoloco-190513-1437.pkl --n_dropout 50 --z_max 30`\n\nWith ground truth matching (only matching people):\n\n\nWithout ground_truth matching (all the detected people): \n\n\n### Images without calibration matrix\nTo accurately estimate distance, the focal length is necessary. \nHowever, it is still possible to test Monoloco on images where the calibration matrix is not available. \nAbsolute distances are not meaningful but relative distance still are. \nBelow an example on a generic image from the web, created with:\n`python3 -m monoloco.run predict --glob docs/surf.jpg --output_types combined --model data/models/monoloco-190513-1437.pkl --n_dropout 50 --z_max 25`\n\n\n\n\n# Webcam\n![\"example](\"docs/webcam_short.gif\")
\n\nMonoLoco can run on personal computers with only CPU and low resolution images (e.g. 256x144) at ~2fps.\nIt support 3 types of visualizations: `front`, `bird` and `combined`.\nMultiple visualizations can be combined in different windows.\n\nThe above gif has been obtained running on a Macbook the command:\n\n```pip3 install opencv-python\npython3 -m monoloco.run predict --webcam --scale 0.2 --output_types combined --z_max 10 --checkpoint resnet50 --model data/models/monoloco-190513-1437.pkl\n```\n\n# Preprocessing\n\n### Datasets\n\n#### 1) KITTI dataset\nDownload KITTI ground truth files and camera calibration matrices for training\nfrom [here](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) and\nsave them respectively into `data/kitti/gt` and `data/kitti/calib`. \nTo extract pifpaf joints, you also need to download training images soft link the folder in `\ndata/kitti/images`\n\n#### 2) nuScenes dataset\nDownload nuScenes dataset from [nuScenes](https://www.nuscenes.org/download) (either Mini or TrainVal), \nsave it anywhere and soft link it in `data/nuscenes`\n\nnuScenes preprocessing requires `pip3 install nuscenes-devkit`\n\n\n### Annotations to preprocess\nMonoLoco is trained using 2D human pose joints. To create them run pifaf over KITTI or nuScenes training images. \nYou can create them running the predict script and using `--networks pifpaf`.\n\n### Inputs joints for training\nMonoLoco is trained using 2D human pose joints matched with the ground truth location provided by\nnuScenes or KITTI Dataset. To create the joints run: `python3 -m monoloco.run prep` specifying:\n1. `--dir_ann` annotation directory containing Pifpaf joints of KITTI or nuScenes. \n\n2. `--dataset` Which dataset to preprocess. For nuscenes, all three versions of the \ndataset are supported: nuscenes_mini, nuscenes, nuscenes_teaser.\n\n### Ground truth file for evaluation\nThe preprocessing script also outputs a second json file called **names-.json** which provide a dictionary indexed\nby the image name to easily access ground truth files for evaluation and prediction purposes.\n\n\n\n# Training\nProvide the json file containing the preprocess joints as argument. \n\nAs simple as `python3 -m monoloco.run --train --joints `\n\nAll the hyperparameters options can be checked at `python3 -m monoloco.run train --help`.\n\n### Hyperparameters tuning\nRandom search in log space is provided. An example: `python3 -m monoloco.run train --hyp --multiplier 10 --r_seed 1`.\nOne iteration of the multiplier includes 6 runs.\n\n\n# Evaluation (KITTI Dataset)\nWe provide evaluation on KITTI for models trained on nuScenes or KITTI. We compare them with other monocular \nand stereo Baselines: \n\n[Mono3D](https://www.cs.toronto.edu/~urtasun/publications/chen_etal_cvpr16.pdf), \n[3DOP](https://xiaozhichen.github.io/papers/nips15chen.pdf), \n[MonoDepth](https://arxiv.org/abs/1609.03677) and our \n[Geometrical Baseline](monoloco/eval/geom_baseline.py).\n\n* **Mono3D**: download validation files from [here](http://3dimage.ee.tsinghua.edu.cn/cxz/mono3d) \nand save them into `data/kitti/m3d`\n* **3DOP**: download validation files from [here](https://xiaozhichen.github.io/) \nand save them into `data/kitti/3dop`\n* **MonoDepth**: compute an average depth for every instance using the following script \n[here](https://github.com/Parrotlife/pedestrianDepth-baseline/tree/master/MonoDepth-PyTorch) \nand save them into `data/kitti/monodepth`\n* **GeometricalBaseline**: A geometrical baseline comparison is provided. \nThe average geometrical value for comparison can be obtained running:\n`python3 -m monoloco.run eval --geometric \n--model data/models/monoloco-190719-0923.pkl --joints data/arrays/joints-nuscenes_teaser-190717-1424.json`\n\n\nThe following results are obtained running:\n`python3 -m monoloco.run eval --model data/models/monoloco-190719-0923.pkl --generate\n--dir_ann `\n\n\n\n\n\n\n\n",
"description_content_type": "text/markdown",
"docs_url": null,
"download_url": "",
"downloads": {
"last_day": -1,
"last_month": -1,
"last_week": -1
},
"home_page": "https://github.com/vita-epfl/monoloco",
"keywords": "",
"license": "GNU AGPLv3",
"maintainer": "",
"maintainer_email": "",
"name": "monoloco",
"package_url": "https://pypi.org/project/monoloco/",
"platform": "",
"project_url": "https://pypi.org/project/monoloco/",
"project_urls": {
"Homepage": "https://github.com/vita-epfl/monoloco"
},
"release_url": "https://pypi.org/project/monoloco/0.4.6/",
"requires_dist": [
"openpifpaf",
"tabulate",
"nuscenes-devkit ; extra == 'prep'",
"pylint ; extra == 'test'",
"pytest ; extra == 'test'"
],
"requires_python": "",
"summary": "MonoLoco: Monocular 3D Pedestrian Localization and Uncertainty Estimation",
"version": "0.4.6"
},
"last_serial": 5720570,
"releases": {
"0.4.0": [
{
"comment_text": "",
"digests": {
"md5": "d97a41e9985b3129c2f1224683fcfde9",
"sha256": "05e22c0d9a68d4ac76f537cf889fa07277991ef0d35fe6b5ca9aea4a0d9ed1f3"
},
"downloads": -1,
"filename": "monoloco-0.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d97a41e9985b3129c2f1224683fcfde9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 70350,
"upload_time": "2019-07-19T14:59:31",
"url": "https://files.pythonhosted.org/packages/72/84/b2be0c61fb7fa4533a0226543e1347c4008f0af0a86fe3939eb6b5ae715c/monoloco-0.4.0-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "8d7525f2140beb6604ffe94b58f42f61",
"sha256": "c6a83edbca189d219407f6e217f8a41c3bf4d56cae34c307b77946e27e92e68f"
},
"downloads": -1,
"filename": "monoloco-0.4.0.tar.gz",
"has_sig": false,
"md5_digest": "8d7525f2140beb6604ffe94b58f42f61",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 49869,
"upload_time": "2019-07-19T14:59:33",
"url": "https://files.pythonhosted.org/packages/18/49/665bc9a8621514914e0ab45477d51f45efa5d84e7434c7d575e48dd20e33/monoloco-0.4.0.tar.gz"
}
],
"0.4.1": [
{
"comment_text": "",
"digests": {
"md5": "4a56ea0fee1e4fbe2d66dd5a9b8b4c2e",
"sha256": "398be04ea1e804c147dc94ac705a577a06f3bb2d707a820fb610a531168d6a8b"
},
"downloads": -1,
"filename": "monoloco-0.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4a56ea0fee1e4fbe2d66dd5a9b8b4c2e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 70500,
"upload_time": "2019-07-19T15:28:41",
"url": "https://files.pythonhosted.org/packages/d6/47/bb4426b41224d15b27ee554006d82c726fbb17b5580834ca711d1d9e3459/monoloco-0.4.1-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "92aac95d6a242faeb5fa40b83a13b8aa",
"sha256": "02396747651f022a7669ed9799b245150c653ccea6d82d79cf0fc4170c4569b9"
},
"downloads": -1,
"filename": "monoloco-0.4.1.tar.gz",
"has_sig": false,
"md5_digest": "92aac95d6a242faeb5fa40b83a13b8aa",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 50016,
"upload_time": "2019-07-19T15:28:42",
"url": "https://files.pythonhosted.org/packages/10/74/da41673a6c0ad71ce9683273b7499482dc55a3c3cfbefe044762c2cdcda3/monoloco-0.4.1.tar.gz"
}
],
"0.4.2": [
{
"comment_text": "",
"digests": {
"md5": "d266965f6c5c8f5398e9431cb1822b6d",
"sha256": "daf1ec84df5d81b82a5486c5769fd19f06862abb4e8e7e69c28182e56548a36f"
},
"downloads": -1,
"filename": "monoloco-0.4.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d266965f6c5c8f5398e9431cb1822b6d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 83482,
"upload_time": "2019-07-26T08:46:38",
"url": "https://files.pythonhosted.org/packages/4e/16/5bf63a96002087c1f4b724f905c563e1935380ed89fa1ea734c2c74e405c/monoloco-0.4.2-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "74e53ecb68a20b09d1d4619d94793e1f",
"sha256": "36dd77c24e588422d2dd11a2bec123f615c37029c8c6f11a4ead1c90423f4063"
},
"downloads": -1,
"filename": "monoloco-0.4.2.tar.gz",
"has_sig": false,
"md5_digest": "74e53ecb68a20b09d1d4619d94793e1f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 49930,
"upload_time": "2019-07-26T08:46:40",
"url": "https://files.pythonhosted.org/packages/d7/4c/a81ed1dbbe7cd90cf7c83ac231ed2716a89738a2d5df0dddddae86ecc62c/monoloco-0.4.2.tar.gz"
}
],
"0.4.3": [
{
"comment_text": "",
"digests": {
"md5": "82dd51d296e93b85aa7779afa7686937",
"sha256": "7be761d12a50bab5820f5ab29d70859990641513664b66007dbd70fa77dcf4d5"
},
"downloads": -1,
"filename": "monoloco-0.4.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "82dd51d296e93b85aa7779afa7686937",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 69467,
"upload_time": "2019-07-26T10:35:18",
"url": "https://files.pythonhosted.org/packages/75/7a/d42b6a2e093074c797bab2b875bb75c77f1de6b345d92f0177adab38aaf7/monoloco-0.4.3-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "5ebf9e35e8d2b8d6044f3752cf751ee1",
"sha256": "fb4f1976b95046cbd4b99d97e71d93e01a7df2f6911a96e5f638533339c109de"
},
"downloads": -1,
"filename": "monoloco-0.4.3.tar.gz",
"has_sig": false,
"md5_digest": "5ebf9e35e8d2b8d6044f3752cf751ee1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 49877,
"upload_time": "2019-07-26T10:35:20",
"url": "https://files.pythonhosted.org/packages/91/85/02a5b46c560d5767c05a1d0d0a0a32ea111a0a19065794a604e306970b0e/monoloco-0.4.3.tar.gz"
}
],
"0.4.4": [
{
"comment_text": "",
"digests": {
"md5": "57e1ba338ad45312e6833a5bf5231acd",
"sha256": "0de49c258d889ae950b5a55dbc944889aad6b515fa81fa3be04696050e61a208"
},
"downloads": -1,
"filename": "monoloco-0.4.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "57e1ba338ad45312e6833a5bf5231acd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 69352,
"upload_time": "2019-08-07T10:20:42",
"url": "https://files.pythonhosted.org/packages/bb/74/f304ab028d68862256db1b05c286f1e5ab9dace2573728ff31c34c106cd3/monoloco-0.4.4-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "21e43d3fd3ed32ba79597f410196f73a",
"sha256": "3b329b0d8fe0d6f914b8809f5c6f59bbcac04b9aa5eadf7950d977cc9981f9e1"
},
"downloads": -1,
"filename": "monoloco-0.4.4.tar.gz",
"has_sig": false,
"md5_digest": "21e43d3fd3ed32ba79597f410196f73a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 49755,
"upload_time": "2019-08-07T10:20:43",
"url": "https://files.pythonhosted.org/packages/3d/92/f19f828fa8fadcebb9617af787f4c278f4f502b55828243c5e9ab5d2bbc6/monoloco-0.4.4.tar.gz"
}
],
"0.4.5": [
{
"comment_text": "",
"digests": {
"md5": "3376f9b226e609cbaa8f33d4860c4354",
"sha256": "01bc3bd927992fb4cb20b4d039831ce90c8e5344af488c2c91b35906736ad834"
},
"downloads": -1,
"filename": "monoloco-0.4.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3376f9b226e609cbaa8f33d4860c4354",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 69305,
"upload_time": "2019-08-07T16:00:48",
"url": "https://files.pythonhosted.org/packages/ce/71/c712c56176c84b0ec13c803be121ee62394347031ac1bae31140d3e817fa/monoloco-0.4.5-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "f3fe3855d79ac8736c948084a8ea50e6",
"sha256": "c44d0bc293fef7fc4e4c3cbe6b6e7b519ad0a502b177621cb34d66cb7c8947ea"
},
"downloads": -1,
"filename": "monoloco-0.4.5.tar.gz",
"has_sig": false,
"md5_digest": "f3fe3855d79ac8736c948084a8ea50e6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 49714,
"upload_time": "2019-08-07T16:00:49",
"url": "https://files.pythonhosted.org/packages/e8/68/9786f72c099fa270fd0282a79d4ef56649607669014b2f7d2106e7506966/monoloco-0.4.5.tar.gz"
}
],
"0.4.6": [
{
"comment_text": "",
"digests": {
"md5": "0a77775f2a83679aa5eebfc869ea209f",
"sha256": "08e6516a5229fa52baf254faf8b47c617960365db18e0227557591257633ba40"
},
"downloads": -1,
"filename": "monoloco-0.4.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0a77775f2a83679aa5eebfc869ea209f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 73066,
"upload_time": "2019-08-23T12:41:08",
"url": "https://files.pythonhosted.org/packages/1b/c9/960266cc8011f1d233623579e9eaf923ea7759507c0e72f3846a9c86f6d8/monoloco-0.4.6-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "9a8722494961c1cdfb5c387650635f56",
"sha256": "e2aa7e91724d3b959ca015176bce40f99db2a92c334b2f2aefcc3b27136ee765"
},
"downloads": -1,
"filename": "monoloco-0.4.6.tar.gz",
"has_sig": false,
"md5_digest": "9a8722494961c1cdfb5c387650635f56",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 51500,
"upload_time": "2019-08-23T12:41:09",
"url": "https://files.pythonhosted.org/packages/77/70/e56f9ffc4a7c43bed97667afc32bc2de63a9a541baa78c29f79904f50474/monoloco-0.4.6.tar.gz"
}
]
},
"urls": [
{
"comment_text": "",
"digests": {
"md5": "0a77775f2a83679aa5eebfc869ea209f",
"sha256": "08e6516a5229fa52baf254faf8b47c617960365db18e0227557591257633ba40"
},
"downloads": -1,
"filename": "monoloco-0.4.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0a77775f2a83679aa5eebfc869ea209f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 73066,
"upload_time": "2019-08-23T12:41:08",
"url": "https://files.pythonhosted.org/packages/1b/c9/960266cc8011f1d233623579e9eaf923ea7759507c0e72f3846a9c86f6d8/monoloco-0.4.6-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "9a8722494961c1cdfb5c387650635f56",
"sha256": "e2aa7e91724d3b959ca015176bce40f99db2a92c334b2f2aefcc3b27136ee765"
},
"downloads": -1,
"filename": "monoloco-0.4.6.tar.gz",
"has_sig": false,
"md5_digest": "9a8722494961c1cdfb5c387650635f56",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 51500,
"upload_time": "2019-08-23T12:41:09",
"url": "https://files.pythonhosted.org/packages/77/70/e56f9ffc4a7c43bed97667afc32bc2de63a9a541baa78c29f79904f50474/monoloco-0.4.6.tar.gz"
}
]
}