{ "info": { "author": "NVIDIA", "author_email": "duncan@nvidia.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python", "Topic :: Scientific/Engineering :: Artificial Intelligence" ], "description": "# TensorFlow Determinism\n\nThis repository serves three purposes:\n\n 1. Provide up-to-date information (in this file) about non-determinism\n sources and solutions in TensorFlow and beyond, with a focus on determinism\n when running on GPUs.\n 2. Provide a patch to attain various levels of GPU-specific determinism in\n stock TensorFlow, via the installation of the `tensorflow-determinism` pip\n package.\n 3. Be the location where a TensorFlow determinism debug tool will be released\n as part of the `tensorflow-determinism` pip package.\n\nFor more information, please watch the video of the GTC 2019 talk\n[_Determinism in Deep Learning_][1]. The desciption under that video also\nincludes links to the slides from the talk and to a poster presentation on this\ntopic.\n\n## Installation\n\nUse `pip` to install:\n\n```\npip install tensorflow-determinism\n```\n\nThis will install a package that can be imported as `tfdeterminism`. The\ninstallation of `tensorflow-determinism` will not automatically install\nTensorFlow. The intention of this is to allow you to install your chosen\nversion of TensorFlow. You will need to install your chosen version of\nTensorFlow before you can import and use `tfdeterminism`.\n\n## Deterministic TensorFlow Solutions\n\nThere are currently two main ways to access GPU-deterministic functionality in\nTensorFlow for most deep learning applications. The first way is to use an\nNVIDIA NGC TensorFlow container. The second way is to use version 1.14, 1.15,\nor 2.0 of stock TensorFlow with GPU support, plus the application of a patch\nsupplied in this repo.\n\nThe longer-term intention and plan is to upstream all solutions into stock\nTensorFlow.\n\nDeterminism is not guaranteed when XLA JIT compilation is enabled.\n\n### NVIDIA NGC TensorFlow Containers\n\nNGC TensorFlow containers, starting with version 19.06, implement\nGPU-deterministic TensorFlow functionality. In Python code running inside the\ncontainer, this can be enabled as follows:\n\n```\nimport tensorflow as tf\nimport os\nos.environ['TF_DETERMINISTIC_OPS'] = '1'\n# Now build your graph and train it\n```\n\nThe following table shows which version of TensorFlow each NGC container\nversion is based on:\n\n NGC Container Version | TensorFlow Version |\n:----------------------|:-------------------|\n 19.06 | 1.13 |\n 19.07 - 19.09 | 1.14 |\n\nFor information about pulling and running the NVIDIA NGC containers, see [these\ninstructions][2].\n\n### Stock TensorFlow\n\nVersions 1.14, 1.15, and 2.0 of stock TensorFlow implement a reduced form of GPU\ndeterminism, which must be supplemented with a patch provided in this repo.\nThe following Python code is running on a machine in which `pip` package\n`tensorflow-gpu=2.0.0` has been installed correctly and on which\n`tensorflow-determinism` has also been installed (as shown in the\n[installation](#installation) section above).\n\n```\nimport tensorflow as tf\nfrom tfdeterminism import patch\npatch()\n# use tf as normal\n```\n\nStock TensorFlow with GPU support can be installed as follows:\n\n```\npip install tensorflow-gpu=2.0.0\n```\n\nThe TensorFlow project includes [detailed instructions][3] for installing\nTensorFlow with GPU support.\n\n### Additional Ingredients in the Determinism Recipe\n\nYou'll also need to set any and all appropriate random seeds:\n\n```\nos.environ['PYTHONHASHSEED']=str(SEED)\nrandom.seed(SEED)\nnp.random.seed(SEED)\ntf.set_random_seed(SEED)\n```\n\nIf you're using Horovod for multi-GPU training, you may need to disable Tensor\nFusion (assuming that the non-determinism associated with Tensor Fusion has not\nyet been resolved):\n\n```\nos.environ['HOROVOD_FUSION_THRESHOLD']='0'\n```\n\n## Detailed Status of Determinism in TensorFlow and Beyond\n\nConfirmed and likely sources of non-determinism, along with any existing\nsolutions, are being tracked here.\n\n### GPU-Specific Sources of Non-Determinism\n\n#### Historic GPU-Specific Sources of Non-Determinism\n\nIn the past, `tf.math.reduce_sum` and `tf.math.reduce_mean` operated\nnon-deterministically when running on a GPU. This was resolved before\nTensorFlow version 1.12. These ops now function deterministically\nby default when running on a GPU.\n\n#### Confirmed Current GPU-Specific Sources of Non-Determinism (With Solutions)\n\n Source | NGC 19.06+ / TF 2.1 | TF 1.14, 1.15, 2.0 |\n:-----------------------------------------------|:--------------------|:--------------------|\n TF auto-tuning of cuDNN convolution algorithms | TCD or TDO | TCD or TDP |\n cuDNN convolution backprop to weight gradients | TCD or TDO | TCD or TDP |\n cuDNN convolution backprop to data gradients | TCD or TDO | TCD or TDP |\n cuDNN max-pooling backprop | TCD or TDO | TCD or TDP |\n `tf.nn.bias_add` backprop (see XLA note) | TDO | TDP |\n `tf.image.resize_bilinear` fwd and bwd | NS1 | NS1 |\n\nKey to the solutions refenced above:\n\n Solution | Description |\n:---------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n TCD | Set environment variable `TF_CUDNN_DETERMINISTIC` to '1' or 'true'. Also *do not* set environment variable `TF_USE_CUDNN_AUTOTUNE` at all (and particularly *do not* set it to '0' or 'false'). |\n TDO | Set environment variable `TF_DETERMINISTIC_OPS` to '1' or 'true'. Also *do not* set environment variable `TF_USE_CUDNN_AUTOTUNE` at all (and particularly *do not* set it to '0' or 'false'). |\n TDP | Apply `tfdeterminism.patch`. Note that solution TDO will be in stock TensorFlow v2.1 (see [PR 31465](https://github.com/tensorflow/tensorflow/pull/31465)). |\n NS1 | There is currently no solution available for this, but one is under development. |\n\nNotes:\n * XLA: These solutions will not work when XLA JIT compilation is enabled.\n\n#### Other Possible GPU-Specific Sources of Non-Determinism\n\nGoing beyond the above-mentioned sources, in version 1.12 of TensorFlow (and\nalso in the master branch on 2019-03-03, afer release 1.31.1), the following\nfiles call CUDA `atomicAdd` either directly or indirectly. This makes them\ncandidates for the injection of non-determinism.\n\n* `crop_and_resize_op_gpu.cu.cc`\n* `scatter_functor_gpu.cu.h`\n* `scatter_nd_op_gpu.cu.cc`\n* `sparse_tensor_dense_matmul_op_gpu.cu.cc`\n* `resize_nearest_neighbor_op_gpu.cu.cc`\n* `segment_reduction_ops.h`\n* `segment_reduction_ops_gpu.cu.cc`\n* `dilation_ops_gpu.cu.cc`\n* `maxpooling_op_gpu.cu.cc`\n* `svd_op_gpu.cu.cc`\n* `cuda_kernel_helper_test.cu.cc`\n* `depthwise_conv_op_gpu.h`\n* `resampler_ops_gpu.cu.cc`\n* `histogram_op_gpu.cu.cc`\n* `stateful_random_ops_gpu.cu.cc`\n\nUnless you are using TensorFlow ops that depend on these files (i.e. ops with\nsimilar names), then your model will not be affected by these potential sources\nof non-determinism.\n\nBeyond `atomicAdd`, there are ten other CUDA [atomic functions][4] whose use\ncould lead to the injection of non-determinism, such as `atomicCAS` (the most\ngeneric, atomic compare and swap). Note also that the word 'atomic' was present\nin 167 files in the TensorFlow repo and some of these may be related to the use\nof CUDA atomic operations. It's important to remember that it's possible to use\nCUDA atomic operations without injecting non-determinism, and that, therefore,\nwhen CUDA atomic operations are present in op code, it doesn't guarantee that\nthe op injects non-determinism into the computation.\n\n### Sources of Non-Determinism in TensorFlow Unrelated to GPU\n\n* [Issue 29101](https://github.com/tensorflow/tensorflow/issues/29101): Random\n seed not set in graph context of `Dataset#map`. This may have been resolved\n in version 1.14 of TensorFlow.\n* `tf.data.Dataset` with more than one worker. The work-around is to use only\n one worker.\n\n### Sources of Non-Determinism Beyond TensorFlow\n\n* TensorRT timing-based kernel schedule. Each time an inference engine is\n generated, it could be slightly different, particularly if there is varying\n load on the machine used to run TensorRT. There is a solution planned for\n this.\n* Horovod Tensor Fusion. Work-around: disable Tensor Fusion by setting the\n environment variable `HOROVOD_FUSION_THRESHOLD` to '0'. This issue may have\n been resolved by Horovod\n [pull-request 1130](https://github.com/horovod/horovod/pull/1130) (not yet\n confirmed).\n\n## Relevant Links\n\nThis section catalogs relevant links.\n\n### TensorFlow Issues\n\nNumber | Title | Updated |\n--------------------------------------------------------------:|:--------------------------------------------------------------------------------------|:-----------|\n [2652](https://github.com/tensorflow/tensorflow/issues/2652) | Backward pass of broadcasting on GPU is non-deterministic | 2019-10-08 |\n [2732](https://github.com/tensorflow/tensorflow/issues/2732) | Mention that GPU reductions are nondeterministic in docs | 2019-10-08 |\n[13932](https://github.com/tensorflow/tensorflow/issues/13932) | Non-determinism from `tf.data.Dataset.map` with random ops | |\n[16889](https://github.com/tensorflow/tensorflow/issues/16889) | Problems Getting TensorFlow to behave Deterministically | 2019-10-08 |\n[18096](https://github.com/tensorflow/tensorflow/issues/18096) | Feature Request: Support for configuring deterministic options of cuDNN conv routines | 2019-10-08 |\n[29101](https://github.com/tensorflow/tensorflow/issues/29101) | Random seed not set in graph context of `Dataset#map` | |\n\n### TensorFlow Pull Requests\n\nNumber | Title | Status | Updated |\n-------------------------------------------------------------:|:-------------------------------------------------------------|:--------------------|:-----------|\n[10636](https://github.com/tensorflow/tensorflow/pull/10636) | Non-determinism Docs | closed (not merged) | 2019-10-08 |\n[24273](https://github.com/tensorflow/tensorflow/pull/24273) | Enable dataset.map to respect seeds from the outer context | closed (not merged) | N/A |\n[24747](https://github.com/tensorflow/tensorflow/pull/24747) | Add cuDNN deterministic env variable (only for convolution). | merged pre-1.14 | N/A |\n[25269](https://github.com/tensorflow/tensorflow/pull/25269) | Add deterministic cuDNN max-pooling | merged pre-1.14 | N/A |\n[25796](https://github.com/tensorflow/tensorflow/pull/25796) | Added tests for `TF_CUDNN_DETERMINISTIC` | merged pre-1.14 | N/A |\n[29667](https://github.com/tensorflow/tensorflow/pull/29667) | Add release note about `TF_CUDNN_DETERMINISTIC` | merged into r1.14 | N/A |\n[31389](https://github.com/tensorflow/tensorflow/pull/31389) | Enhance release notes related to `TF_CUDNN_DETERMINISTIC` | merged into r1.14 | N/A |\n[31465](https://github.com/tensorflow/tensorflow/pull/31465) | Add GPU-deterministic `tf.nn.bias_add` | merged pre-2.1 | N/A |\n[32979](https://github.com/tensorflow/tensorflow/pull/32979) | Fix typo in release note | closed (not merged) | N/A |\n[33483](https://github.com/tensorflow/tensorflow/pull/33483) | Fix small typo in v2.0.0 release note | | N/A |\n\n### Miscellaneous\n\n* Two Sigma: [A Workaround for Non-Determinism in\n TensorFlow](http://bit.ly/two-sigma-determinism)\n* Keras [issue 12800](https://github.com/keras-team/keras/issues/12800):\n Unable to get reproducible results using Keras with TF backend on GPU (updated\n on 2019-10-08)\n* PyTorch [Reproducibility](http://bit.ly/pytorch-determinism) (from the\n official documentation)\n* Chainer [PR 2710](https://github.com/chainer/chainer/pull/2710): cuDNN\n Deterministic mode\n* Stack Overflow: [Tensorflow: Different results with the same random seed][501]\n* Stack Overflow: [Are tensorflow random values guaranteed to be the same\n inside a single run? (comment)][502] (updated 2019-10-10).\n\n[501]: https://stackoverflow.com/questions/54047654/tensorflow-different-results-with-the-same-random-seed\n[502]: https://stackoverflow.com/questions/52213325/are-tensorflow-random-values-guaranteed-to-be-the-same-inside-a-single-run#comment91376212_52213325\n\n[1]: http://bit.ly/determinism-in-deep-learning\n[2]: https://ngc.nvidia.com/catalog/containers/nvidia:tensorflow\n[3]: https://www.tensorflow.org/install/gpu\n[4]: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#atomic-functions", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/NVIDIA/tensorflow-determinism", "keywords": "tensorflow gpu deep-learning determinism", "license": "Apache 2.0", "maintainer": "", "maintainer_email": "", "name": "tensorflow-determinism", "package_url": "https://pypi.org/project/tensorflow-determinism/", "platform": "", "project_url": "https://pypi.org/project/tensorflow-determinism/", "project_urls": { "Homepage": "https://github.com/NVIDIA/tensorflow-determinism" }, "release_url": "https://pypi.org/project/tensorflow-determinism/0.3.0/", "requires_dist": null, "requires_python": "", "summary": "Tracking, debugging, and patching non-determinism in TensorFlow", "version": "0.3.0", "yanked": false, "yanked_reason": null }, "last_serial": 6026604, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "8ea47a56c340cf39a5791dc66c2142cd", "sha256": "f066532f37554af4e8bc44668bfb65f5208f4ad5f937d8043567f417b03df31e" }, "downloads": -1, "filename": "tensorflow-determinism-0.1.0.tar.gz", "has_sig": false, "md5_digest": "8ea47a56c340cf39a5791dc66c2142cd", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7218, "upload_time": "2019-09-20T00:01:10", "upload_time_iso_8601": "2019-09-20T00:01:10.061941Z", "url": "https://files.pythonhosted.org/packages/7f/12/f8aafd9e5e2a3d8a671a1e2966faea282160ebb717e97a5871934a023577/tensorflow-determinism-0.1.0.tar.gz", "yanked": false, "yanked_reason": null } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "1249f12f52867914026a6e9b5af02418", "sha256": "1e364b142acd96ec8a8fca6dee32ef04e1c5f5ca20bd83fc9ec37f7dbf2fb5b5" }, "downloads": -1, "filename": "tensorflow-determinism-0.2.0.tar.gz", "has_sig": false, "md5_digest": "1249f12f52867914026a6e9b5af02418", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 10255, "upload_time": "2019-10-07T22:48:47", "upload_time_iso_8601": "2019-10-07T22:48:47.783654Z", "url": "https://files.pythonhosted.org/packages/9b/28/4aa0d8f183d13d3477626e931a3363651bb8e291165d58962a39c57ae7fb/tensorflow-determinism-0.2.0.tar.gz", "yanked": false, "yanked_reason": null } ], "0.3.0": [ { "comment_text": "", "digests": { "md5": "69138227a4773ea74b4fb5fee9b7ba9d", "sha256": "563316b785614df300bbdec9b3c45a94d212761aec796cf1aafcbb2f69fdeb01" }, "downloads": -1, "filename": "tensorflow-determinism-0.3.0.tar.gz", "has_sig": false, "md5_digest": "69138227a4773ea74b4fb5fee9b7ba9d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12300, "upload_time": "2019-10-24T23:53:12", "upload_time_iso_8601": "2019-10-24T23:53:12.311321Z", "url": "https://files.pythonhosted.org/packages/76/56/79d74f25b326d8719753172496abc524980fa67d1d98bb247021376e370a/tensorflow-determinism-0.3.0.tar.gz", "yanked": false, "yanked_reason": null } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "69138227a4773ea74b4fb5fee9b7ba9d", "sha256": "563316b785614df300bbdec9b3c45a94d212761aec796cf1aafcbb2f69fdeb01" }, "downloads": -1, "filename": "tensorflow-determinism-0.3.0.tar.gz", "has_sig": false, "md5_digest": "69138227a4773ea74b4fb5fee9b7ba9d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12300, "upload_time": "2019-10-24T23:53:12", "upload_time_iso_8601": "2019-10-24T23:53:12.311321Z", "url": "https://files.pythonhosted.org/packages/76/56/79d74f25b326d8719753172496abc524980fa67d1d98bb247021376e370a/tensorflow-determinism-0.3.0.tar.gz", "yanked": false, "yanked_reason": null } ], "vulnerabilities": [] }