{ "info": { "author": "Matthew Rocklin", "author_email": "mrocklin@gmail.com", "bugtrack_url": null, "classifiers": [], "description": "Dask-Tensorflow\n===============\n\n.. |Build Status| image:: https://travis-ci.org/mrocklin/dask-tensorflow.svg?branch=master\n :target: https://travis-ci.org/mrocklin/dask-tensorflow\n\nStart TensorFlow clusters from Dask\n\nExample\n-------\n\nGiven a Dask cluster\n\n.. code-block:: python\n\n from dask.distributed import Client\n client = Client('scheduler-address:8786')\n\nGet a TensorFlow cluster, specifying groups by name\n\n.. code-block:: python\n\n from dask_tensorflow import start_tensorflow\n tf_spec, dask_spec = start_tensorflow(client, ps=2, worker=4)\n\n >>> tf_spec\n {'worker': ['192.168.1.100:2222', '192.168.1.101:2222',\n '192.168.1.102:2222', '192.168.1.103:2222'],\n 'ps': ['192.168.1.104:2222', '192.168.1.105:2222']}\n\nThis creates a ``tensorflow.train.Server`` on each Dask worker and sets up a\nQueue for data transfer on each worker. These are accessible directly as\n``tensorflow_server`` and ``tensorflow_queue`` attributes on the workers.\n\nMore Complex Workflow\n---------------------\n\nTypically then we set up long running Dask tasks that get these servers and\nparticipate in general TensorFlow compuations.\n\n.. code-block:: python\n\n from dask.distributed import worker_client\n\n def ps_function(self):\n with worker_client() as c:\n tf_server = c.worker.tensorflow_server\n tf_server.join()\n\n ps_tasks = [client.submit(ps_function, workers=worker, pure=False)\n for worker in dask_spec['ps']]\n\n def worker_function(self):\n with worker_client() as c:\n tf_server = c.worker.tensorflow_server\n\n # ... use tensorflow as desired ...\n\n worker_tasks = [client.submit(worker_function, workers=worker, pure=False)\n for worker in dask_spec['worker']]\n\nOne simple and flexible approach is to have these functions block on queues and\nfeed them data from dask arrays, dataframes, etc.\n\n\n.. code-block:: python\n\n def worker_function(self):\n with worker_client() as c:\n tf_server = c.worker.tensorflow_server\n queue = c.worker.tensorflow_queue\n\n while not stopping_condition():\n batch = queue.get()\n # train with batch\n\nAnd then dump blocks of numpy and pandas dataframes to these queues\n\n.. code-block:: python\n\n from distributed.worker_client import get_worker\n def dump_batch(batch):\n worker = get_worker()\n worker.tensorflow_queue.put(batch)\n\n\n import dask.dataframe as dd\n df = dd.read_csv('hdfs:///path/to/*.csv')\n # clean up dataframe as necessary\n partitions = df.to_delayed() # delayed pandas dataframes\n client.map(dump_batch, partitions)\n\n\n", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "", "keywords": "", "license": "BSD", "maintainer": "", "maintainer_email": "", "name": "dask-tensorflow", "package_url": "https://pypi.org/project/dask-tensorflow/", "platform": "", "project_url": "https://pypi.org/project/dask-tensorflow/", "project_urls": null, "release_url": "https://pypi.org/project/dask-tensorflow/0.0.2/", "requires_dist": [ "dask", "distributed", "tensorflow" ], "requires_python": "", "summary": "Interactions between Dask and Tensorflow", "version": "0.0.2" }, "last_serial": 3477998, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "d80264571f2c4aa18d1548cbd8317376", "sha256": "2fda52e365b86afc67b544bc85421e841ddb273eae701bad0dec98013f0b8f06" }, "downloads": -1, "filename": "dask-tensorflow-0.0.1.tar.gz", "has_sig": false, "md5_digest": "d80264571f2c4aa18d1548cbd8317376", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3352, "upload_time": "2017-10-23T20:33:14", "url": "https://files.pythonhosted.org/packages/36/a1/a7a233185381a117cf1716278524d73d07fb5c1d7ed5ea90ed96dc467e86/dask-tensorflow-0.0.1.tar.gz" } ], "0.0.2": [ { "comment_text": "", "digests": { "md5": "944c7b4c6cde97705bc54036fff531ff", "sha256": "b88b6f330f846bb1a9d58cfdd612ca5a3d3ef4874e3e81405395e6104d8ad2d9" }, "downloads": -1, "filename": "dask_tensorflow-0.0.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "944c7b4c6cde97705bc54036fff531ff", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 5307, "upload_time": "2018-01-10T16:03:24", "url": "https://files.pythonhosted.org/packages/00/a5/6cd58713aacf16fc8ef801e3020894a1faba7710c19c047c3e9582081b20/dask_tensorflow-0.0.2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "0a27b2144c4f550cafd6ddb98a29d018", "sha256": "9687ca632ca9769ccc2c8f99300ebfc675b19ef6369a57e33c00e7178f412d2e" }, "downloads": -1, "filename": "dask-tensorflow-0.0.2.tar.gz", "has_sig": false, "md5_digest": "0a27b2144c4f550cafd6ddb98a29d018", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4762, "upload_time": "2018-01-10T16:03:26", "url": "https://files.pythonhosted.org/packages/9c/6e/b09e3fa80760aef4bb77ed5cee6df94ef21dec347da6be0fea9505f9d792/dask-tensorflow-0.0.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "944c7b4c6cde97705bc54036fff531ff", "sha256": "b88b6f330f846bb1a9d58cfdd612ca5a3d3ef4874e3e81405395e6104d8ad2d9" }, "downloads": -1, "filename": "dask_tensorflow-0.0.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "944c7b4c6cde97705bc54036fff531ff", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 5307, "upload_time": "2018-01-10T16:03:24", "url": "https://files.pythonhosted.org/packages/00/a5/6cd58713aacf16fc8ef801e3020894a1faba7710c19c047c3e9582081b20/dask_tensorflow-0.0.2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "0a27b2144c4f550cafd6ddb98a29d018", "sha256": "9687ca632ca9769ccc2c8f99300ebfc675b19ef6369a57e33c00e7178f412d2e" }, "downloads": -1, "filename": "dask-tensorflow-0.0.2.tar.gz", "has_sig": false, "md5_digest": "0a27b2144c4f550cafd6ddb98a29d018", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4762, "upload_time": "2018-01-10T16:03:26", "url": "https://files.pythonhosted.org/packages/9c/6e/b09e3fa80760aef4bb77ed5cee6df94ef21dec347da6be0fea9505f9d792/dask-tensorflow-0.0.2.tar.gz" } ] }