{ "info": { "author": "Nikhil Kothari", "author_email": "nikhilk@twitter", "bugtrack_url": null, "classifiers": [], "description": "Introduction to TensorFX\n========================\n\nTensorFX is an end to end application framework to simplifies machine\nlearning with `TensorFlow `__ - both training\nmodels and using them for prediction. It is designed from the ground up\nto make the mainline scenarios simple with higher level building blocks,\nwhile ensuring custom or complex scenarios remain possible by preserving\nthe flexibility of TensorFlow APIs.\n\nThere are some important principles that shape the design of the\nframework:\n\n1. **Simple, consistent set of usage patterns** Local or cloud, single\n node or distributed execution, in-memory data or big data sharded\n across files, you should have to write code once, in a single way\n regardless of how the code executes.\n\n2. **A Toolbox with Useful Abstractions** The right entrypoint for the\n task at hand, starting with off-the-shelf algorithms that let you\n focus on feature engineering and hyperparam tuning. If you need to\n solve something unqiue, you can focus on building TensorFlow graphs,\n rather than infrastructure code (distributed cluster setup,\n checkpointing, logging, exporting models etc.).\n\n3. **Declarative** Using YAML, JSON, and simplified Python interfaces to\n minimize the amount of boilerplate code.\n\nOK, enough context... here is some information to get you started.\n\nGetting Started\n---------------\n\nOnce you have a Python environment (recommendation: use Miniconda),\ninstallation is straightforward:\n\n::\n\n pip install tensorflow\n pip install tensorfx\n\nNote that TensorFX depends on TensorFlow 1.0, and supporting libraries\nsuch as numpy and pandas.\n\nDocumentation\n-------------\n\nDocumentation is at https://tensorlab.github.io/tensorfx/. This includes\nAPI reference topics, as well as conceptual and how-to topics. They are\na work-in-progress, but check them out! There are a few samples that\ndemonstrate how to get started as well in the repository. Likewise, more\nto be added over time.\n\nContributions and Development\n-----------------------------\n\nWe welcome contributions in form of ideas, issues, samples as well as\ncode. Since the project is at a super-early stage, and evolving rapidly,\nits best to start a discussion by filing an issue for any contribution.\n\nBuilding and Testing\n~~~~~~~~~~~~~~~~~~~~\n\nIf you want to develop within the repository, clone it, and run the\nfollowing commands:\n\n::\n\n # Install requirements and setup envionment\n source init.sh install\n\n # Build and Test\n ./build.sh test\n\nRelated Links\n~~~~~~~~~~~~~\n\n- Development workflow [TODO: Add wiki entry]\n\nHello World - Iris Classification Model\n---------------------------------------\n\nThis sample here is a quick 5-minute introduction to using TensorFX.\nHere is the code for building a feed-forward neural network\nclassification model for the `iris\ndataset `__.\n\n::\n\n import tensorfx as tfx\n import tensorfx.models.nn as nn\n\n # Hyperparameters, training parameters, and data\n args, job = nn.FeedForwardClassificationArguments.parse(parse_job=True)\n dataset = tfx.data.CsvDataSet(args.data_schema,\n train=args.data_train,\n eval=args.data_eval,\n metadata=args.data_metadata,\n features=args.data_features)\n\n # Instantiating the model builder\n classification = nn.FeedForwardClassification(args, dataset)\n\n # Training\n trainer = tfx.training.ModelTrainer()\n model = trainer.train(classification, job)\n\n # Prediction\n instances = [\n '6.3,3.3,6,2.5', # virginica\n '4.4,3,1.3,0.2', # setosa\n '6.1,2.8,4.7,1.2' # versicolor\n ]\n predictions = model.predict(instances)\n\nHere's an outline steps to perform for basic usage of what TensorFX\noffers:\n\n1. Parse (or build) an Arguments object, usually from the command-line\n to define hyperparameters. This object corresponds to the kind of\n model you are training, so, ``FeedForwardClassificationArguments`` in\n this case.\n2. Create a DataSet to reference training and evaluation data, along\n with supporting configuration - namely - schema, metadata, and\n features (more on these below).\n3. Initialize the model builder - in this case\n ``FeedForwardClassification``.\n4. Initialize the model trainer, and invoke ``train()`` which runs the\n training process to return a model.\n5. Load some instances you want to run through the model and call\n ``predict()``.\n\nSchema - schema.yaml\n^^^^^^^^^^^^^^^^^^^^\n\nThe schema describes the structure of your data. This can be defined\nprogrammatically, but is conveniently expressible in declarative YAML\nform, and placed alongside training data.\n\n::\n\n fields:\n - name: species\n type: discrete\n - name: petal_length\n type: numeric\n - name: petal_width\n type: numeric\n - name: sepal_length\n type: numeric\n - name: sepal_width\n type: numeric\n\nMetadata - metadata.json\n^^^^^^^^^^^^^^^^^^^^^^^^\n\nMetadata is the result of analyzing training data, based on type\ninformation in the schema. Iris is a tiny dataset, so metadata is\nreadily producable using simple python code looping over the data. For\nreal-world and large datasets, you'll find Spark and BigQuery (on Google\nCloud Platform) as essential data processing runtimes. Stay tuned -\nTensorFX will provide support for these capabilities out of the box.\n\n::\n\n {\n \"species\": { \"entries\": [\"setosa\", \"virginica\", \"versicolor\"] },\n \"petal_length\": { \"min\": 4.3, \"max\": 7.9 },\n \"petal_width\": { \"min\": 2.0, \"max\": 4.4 },\n \"sepal_length\": { \"min\": 1.1, \"max\": 6.9 },\n \"sepal_width\": { \"min\": 0.1, \"max\": 2.5 }\n }\n\nFeatures - features.yaml\n^^^^^^^^^^^^^^^^^^^^^^^^\n\nLike schema, features can also be defined programmatically, or expressed\nin YAML. Features describe the set of inputs that your models operate\nover, and how they are produced by applying transformations to the\nfields in your data. These transformations are turned into TensorFlow\ngraph constructs and applied consistently to both training and\nprediction data.\n\nIn this particular example, the FeedForwardClassification model requires\ntwo features: X defining the values the model uses for producing\ninferences, and Y, the target label that the model is expected to\npredict which are defined as follows:\n\n::\n\n features:\n - name: X\n type: concat\n features:\n - name: petal_width\n type: scale\n - name: petal_length\n type: scale\n - name: sepal_width\n type: log\n - name: sepal_length\n type: log\n - name: Y\n type: target\n fields: species\n\nRunning the Model\n^^^^^^^^^^^^^^^^^\n\nThe python code in the sample can be run directly, or using a ``train``\ntool, as shown:\n\n::\n\n python -m tensorfx.tools.train \\\n --module iris.trainer.main \\\n --output /tmp/tensorfx/iris/csv \\\n --data-train iris/data/train.csv \\\n --data-eval iris/data/eval.csv \\\n --data-schema iris/data/schema.yaml \\\n --data-metadata iris/data/metadata.json \\\n --data-features iris/features.yaml \\\n --log-level-tensorflow ERROR \\\n --log-level INFO \\\n --batch-size 5 \\\n --max-steps 2000 \\\n --checkpoint-interval-secs 1 \\\n --hidden-layers:1 20 \\\n --hidden-layers:2 10\n\nOnce the training is complete, you can list the contents of the output\ndirectory. You should see the model (the prediction graph, and learnt\nvariables) in the ``model`` subdirectory, alongside checkpoints, and\nsummaries.\n\n::\n\n ls -R /tmp/tensorfx/iris/csv\n checkpoints job.yaml model summaries\n\n /tmp/tensorfx/iris/csv/checkpoints:\n checkpoint model.ckpt-2000.index\n model.ckpt-1.data-00000-of-00001 model.ckpt-2000.meta\n model.ckpt-1.index model.ckpt-2001.data-00000-of-00001\n model.ckpt-1.meta model.ckpt-2001.index\n model.ckpt-1562.data-00000-of-00001 model.ckpt-2001.meta\n model.ckpt-1562.index model.ckpt-778.data-00000-of-00001\n model.ckpt-1562.meta model.ckpt-778.index\n model.ckpt-2000.data-00000-of-00001 model.ckpt-778.meta\n\n /tmp/tensorfx/iris/csv/model:\n saved_model.pb variables\n\n /tmp/tensorfx/iris/csv/model/variables:\n variables.data-00000-of-00001 variables.index\n\n /tmp/tensorfx/iris/csv/summaries:\n eval prediction train\n\n /tmp/tensorfx/iris/csv/summaries/eval:\n events.out.tfevents.1488351760\n events.out.tfevents.1488352853\n\n /tmp/tensorfx/iris/csv/summaries/prediction:\n events.out.tfevents.1488351765\n\n /tmp/tensorfx/iris/csv/summaries/train:\n events.out.tfevents.1488351760\n events.out.tfevents.1488352852\n\nSummaries are TensorFlow events logged during training. They can be\nobserved while the training job is running (which is essential when\nrunning a long or real training job) to understand how your training is\nprogressing, or how the model is converging (or not!).\n\n::\n\n tensorboard --logdir /tmp/tensorfx/iris/csv\n\nThis should bring up TensorBoard. Its useful to see the graph structure,\nmetrics and other tensors that are automatically published.\n\n**Training Graph**\n\n.. figure:: https://tensorlab.github.io/tensorfx/_static/images/intro-graph.jpg\n :alt: Graphs in TensorBoard\n\n Graphs in TensorBoard\n**Training Metrics -- Accuracy, Loss and Throughput**\n\n.. figure:: https://tensorlab.github.io/tensorfx/_static/images/intro-metrics.jpg\n :alt: Metrics in TensorBoard\n\n Metrics in TensorBoard\n**Model Variables -- Weights, Gradients, etc.**\n\n.. figure:: https://tensorlab.github.io/tensorfx/_static/images/intro-watch.jpg\n :alt: Watchin Learnt Variables\n\n Watchin Learnt Variables\nAs you can see, the out-of-box model takes care of a number of details.\nThe same code can be run on a single machine, or in a cluster (of\ncourse, iris is too simple of a problem to need that).\n", "description_content_type": null, "docs_url": null, "download_url": null, "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/TensorLab/tensorfx", "keywords": "TensorLab,TensorFlow,Machine Learning,Deep Learning,Google", "license": "Apache Software License", "maintainer": null, "maintainer_email": null, "name": "tensorfx", "package_url": "https://pypi.org/project/tensorfx/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/tensorfx/", "project_urls": { "Homepage": "https://github.com/TensorLab/tensorfx" }, "release_url": "https://pypi.org/project/tensorfx/0.1.3/", "requires_dist": null, "requires_python": null, "summary": "TensorFX Framework", "version": "0.1.3" }, "last_serial": 2691164, "releases": { "0.1.3": [ { "comment_text": "", "digests": { "md5": "879fbf8fb87fef4c86e1b248f2c1783f", "sha256": "8f8114755dd678fdc839fa7a53b45a61b4872fb77752144d403903c1dd8bd459" }, "downloads": -1, "filename": "tensorfx-0.1.3.tar.gz", "has_sig": false, "md5_digest": "879fbf8fb87fef4c86e1b248f2c1783f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 27780, "upload_time": "2017-03-08T10:22:48", "url": "https://files.pythonhosted.org/packages/7b/af/f5e8e9e72f17e872ac583e06724ab15bfb25e0d0ad4c389d3d8b992f3f94/tensorfx-0.1.3.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "879fbf8fb87fef4c86e1b248f2c1783f", "sha256": "8f8114755dd678fdc839fa7a53b45a61b4872fb77752144d403903c1dd8bd459" }, "downloads": -1, "filename": "tensorfx-0.1.3.tar.gz", "has_sig": false, "md5_digest": "879fbf8fb87fef4c86e1b248f2c1783f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 27780, "upload_time": "2017-03-08T10:22:48", "url": "https://files.pythonhosted.org/packages/7b/af/f5e8e9e72f17e872ac583e06724ab15bfb25e0d0ad4c389d3d8b992f3f94/tensorfx-0.1.3.tar.gz" } ] }