{ "info": { "author": "Steffen Grohsschmiedt", "author_email": "steffen@logicalclocks.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 3", "Topic :: Utilities" ], "description": "===================\nhopsworks-cloud-sdk\n===================\n\n|Downloads| |PypiStatus| |PythonVersions|\n\n.. |Downloads| image:: https://pepy.tech/badge/hopsworks-cloud-sdk\n :target: https://pepy.tech/project/hopsworks-cloud-sdk\n.. |PypiStatus| image:: https://img.shields.io/pypi/v/hopsworks-cloud-sdk.svg\n :target: https://pypi.org/project/hopsworks-cloud-sdk\n.. |PythonVersions| image:: https://img.shields.io/pypi/pyversions/hopsworks-cloud-sdk.svg\n :target: https://travis-ci.org/hopsworks-cloud-sdk\n\n`hopsworks-cloud-sdk` is an SDK to integrate existing cloud solutions such as Amazon SageMaker our Databricks with the Hopsworks platform.\n\nIt enables accessing the Hopsworks feature store from SageMaker and Databricks notebooks.\n\n-----------\nQuick Start\n-----------\n\nEnsure that your Hopsworks installation is set up correctly: `Setting up Hopsworks for the cloud `_\n\nTo Install:\n\n>>> pip install hopsworks-cloud-sdk\n\nSample usage:\n\n>>> from hops import featurestore\n>>> featurestore.connect('ec2-w-x-y-z.us-east-2.compute.amazonaws.com', 'my_hopsworks_project')\n>>> features_df = featurestore.get_features([\"my_feature_1\", \"my_feature_2\"])\n\n--------\nExamples\n--------\n\n`Examples for using the Cloud SDK on SageMaker `_\n\n------------------------------------\nDocumentation\n------------------------------------\n\nAPI for the Hopsworks Feature Store\n--------------------------------------------------------------------\nHopsworks has a data management layer for machine learning, called a feature store.\nThe feature store enables simple and efficient versioning, sharing, governance and definition of features that can be used to both train machine learning models or to serve inference requests.\nThe featurestore serves as a natural interface between data engineering and data science.\n\n**Reading from the featurestore**:\n\n.. code-block:: python\n\n from hops import featurestore\n features_df = featurestore.get_features([\"team_budget\", \"average_attendance\", \"average_player_age\"])\n\n**Integration with Sci-kit Learn**:\n\n.. code-block:: python\n\n from hops import featurestore\n train_df = featurestore.get_featuregroup(\"iris_features\", dataframe_type=\"pandas\")\n x_df = train_df[['sepal_length', 'sepal_width', 'petal_length', 'petal_width']]\n y_df = train_df[[\"label\"]]\n X = x_df.values\n y = y_df.values.ravel()\n iris_knn = KNeighborsClassifier()\n iris_knn.fit(X, y)\n\n**Integration with Tensorflow**:\n\n.. code-block:: python\n\n from hops import featurestore\n feature_list = [\"team_budget\", \"average_attendance\", \"average_player_age\",\n \"team_position\", \"sum_attendance\",\n \"average_player_rating\", \"average_player_worth\", \"sum_player_age\",\n \"sum_player_rating\", \"sum_player_worth\", \"sum_position\",\n \"average_position\"\n ]\n\n latest_version = featurestore.get_latest_training_dataset_version(\"team_position_prediction\")\n featurestore.create_training_dataset(\n features = feature_list,\n training_dataset = \"team_position_prediction\",\n descriptive_statistics = False,\n feature_correlation = False,\n feature_histograms = False,\n cluster_analysis = False,\n training_dataset_version = latest_version + 1\n )\n\n def create_tf_dataset():\n dataset_dir = featurestore.get_training_dataset_path(\"team_position_prediction\")\n input_files = tf.gfile.Glob(dataset_dir + \"/part-r-*\")\n dataset = tf.data.TFRecordDataset(input_files)\n tf_record_schema = ... # Add tf schema\n feature_names = [\"team_budget\", \"average_attendance\", \"average_player_age\", \"sum_attendance\",\n \"average_player_rating\", \"average_player_worth\", \"sum_player_age\", \"sum_player_rating\", \"sum_player_worth\",\n \"sum_position\", \"average_position\"\n ]\n label_name = \"team_position\"\n\n def decode(example_proto):\n example = tf.parse_single_example(example_proto, tf_record_schema)\n x = []\n for feature_name in feature_names:\n x.append(example[feature_name])\n y = [tf.cast(example[label_name], tf.float32)]\n return x,y\n\n dataset = dataset.map(decode).shuffle(SHUFFLE_BUFFER_SIZE).batch(BATCH_SIZE).repeat(NUM_EPOCHS)\n return dataset\n\n tf_dataset = create_tf_dataset()\n\n**Feature Visualizations**:\n\n.. _feature_plots1.png: imgs/feature_plots1.png\n.. figure:: imgs/feature_plots1.png\n :alt: Visualizing feature distributions\n :target: `feature_plots1.png`_\n :align: center\n :scale: 75 %\n :figclass: align-center\n\n\n.. _feature_plots2.png: imgs/feature_plots2.png\n.. figure:: imgs/feature_plots2.png\n :alt: Visualizing feature correlations\n :target: `feature_plots2.png`_\n :align: center\n :scale: 75 %\n :figclass: align-center\n\n------------------------\nDevelopment Instructions\n------------------------\n\nFor development details such as how to test and build docs, see this reference: Development_.\n\n.. _Development: ./Development.rst", "description_content_type": "", "docs_url": null, "download_url": "http://snurran.sics.se/hops/hopsworks-cloud-sdk/hops-1.0.0.3.tar.gz", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/logicalclocks/hopsworks-cloud-sdk", "keywords": "Hopsworks,SageMaker,Databricks", "license": "Apache License 2.0", "maintainer": "", "maintainer_email": "", "name": "hopsworks-cloud-sdk", "package_url": "https://pypi.org/project/hopsworks-cloud-sdk/", "platform": "", "project_url": "https://pypi.org/project/hopsworks-cloud-sdk/", "project_urls": { "Download": "http://snurran.sics.se/hops/hopsworks-cloud-sdk/hops-1.0.0.3.tar.gz", "Homepage": "https://github.com/logicalclocks/hopsworks-cloud-sdk" }, "release_url": "https://pypi.org/project/hopsworks-cloud-sdk/1.0.0.3/", "requires_dist": null, "requires_python": "", "summary": "An SDK to integrate cloud solutions such as SageMaker and Databricks with Hopsworks.", "version": "1.0.0.3" }, "last_serial": 5948584, "releases": { "1.0.0.0": [ { "comment_text": "", "digests": { "md5": "52dc89af5678d639a3643cecfda7d16f", "sha256": "713cc6933eccee34e98a57424fdacfad69d9e69cbd72cc1abe2ccdba300b4af0" }, "downloads": -1, "filename": "hopsworks-cloud-sdk-1.0.0.0.tar.gz", "has_sig": false, "md5_digest": "52dc89af5678d639a3643cecfda7d16f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 39460, "upload_time": "2019-10-07T14:38:05", "url": "https://files.pythonhosted.org/packages/21/b1/f473304a61f53b7fda037a395cdab9d25131a7a50bb7add6473d86a70237/hopsworks-cloud-sdk-1.0.0.0.tar.gz" } ], "1.0.0.1": [ { "comment_text": "", "digests": { "md5": "931a3fa63eb75036c8efa12729a0e120", "sha256": "f1242361dd6d9032832589a71508b7a42d874ba435bdd9e04fd2fec848cac655" }, "downloads": -1, "filename": "hopsworks-cloud-sdk-1.0.0.1.tar.gz", "has_sig": false, "md5_digest": "931a3fa63eb75036c8efa12729a0e120", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 39527, "upload_time": "2019-10-08T14:13:36", "url": "https://files.pythonhosted.org/packages/cf/c1/0e1a85ec172162ae04734afc76c962a08f8d6c2bc3bd7cb3b963556e213f/hopsworks-cloud-sdk-1.0.0.1.tar.gz" } ], "1.0.0.2": [ { "comment_text": "", "digests": { "md5": "b6bc64fe1ebf54ad24bc085e73a9d0f1", "sha256": "cd25c9eee9d0ba9538e9712e6eabc007515283b5fe51805ef55c78a65d9344a0" }, "downloads": -1, "filename": "hopsworks-cloud-sdk-1.0.0.2.tar.gz", "has_sig": false, "md5_digest": "b6bc64fe1ebf54ad24bc085e73a9d0f1", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 39538, "upload_time": "2019-10-09T08:42:01", "url": "https://files.pythonhosted.org/packages/6b/e9/d18b00206e2d8faa5a3e8c4c9dae93418c1a32c67480636e5a2b3a75eb23/hopsworks-cloud-sdk-1.0.0.2.tar.gz" } ], "1.0.0.3": [ { "comment_text": "", "digests": { "md5": "b20ce086867ef9c86636639a284e0790", "sha256": "216ca4b0dd20e925ac57e5230af428ea7d79fdf5667d80e498c95d123ada6e8d" }, "downloads": -1, "filename": "hopsworks-cloud-sdk-1.0.0.3.tar.gz", "has_sig": false, "md5_digest": "b20ce086867ef9c86636639a284e0790", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 39533, "upload_time": "2019-10-09T08:59:27", "url": "https://files.pythonhosted.org/packages/a7/da/dadea224e0afc8d36ac4c6cea1829b7f29c5a9647ef7e5472b5c3e64812e/hopsworks-cloud-sdk-1.0.0.3.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "b20ce086867ef9c86636639a284e0790", "sha256": "216ca4b0dd20e925ac57e5230af428ea7d79fdf5667d80e498c95d123ada6e8d" }, "downloads": -1, "filename": "hopsworks-cloud-sdk-1.0.0.3.tar.gz", "has_sig": false, "md5_digest": "b20ce086867ef9c86636639a284e0790", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 39533, "upload_time": "2019-10-09T08:59:27", "url": "https://files.pythonhosted.org/packages/a7/da/dadea224e0afc8d36ac4c6cea1829b7f29c5a9647ef7e5472b5c3e64812e/hopsworks-cloud-sdk-1.0.0.3.tar.gz" } ] }