{
    "info": {
        "author": "Abhijit Balaji",
        "author_email": "balaabhijit5@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3"
        ],
        "description": "# Model-Server\n\nA Pure `python-3` based flexible gRPC server for hosting Deep Learning, Machine Learning models trained on any framework!\n\n## Installation\n\n### Method 1:\nInstalling from [python pip](https://pypi.org/project/model-server)\n\n`pip3 install model-server`\n\n### Method 2:\nCreating wheel from github\n\n1. clone the repository\n2. Run `bash create_pip_wheel_and_upload.sh`. This will prompt for userid and password. You can `ctrl-c` this\n3. Then install the created wheel\n\n### Method 3:\n\nNo installation. Using the source code directly.\n\nIf this is the case, you need to compile the protobufs. Run `bash compile_protobufs.sh`. Then add the project root to your `$PYTHONPATH`.\n\n### Note:\nMethod 2 and 3 requires `libprotoc>=3.6.0`\n\n\n## Why Model-Server?\n\nTaking deep learning models to production at scale is not a very straight forward process. If you are using Tensorflow then you have [Tensorflow Serving](https://www.tensorflow.org/tfx/guide/serving). But, if you are using other frameworks like [PyTorch](https://pytorch.org/), [MXNet](https://mxnet.apache.org/), [scikit-learn](https://scikit-learn.org/stable/) etc. Taking your model to production is not very straight forward (Flask, Django and other ReST frameworks). Ideally you should be able to extend Tensorflow Serving to support models from other frameworks also but, this is extremly cumbersome! Thus, to bridge this gap we have [Model-Server](https://abhijit-2592.github.io/model-server/): A high performance framework neutral serving solution! **The idea is:** if you are able to train your model in `python` you should be  able to deploy at scale using pure `python`\n\n## Salient Features\n\n [Model-Server](https://abhijit-2592.github.io/model-server/) is heavily inspired from [Tensorflow Serving](https://www.tensorflow.org/tfx/guide/serving)\n\n* **Out of box client side batching support**\n* **Pure python implementation**: You don't need to fiddle around with C++ to have a scalable deployment solution\n* **Framework neutral**:  Using [PyTorch](https://pytorch.org/), [MXNet](https://mxnet.apache.org/) etc? Don't worry! The solution is platform neutral. If you can use a framework to train in `python-3`, [Model-Server](https://abhijit-2592.github.io/model-server/) can deploy it at scale\n* **Single server for multi-framework and multi-models**: You can host multiple models using the same framework or a mixture of multiple [PyTorch](https://pytorch.org/), [MXNet](https://mxnet.apache.org/), [scikit-learn](https://scikit-learn.org/stable/) [Tensorflow](https://www.tensorflow.org/) etc models!\n\n## Getting started\n\nThe core of Model Server is a `Servable`. A servable is nothing but a `python class` containing your model's prediction definition which will be served by the `Model-Server`. All servables must inherit from `model_server.Servable` for the  `Model-Server` to serve it.\n\nTo deploy your model to production with `Model Server`, you just have to write a single `python-3` file containing a `class` which inherits from `model_server.Servable` and has the following two methods:\n\n```python\npredict(self, input_array_dict)\nget_model_info(self, list_of_model_info_dict)\n```\n\nNow run the floowing to start the server in `5001` port\n```bash\npython -m model_server.runserver path_to_custom_servable_file.py\n```\n\nFor more info on  command line arguments:\n```bash\npython -m model_server.runserver --help\n```\n\n\n### A simple example\n\ncreate a file called `simple_servable.py` with the following contents:\n```python\n\nimport numpy as np\nfrom model_server import Servable\n\n\nclass my_custom_servable(Servable):\n    def __init__(self, args):\n        # args contains values from ArgumentParser\n        # Thus you can pass any kwargs via command line and you get them here\n        pass\n\n    def predict(self, input_array_dict):\n        \"\"\"This method is responsible for the gRPC call GetPredictions().\n        All custom servables must define this method.\n\n        Arguments:\n            input_array_dict (dict): The PredictionRequest proto decoded as a python dictionary.\n\n        # example\n        input_array_dict = {\n                           \"input_tensor_name1\": numpy array,\n                           \"input_tensor_name2\": numpy array\n                            }\n\n        Returns:\n            A python dictionary with key (typically output name) and value as numpy array of predictions\n\n        # example\n        output = {\n                   \"output_tensor_name1\": numpy array,\n                   \"output_tensor_name2\": numpy array\n                  }\n        \"\"\"\n        print(input_array_dict)\n        return ({\"output_array1\": np.array([100, 200]).astype(np.float32),\n                 \"output_array2\": np.array([\"foo\".encode(),\"bar\".encode()]).astype(object),  # you can get and pass strings encoded as bytes also\n                 })\n\n    def get_model_info(self, list_of_model_info_dict):\n        \"\"\"This method which is responsible for the call GetModelInfo()\n\n        Arguments:\n            list_of_model_info_dict (list/tuple): A list containing model_info_dicts\n\n        Note:\n            model_info_dict contains the following keys:\n\n            {\n                \"name\": \"model name as string\"\n                \"version\": \"version as string\"\n                \"status\": \"status string\"\n                \"misc\": \"string with miscellaneous info\"\n            }\n\n        Returns:\n            list_of_model_info_dict (dict): containing the model and server info. This is similar to the function input\n        \"\"\"\n        return [{\"name\": \"first_model\", \"version\": 1, \"status\": \"up\"},\n                {\"name\": \"second_model\", \"version\": 2, \"status\": \"up\", \"misc\": \"Other miscellaneous details\"}]\n```\n\nNow run:\n\n```bash\npython -m model_server.runserver path/to/simple_servable.py\n```\nTo start the gRPC server!\n\nNow let's define the client!\n\n```python\nimport grpc\nimport numpy as np\n\nfrom model_server import server_pb2, server_pb2_grpc\nfrom model_server.utils import create_tensor_proto\nfrom model_server.utils import create_predict_request\nfrom model_server.utils import create_array_from_proto\nfrom model_server.utils import create_model_info_proto\n\nchannel = grpc.insecure_channel('localhost:5001')  # default port\n# create a stub (client)\nstub = server_pb2_grpc.ModelServerStub(channel)\ninput_array_dict = {\"input1\":create_tensor_proto(np.array([1,2]).astype(np.uint8)),\n                    \"input2\":create_tensor_proto(np.array([[10.0,11.0], [12.0,13.0]]).astype(np.float32)),\n                    \"input3\":create_tensor_proto(np.array([\"Hi\".encode(), \"Hello\".encode(), \"test\".encode()]).astype(object))\n                   }\n# create the prediction request\npredict_request= create_predict_request(input_array_dict, name=\"simple_call\")\n# make the call\nresponse = stub.GetPredictions(predict_request)\n\n# decode the response\nprint(create_array_from_proto(response.outputs[\"output_array1\"]))\n\n# prints: array([100., 200.], dtype=float32)\n\n# Getting the model status\n\nmodel_info_proto = create_model_info_proto([])  # you can pass an empty list also\nresponse = stub.GetModelInfo(model_info_proto)\n\n```\n\nLook at [examples](https://github.com/Abhijit-2592/model-server/tree/master/examples) folder for further examples\n\n\n## Work in Progress\n\n- Support server side batching and async calls.\n- Provide a gRPC endpoint for [Active Learning](https://en.wikipedia.org/wiki/Active_learning_(machine_learning)) so that you can plug in `Model Server` with your labeling tool and train on fly!\n- Provide a ReST wrapper\n\nFeel free to file issues, provide suggestions and pull requests\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/Abhijit-2592/model-server",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "model-server",
        "package_url": "https://pypi.org/project/model-server/",
        "platform": "",
        "project_url": "https://pypi.org/project/model-server/",
        "project_urls": {
            "Homepage": "https://github.com/Abhijit-2592/model-server"
        },
        "release_url": "https://pypi.org/project/model-server/1.0/",
        "requires_dist": [
            "grpcio (==1.21.1)",
            "numpy (>=1.14.5)",
            "pytest"
        ],
        "requires_python": "",
        "summary": "A Pure `python-3` based flexible gRPC server for hosting Deep Learning, Machine Learning models trained on any framework!",
        "version": "1.0"
    },
    "last_serial": 5434603,
    "releases": {
        "1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fdb2e234ca8410083f780c98cf97bb0e",
                    "sha256": "9b4a7a4058e558cebfe1389a5512d49543d150ab8f56f911642fb08df532c126"
                },
                "downloads": -1,
                "filename": "model_server-1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "fdb2e234ca8410083f780c98cf97bb0e",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 17345,
                "upload_time": "2019-06-22T12:47:45",
                "url": "https://files.pythonhosted.org/packages/dc/d6/9a25302e182e59399286c6b4792aa5b343ccd5d16abddf95ff673a2c7015/model_server-1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "6136f50723605d353eb593f66864e572",
                    "sha256": "bd281149df4e6911b6e0d2f1404468a0731d78adc516487e86b58d949217cfec"
                },
                "downloads": -1,
                "filename": "model-server-1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "6136f50723605d353eb593f66864e572",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11412,
                "upload_time": "2019-06-22T12:47:47",
                "url": "https://files.pythonhosted.org/packages/2f/f1/40e84565063f3778d4ff5f8330e679b2e75d9125e34a4b1cbed6b4d1c1bf/model-server-1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "fdb2e234ca8410083f780c98cf97bb0e",
                "sha256": "9b4a7a4058e558cebfe1389a5512d49543d150ab8f56f911642fb08df532c126"
            },
            "downloads": -1,
            "filename": "model_server-1.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "fdb2e234ca8410083f780c98cf97bb0e",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 17345,
            "upload_time": "2019-06-22T12:47:45",
            "url": "https://files.pythonhosted.org/packages/dc/d6/9a25302e182e59399286c6b4792aa5b343ccd5d16abddf95ff673a2c7015/model_server-1.0-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "6136f50723605d353eb593f66864e572",
                "sha256": "bd281149df4e6911b6e0d2f1404468a0731d78adc516487e86b58d949217cfec"
            },
            "downloads": -1,
            "filename": "model-server-1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "6136f50723605d353eb593f66864e572",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 11412,
            "upload_time": "2019-06-22T12:47:47",
            "url": "https://files.pythonhosted.org/packages/2f/f1/40e84565063f3778d4ff5f8330e679b2e75d9125e34a4b1cbed6b4d1c1bf/model-server-1.0.tar.gz"
        }
    ]
}