{ "info": { "author": "Adam Bronte", "author_email": "adam@bronte.me", "bugtrack_url": null, "classifiers": [], "description": "Pypsark Proxy |Build Status| |PyPi|\n============================\n\n**Under active development. Do not use for production use.**\n\nSeamlessly execute pyspark code on remote clusters.\n\nHow it works\n------------\n\nPyspark proxy is made of up a client and server. The client mimics the\npyspark api but when objects get created or called a request is made to\nthe API server. The calls the API server receives then calls the actual\npyspark APIs.\n\nWhat has been implemented\n-------------------------\n\nCurrently only some basic functionalities with the ``SparkContext``,\n``sqlContext`` and ``DataFrame`` classes have been implemented. See the\n`tests`_ for more on what is currently working.\n\nGetting Started\n---------------\n\nPyspark Proxy requires set up a server where your Spark is located and\nsimply install the package locally where you want to execute code from.\n\nOn Server\n~~~~~~~~~\n\nInstall pyspark proxy via pip:\n\n::\n\n pip install pysparkproxy\n\nStart the server:\n\n::\n\n pyspark-proxy-server start\n\n\nThe server listens on ``localhost:8765`` by default. Check the ``pyspark-proxy-server`` help for additional options.\n\nLocally\n~~~~~~~\n\nInstall pyspark proxy via pip:\n\n::\n\n pip install pysparkproxy\n\nNow you can start a spark context and do some dataframe operations.\n\n::\n\n from pyspark_proxy import SparkContext\n from pyspark_proxy.sql import SQLContext\n\n sc = SparkContext(appName='pyspark_proxy_app')\n\n sc.setLogLevel('ERROR')\n\n sqlContext = SQLContext(sc)\n\n df = sqlContext.read.json('my.json')\n\n print(df.count())\n\nThen use the normal python binary to run this ``python my_app.py``. This\ncode works the same if you were to run it via ``spark-submit``.\n\n.. _tests: https://github.com/abronte/PysparkProxy/tree/master/tests\n.. _example: https://github.com/abronte/PysparkProxy/blob/master/examples/pyspark_proxy_server.py\n\n.. |Build Status| image:: https://travis-ci.org/abronte/PysparkProxy.svg?branch=master\n :target: https://travis-ci.org/abronte/PysparkProxy\n\n.. |PyPi| image:: https://img.shields.io/pypi/v/pysparkproxy.svg\n :target: https://pypi.org/project/PysparkProxy/", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/abronte/PysparkProxy", "keywords": "", "license": "Apache 2.0", "maintainer": "", "maintainer_email": "", "name": "PysparkProxy", "package_url": "https://pypi.org/project/PysparkProxy/", "platform": "", "project_url": "https://pypi.org/project/PysparkProxy/", "project_urls": { "Homepage": "https://github.com/abronte/PysparkProxy" }, "release_url": "https://pypi.org/project/PysparkProxy/0.0.17/", "requires_dist": null, "requires_python": "", "summary": "Seamlessly execute pyspark code on remote clusters", "version": "0.0.17" }, "last_serial": 4488018, "releases": { "0.0.10": [ { "comment_text": "", "digests": { "md5": "1e3a1f5ff94802c54503e040e7085789", "sha256": "c4a11117bafeebb00b5f3eaca1fcaaba5b3d79f0a6d14b4bbdb3bf4e54171b91" }, "downloads": -1, "filename": "PysparkProxy-0.0.10.tar.gz", "has_sig": false, "md5_digest": "1e3a1f5ff94802c54503e040e7085789", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12710, "upload_time": "2018-10-11T21:57:38", "url": "https://files.pythonhosted.org/packages/de/77/3aff9365d03d0394f51a4799bbb11f43d60e0e1f9092f68c78e41a79d63c/PysparkProxy-0.0.10.tar.gz" } ], "0.0.11": [ { "comment_text": "", "digests": { "md5": "bf44d56c3f91c4329357f62844396ab3", "sha256": "0dad8aced548f3987d0e23a7d61357803cfc2bc5b55d4f5335980e710cf8d327" }, "downloads": -1, "filename": "PysparkProxy-0.0.11.tar.gz", "has_sig": false, "md5_digest": "bf44d56c3f91c4329357f62844396ab3", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13770, "upload_time": "2018-11-05T22:39:58", "url": "https://files.pythonhosted.org/packages/f7/ad/fd2230a11f48bf6002103f6c39ca5238b8450c899bd1f6041047b38e3daa/PysparkProxy-0.0.11.tar.gz" } ], "0.0.12": [ { "comment_text": "", "digests": { "md5": "29c473378386b0a104766e0e0bfd215b", "sha256": "74f34ec2e2ef488d4f4c0edcced6c0e1d6dfcf2d5e62d7f4eb7fafc19a4ac813" }, "downloads": -1, "filename": "PysparkProxy-0.0.12.tar.gz", "has_sig": false, "md5_digest": "29c473378386b0a104766e0e0bfd215b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14041, "upload_time": "2018-11-06T20:14:00", "url": "https://files.pythonhosted.org/packages/ad/83/8a5adb5b407f40943b746174b0c35bc062db6723883ff63cd1ca4bdd7a2e/PysparkProxy-0.0.12.tar.gz" } ], "0.0.13": [ { "comment_text": "", "digests": { "md5": "7b1ac4f357f8c99debbd3c3aed609661", "sha256": "b73ba265fec3c20d9a3305a14ad22f649711d180d56295d28afd219b247ab745" }, "downloads": -1, "filename": "PysparkProxy-0.0.13.tar.gz", "has_sig": false, "md5_digest": "7b1ac4f357f8c99debbd3c3aed609661", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 16950, "upload_time": "2018-11-13T04:08:26", "url": "https://files.pythonhosted.org/packages/5b/a5/608122a0e22ac29e483a32b78477b9aa569884975a85407a65400841d8b2/PysparkProxy-0.0.13.tar.gz" } ], "0.0.14": [ { "comment_text": "", "digests": { "md5": "ed6e0fd9ca5f769d765b592a59536cbc", "sha256": "35c35fbf96fedd0deee29ffe9a22ab28b430300773dcf9e9af1e488963602736" }, "downloads": -1, "filename": "PysparkProxy-0.0.14.tar.gz", "has_sig": false, "md5_digest": "ed6e0fd9ca5f769d765b592a59536cbc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17166, "upload_time": "2018-11-13T17:48:19", "url": "https://files.pythonhosted.org/packages/38/4b/0206b167d6c14f747b9975752942f9a00af35e3737c39e3d567ed5029db8/PysparkProxy-0.0.14.tar.gz" } ], "0.0.15": [ { "comment_text": "", "digests": { "md5": "e4be25f75fe02e179558f25b68e04227", "sha256": "69c303f4637258953e20938491b334d8b62010b833d0b7662023ace32cf761ba" }, "downloads": -1, "filename": "PysparkProxy-0.0.15.tar.gz", "has_sig": false, "md5_digest": "e4be25f75fe02e179558f25b68e04227", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17435, "upload_time": "2018-11-13T19:19:15", "url": "https://files.pythonhosted.org/packages/c4/68/f800a956957502f3af2aa227d616bc4bc71dbf412c4372938c645185b39c/PysparkProxy-0.0.15.tar.gz" } ], "0.0.16": [ { "comment_text": "", "digests": { "md5": "a52882217eb739a04bd6859bbf758c93", "sha256": "84b2535c3fea6faac865409c6883ba093c942b5f5113083ff8f2e669829347b3" }, "downloads": -1, "filename": "PysparkProxy-0.0.16.tar.gz", "has_sig": false, "md5_digest": "a52882217eb739a04bd6859bbf758c93", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17704, "upload_time": "2018-11-13T23:06:59", "url": "https://files.pythonhosted.org/packages/43/ca/528652441cf3e55810bb5f5426e9630cc5abad5d20922c056a587ed4c03c/PysparkProxy-0.0.16.tar.gz" } ], "0.0.17": [ { "comment_text": "", "digests": { "md5": "5f7e6e6f89720561156bd1f2ef761f50", "sha256": "137383cbf0c8e8603a4d92d2abdc55ea8ec4fb965b2973c5655cb1ef8ed0e1ad" }, "downloads": -1, "filename": "PysparkProxy-0.0.17.tar.gz", "has_sig": false, "md5_digest": "5f7e6e6f89720561156bd1f2ef761f50", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18442, "upload_time": "2018-11-15T03:15:23", "url": "https://files.pythonhosted.org/packages/12/78/0b481e1fa5bd74a1b5ecb16f1502885408ecacfb36cc1311ac271b62c300/PysparkProxy-0.0.17.tar.gz" } ], "0.0.9": [ { "comment_text": "", "digests": { "md5": "283388ce8167f8259e5262084bd9271b", "sha256": "303c7f432543880a619c11e806a0f09c6afcdf25014147cb68a20d08af1c9265" }, "downloads": -1, "filename": "PysparkProxy-0.0.9.tar.gz", "has_sig": false, "md5_digest": "283388ce8167f8259e5262084bd9271b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11483, "upload_time": "2018-10-08T21:36:36", "url": "https://files.pythonhosted.org/packages/b7/b3/39e0883a6c0468207ac4a2b2ad1ca21c2de365c5bc4864408357c8684b5c/PysparkProxy-0.0.9.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "5f7e6e6f89720561156bd1f2ef761f50", "sha256": "137383cbf0c8e8603a4d92d2abdc55ea8ec4fb965b2973c5655cb1ef8ed0e1ad" }, "downloads": -1, "filename": "PysparkProxy-0.0.17.tar.gz", "has_sig": false, "md5_digest": "5f7e6e6f89720561156bd1f2ef761f50", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18442, "upload_time": "2018-11-15T03:15:23", "url": "https://files.pythonhosted.org/packages/12/78/0b481e1fa5bd74a1b5ecb16f1502885408ecacfb36cc1311ac271b62c300/PysparkProxy-0.0.17.tar.gz" } ] }