{
    "info": {
        "author": "Ganesh N. Sivalingam",
        "author_email": "g.n.sivalingam@gmail.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# Spark acceleration for Scikit-Learn\n\nThis project is a major re-write of the \n[spark-sklearn](https://github.com/databricks/spark-sklearn) project, which \nseems to no longer be under development. It focuses specifically on the \nacceleration of Scikit-Learn's cross validation functionality using PySpark.\n\n### Improvements over spark-sklearn\n`scikit-spark` supports `scikit-learn` versions past 0.19, `spark-sklearn` [have stated that they are probably not \ngoing to support newer versions](https://github.com/databricks/spark-sklearn/issues/113).\n\nThe functionality in `scikit-spark` is based on `sklearn.model_selection` module rather than the \ndeprecated and soon to be removed `sklearn.grid_search`. The new `model_selection` versions \ncontain several nicer features and `scikit-spark` maintains full compatibility.\n\n## Installation\nThe package can be installed through pip:\n```bash\npip install scikit-spark\n```\n\nIt has so far only been tested with Spark 2.2.0 and up, but may work with \nolder versions. \n\n### Supported scikit-learn versions\n- 0.18 untested, likely doesn't work\n- 0.19 supported\n- 0.20 supported\n- 0.21 supported (Python 3 only as sklearn dropped support for Python 2)\n\n## Usage\n\nThe functionality here is meant to as closely resemble using Scikit-Learn as \npossible. By default (with `spark=True`) the `SparkSession` is obtained\ninternally by calling `SparkSession.builder.getOrCreate()`, so the instantiation\nand calling of the functions is the same (You will preferably have already \ncreated a `SparkSession`). \n\nThis example is adapted from the Scikit-Learn documentation. It instantiates\na local `SparkSession`, and distributes the cross validation folds and \niterations using this. In actual use, to get the benefit of this package it \nshould be used distributed across several machines with Spark as running it \nlocally is slower than the `Scikit-Learn` parallelisation implementation.\n\n```python\nfrom sklearn import svm, datasets\nfrom pyspark.sql import SparkSession\n\niris = datasets.load_iris()\nparameters = {'kernel':('linear', 'rbf'), 'C':[0.01, 0.1, 1, 10, 100]}\nsvc = svm.SVC()\n\nspark = SparkSession.builder\\\n    .master(\"local[*]\")\\\n    .appName(\"skspark-grid-search-doctests\")\\\n    .getOrCreate()\n\n# How to run grid search\nfrom skspark.model_selection import GridSearchCV\n\ngs = GridSearchCV(svc, parameters)\ngs.fit(iris.data, iris.target)\n\n# How to run random search\nfrom skspark.model_selection import RandomizedSearchCV\n\nrs = RandomizedSearchCV(spark, svc, parameters)\nrs.fit(iris.data, iris.target)\n```\n\n## Current and upcoming functionality\n- Current\n    - model_selection.RandomizedSearchCV\n    - model_selection.GridSearchCV\n- Upcoming\n    - model_selection.cross_val_predict\n    - model_selection.cross_val_score\n\n*The docstrings are modifications of the Scikit-Learn ones and are still being\nconverted to specifically refer to this project.* \n\n## Performance optimisations \n\n### Reducing RAM usage \n*Coming soon*\n\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/scikit-spark/scikit-spark",
        "keywords": "spark,pyspark,scikit-learn,sklearn,machine learning,random search,grid search",
        "license": "Apache 2.0",
        "maintainer": "",
        "maintainer_email": "",
        "name": "scikit-spark",
        "package_url": "https://pypi.org/project/scikit-spark/",
        "platform": "",
        "project_url": "https://pypi.org/project/scikit-spark/",
        "project_urls": {
            "Homepage": "https://github.com/scikit-spark/scikit-spark"
        },
        "release_url": "https://pypi.org/project/scikit-spark/0.3.0/",
        "requires_dist": [
            "numpy (>=1.13.0)",
            "six (==1.11.0)"
        ],
        "requires_python": "",
        "summary": "Spark acceleration for Scikit-Learn cross validation techniques",
        "version": "0.3.0"
    },
    "last_serial": 6002703,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "769032dbb6ef134346a3a611ee30267b",
                    "sha256": "c502f8cbf39111ca7cba0806c060fe5e2577086c97630f00cf2d110969fd401f"
                },
                "downloads": -1,
                "filename": "scikit_spark-0.1.0-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "769032dbb6ef134346a3a611ee30267b",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 30001,
                "upload_time": "2018-11-01T18:19:42",
                "url": "https://files.pythonhosted.org/packages/97/7e/483ede11897615e82d7798a6525ddb401d3187a86ad3b28af76f867d3ac8/scikit_spark-0.1.0-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ff419d3fec1204ae2695a367be66f2eb",
                    "sha256": "004ef746f966cae95a70a522f8c3c653bebd58b9485150349244713e413ca284"
                },
                "downloads": -1,
                "filename": "scikit_spark-0.1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ff419d3fec1204ae2695a367be66f2eb",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 19789,
                "upload_time": "2018-11-01T18:19:44",
                "url": "https://files.pythonhosted.org/packages/dd/d7/2ef215c4c7d4edc567a49d5d1d66637e84a7944520fcd1eb783a01bbceac/scikit_spark-0.1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "8373913b2c1228076042adaa0db7c234",
                    "sha256": "b304088b1ec700149328d287332c81dfeadf4d22359e3cb7ea1a7a0ec394a8f3"
                },
                "downloads": -1,
                "filename": "scikit-spark-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "8373913b2c1228076042adaa0db7c234",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17586,
                "upload_time": "2018-11-01T18:19:45",
                "url": "https://files.pythonhosted.org/packages/e4/87/c24c2828945f60e201d0a50c0633655885a558aa87b062452bd19a2dd7c5/scikit-spark-0.1.0.tar.gz"
            }
        ],
        "0.1.0rc2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ee8c9c4083d2eedba6aea7e0f2c2ea55",
                    "sha256": "14515ce5ceb98768a42b959604eec2955b8ce563c87f2cc82fc74e2066f19010"
                },
                "downloads": -1,
                "filename": "scikit_spark-0.1.0rc2-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "ee8c9c4083d2eedba6aea7e0f2c2ea55",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 28066,
                "upload_time": "2018-07-22T05:54:29",
                "url": "https://files.pythonhosted.org/packages/a4/d4/32f0d3edabc33d0668c81c22704e840289a3f6229b01907015c87848cac5/scikit_spark-0.1.0rc2-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "7d99fedb8b369f2f05103b4067dc1bf7",
                    "sha256": "676133c2a5aa5c73914abf5cac27c74e36d7cc06bd178a69da3341558fabc8b2"
                },
                "downloads": -1,
                "filename": "scikit-spark-0.1.0rc2.tar.gz",
                "has_sig": false,
                "md5_digest": "7d99fedb8b369f2f05103b4067dc1bf7",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15215,
                "upload_time": "2018-07-22T05:54:30",
                "url": "https://files.pythonhosted.org/packages/d6/0d/91f304c9feb2280b4e4f43831781a7bdc32ceea69561daea2a7d27363399/scikit-spark-0.1.0rc2.tar.gz"
            }
        ],
        "0.1.0rc4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7bcb6650d62e525700a01b07e58a573c",
                    "sha256": "b4789c41c8c99ed3aa4cbc0e510965d4cc8679da22e5651907109abfcaffb68b"
                },
                "downloads": -1,
                "filename": "scikit_spark-0.1.0rc4-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "7bcb6650d62e525700a01b07e58a573c",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 29628,
                "upload_time": "2018-07-22T07:51:24",
                "url": "https://files.pythonhosted.org/packages/ad/50/a36b3cbb4fd0e1cd00120b430aabce41dd0de335d5d0ad22d409c3535773/scikit_spark-0.1.0rc4-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d6ad0ea0675087e12ba942b0f5e1df6a",
                    "sha256": "56507bddf8627347658c899a5a8af388ffb0ed1acd308429375e8ac011e25773"
                },
                "downloads": -1,
                "filename": "scikit-spark-0.1.0rc4.tar.gz",
                "has_sig": false,
                "md5_digest": "d6ad0ea0675087e12ba942b0f5e1df6a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15917,
                "upload_time": "2018-07-22T07:51:25",
                "url": "https://files.pythonhosted.org/packages/e6/d8/25f7962aaa0f30ada6110a7670642ea58305a4402de6dd79051296d90803/scikit-spark-0.1.0rc4.tar.gz"
            }
        ],
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "59aa199aae107214d4aa94b6042e07f7",
                    "sha256": "1596f0ae99f666b284d765997915e7987170ad5b4b62fe5713723a610d36f2a1"
                },
                "downloads": -1,
                "filename": "scikit_spark-0.2.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "59aa199aae107214d4aa94b6042e07f7",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 46323,
                "upload_time": "2019-10-19T10:49:26",
                "url": "https://files.pythonhosted.org/packages/12/6e/20ecfe9dc9f95963827cc581d38e1fca8d5897f1e7b1ccb2dcb2cd75fb9c/scikit_spark-0.2.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "1a79de57b476478ff3aa36b2ebdc730e",
                    "sha256": "3539e4f5490acd106825cf2819ca07e312ed7e57533e092cd7a4f0fc50562bea"
                },
                "downloads": -1,
                "filename": "scikit-spark-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "1a79de57b476478ff3aa36b2ebdc730e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 26309,
                "upload_time": "2019-10-19T10:49:28",
                "url": "https://files.pythonhosted.org/packages/e8/e4/312fecf0cb4e1518dc4c62d29b71a52c9351fc6053534a98cb2dc7bcd08f/scikit-spark-0.2.0.tar.gz"
            }
        ],
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6490e29089615e28bc3376c0d20fe8f4",
                    "sha256": "bcf08eb9ce9bdc3b1e9f27a9b40b510c0b0d96db6d1d3f7aa9b93febf01f582c"
                },
                "downloads": -1,
                "filename": "scikit_spark-0.3.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "6490e29089615e28bc3376c0d20fe8f4",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 56873,
                "upload_time": "2019-10-20T11:46:29",
                "url": "https://files.pythonhosted.org/packages/d8/28/1047c01bc7e79cbb66c053816b5acb00bbef16d22649683c33ab213a64a6/scikit_spark-0.3.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "6dd042a12a20e9c0f05cd4bb2c51c69e",
                    "sha256": "00f51ebd472cc6ab39f6e7ba282a02e7559d83b663ad72d55c68a517ea6f5672"
                },
                "downloads": -1,
                "filename": "scikit-spark-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "6dd042a12a20e9c0f05cd4bb2c51c69e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 32456,
                "upload_time": "2019-10-20T11:46:31",
                "url": "https://files.pythonhosted.org/packages/e9/e4/2c48fe02f211a42ed07ba553e7e8db30789c139114f8670e0e4edea75092/scikit-spark-0.3.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "6490e29089615e28bc3376c0d20fe8f4",
                "sha256": "bcf08eb9ce9bdc3b1e9f27a9b40b510c0b0d96db6d1d3f7aa9b93febf01f582c"
            },
            "downloads": -1,
            "filename": "scikit_spark-0.3.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "6490e29089615e28bc3376c0d20fe8f4",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 56873,
            "upload_time": "2019-10-20T11:46:29",
            "url": "https://files.pythonhosted.org/packages/d8/28/1047c01bc7e79cbb66c053816b5acb00bbef16d22649683c33ab213a64a6/scikit_spark-0.3.0-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "6dd042a12a20e9c0f05cd4bb2c51c69e",
                "sha256": "00f51ebd472cc6ab39f6e7ba282a02e7559d83b663ad72d55c68a517ea6f5672"
            },
            "downloads": -1,
            "filename": "scikit-spark-0.3.0.tar.gz",
            "has_sig": false,
            "md5_digest": "6dd042a12a20e9c0f05cd4bb2c51c69e",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 32456,
            "upload_time": "2019-10-20T11:46:31",
            "url": "https://files.pythonhosted.org/packages/e9/e4/2c48fe02f211a42ed07ba553e7e8db30789c139114f8670e0e4edea75092/scikit-spark-0.3.0.tar.gz"
        }
    ]
}