{
    "info": {
        "author": "Aleksandrs Krivickis",
        "author_email": "aleksandrs.krivickis@gmail.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# Spark Safe Delta\n    Combination of tools that allow more convenient use of PySpark within Azure DataBricks environment.\n\n## I. Package contents:\n### 1.delta_write_safe\n    Tool that allows to automatically update schema of DataBricks Delta in case of Changes in data structure\n\n### 2.write_data_mysql\n    Method writes data into MySQL and takes care of repartitioning in case if it's necessary.\n\nDependencies:\n   \n    1. MySQL connector Java 8_0_13\n    dbfs:/FileStore/jars/7b863f06_67cf_4a51_8f3b_67d414d808b3-Barnymysql_connector_java_8_0_13_4ac45-2f7c7.jar\n    \n    http://dev.mysql.com/doc/connector-j/en/\n    https://mvnrepository.com/artifact/mysql/mysql-connector-java\n\nBy default, it relies on constant variables outside of method that define MySQL credentials, that can be also specified as a parameters:\n\n    * MYSQL_URL\n    * MYSQL_DRIVER\n    * MYSQL_USER\n    * MYSQL_PASSWORD\n    * MYSQL_SSL_CA_PATH\n    * MYSQL_QUERY_TIMEOUT\n\nMethod Parameters:\n\n    * p_spark_dataframe - dataframe to write\n    * p_mysql_db_name - name of database to write to\n    * p_mysql_table_name - name of table to write to\n    * p_num_partitions - amount of partitions, if -1, runs with default amount of partitions defined in spark environment or specific delta\n\nMethod default parameters:\n\n    p_num_partitions=-1\n    url=MYSQL_URL,\n    driver=MYSQL_DRIVER,\n    user=MYSQL_USER,\n    password=MYSQL_PASSWORD,\n    ssl_ca=MYSQL_SSL_CA_PATH,\n    queryTimeout=MYSQL_QUERY_TIMEOUT\n\nUsage example:\n\n    #MySQL settings defined outside of a method below:\n    MYSQL_DRIVER = \"com.mysql.jdbc.Driver\"\n    MYSQL_URL = \"jdbc:mysql://hostname:port/database?useUnicode=true&characterEncoding=utf-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false\"\n    MYSQL_QUERY_TIMEOUT = 0\n    \n    MYSQL_USER = \"user@namespace\"\n    MYSQL_PASSWORD = \"example_password\"\n    MYSQL_SSL_CA_PATH = \"/mnt/alex-experiments-blob/certs/cert.txt\"\n    \n    #Method execution itself\n    write_data_mysql(p_spark_dataframe=target_data, p_mysql_dbtable=destination_db_name_column_name_construct)\n\n### 3.remove_columns\n\n    remove_columns() method removes columns from a specified dataframe.\n    It will silently return a result even if user specifies column that doesn't exist.\n    Usage example: destination_df = remove_columns(source_df, \"SequenceNumber;Body;Non-existng-column\")\n\n### 4.read_mysql\n\n    Method allows fetch the table, or a query as a Spark DataFrame.\n    Returnws Spark DataFrame as a result.\n\n    # Example usage:\n    read_mysql(table_name=customers)\n    read_mysql(table_name=h2.customers)\n    read_mysql(table_name=h2.customers, url=MYSQL_URL, driver=MYSQL_DRIVER, user=MYSQL_USER, password=MYSQL_PASSWORD, ssl_ca=MYSQL_SSL_CA_PATH, queryTimeout=MYSQL_QUERY_TIMEOUT)\n\n### 4.list_available_mysql_tables\n\n    Method allows to list all the tables that available to a particular user.\n    Returnws Spark DataFrame as a result\n\n\n## Package sample usage:\n\n    #!/usr/bin/env python\n    \n    from sparksafedelta import sparksafedelta\n    sparksafedelta.delta_write_safe(sp_df_to_write, SP_CONTEXT, DATABRICKS_TABLE_NAME)",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://pypi.python.org/pypi/SparkSafeDelta/",
        "keywords": "",
        "license": "LICENSE.txt",
        "maintainer": "",
        "maintainer_email": "",
        "name": "SparkSafeDelta",
        "package_url": "https://pypi.org/project/SparkSafeDelta/",
        "platform": "",
        "project_url": "https://pypi.org/project/SparkSafeDelta/",
        "project_urls": {
            "Homepage": "http://pypi.python.org/pypi/SparkSafeDelta/"
        },
        "release_url": "https://pypi.org/project/SparkSafeDelta/0.4.0/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Combination of tools that allow more convenient use of PySpark within Azure DataBricks environment.",
        "version": "0.4.0"
    },
    "last_serial": 5149177,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a0dc31d006f354ed9519bbd31a2b923a",
                    "sha256": "24c4bde3e3549af851db1ccefed7c26750c49bdfa28694a32ae4b7a8de528916"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "a0dc31d006f354ed9519bbd31a2b923a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1781,
                "upload_time": "2019-03-08T07:23:44",
                "url": "https://files.pythonhosted.org/packages/28/05/ea41271be7cc5a579c0ca996935c044de26cb42df70c3b4188845ffb5efe/SparkSafeDelta-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "22ae60b4225246d725869cd24b226d98",
                    "sha256": "23068e0882989d7c18d1166a1f8fabf628f5fba0d40e29581d129e66b080ad6e"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "22ae60b4225246d725869cd24b226d98",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1728,
                "upload_time": "2019-03-08T12:28:28",
                "url": "https://files.pythonhosted.org/packages/df/34/1e11419803ab5c5d688f77c78a67a6af1015a91adc712cff8e39a9f2c116/SparkSafeDelta-0.1.1.tar.gz"
            }
        ],
        "0.1.14": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7e77dddeda32607f326717f9a1abfaa0",
                    "sha256": "c0f51b3f6110288bf9e1677050c100242164f2e6042bded58349291a8a51744e"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.1.14.tar.gz",
                "has_sig": false,
                "md5_digest": "7e77dddeda32607f326717f9a1abfaa0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8704,
                "upload_time": "2019-04-02T14:13:27",
                "url": "https://files.pythonhosted.org/packages/e6/e6/7c2d7373a5eb90badb6a4181c742319f905c8c762e0b12ce27811c2766a5/SparkSafeDelta-0.1.14.tar.gz"
            }
        ],
        "0.1.3.post1.dev0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "80e2015210ad2e40210af8c41c719e6c",
                    "sha256": "b28a877a5ce8a4f04cc74bac65cd4d5bd701acf72d962d412e3c59225331c33c"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.1.3.post1.dev0.tar.gz",
                "has_sig": false,
                "md5_digest": "80e2015210ad2e40210af8c41c719e6c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17985,
                "upload_time": "2019-03-17T17:33:25",
                "url": "https://files.pythonhosted.org/packages/b0/44/857a9aa88a316f50b8dc1096154b5df90f8de86c44d74e58ae406168f78d/SparkSafeDelta-0.1.3.post1.dev0.tar.gz"
            }
        ],
        "0.1.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "25c03105bbf4ac53e9adbee448d24b3d",
                    "sha256": "f45964714f436ae969f0243dea6cd6304cea7e405b8f6a49c0d6a027a8d735d1"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.1.9.tar.gz",
                "has_sig": false,
                "md5_digest": "25c03105bbf4ac53e9adbee448d24b3d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7092,
                "upload_time": "2019-03-17T17:52:55",
                "url": "https://files.pythonhosted.org/packages/5b/65/5f91e617ed31696f052ffed7610dc8c39a20c3c36f1c22e44f8fea74cbfe/SparkSafeDelta-0.1.9.tar.gz"
            }
        ],
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f7ebeb9cf65f20765e850e13a5a2b7a4",
                    "sha256": "56d2b8255dd9770f3926a78c39ef3fcac569eb7bf64194d2abfb2666163cd2c3"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "f7ebeb9cf65f20765e850e13a5a2b7a4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3211,
                "upload_time": "2019-04-02T14:18:48",
                "url": "https://files.pythonhosted.org/packages/49/1a/79c86a05bb6055eb3483cfcbb9ae1eb3d2042cda015bbfd1465ebf877ac1/SparkSafeDelta-0.3.0.tar.gz"
            }
        ],
        "0.3.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "cea3243f8a03eec4a191f5cbe65686b7",
                    "sha256": "204e6ee8efc494057392c29eeb4561a7b61dc44f9f10dfd4df4f638235f7fa0b"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.3.1.tar.gz",
                "has_sig": false,
                "md5_digest": "cea3243f8a03eec4a191f5cbe65686b7",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3186,
                "upload_time": "2019-04-02T14:23:18",
                "url": "https://files.pythonhosted.org/packages/7a/08/1b221c55d18be7003f495b5f9a1c77e6662e1f42a6d8331ec4764b527e3e/SparkSafeDelta-0.3.1.tar.gz"
            }
        ],
        "0.3.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "888c8d693066bd0ff412d61b06959549",
                    "sha256": "4eba9f60be6f0802ad072ff40925ee8b8d0053092431fcfde164fa02e1c3f124"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.3.2.tar.gz",
                "has_sig": false,
                "md5_digest": "888c8d693066bd0ff412d61b06959549",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3188,
                "upload_time": "2019-04-02T14:25:24",
                "url": "https://files.pythonhosted.org/packages/7c/46/fdea1f5a59ae5014ec92f653c160c862dbe9b3db478c8b0aaf4017bb3c38/SparkSafeDelta-0.3.2.tar.gz"
            }
        ],
        "0.3.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "88abeb17c362ef764c49470572b220b3",
                    "sha256": "5e8a33f936f5d0d1cab3f8fb1c2a604049ccd2c60a5e7f8cd0b9909ffb54798f"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.3.3.tar.gz",
                "has_sig": false,
                "md5_digest": "88abeb17c362ef764c49470572b220b3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3418,
                "upload_time": "2019-04-02T14:31:53",
                "url": "https://files.pythonhosted.org/packages/0f/86/7495b474fe6da90ec6a7fb94d1f68d0be9ff29d2c7a31ea6b3b89b714ec4/SparkSafeDelta-0.3.3.tar.gz"
            }
        ],
        "0.3.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "da36bec05f58a6407e6fa67b88198b1f",
                    "sha256": "1218e523df5a0c72113020b9aafcdbfcc4a56e0924b0826226e10d89b4af2300"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.3.4.tar.gz",
                "has_sig": false,
                "md5_digest": "da36bec05f58a6407e6fa67b88198b1f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3190,
                "upload_time": "2019-04-02T14:37:56",
                "url": "https://files.pythonhosted.org/packages/c5/1f/09f67396bfd25ec9a663033df4dde2b9a16371f2c5a62bd79a3b92b84d12/SparkSafeDelta-0.3.4.tar.gz"
            }
        ],
        "0.3.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "21077abc18d76a6a685507ca259e908a",
                    "sha256": "bc89ba65ecd0e40f5949c0a520e82982a5cd6fc2b2423245bd1f3a7e71f334bd"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.3.5.tar.gz",
                "has_sig": false,
                "md5_digest": "21077abc18d76a6a685507ca259e908a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3198,
                "upload_time": "2019-04-16T09:27:12",
                "url": "https://files.pythonhosted.org/packages/ca/4b/aed897899b5929d3d3ceb0c54b26ba30213ba91e2415a158c2a61b57acbe/SparkSafeDelta-0.3.5.tar.gz"
            }
        ],
        "0.3.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b20407937cc83f5c56611b07fb3945ca",
                    "sha256": "b6b90f37aedd6f783b9b775d67f6c213125ac44bb5c2a4da7e8dc1cf9b34bc61"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.3.6.tar.gz",
                "has_sig": false,
                "md5_digest": "b20407937cc83f5c56611b07fb3945ca",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3423,
                "upload_time": "2019-04-16T09:29:41",
                "url": "https://files.pythonhosted.org/packages/07/38/238afbbea09f03c8076bfa815dcd01911af1913d5278e637ef58aeecc29b/SparkSafeDelta-0.3.6.tar.gz"
            }
        ],
        "0.3.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "146daffe77a0a53779f9e5bab29d14fd",
                    "sha256": "eb8d71d02bbd2ac2c5b8fa9d714a43010af0942f06ebf57660a10014040e830d"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.3.7.tar.gz",
                "has_sig": false,
                "md5_digest": "146daffe77a0a53779f9e5bab29d14fd",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3415,
                "upload_time": "2019-04-16T09:32:37",
                "url": "https://files.pythonhosted.org/packages/be/ac/5c92fae85c9a074ee0b5c85688ba3ee64af69854cfbca557de3d21d3878b/SparkSafeDelta-0.3.7.tar.gz"
            }
        ],
        "0.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "079c016a11ddc893c94f34448e56d402",
                    "sha256": "1b86c2c93e39da1c69787bcb5e6203613dd7eff1076cbc8f1ea287adf346ea98"
                },
                "downloads": -1,
                "filename": "SparkSafeDelta-0.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "079c016a11ddc893c94f34448e56d402",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3423,
                "upload_time": "2019-04-16T09:46:42",
                "url": "https://files.pythonhosted.org/packages/44/6d/3201b83177601020f9482eda7181d219df05056c2ab9e0112c75e4edb6ce/SparkSafeDelta-0.4.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "079c016a11ddc893c94f34448e56d402",
                "sha256": "1b86c2c93e39da1c69787bcb5e6203613dd7eff1076cbc8f1ea287adf346ea98"
            },
            "downloads": -1,
            "filename": "SparkSafeDelta-0.4.0.tar.gz",
            "has_sig": false,
            "md5_digest": "079c016a11ddc893c94f34448e56d402",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 3423,
            "upload_time": "2019-04-16T09:46:42",
            "url": "https://files.pythonhosted.org/packages/44/6d/3201b83177601020f9482eda7181d219df05056c2ab9e0112c75e4edb6ce/SparkSafeDelta-0.4.0.tar.gz"
        }
    ]
}