{
    "info": {
        "author": "Roland Stevenson",
        "author_email": "roland@rmg-services.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# PyCondusco\n\n## Overview\n\npycondusco lets you run a function iteratively, passing it the rows of a dataframe or the results of a query.\n\nWe call the functions pycondusco runs pipelines, and define a pipeline as a function that accepts a list of parameters and does something based on the values of the parameters.\n\nThe most common use case for pycondusco are data pipelines.  For data pipelines that primarily run SQL queries, we can template queries with a library (ie. [pystache](https://github.com/defunkt/pystache)), so that parametrized values are separated from the query logic.  We can then render the query with the appropriate values:\n\n```\nimport pycondusco\nfrom pycondusco.run_pipeline import run_pipeline\nimport pystache\n\njson_string = '{\"first_name\": \"First\", \"last_name\":\"Last\"}'\n\nparams = [\n    {\n        'k1':'v1',\n        'k2':'v2',\n    },\n    {\n        'k1':'v1',\n        'k2': json_string,\n    },\n]\n\ndef pipeline(params):\n    print pystache.render('k1 value is {{k1}}, k2 is {{k2}}',params)\n\nrun_pipeline(pipeline,params)\n```\n\n\npycondusco provides the following extensions in functionality to the above design pattern:\n - the user can provide a query and each row of results is iteratively passed to the pipeline\n - any JSON-string parameter will be converted to an object before being passed to the pipeline\n\n\n## Functions\n\n|function|description|\n|:--------------|:--------------|\n|run_pipeline(pipeline, parameters)| iteratively pass each row of parameters to a pipeline, converting any JSON parameters to objects|\n|run_pipeline_gbq(pipeline, query, project)|calls run_pipeline with the results of query executed via bigquery|\n\n\n## Installation\n\n```\npip install pycondusco\nexport GOOGLE_APPLICATION_CREDENTIALS=\"<FILE_PATH_TO_CREDENTIALS>\"\n```\n\n## Features\n\n*   Name-based substitution of query-results including JSON into pipelines, iterating through rows of parameters dataframe:\n```\nimport pystache\nfrom google.cloud import bigquery\nimport pycondusco\nfrom pycondusco.run_pipeline_gbq import run_pipeline_gbq\n\nclient = bigquery.Client()\n\ndef pipeline(params):\n    query = \"\"\"\n      SELECT\n        {{#list}}\n          SUM(CASE WHEN author.name ='{{name}}' THEN 1 ELSE 0 END) as n_{{name_clean}},\n        {{/list}}\n        repo_name\n      FROM `bigquery-public-data.github_repos.sample_commits`\n      GROUP BY repo_name\n    \"\"\"\n\n    query_job = client.query(pystache.render(query, params))\n    results = query_job.result()  # Waits for job to complete.\n    for row in results:\n        print(dict(row.items()))\n\n\nquery = \"\"\"\n   SELECT CONCAT('[',\n   STRING_AGG(\n     CONCAT('{\\\"name\\\":\\\"',name,'\\\",'\n       ,'\\\"name_clean\\\":\\\"', REGEXP_REPLACE(name, r'[^[:alpha:]]', ''),'\\\"}'\n     )\n   ),\n   ']') as list\n   FROM (\n     SELECT author.name,\n       COUNT(commit) n_commits\n     FROM `bigquery-public-data.github_repos.sample_commits`\n     GROUP BY 1\n     ORDER BY 2 DESC\n     LIMIT 10\n   )\n\"\"\"\n\nrun_pipeline_gbq(pipeline, client, query)\n```\n\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/ras44/pycondusco",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "pycondusco",
        "package_url": "https://pypi.org/project/pycondusco/",
        "platform": "",
        "project_url": "https://pypi.org/project/pycondusco/",
        "project_urls": {
            "Homepage": "https://github.com/ras44/pycondusco"
        },
        "release_url": "https://pypi.org/project/pycondusco/0.1.0/",
        "requires_dist": [
            "google-cloud-bigquery (>=1.0.0)"
        ],
        "requires_python": "",
        "summary": "pycondusco lets you run a function iteratively, passing it the rows of a dataframe or the results of a query.",
        "version": "0.1.0"
    },
    "last_serial": 4787795,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9d4284625249b5aafbfadfba37cb7cf1",
                    "sha256": "483a4af16fdbe150ba6f42a9683ff027a06b88e287084ea24b1dce7b4f58d1eb"
                },
                "downloads": -1,
                "filename": "pycondusco-0.1.0-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "9d4284625249b5aafbfadfba37cb7cf1",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 19659,
                "upload_time": "2019-02-06T17:46:39",
                "url": "https://files.pythonhosted.org/packages/9a/f8/c377eb8d0d8165109dfa7836095a7f353df31743e910bfae2ed9066212c8/pycondusco-0.1.0-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "216a4f2ecf2e30102cb4c2831df40338",
                    "sha256": "a755c6119ec94019be7a238d6018c429fbefbee7e877807f41d0dd5db911d44d"
                },
                "downloads": -1,
                "filename": "pycondusco-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "216a4f2ecf2e30102cb4c2831df40338",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3879,
                "upload_time": "2019-02-06T17:46:42",
                "url": "https://files.pythonhosted.org/packages/68/fd/fa8bb5196a5f365693aadedc2af43374c40135fc18234129a6a0833d98f1/pycondusco-0.1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "9d4284625249b5aafbfadfba37cb7cf1",
                "sha256": "483a4af16fdbe150ba6f42a9683ff027a06b88e287084ea24b1dce7b4f58d1eb"
            },
            "downloads": -1,
            "filename": "pycondusco-0.1.0-py2-none-any.whl",
            "has_sig": false,
            "md5_digest": "9d4284625249b5aafbfadfba37cb7cf1",
            "packagetype": "bdist_wheel",
            "python_version": "py2",
            "requires_python": null,
            "size": 19659,
            "upload_time": "2019-02-06T17:46:39",
            "url": "https://files.pythonhosted.org/packages/9a/f8/c377eb8d0d8165109dfa7836095a7f353df31743e910bfae2ed9066212c8/pycondusco-0.1.0-py2-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "216a4f2ecf2e30102cb4c2831df40338",
                "sha256": "a755c6119ec94019be7a238d6018c429fbefbee7e877807f41d0dd5db911d44d"
            },
            "downloads": -1,
            "filename": "pycondusco-0.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "216a4f2ecf2e30102cb4c2831df40338",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 3879,
            "upload_time": "2019-02-06T17:46:42",
            "url": "https://files.pythonhosted.org/packages/68/fd/fa8bb5196a5f365693aadedc2af43374c40135fc18234129a6a0833d98f1/pycondusco-0.1.0.tar.gz"
        }
    ]
}