{
    "info": {
        "author": "Sean Hammond",
        "author_email": "ckanapi-exporter@seanh.cc",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
            "Programming Language :: Python :: 2.7"
        ],
        "description": "[![Build Status](https://travis-ci.org/ckan/ckanapi-exporter.svg)](https://travis-ci.org/ckan/ckanapi-exporter)\n[![Coverage Status](https://img.shields.io/coveralls/ckan/ckanapi-exporter.svg)](https://coveralls.io/r/ckan/ckanapi-exporter)\n[![Latest Version](https://img.shields.io/pypi/v/ckanapi-exporter.svg)](https://pypi.python.org/pypi/ckanapi-exporter/)\n[![Supported Python versions](https://img.shields.io/pypi/pyversions/ckanapi-exporter.svg)](https://pypi.python.org/pypi/ckanapi-exporter/)\n[![Development Status](https://img.shields.io/pypi/status/ckanapi-exporter.svg)](https://pypi.python.org/pypi/ckanapi-exporter/)\n[![License](https://img.shields.io/pypi/l/ckanapi-exporter.svg)](https://pypi.python.org/pypi/ckanapi-exporter/)\n\nckanapi-exporter\n================\n\nAn API client (usable as a command-line script or as a Python library) for\nexporting dataset metadata from CKAN sites to Excel-compatible CSV files.\n\n\nInstallation\n------------\n\nTo install run:\n\n    pip install ckanapi-exporter\n\n\nUsage\n-----\n\n```bash\nckanapi-exporter --url 'https://demo.ckan.org' \\\n    --column \"Title\" --pattern '^title$' > output.csv\n```\n\nThis searches each dataset on demo.ckan.org for fields matching the\n[regular expression](https://docs.python.org/2/howto/regex.html#regex-howto)\n`^title$` (the `--pattern` argument) and puts the values into a\ncolumn called \"Title\" in the CSV file (the `--column` argument).  It'll create\nan `output.csv` file something like this:\n\n<table>\n  <tr>\n    <th>Title</th>\n  </tr>\n  <tr>\n    <td>Senior Salaries Information</td>\n  </tr>\n  <tr>\n    <td>Demo Data for Open Data in 1 Day - Spending Over \u00a3500</td>\n  </tr>\n  <tr>\n    <td>UK Cat Burglaries</td>\n  </tr>\n  <tr>\n    <td>...</td>\n  </tr>\n</table>\n\nYou can add as many columns as you want: just add a `--column` and a\n`--pattern` argument for each column. The title of the column in the CSV file\ncan be anything you want - it doesn't have to match the name of the field in\nCKAN. Let's add a second column titled \"Rights\" that contains the\n`license_title` fields from the datasets:\n\n```bash\nckanapi-exporter --url 'https://demo.ckan.org' \\\n    --column \"Title\" --pattern '^title$' \\\n    --column \"Rights\" --pattern '^license_title$' > output.csv\n```\n\n<table>\n  <tr>\n    <th>Title</th>\n    <th>Rights</th>\n  </tr>\n  <tr>\n    <td>Senior Salaries Information</td>\n    <td>Creative Commons Attribution</td>\n  </tr>\n  <tr>\n    <td>Demo Data for Open Data in 1 Day - Spending Over \u00a3500</td>\n    <td>Creative Commons CCZero</td>\n  </tr>\n  <tr>\n    <td>UK Cat Burglaries</td>\n    <td>UK Open Government Licence (OGL)</td>\n  </tr>\n  <tr>\n    <td>...</td>\n    <td>...</td>\n  </tr>\n</table>\n\n### API Parameters\nThe ckanapi-exporter calls the [`package_search`](http://docs.ckan.org/en/latest/api/index.html#ckan.logic.action.get.package_search)\nAPI action and you can pass in related query parameters by using the `--params`\nargument and passing in a string formated as a dictionary. Each key: value pair\nrepresents a query passed to the API call.\n\nFor example if you wanted to only export datasets between a date range you can\npass in the `fq` (filtered query) parameter and use `metadata_created` to filter\nthe results.\n\n```bash\nckanapi-exporter --url 'https://demo.ckan.org' \\\n    --params \"{'fq':'metadata_created:[2017-01-01T00:00:00Z TO 2017-01-31T23:59:99.999Z]'}\" \\\n    --column \"Title\" --pattern '^title$' \\\n    --column \"Rights\" --pattern '^license_title$' > output.csv\n```\n\n\n### Transformations\n\nYou can apply certain transformations to the values from the datasets.\nFor example, let's add a third column with the first 50 characters of each\ndataset's description (the `notes` field in the CKAN API):\n\n```bash\nckanapi-exporter --url 'https://demo.ckan.org' \\\n    --column \"Title\" --pattern '^title$' \\\n    --column \"Rights\" --pattern '^license_title$' \\\n    --column \"Description\" --pattern '^notes$' --max-length 50 > output.csv\n```\n\n<table>\n  <tr>\n    <th>Title</th>\n    <th>Rights</th>\n    <th>Description</th>\n  </tr>\n  <tr>\n    <td>Senior Salaries Information</td>\n    <td>Creative Commons Attribution</td>\n    <td>Demo information about senior salaries from 11/04/</td>\n  </tr>\n  <tr>\n    <td>Demo Data for Open Data in 1 Day - Spending Over \u00a3500</td>\n    <td>Creative Commons CCZero</td>\n    <td>Data on spending over \u00a3500 generated for Open Data</td>\n  </tr>\n  <tr>\n    <td>UK Cat Burglaries</td>\n    <td>UK Open Government Licence (OGL)</td>\n    <td>A record of cat burgalries, listing the cat names,</td>\n  </tr>\n  <tr>\n    <td>...</td>\n    <td>...</td>\n    <td>...</td>\n  </tr>\n</table>\n\n\n### Exporting Resource Fields\n\nLet's add a column containing the formats of each datasets' resources:\n\n```bash\nckanapi-exporter --url 'https://demo.ckan.org' \\\n    --column \"Title\" --pattern '^title$' \\\n    --column \"Rights\" --pattern '^license_title$' \\\n    --column \"Description\" --pattern '^notes$' --max-length 50 \\\n    --column Formats --pattern '^resources$' '^format$' > output.csv\n```\n\nThis time the pattern has two arguments: `--pattern '^resources$' '^format$'`.\nThis means find the \"resources\" field of each dataset and then find the\n\"format\" field of each resource. When a dataset has more than one resource\nthe formats will be combined into a quoted, comma-separated list in a single\ntable cell. It'll create a CSV file something like this:\n\n<table>\n  <tr>\n    <th>Title</th>\n    <th>Rights</th>\n    <th>Description</th>\n    <th>Formats</th>\n  </tr>\n  <tr>\n    <td>Senior Salaries Information</td>\n    <td>Creative Commons Attribution</td>\n    <td>Demo information about senior salaries from 11/04/</td>\n    <td>XLSX, CSV</td>\n  </tr>\n  <tr>\n    <td>Demo Data for Open Data in 1 Day - Spending Over \u00a3500</td>\n    <td>Creative Commons CCZero</td>\n    <td>Data on spending over \u00a3500 generated for Open Data</td>\n    <td>CSV, CSV, CSV, CSV</td>\n  </tr>\n  <tr>\n    <td>UK Cat Burglaries</td>\n    <td>UK Open Government Licence (OGL)</td>\n    <td>A record of cat burgalries, listing the cat names,</td>\n    <td>JPEG, CSV, CSV</td>\n  </tr>\n  <tr>\n    <td>...</td>\n    <td>...</td>\n    <td>...</td>\n    <td>...</td>\n  </tr>\n</table>\n\nCSV is repeated a lot because lots of the datasets have multiple CSV resources.\nYou can add the `--deduplicate` option to the column to remove the duplication:\n\n```bash\nckanapi-exporter --url 'https://demo.ckan.org' \\\n    --column \"Title\" --pattern '^title$' \\\n    --column \"Rights\" --pattern '^license_title$' \\\n    --column \"Description\" --pattern '^notes$' --max-length 50 \\\n    --column Formats --pattern '^resources$' '^format$' --deduplicate \\\n    > output.csv\n```\n\n<table>\n  <tr>\n    <th>Title</th>\n    <th>Rights</th>\n    <th>Description</th>\n    <th>Formats</th>\n  </tr>\n  <tr>\n    <td>Senior Salaries Information</td>\n    <td>Creative Commons Attribution</td>\n    <td>Demo information about senior salaries from 11/04/</td>\n    <td>XLSX, CSV</td>\n  </tr>\n  <tr>\n    <td>Demo Data for Open Data in 1 Day - Spending Over \u00a3500</td>\n    <td>Creative Commons CCZero</td>\n    <td>Data on spending over \u00a3500 generated for Open Data</td>\n    <td>CSV</td>\n  </tr>\n  <tr>\n    <td>UK Cat Burglaries</td>\n    <td>UK Open Government Licence (OGL)</td>\n    <td>A record of cat burgalries, listing the cat names,</td>\n    <td>JPEG, CSV</td>\n  </tr>\n  <tr>\n    <td>...</td>\n    <td>...</td>\n    <td>...</td>\n    <td>...</td>\n  </tr>\n</table>\n\n\n### Exporting Dataset Extras\n\nLet's add a column with the values of the \"Next Update\" extra from each\ndataset. Dataset publishers have been inconsistent with naming this column,\nit's sometimes \"Next Update\" and sometimes \"next update\", \"Next update day\",\n\"Next Update Time\" etc. We'll use a regular expression that matches all of\nthese possible names and combine them into a single \"Next Update\" column:\n\n```bash\nckanapi-exporter --url 'https://demo.ckan.org' \\\n    --column \"Title\" --pattern '^title$' \\\n    --column \"Rights\" --pattern '^license_title$' \\\n    --column \"Description\" --pattern '^notes$' --max-length 50 \\\n    --column Formats --pattern '^resources$' '^format$' --deduplicate \\\n    --column \"Next Update\" --pattern '^extras$' '^next update.*' --unique \\\n    > output.csv\n```\n\nThe two-part pattern `'^extras$' '^next update.*'` means to look in the\n\"extras\" field of each dataset for extras whose name matches\n`^next update.*`. We're expecting each dataset to have only one matching\nextra so we add the `--unique` argument which will crash if a dataset has more\nthan one extra matching the pattern.\n\nBy default patterns are matched case-insensitively and whitespace is stripped\nfrom field names before matching. To match case-sensitively and without\nstripping whitespace add `--case-sensitive --strip false` to the column.\n\nWe can also find multiple extras and combine them into a single column.\nFor example, let's say our datasets have a \"contributor\" extra\n(sometimes spelled \"contributor\", sometimes \"Contributor\"). Some datasets have\nmultiple extras named \"Contributor 1\", \"Contributor 2\" etc. We can find all of\nthese contributor extras and combine them into a single quoted, comma-separated\nlist with a pattern like this:\n\n    --column Contributors --pattern '^extras$' '^contributor.*'\n\n\n### Using a columns.json File\n\nYou can specify your columns in a `columns.json` file instead of on the command\nline. Here's an example of the format:\n\n```json\n{\n  \"Data Owner\": {\n      \"pattern\": \"^author$\",\n      \"unique\": true,\n      \"case_sensitive\": true\n  },\n  \"Delivery Unit\": {\n      \"pattern\": [\"^extras$\", \"^Delivery Unit$\"],\n      \"unique\": true\n  },\n  \"Contributor\": {\n      \"pattern\": [\"^extras$\", \"^Contributor.*\"]\n  },\n  \"Description\": {\n      \"pattern\": \"^notes$\",\n      \"unique\": true,\n      \"case_sensitive\": true,\n      \"max_length\": 255\n  },\n  \"Format\": {\n      \"pattern\": [\"^resources$\", \"^format$\"],\n      \"case_sensitive\": true,\n      \"deduplicate\": true\n  }\n}\n```\n\nThen tell ckanapi-exporter to read the column options from this file instead of\ngiving them on the command line:\n\n```bash\nckanapi-exporter --url 'https://demo.ckan.org' --columns columns.json > output.csv\n```\n\nFor a working example `columns.json` file that you can use against demo.ckan.org,\nsee [test_columns.json](ckanapi_exporter/test_columns.json).\n\nckanapi-exporter is a thin wrapper around\n[losser](https://github.com/ckan/losser), hooking it up to the CKAN API.\nFor more documentation of the filtering and transforming options run\n`ckanapi-exporter --help` or read losser's docs.\n\n\nUsing as a Python Library\n-------------------------\n\nYou can also import ckanapi-exporter in Python and use it from your CKAN API\nclient or plugin:\n\n```python\nimport ckanapi_exporter.exporter as exporter\ncsv_string = exporter.export('https://demo.ckan.org', 'columns.json')\n```\n\nReturns a UTF8-encoded string.\n\nThe second argument can be either the filename of the columns.json file as a\nstring, or a list of dictionaries (equivalent to the contents of columns.json\nfile after loading the JSON).\n\n\nDevelopment\n-----------\n\nTo install for development, create and activate a Python virtual environment\nthen do:\n\n```bash\ngit clone https://github.com/ckan/ckanapi-exporter.git\ncd ckanapi-exporter\npython setup.py develop\npip install -r dev-requirements.txt\n```\n\nTo run the tests do:\n\n    nosetests",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/ckan/ckanapi-exporter",
        "keywords": "ckan csv export excel api",
        "license": "AGPL",
        "maintainer": "",
        "maintainer_email": "",
        "name": "ckanapi-exporter",
        "package_url": "https://pypi.org/project/ckanapi-exporter/",
        "platform": "",
        "project_url": "https://pypi.org/project/ckanapi-exporter/",
        "project_urls": {
            "Homepage": "https://github.com/ckan/ckanapi-exporter"
        },
        "release_url": "https://pypi.org/project/ckanapi-exporter/0.1.0/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Export dataset metadata from CKAN to Excel",
        "version": "0.1.0"
    },
    "last_serial": 5006452,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a00530d7c990cde3280727045350ac3f",
                    "sha256": "7d36f5a6a7c033677e631de8d81d33b2e5a532ce01f019b36c84d9df9a6e8280"
                },
                "downloads": -1,
                "filename": "ckanapi-exporter-0.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "a00530d7c990cde3280727045350ac3f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3738,
                "upload_time": "2014-10-22T19:51:12",
                "url": "https://files.pythonhosted.org/packages/df/9d/033dd92f2f7ae3a1db45fe60cd308c6da97a0fc5aeb5ca7e87b00c229fed/ckanapi-exporter-0.0.1.tar.gz"
            }
        ],
        "0.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "adb37e2249d76171a5ab0a397854623a",
                    "sha256": "8dbed1fa7ef964ae7da2985ecb13c8df5199b9e67be1b8470e75aed7e23508d2"
                },
                "downloads": -1,
                "filename": "ckanapi-exporter-0.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "adb37e2249d76171a5ab0a397854623a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6352,
                "upload_time": "2014-11-06T19:30:20",
                "url": "https://files.pythonhosted.org/packages/da/b3/478cd782ae1d6183f3236c9c64ac1d3883b4d3307bdf0a7aad19713e9d4b/ckanapi-exporter-0.0.2.tar.gz"
            }
        ],
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "60e1a4087392e3ece0907a3f1f883d4c",
                    "sha256": "398ecf927f560406f10a02325c79088cfb60f5c32e69cb53b200482a273a1284"
                },
                "downloads": -1,
                "filename": "ckanapi-exporter-0.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "60e1a4087392e3ece0907a3f1f883d4c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6446,
                "upload_time": "2014-11-06T19:42:36",
                "url": "https://files.pythonhosted.org/packages/73/e9/e933b954e0d86051e5e1e3dcef6b81497dba3238cc69a6472336febf79d4/ckanapi-exporter-0.0.3.tar.gz"
            }
        ],
        "0.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7ea5e355d27288b6073fc557c50898a3",
                    "sha256": "095d1caa70c022da6eefd4b81340bf760f832e6be4e25f6d195295e56972713b"
                },
                "downloads": -1,
                "filename": "ckanapi-exporter-0.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "7ea5e355d27288b6073fc557c50898a3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8293,
                "upload_time": "2014-11-09T15:50:35",
                "url": "https://files.pythonhosted.org/packages/d0/ad/9f752b853bda6033fa92a99a872a80d505bbb819bf853068c3a468ac3c09/ckanapi-exporter-0.0.4.tar.gz"
            }
        ],
        "0.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e50c53a03911e6456b5f0a2b4b88f014",
                    "sha256": "dfecd1d04f68c2e9d10d029edd18d1798025ee25da3731468cd2069e09215a53"
                },
                "downloads": -1,
                "filename": "ckanapi-exporter-0.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "e50c53a03911e6456b5f0a2b4b88f014",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8304,
                "upload_time": "2014-11-20T10:37:02",
                "url": "https://files.pythonhosted.org/packages/68/7a/5ba2ca49bc53dbd06e8c1bbde3477e77e706be42b94abce393c21519afbd/ckanapi-exporter-0.0.5.tar.gz"
            }
        ],
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b6ddac36be93482689a9572d46b93e1a",
                    "sha256": "b0e1d0ec23b6940ab020f96df909820d7948b1ec6d3716ff90453127f2807c2d"
                },
                "downloads": -1,
                "filename": "ckanapi-exporter-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "b6ddac36be93482689a9572d46b93e1a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8139,
                "upload_time": "2019-03-30T08:51:19",
                "url": "https://files.pythonhosted.org/packages/c1/9a/c2a7f0dc0da61d45be7c065514b2607b2397b16a33f5940ec34029f7e856/ckanapi-exporter-0.1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "b6ddac36be93482689a9572d46b93e1a",
                "sha256": "b0e1d0ec23b6940ab020f96df909820d7948b1ec6d3716ff90453127f2807c2d"
            },
            "downloads": -1,
            "filename": "ckanapi-exporter-0.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "b6ddac36be93482689a9572d46b93e1a",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 8139,
            "upload_time": "2019-03-30T08:51:19",
            "url": "https://files.pythonhosted.org/packages/c1/9a/c2a7f0dc0da61d45be7c065514b2607b2397b16a33f5940ec34029f7e856/ckanapi-exporter-0.1.0.tar.gz"
        }
    ]
}