{
    "info": {
        "author": "Jonas Hagen",
        "author_email": "jonas.hagen@iap.unibe.ch",
        "bugtrack_url": null,
        "classifiers": [
            "Operating System :: OS Independent"
        ],
        "description": "\n# databird\n\nPeriodically retrieve data from different sources.\n\nThe `databird` package only provides a framework to plan and run the tasks needed to keep a local data-file-store up do date with various remote sources.\nThe remote sources can be anything (e.g. FTP Server, ECMWF, HTTP Api, SQL database, ...), as long as there is a *databird-driver* available for the specific source.\n\n## Usage\n\nDatabird is configured with configuration files and invoked by\n\n```\n$ databird retrieve -c /etc/databird/databird.conf\n\n# or (as the above is the default)\n$ databird retrieve\n```\n\nYou can store the configuration files anywhere and for example run the above command periodically as cron job.\n\nAlso, some rq workers are required:\n\n```\n$ rq worker databird\n```\n\nThis will start one worker. You should use a supervisor to start multiple workers.\n\n## Configuration\n\nThe following example configuration defines a repository, which is populated with daily GNSS data from [ftp://cddis.nasa.gov/gnss/data/daily/](ftp://cddis.nasa.gov/gnss/data/daily/).\n\nThe main configuration file (usually `databird.conf`) could look like that:\n\n```yml\ngeneral:\n  root: /data/repos # root path for data repositories\n  num-workers: 16   # max number of async workers\n  include: \"databird.conf.d/*.conf\"  # include config files\n```\n\nGenerally you can configure anything in any file, as all configuration files are merged to one configuration tree. The `include` option is an exception, as it can only be declared in the top config file.\n\nThen in `databird.conf.d/cddis.conf` you can configure a profile and a repository:\n\n```yml\nprofiles:\n  nasa_cddis:\n    driver: standard.FtpDriver\n    configuration:\n      host: cddis.nasa.gov\n      user: anonymous\n      password: \"\"\n      tls: False\n\nrepositories:\n  nasa_gnss:\n    description: Data from NASAs Archive of Space Geodesy Data\n    profile: nasa_cddis\n    period: 1 day\n    delay: 2 days\n    start: 2019-01-01\n    targets:\n      status: \"{time:%Y}/cddis_gnss_{iso_date}.status\"\n    configuration:\n      user: anonymous  # this could override 'user' from profile\n      root: \"/gnss/data/daily\"\n      patterns:\n        status: \"{time:%Y}/{time:%j}/{time:%y%j}.status\"\n```\n\nWhen calling databird with this configuration the following is achieved:\n\n* A repository in the folder `/data/repos/nasa_gnss/` is created\n* For every day, a file like `2019/nasa_gnss_2019-01-20.status` is expected\n* If that file is missing, retrieve it from `ftp://cddis.nasa.gov/gnss/data/daily/2019/020/19020.status`\n* If there are many files missing, the data is retrieved asynchronously\n\nThis example used the `standard.FTPDriver`.\n\n## Monitoring\n\nUse `databird webmonitor [PORT]` to start the web interface.\n\nSince databird uses RQ for managing jobs, you also check the options at [RQ/docs/monitoring](https://python-rq.org/docs/monitoring/).\n\n## Drivers\n\nAnyone can write drivers (see below). Currently, the following drivers are available:\n\nIncluded:\n\n* `standard.FilesystemDriver`: Retrieve data from the local filesystem\n* `standard.CommandDriver`: Run an arbitrary shell command\n* `standard.FtpDriver`: Retrieve data from an FTP server\n\nClimate:\n\n* `climate.EcmwfDriver`: Retrieve data from the European Centre for Medium-Range Weather Forecasts (ECMWF) via their API\n* `climate.C3SDriver`: Retrieve data from the Copernicus Climate Change Service (C3S) via their API\n* `climate.GesDiscDriver`: Retrieve data from the NASA EarthData GES DISC service.\n\n\n## Development\n\n1. Create a Python environment and activate it\n   ``` shell\n   $ python3 -m venv . && source bin/activate\n   ```\n2. Install the development environment:\n   ``` shell\n   (databird) $ pip install -r requirements-dev.txt\n   ```\n\n### Writing a new driver\n\nDrivers are published in a namespace package `databird-drivers`. Everyone can develop drivers and share them.\n\nInstall `databird` and run mr.bob to create a new driver package:\n\n```\n(databird) $ cd $HOME/projects\n(databird) $ python -m mrbob.cli databird.blueprints:driver\n```\n\nAfter answering some questions, a new directory `databird-driver-<chosen_name>` is created.\nLets asume `<chosen_name> = foo`, then your driver is usually implemented in `databird/drivers/foo/foo.py` in a class named `FooDriver()`.\nUntil more documentation is available, you have to look at the code to figure out how to write a driver.\n\nOther people will be able to use it with `driver: foo.FooDriver`.\n\nTell me if you wrote a new driver, so I can include it in the list.\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "https://github.com/jonas-hagen/databird/archive/databird-0.7.1.tar.gz",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/jonas-hagen/databird",
        "keywords": "",
        "license": "MIT",
        "maintainer": "Jonas Hagen",
        "maintainer_email": "jonas.hagen@iap.unibe.ch",
        "name": "databird",
        "package_url": "https://pypi.org/project/databird/",
        "platform": "",
        "project_url": "https://pypi.org/project/databird/",
        "project_urls": {
            "Download": "https://github.com/jonas-hagen/databird/archive/databird-0.7.1.tar.gz",
            "Homepage": "https://github.com/jonas-hagen/databird"
        },
        "release_url": "https://pypi.org/project/databird/0.7.1/",
        "requires_dist": [
            "click",
            "dict-recursive-update",
            "flask",
            "frozendict",
            "mr.bob",
            "redis",
            "rq",
            "rq-dashboard",
            "ruamel.yaml"
        ],
        "requires_python": ">=3.5.*, <4",
        "summary": "Keeps a local data repository up to date with different remote data sources.",
        "version": "0.7.1"
    },
    "last_serial": 5645042,
    "releases": {
        "0.6.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7d4eaf23721fccf84db01cfbac088212",
                    "sha256": "994646af8af5c72ad1f6bada725d1062ed7017a93fa14ea4e1a02c26f69f2976"
                },
                "downloads": -1,
                "filename": "databird-0.6.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "7d4eaf23721fccf84db01cfbac088212",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.5.*, <4",
                "size": 13139,
                "upload_time": "2019-03-28T16:07:36",
                "url": "https://files.pythonhosted.org/packages/6a/81/e6bc4b0df862e44356c7b75ac0535f1be379a41bd83e024ff472b9d4fedf/databird-0.6.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "7be4f62c98ad0922f1c032be6b6714ea",
                    "sha256": "f1b90c2374bdf07b9e688f1cc7f19bc80ca0cb3f7c99d201e5b3ca1f07967fa7"
                },
                "downloads": -1,
                "filename": "databird-0.6.2.tar.gz",
                "has_sig": false,
                "md5_digest": "7be4f62c98ad0922f1c032be6b6714ea",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5.*, <4",
                "size": 9009,
                "upload_time": "2019-03-28T16:07:38",
                "url": "https://files.pythonhosted.org/packages/ea/0c/53cc3782edfcb79ba94c39ec14a2e9e9e04877c2a8f84c59df9bab0fa302/databird-0.6.2.tar.gz"
            }
        ],
        "0.6.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "032e7a333a9c2a84eebbf17ee82c0ba6",
                    "sha256": "ace53e6656450606049e0a8c1ea19abdf009bad2146ce242623d5c7647af90f6"
                },
                "downloads": -1,
                "filename": "databird-0.6.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "032e7a333a9c2a84eebbf17ee82c0ba6",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.5.*, <4",
                "size": 13087,
                "upload_time": "2019-03-28T16:15:27",
                "url": "https://files.pythonhosted.org/packages/7c/8e/b89f89477c9fe08e086bad3f162b22cba5cec6411ef1e5c31306a479f1f1/databird-0.6.3-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "9d4a96bf8f690199a4704c42f4d492d0",
                    "sha256": "44865c8822b5c92c21646cc19eb96545e55ad7e82fdae01d8fb668414d87bafc"
                },
                "downloads": -1,
                "filename": "databird-0.6.3.tar.gz",
                "has_sig": false,
                "md5_digest": "9d4a96bf8f690199a4704c42f4d492d0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5.*, <4",
                "size": 9040,
                "upload_time": "2019-03-28T16:15:30",
                "url": "https://files.pythonhosted.org/packages/56/bd/e2268b7c8c568e49b794757be9764bbfe12874bfeac6fd38396f442c1166/databird-0.6.3.tar.gz"
            }
        ],
        "0.7.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e99ec7742383a7cddadc3656728a6f02",
                    "sha256": "762f125a6cdfab6c2025ccf6faa889507d806cfefe4fd37caa1f7ee04fb91d73"
                },
                "downloads": -1,
                "filename": "databird-0.7.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "e99ec7742383a7cddadc3656728a6f02",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.5.*, <4",
                "size": 15780,
                "upload_time": "2019-08-06T12:45:17",
                "url": "https://files.pythonhosted.org/packages/d5/0d/72619297188a7584de88ba92789fd48ddf07f1b2cf8e0c948171d845acee/databird-0.7.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c01527dac69ffc12dd1c0dbd027c06bd",
                    "sha256": "cb41da2041ef4d3ed6d103f689bdeec4d496510d0202a62a013cbfb917385962"
                },
                "downloads": -1,
                "filename": "databird-0.7.0.tar.gz",
                "has_sig": false,
                "md5_digest": "c01527dac69ffc12dd1c0dbd027c06bd",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5.*, <4",
                "size": 13074,
                "upload_time": "2019-08-06T12:45:19",
                "url": "https://files.pythonhosted.org/packages/1e/4d/7cf6f95d1b088eaf0d0c7bdb3b0e3465aa1bea99a3ec40ad1ff5f19c7c44/databird-0.7.0.tar.gz"
            }
        ],
        "0.7.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5a9549108cd68c791f281fe1d178ac3b",
                    "sha256": "2d38f1fa6b05801d8f38b9cbad078fb2678825c022cb72f8ea17be9c387bcec7"
                },
                "downloads": -1,
                "filename": "databird-0.7.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "5a9549108cd68c791f281fe1d178ac3b",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.5.*, <4",
                "size": 16263,
                "upload_time": "2019-08-07T13:18:51",
                "url": "https://files.pythonhosted.org/packages/69/1e/ee90022a41bea22a148714646137cc86dce2e6d26590d36d210760fe71bd/databird-0.7.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "7e6c43af9f31807c37b4c42158a4a0fd",
                    "sha256": "7e18bdc21768ac50b076aa8fd2bbc66662b9b1f38a3a1d463b2a20cb5f3c6f1d"
                },
                "downloads": -1,
                "filename": "databird-0.7.1.tar.gz",
                "has_sig": false,
                "md5_digest": "7e6c43af9f31807c37b4c42158a4a0fd",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5.*, <4",
                "size": 13494,
                "upload_time": "2019-08-07T13:18:53",
                "url": "https://files.pythonhosted.org/packages/f9/11/279e51ab7bfea44627d6a1e32d8df76164d60a1e44139866de6436912796/databird-0.7.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "5a9549108cd68c791f281fe1d178ac3b",
                "sha256": "2d38f1fa6b05801d8f38b9cbad078fb2678825c022cb72f8ea17be9c387bcec7"
            },
            "downloads": -1,
            "filename": "databird-0.7.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "5a9549108cd68c791f281fe1d178ac3b",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.5.*, <4",
            "size": 16263,
            "upload_time": "2019-08-07T13:18:51",
            "url": "https://files.pythonhosted.org/packages/69/1e/ee90022a41bea22a148714646137cc86dce2e6d26590d36d210760fe71bd/databird-0.7.1-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "7e6c43af9f31807c37b4c42158a4a0fd",
                "sha256": "7e18bdc21768ac50b076aa8fd2bbc66662b9b1f38a3a1d463b2a20cb5f3c6f1d"
            },
            "downloads": -1,
            "filename": "databird-0.7.1.tar.gz",
            "has_sig": false,
            "md5_digest": "7e6c43af9f31807c37b4c42158a4a0fd",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.5.*, <4",
            "size": 13494,
            "upload_time": "2019-08-07T13:18:53",
            "url": "https://files.pythonhosted.org/packages/f9/11/279e51ab7bfea44627d6a1e32d8df76164d60a1e44139866de6436912796/databird-0.7.1.tar.gz"
        }
    ]
}