{
    "info": {
        "author": "",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "<!---\nCopyright 2019 RStudio, Inc.\nAll rights reserved.\n\nUse of this source code is governed by a BSD 2-Clause\nlicense that can be found in the LICENSE_BSD file.\n-->\n\n[![Build Status](https://travis-ci.org/ursa-labs/ursabot.svg?branch=master)](https://travis-ci.org/ursa-labs/ursabot)\n\n# Ursa Labs' buildbot configuration for Apache Arrow\n\nUrsabot is a continous integration framework based on the\n[buildbot][buildbot-docs] framework. The primary focus of ursabot is to\nexecute various builds benchmark and packaging tasks for\n[Apache Arrow][arrow-url] however `ursabot` can be used for arbitrary projects.\n\n## Notable features\n\n- a standalone project abstraction to make the project configurations module\n  and reusable, and a less verbose master configuration supporting multiple\n  projects\n- locally reproducible builds via command line interface\n- attachable interactive shells to the docker workers in case of build failures\n- local source mounting for docker workers\n- declerative builder configuration and a docker builder which makes it easier\n  to work with docker latent workers\n- extended github hook to drive buildbot via github comments\n- click based comment parser\n- improved change filter to filter changes based on build properties\n- reimplemented github reporters: status-, comment- and review reporters\n- easily extensible formatter classes to use with the reimplemented reporters\n- steps implemented based on new-style ShellCommand step\n- a token rotator to use multiple github tokens with github services\n- a docker image tool to maintain and build hierachical docker images\n- command line interface and additional utilities\n\n## Driving Ursabot\n\nAllowing PR reviewers to request additional checks on demand within the review\nprocess makes it easier for us to apply extra scrutiny at review time while\nalso conserving CI bandwidth by using human expertise to know which checks are\nneeded.\n\n### via Comments\n\nUrsabot receives github events through a webhook. It listens on pull request\ncomments mentioning @ursabot. It follows the semantics of a command line\ninterface, to see the available commands add a comment on the pull request:\n`@ursabot --help`.\n\nThe @ursabot GitHub user will respond or [react][github-reactions] that it has\nstarted a build for you. The command parser is implemented in\n[commands.py](commands.py).\n\nCurrently available commands:\n\n  - `@ursabot build`: Triggers all the ursabot tests. These tests are run\n    automatically, but this is a convinient way to force a re-build.\n  - `@ursabot benchmark`: Triggers C++ benchmarks and sends back the results as\n    a github comment and highlights the regressions.\n  - `@ursabot crossbow test cpp-python`: Triggers the `cpp-python` test group\n    defined in [test.yml][crossbow-tests] and responds with a URL pointing to\n    submitted crossbow branches at the github UI showing the build statuses.\n  - `@ursabot crossbow package -g wheel -g conda`: Triggers the `wheel` and\n    `conda` crossbow packaging groups defined in [tasks.yml][crossbow-tasks].\n  - `@ursabot crossbow package wheel-win-cp35m wheel-win-cp36m`: Triggers only\n    two tasks passed explicitly.\n\nNote that the commands won't trigger any builds if the commit message contains\na skip pattern, like `[skip ci]` or `[ci skip]`. In order to drive ursabot\nthe user must have either 'OWNER', 'MEMBER' or 'CONTRIBUTOR\n[roles][github-author-association].\n\n### via the Web UI\n\nYou can also initiate a build for a specific architecture/configuration in the\n[buildbot UI][ursabot-url]. Navigate to [Builds > Builders][ursabot-builders],\nselect a builder, and click `Build apache/arrow` buttin at the top right. This\ntriggers the force schedulers where you can specify a branch and/or commit to\nbuild. In the future specialized builders will have different fields to provide\nthe neccessary information.\n\n### via CLI\n\nBuildbot supports submitting local patches directly to the cluster and\ntriggering specific builders. The `TryScheduler` is a really handy way to test\nlocal changes without polluting the git history:\n\n```bash\nbuildbot try \\\n  --connect=pb \\\n  --master=... \\\n  --username=... \\\n  --passwd=... \\\n  --get-builder-names\n```\n\nIf someone wants to use this feature then please raise an issue, because it\nrequires custom credentials.\n\n## Install ursabot and the CLI\n\nRunning it locally helps with the development and testing new feature and/or\ndebugging issues without touching the production instance.\n\nInstallation requires at least Python 3.6:\n\n```bash\ncd /path/to/ursabot\npip install -e .\n```\n\nNow the `ursabot` command is available which looks for a `master.cfg` file in\nthe current directory. `master.cfg` can be passed explicitly via the `--config`\noption:\n\n```bash\nursabot -c path/to/master.cfg\n```\n\nDescribe the loaded master configuration:\n\n```bash\nursabot desc\n```\n\nDescribe the loaded project configuration:\n\n```bash\nursabot project desc  # for master configs with a single project\nursabot project -p arrow desc  # for master configs with multiple projects\n```\n\n## How to validate the configurations\n\nThe `checkconfig` command runs sanity checks and various validations on the\nmaster configuration. Most of the time is `checkconfig` passes then the master\ncan be run successfully (unless there are some variables only available at\nruntime).\n\n```bash\nursabot checkconfig\n```\n\n`ursabot` command loads `master.cfg` from the current directory by default, but\n`--config` argument can be passed to explicitly define a configuration file.\n\n```bash\nursabot -c arrow/master.cfg checkconfig\n```\n\nThe top-level `master.cfg` contains the production configuration for \nci.ursalabs.org so it requires additional dependencies like `pass`. \nTo install `pass`:\n\n```bash\nwhich apt && sudo -H apt install -V -y pass\nwhich brew && brew install pass\n```\n\n## Run a local instance of Ursabot\n\nAfter installation master's database must be initialized:\n\n```bash\nursabot -v upgrade-master\n```\n\nStart/stop/restart the master:\n\n```bash\nursabot -v start|stop|restart\n```\n\nDefine the configuration environment (prod|test) and start the service:\n\n```bash\nexport URSABOT_ENV=test  # this is the default\nbuildbot restart ursabot\ntail -f ursabot/twisted.log\n```\n\nThen open `http://localhost:8100` in the browser.\n\n## Commands for local reproducibility\n\nBuilders can be run locally without the web interface using the\n`ursabot project build` command.\n\nTesting `AMD64 Conda C++` builder on master:\n\n```bash\nursabot project build 'AMD64 Conda C++'\n```\n\nTesting `AMD64 Conda C++` builder with github pull request number 140:\n\n```bash\nursabot project build -pr 140 'AMD64 Conda C++'\n```\n\nTesting `AMD64 Conda C++` with local repository:\n\n```bash\nursabot project build -s ~/Workspace/arrow:. 'AMD64 Conda C++'\n```\n\nWhere `~/Workspace/arrow` is the path of the local Arrow repository and `.`\nis the destination directory under the worker's build directory (in this case:\n`/buildbot/AMD64_Conda_C__/.`)\n\nPassing multiple buildbot properties for the build:\n\n```bash\nursabot project build -p prop=value -p myprop=myvalue 'AMD64 Conda C++'\n```\n\n### Attach on failure\n\nUrsabot supports debugging failed builds with attach attaching ordinary shells\nto the still running workers - where the build has previously failed.\n\nUse the `--attach-on-failure` or `-a` flags.\n\n```bash\nursabot project build --attach-on-failure `AMD64 Conda C++`\n```\n\n## Configuring Ursabot\n\nThe buildmaster configuration happens in the `master.cfg` files. Originally\nbuildbot loads the dictionary called `BuildmasterConfig`, but to make it more\nflexible and moduler ursabot introduces the `ProjectConfig` and `MasterConfig`\nabstractions.\n`ProjectConfig` contains all the relevant information for testing a project\nlike Apache Arrow or Ursabot itself. `ProjectConfig` can be run alone, it must\nbe passed to a `MasterConfig` object which provides a thin abstraction over\nthe original buildbot `BuildmasterConfig`. One `MasterConfig` can\n[contain multiple][multiple-configs] `ProjectConfig` objects.\n[Including other project configurations][Including-configs] makes it possible\nto maintain the project relevant settings within the projects' repositories\ninstead of a decoupled one dedicated for the buildmaster.\n\n\n### Adding a new build(er)s\n\nThe closest abstraction to the traditional yaml based CI configs in ursabot are\nthe Builders. In the simplest case a builder is defined by a sequence of steps\nwhich are executed as shell commands on the worker.\nThe following example builder presumes, that `apt-get` and `git` is available\non the worker.\n\n```python\nfrom buildbot.plugins import util, worker\nfrom ursabot.steps import ShellCommand\nfrom ursabot.builders import Builder\nfrom ursabot.schedulers import AnyBranchScheduler\n\n\nrepo = 'https://github.com/example/repo'\n\n\nclass TestBuilder(Builder):\n    tags = ['example-build', 'arbitrary-tag']\n    steps = [\n        GitHub(\n            name='Clone the test repository',\n            repourl=repo,\n            mode='full'\n        ),\n        ShellCommand(\n            name='Install dependencies',\n            command=['apt-get', 'install', '-y'],\n            args=['my', 'packages']\n        ),\n        ShellCommand(\n            name='Execute tests',\n            command=['my-custom-test-runner', util.Property('test-selector')]\n        )\n    ]\n\n\n# in the master.cfg\nlocal_worker = worker.LocalWorker('my-local-worker')\nsimple_builder = TestBuilder(\n    workers=[local_worker],\n    properties={\n        'test-selector': 'all'\n    }\n)\nscheduler = AnyBranchScheduler(\n    name='my-scheduler-name',\n    builders=[simple_builder]\n)\n\nproject = ProjectConfig([\n    name='example/repo',\n    repo='https://github.com/example/repo'\n    workers=[local_worker],\n    builders=[simple_builder],\n    schedulers=[scheduler]\n])\n\nmaster = MasterConfig(\n    title='TestConfig',\n    projects=[project]\n)\n```\n\nThe `DockerBuilder` provides more flexibility, faster builds and better worker\nisolation, Ursabot uses `DockerBuilders` extensively.\n\n```python\nfrom ursabot.docker import DockerImage\nfrom ursabot.builders import DockerBuilder\nfrom ursabot.workers import DockerLatentWorker\n\n\nminiconda = DockerImage(\n    'conda',\n    base='continuumio/miniconda3',\n    arch='amd64',\n    os='debian-9'\n)\n\n\nclass TestDockerBuilder(DockerBuilder):\n    tags = ['build-within-docker-container']\n    steps = [\n        # checkout the source code\n        GitHub(args0),\n        # execute arbitrary commands\n        ShellCommand(args1),\n        ShellCommand(args2),\n        # ...\n    ]\n\n\ndocker_worker = DockerLatentWorker(\n    name='my-docker-worker'\n    arch='amd64'\n    password=None,\n    max_builds=2\n)\n\n# instantiates builders based on the available workers, the Builder's\n# images and the workers are matched based on their architecture\ndocker_builders = TestDockerBuilder.combine_with(\n    workers=[docker_worker],\n    images=[miniconda]\n)\n\nscheduler = AnyBranchScheduler(\n    name='my-scheduler-name',\n    builders=docker_builders\n)\n\nproject = ProjectConfig([\n    name='example/repo',\n    repo='https://github.com/example/repo'\n    images=[miniconda],\n    workers=[docker_worker],\n    builders=docker_builders,\n    schedulers=[scheduler]\n])\n\nmaster = MasterConfig(\n    title='TestConfig',\n    projects=[project]\n)\n```\n\n## Define docker images\n\nArrow supports multiple platforms, has a wide variety of features thus a lot of\ndependencies. Installing them in each build would be time and resource\nconsuming, so ursabot ships docker images for reusability.\n\nThere is a small docker utility in `ursabot.docker` module to define\nhierachical images. It uses a DSL implemented in python instead of plain\nDockerfiles. A small example to demonstrate it:\n\n```python\nfrom ursabot.docker import DockerImage, ImageCollection\nfrom ursabot.docker import RUN, ENV, CMD, ADD, apt, conda\n\n\nminiconda = DockerImage(\n    name='conda',\n    base='continuumio/miniconda3',\n    arch='amd64',\n    os='debian-9'\n)\npandas = DockerImage(\n    name='pandas',\n    base=miniconda,\n    steps=[\n        RUN(conda('pandas'))\n    ]\n)\npyarrow = DockerImage(\n    name='pyarrow',\n    base=miniconda,\n    steps=[\n        RUN(conda('pyarrow'))\n    ]\n)\n\nimages = ImageCollection([miniconda, pandas, pyarrow])\n\n# create a docker image for each of the previous ones running jupyter notebook\njupyter_steps = [\n    RUN(conda('jupyter')),\n    CMD([\n        'jupyter', 'notebook',\n        '--ip', '0.0.0.0',\n        '--no-browser',\n        '--allow-root'\n    ])\n]\nimages.extend(\n    DockerImage(\n        name=image.name,\n        base=image,\n        tag='jupyter',\n        steps=jupyter_steps\n    )\n    for image in images\n])\n\n# build all of the images in topological order\nimages.build()\n\n# filter the images\nprint(images.filter(name='pyarrow', tag='jupyter'))\n```\n\nTry running jupyter with pyarrow pre-installed:\n\n```bash\ndocker run -p 8888:8888 amd64-debian-9-pyarrow:jupyter\n```\n\nUrsabot has a CLI interface to build the docker images:\n\n```bash\nursabot docker build --help\n```\n\nTo list Arrow C++ `amd64` `conda` `cpp` images:\n\n```bash\nursabot --verbose docker --arch amd64 --variant conda --name cpp list\n```\n\nAdditional filtering:\n\n```bash\nursabot docker --arch amd64 list\nursabot docker --arch amd64 --variant conda list\nursabot docker --arch amd64 --variant conda --name cpp list\nursabot docker --arch amd64 --variant conda --name cpp --tag worker list\nursabot docker --arch amd64 --variant conda --name cpp --os debian-9 list\n```\n\nTo build and push Arrow C++ `amd64` `conda` `cpp` images:\n\n```bash\nursabot --verbose docker --arch amd64 --variant conda --name cpp build --push\n```\n\nTo build and push all `arm64v8` `alpine` images:\n\n```bash\nursabot --verbose \\\n  docker --docker-host tcp://arm-machine:2375 --arch arm64v8 --os alpine-3.9 \\\n  build --push\n```\n\n## Developing Ursabot\n\nBuildbot doesn't distribute its testing suite with binary wheels, so in order\nto run the unit tests buildbot must be installed from source:\n\n```bash\npip install --no-binary buildbot -e .\npytest -v ursabot\n```\n\n### Pre-commit hooks\n\nInstall [pre-commit](https://pre-commit.com/) then to setup the git\n[hooks](.pre-commit-config.yaml) run `pre-commit install`.\n\n\n### Adding new workers to ci.ursalabs.org\n\nAdding docker latent workers requires a worker entry in the `workers.yaml` configuration.\nName, architecture and a docker host (accessable by the buildmaster) are\nrequired, see an example in [workers.yaml](workers.yaml).\nAdding non-docker workers are also possible, but must register them in the\n[master.cfg](master.cfg).\n\n\n## Possible further improvements\n\n\nThese have been discussed and would be valuable, but they are definitely\n\"nice to haves\" and should be deferred until the primary goals are met.\n\n- Database for storing benchmark results\n- Central station for hosting the build artifacts\n- Dashboard showing build health across all platforms and configurations\n\nMore closely Ursabot related:\n\n- Multi-master setup for scaling\n- Setup WAMP/Crossbar to restart the buildmaster without cancelling the running\n  builds\n- Windows containers and workers (docker in virtualized nodes)\n\n\n[arrow-repo]: https://github.com/apache/arrow\n[arrow-url]: https://arrow.apache.org\n[archery-readme]: https://github.com/apache/arrow/tree/master/dev/archery\n[crossbow-readme]: https://github.com/apache/arrow/tree/master/dev/tasks\n[crossbow-repo]: https://github.com/ursa-labs/crossbow\n[crossbow-tests]: https://github.com/apache/arrow/blob/master/dev/tasks/tests.yml#L18\n[crossbow-tasks]: https://github.com/apache/arrow/blob/master/dev/tasks/tasks.yml#L18\n[ursabot-repo]: https://github.com/ursa-labs/ursabot\n[ursabot-url]: https://ci.ursalabs.org\n[ursabot-builders]: https://ci.ursalabs.org/#/builders\n[buildbot-docs]: https://docs.buildbot.net\n[github-reactions]: https://help.github.com/en/articles/about-conversations-on-github#reacting-to-ideas-in-comments\n[github-author-association]: https://developer.github.com/v4/enum/commentauthorassociation/\n[multiple-configs]: master.cfg#L137\n[including-configs]: master.cfg#L36\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://github.com/ursa-labs/ursabot",
        "keywords": "ursabot,buildbot,ci,continuous-integration",
        "license": "",
        "maintainer": "Ursa-Labs team",
        "maintainer_email": "team@ursalabs.org",
        "name": "ursabot",
        "package_url": "https://pypi.org/project/ursabot/",
        "platform": "",
        "project_url": "https://pypi.org/project/ursabot/",
        "project_urls": {
            "Homepage": "http://github.com/ursa-labs/ursabot"
        },
        "release_url": "https://pypi.org/project/ursabot/0.1.0/",
        "requires_dist": [
            "buildbot-console-view",
            "buildbot-grid-view",
            "buildbot-waterfall-view",
            "buildbot-worker",
            "buildbot-www",
            "buildbot",
            "click",
            "distro",
            "docker-map",
            "docker",
            "dockerpty",
            "python-dotenv",
            "ruamel.yaml",
            "tabulate",
            "toolz",
            "toposort",
            "treq",
            "twisted[tls]",
            "typeguard"
        ],
        "requires_python": ">=3.6",
        "summary": "Extension for the Buildbot continuous integration tool",
        "version": "0.1.0"
    },
    "last_serial": 5741975,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "86a15320e351a9e87093019d3dfbcfb6",
                    "sha256": "baf7cbd59d7d2ecbae7a1defdd68b6915f52aaae5b52aa41e4b3d641b4ca6cef"
                },
                "downloads": -1,
                "filename": "ursabot-0.1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "86a15320e351a9e87093019d3dfbcfb6",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 85876,
                "upload_time": "2019-08-28T11:30:31",
                "url": "https://files.pythonhosted.org/packages/ae/b9/42d9b44419048e5d68965cb11a034ad0ae5399d2a15ffd0379d3cf807ed9/ursabot-0.1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "63f9e9716ce7eda3686e2bd8e1ca9dec",
                    "sha256": "91c4577083955af239d6deb938c28b0dd49d9577759866dba0abb5e56be2ccd3"
                },
                "downloads": -1,
                "filename": "ursabot-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "63f9e9716ce7eda3686e2bd8e1ca9dec",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 100990,
                "upload_time": "2019-08-28T11:30:34",
                "url": "https://files.pythonhosted.org/packages/96/95/5647ccc3ef828875767d91699f30238ac29b310c7bf8f9e1e1c43ce8d1d3/ursabot-0.1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "86a15320e351a9e87093019d3dfbcfb6",
                "sha256": "baf7cbd59d7d2ecbae7a1defdd68b6915f52aaae5b52aa41e4b3d641b4ca6cef"
            },
            "downloads": -1,
            "filename": "ursabot-0.1.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "86a15320e351a9e87093019d3dfbcfb6",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.6",
            "size": 85876,
            "upload_time": "2019-08-28T11:30:31",
            "url": "https://files.pythonhosted.org/packages/ae/b9/42d9b44419048e5d68965cb11a034ad0ae5399d2a15ffd0379d3cf807ed9/ursabot-0.1.0-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "63f9e9716ce7eda3686e2bd8e1ca9dec",
                "sha256": "91c4577083955af239d6deb938c28b0dd49d9577759866dba0abb5e56be2ccd3"
            },
            "downloads": -1,
            "filename": "ursabot-0.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "63f9e9716ce7eda3686e2bd8e1ca9dec",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6",
            "size": 100990,
            "upload_time": "2019-08-28T11:30:34",
            "url": "https://files.pythonhosted.org/packages/96/95/5647ccc3ef828875767d91699f30238ac29b310c7bf8f9e1e1c43ce8d1d3/ursabot-0.1.0.tar.gz"
        }
    ]
}