{
    "info": {
        "author": "Adi Roiban",
        "author_email": "adi.roiban@chevah.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "Programming Language :: Python"
        ],
        "description": "buildbot-configuration-builder\n##############################\n\nA configuration layer on top of Buildbot configuration dict.\n\nYou pass a simplified dict and it will return a new dict with all objects\nrequired for Buildbot configuration.\n\nThis project tries to simplify buildbot configuration by defining a set\nof common rules to be used by all builders. It reduced flexibility in favor\nor simplicity.\n\nSteps are just shell commands which take arguments and environment variables.\n\nIt provide a few specialized steps (ex to get the source code) but is not\ndesigned for custom steps written in Python.\n\nOnce the buildbot configuration is defined as a simple dict it should be\neasy for any member of the team to maintain it (add slaves or builders).\n\nFor configuration the Buildbot you will still need to read buildbot\ndocumentation.\n\nAt this stage it is designed to be used for private buildmasters.\n\nIt provides the following features:\n\n* Get single\n* Multiple projects\n* Slaves definition\n* Multiple slaves for a builder.\n* HTTP web status\n* Try schedulers with username/password authentication\n* Send build results to GitHub Pull Request state\n* Send email notifications\n\nI plan to add:\n\n* Run steps in a Python virtualenv\n* All defining Python version for virtualenv\n* Scheduler based on GitHub PR commits\n* Scheduler based on GitHub commits, to replace polling scheduler.\n* IRC status... for example when master builds fail\n\nIt does not provide (patches welcomed):\n\n* SVN, BZR, CVS...etc\n* Get code from multiple sources\n\n\nTravis CI comparison\n====================\n\nThis project was inspired by Travis CI simple configuration file.\nI wanted to have something similar for Buildbot.\n\nThere are already a few project which can convert a travis.yml into a\nbuildmaster configuration so this project is not designed this purpose.\n\nI wanted multiple projects on a single Buildbot instance and with slaves\non multiple operating systems.\n\nI also wanted the powers of `buildbot try` which allows to test a patch\nwithout committing the changes.\n\n\nBuilders\n========\n\nThe reason why you ended up with buildbot is to get a builder which will\nrun a set of steps, on a particular slave with option environment variables.\n\nNormal buildbot builder are magically created based on an `environment` and a\nset of steps. `environments` are created based on a set of slaves and a set\nof environment variables.\n\nContinue reading for info about slaves, environments, projects, steps.\n\n\nSlaves\n======\n\nA slave is the normal buildslave which has a name, password, max_builds.\nThey a defined using the `slaves` root key.\n\nYou have the optional `DEFAULT` slave definition with settings for all slaves.\n\nI hope that slaves are simple to understand, so here is an example::\n\n    {\n    OTHER_CONFIGS: {},\n\n    'slaves': {\n        # `DEFAULT` values are applied to all slaves and can be overwritten\n        # for each slave.\n        DEFAULT: {\n            'password': 'password',\n            'max_builds': 1,\n            'notify_on_missing': 'infrastructure@chevah.com',\n            },\n\n        # We have 2 linux slaved with default configuration.\n        'bs1c-lnx-ubuntu1404-x64-29': {},\n        'bs1i-lnx-debian7-x86-37': {},\n\n        # This is a windows slave which uses a different password.\n        'bs1i-win-2012-x64-34': {\n            'password': 'other-password'\n            },\n\n        # We have some linux slaves which can run many tasks.\n        'bsmeta1b-lnx-ubuntu1204-x86-30': {'max_builds': 100},\n        'bsmeta2i-lnx-ubuntu1204-x86-32': {'max_builds': 100},\n\n        },\n    }\n\n\nEnvironments\n============\n\nThe concept of `environment` is inspired by Travis CI environments.\n\nYou create an `environment` by associated a set of (similar) slaves with\na set of environment variables.\n\nA slave can be part of multiple environments.\n\nEnvironments are latest combined with projects and steps to create a builder.\n\nYou will define in a single place all the environments used by your projects.\nIn this way you can share similar environments (builders) between multiple\nprojects.\nLater, in project definition you can choose what environments are associated\nwith a specific project.\n\nThe final builder generated from environment and the project will be named:\n`PROJECTNAME-ENVNAME`\n\nAll values from the environment dictionary are copied to builders, with\nthe exception of the `slaves` list.\n\nThe builder will run only on one of the slaves associated with the environment,\nbased on a random rule.\n\nTo increase availability you can define multiple slaves for an environment,hence for a builder.\n\nYou have the optional `DEFAULT` environment definition with settings for all\nenvironments.\n\nIn addition, the following environment values are always set:\n\n* `CI=true`\n* `BUILDBOT=true`\n* `COMMIT=current_revision`\n* `BRANCH=name_of_the_branch`\n* `BUILD_NUMBER=buildbot_build_number`\n* `BUILDER_NAME=name_of_buildbot_builder`\n* `BUILD_DIR=directory_where_test_is_executed`\n* `TEST_ENVIRONMENT=name_of_builders_environment`\n* `TEST_ARGUMENTS=builder_test_property`\n\n`TEST_ARGUMENTS` are not present in builds triggered by gatekeepers. This is\ndone on purpose to avoid variable builds for gatekeepers.\n`TEST_ARGUMENTS` are copied in group builders, as well as skip steps.\n\nThe following environment variables are set if there is a property with the\nsame name, but in lowercase:\n\n* `GITHUB_TOKEN`\n* `GITHUB_PULL_ID`\n* `CODECOV_TOKEN`\n* `TEST_AUTHOR`\n\n\nExample::\n\n    {\n    OTHER_CONFIGS: {},\n    'environments': {\n        # These env args are applied to all environments.\n        DEFAULT: {\n            'TEST_TYPE': 'normal'\n            },\n        # We want to run tests on Linux\n        # but you can create environments based on distro / CPU, etc\n        # When combined with project named `brink` it will create a\n        # builder named `brink-linux`.\n        'linux': {\n            'slaves: [\n                'bs1c-lnx-ubuntu1404-x64-29',\n                'bs1a-lnx-centos7-x64-31'\n                ],\n            },\n        # This will generated a special builder which will overwrite the\n        # default environment variable `TEST_TYPE` and also add a new one.\n        # For project `brink` it will create builder `brink-leaks`.\n        'leaks': {\n            'slaves': [\n                'bs1c-lnx-ubuntu1404-x64-29',\n                'bs1i-win-2012-x64-34',\n                ]\n            'TEST_TYPE': 'leaks',\n            'CHEVAH_GC': 'yes',\n            },\n\n        # In case an environment has no extra environment variables you can\n        # defined it as a list of slaves.\n        # This will create builder `brink-meta-director`.\n        'meta-director': [\n            'bsmeta1b-lnx-ubuntu1204-x86-30',\n            'bsmeta2i-lnx-ubuntu1204-x86-32',\n            ],\n        },\n    }\n\n\nProjects\n========\n\nYou can use the same Buildbot installation for multiple projects.\nProjects don't have direct access to slaves but rather use the environments\nto run project's code on a slave.\n\nA project defines a set of project specific options:\n\n* `repo` - the url used to get project source\n* `github_slug` - used to publish GitHub commit status\n* `poll_interval` - number of seconds to wait for change source scheduler\n\nIt also defines a set of steps, a set of groups and a set of gatekeepers which\nare explained later.\n\nA `DEFAULT` project can be used do define default values for all projects.\nThese values will be used when a project does not specify a specific\nconfiguration.\n\nHere is an example for defining 2 projects. Steps, groups and gatekeepers\nare explained later::\n\n    {\n    OTHER_CONFIGS: {}\n    'projects': {\n        DEFAULT: {\n            'steps': [DEFAULT_STEPS_FOR_ALL_PROJECTS],\n\n            'poll_interval': 60,\n            },\n        # `brink` project will use the default steps\n        # will will check for changes on master much often.\n        'brink': {\n            'repo': 'http://git.chevah.com/brink.git',\n            'github_slug': 'chevah/brink',\n            'poll_interval': 30,\n            'groups': BRINK_GROUPS_EXPLAINED_LATER,\n            'gatekeepers': BRINK_GK_EXPLAINED_LATER,\n            },\n        # `compat` project has a different set of steps.\n        'compat': {\n            'repo': 'http://git.chevah.com/compat.git',\n            'github_slug': 'chevah/compat',\n            'steps': [COMPAT_SPECIFIC_STEPS],\n            'groups': COMPAT_GROUPS_EXPLAINED_LATER,\n            'gatekeepers': COMPAT_GK_EXPLAINED_LATER,\n            },\n        },\n    }\n\n\nSteps\n=====\n\nThe following step types are available\n\n* SLAVE_COMMAND - execute a shell command on slave\n* SOURCE_COMMAND - get project code or apply patch\n* MERGE_COMMAND - merge code with a branch\n\nDefault step type is SLAVE_COMMAND.\n\nYou can conditionally execute a step by using the `optional` configuration.\nIn this case it will be executed only when when `force_STEPNAME` property\nis present on the builder and is not false.\n\nThe same set of steps are executed for all builders. In order to run\ndifferent tests based on different environments/builders you should dispatch\nthem bases on environment variable.\n\nFor example to run unit tests, pyflakes checker and documentation builder a\nsingle `run_ci` shell command is used which should dispatch a specialized\ncommand based on the environment variables.\n\nSteps are defined inside the project's `steps` key::\n\n    from chevah.buildbot_configuration_builder.builder import (\n        MERGE_COMMAND,\n        SOURCE_COMMAND,\n        )\n\n    {\n    OTHER_CONFIGS: {}\n    'projects': {\n        'brink': {\n            'steps': [\n                # Get source based on project settings.\n                {'type': SOURCE_COMMAND},\n                # Merge with master.\n                {'type': MERGE_COMMAND , 'branch': 'master'},\n                # Option clean the build folder.\n                {\n                    'name': 'clean',\n                    'command': ['bash','./paver.sh', 'clean'],\n                    # This step is only executed when `force_clean=1`\n                    # builder property is defined.\n                    'optional': True,\n                    },\n                # Build dependencies\n                {\n                    'name': 'deps',\n                    'command': ['bash','./paver.sh', 'deps'],\n                    'add_environment': {\n                        'SOME_VAR': 'some-value',\n                        'OTHER_VAR': Interpolate('%(prop:builder)s'),\n                        },\n                    },\n                # Run tests\n                {\n                    'name': 'test',\n                    'command': ['bash', './paver.sh', 'run_ci'],\n                    },\n                ],\n            },\n        },\n    }\n\n\nGroups\n======\n\nYou can group multiple builders into a group.\nFor example you can create a group named `pr` which will trigger all builders\nrequired to validate a pull request commit and another group named\n`post-commit` which will trigger all builders required to check the committed\ncode. Or you can group them in 'supported' and 'experimental' builders.\n\nFor each member of a group a dedicated builder is created. This builder will\nexecute the steps associated with this project. The builder will be named\nPROJECT_NAME-ENV_NAME.\n\nA builder can be part of multiple groups.\n\nThe group will have a dedicated group builder which will trigger a build for\nall builders form the group and report the results once all builders are done.\n\nThe builder associated with GROUP_NAME for PROJECT_NAME will be named\nPROJECT_NAME-group-GROUP_NAME.\n\nGroups are defined inside the project's `groups` key::\n\n    {\n    OTHER_CONFIGS: {}\n    'projects': {\n        'brink': {\n            'groups': {\n                # This will create a builder named `brink-group-post-commit`\n                # and we can use this builder to trigger multiple builders.\n                'post-commit': [\n                    # Here is a list with environment names, NOT slave names.\n                    # This will created the following builders:\n                    # brink-leaks, brink-linux-x86, brink-windows-x64\n                    'leaks',\n                    'linux-x86',\n                    'windows-x64'\n                    ],\n                # This will create a builder named `brink-group-supported`\n                # Since `leaks` builder is very slow we don't run to check\n                # if a changes is ready for review.\n                'supported': [\n                    'linux-x86',\n                    'windows-x64',\n                    ],\n                # This will create a group named `brink-group-unstable`.\n                'unstable': [\n                    'solaris-x86',\n                    'freebsd-x64',\n                    ],\n                },\n            },\n        },\n    }\n\n\nGatekeepers\n===========\n\nGatekeepers are specialized builders which will not use the project's steps.\n\nThe following step types are available:\n\n* SEQUENTIAL_GROUP - run all builders from a group one after another and wait\n  for all builders to end.\n* PARALLEL_GROUP - run all builders in parallel and wait for all to end.\n* MASTER_COMMAND - run a shell command on master\n* DIRECTORY_UPLOAD - upload a folder on master\n\nBy default gatekeeper builders are triggered by try schedulers. You can\nchange this to trigger the builder based on changes on a branch using\n`'scheduler': 'master'` option.\n\n\nGatekeepers are defined inside the project's `gatekeepers` key::\n\n    {\n    OTHER_CONFIGS: {}\n    'projects': {\n        'brink': {\n            'gatekeepers': {\n                # Post commit builder which will run all tests\n                # in sequential mode.\n                'master': {\n                    # Run this builder after changes are done on master branch.\n                    'scheduler': 'master',\n                    'stable_timer': 300,\n                    'steps': [\n                        {\n                        # Trigger each builde from a group, one after another.\n                        'type': SEQUENTIAL_GROUP,\n                        # Name of the step show in logs.\n                        'name': 'step name all',\n                        # Name of the triggered group.\n                        'target': 'post-commit',\n                        },\n                        {\n                        # Upload local folder `dist` on buildmaster.\n                        'type': DIRECTORY_UPLOAD,\n                        'name': 'upload_production',\n                        'source': 'dist',\n                        'destination': '/srv/buildmaster/upload/production',\n                        'optional': True,\n                        },\n                        {\n                        # Execute shell command on buildmaster.\n                        'type': MASTER_COMMAND,\n                        'name': 'Fix permissions',\n                        'command': ['chmod', '-R', '755', '/srv/buildmaster/upload'],\n                        },\n                        ],\n                    'notifications': {\n                        # Once tests on master are done notify everyone on\n                        # both success or failure.\n                        'email_all': ['dev@domain.com'],\n                        },\n                    },\n\n                # Trigger tests and send result to GitHub for pull request\n                # status when asked by buildbot try.\n                'review': {\n                    'steps': [{\n                        'type': PARALLEL_GROUP,\n                        'name': 'check review step',\n                        'target': 'supported',\n                        'set_properties': {\n                            'codecov_token': '1234',\n                            },\n                        'copy_properties': ['github_pull_id'],\n                        }],\n                    'notifications': {\n                        'email_all': [INTERESTED_USERS],\n                        },\n                    'github_send_status': True,\n                    },\n\n                # Trigger tests and merge in master on success\n                # When asked by buildbot try.\n                'merge': {\n                    'steps':  [\n                        {'type': SOURCE_COMMAND},\n                        {\n                            'type': PARALLEL_GROUP,\n                            'name': 'all',\n                            'target': 'all',\n                            },\n                        {\n                            'name': 'merge-commit',\n                            'command': ['bash', './paver.sh', 'merge_commit'],\n                            },\n                        ],\n                    'notifications': {\n                        'email_passing': ['dev@domain.com'],\n                        'email_error': [INTERESTED_USERS],\n                        },\n                    },\n                },\n            },\n        },\n    }\n\n\nGlobal buildmaster configuration\n================================\n\nYou can directly define buildmaster configuration by using the `global` key\nfrom root dict::\n\n    {\n    OTHER_CONFIGS: {}\n    'global': {\n        'title': 'ACME Project Buildbot',\n        'db_url': 'sqlite:///state.sqlite',\n        'buildbotURL': 'http://build.domain.com/',\n        'titleURL': 'http://www.domain.com',\n\n        'slavePortnum': 10089,\n\n        'changeHorizon': 500,\n        'buildHorizon': 500,\n        'eventHorizon': 50,\n        'logHorizon': 50,\n        'buildCacheSize': 15,\n\n        'logCompressionMethod': 'gz',\n\n        'mergeRequests': True,\n        },\n    }\n\n\nWeb status\n==========\n\nWeb status settings are defined in the `web` key of the root dict::\n\n    {\n    OTHER_CONFIGS: {}\n    'web': {\n        'interface': '0.0.0.0',\n        'port': 10088,\n        'htpasswd': '/srv/www/buildbot.passwd',\n        'authorization': {\n            'gracefulShutdown': True,\n            'forceBuild': True,\n            'forceAllBuilds': True,\n            'pingBuilder': True,\n            'stopBuild': True,\n            'stopAllBuilds': True,\n            'cancelPendingBuild': True,\n            },\n        },\n    }\n\n\nEmail notifications\n===================\n\nGatekeeper builders can be configured to send email notifications.\n\nEmail settings are defined in the `email` key of the root dict::\n\n    def lookup(user):\n        \"\"\"\n        Called with each responsible user of the build and should\n        return the full email address.\n        \"\"\"\n        return user + '@domain.tld'\n\n    {\n    OTHER_CONFIGS: {}\n    'email': {\n        'server': {\n            'fromaddr': 'buildbot@domain.com',\n            'relayhost': 'smtp.domain.com',\n            'useTls': True,\n            'smtpPort': 587,\n            'smtpUser': 'stmp-user',\n            'smtpPassword': 'stmp-pass',\n            },\n\n        # This is optional and use to convert buildbot users to\n        # email addresses.\n        'user_to_email_mapper': lookup,\n        # Available placeholders for subject:\n        #  result, projectName, title, builder\n        'subject': '%(result)s %(builder)s',\n        },\n    }\n\n\nTry schedulers and triggers\n===========================\n\nEach builder has an associated Try scheduler so you can use `buildbot try`\nto run a patch or a revision.\n\nGroup builders and gatekeeper builders also have an associated try scheduler.\n\nThey are defined in the `try_scheduler` root key::\n\n    {\n    OTHER_CONFIGS: {}\n    'try_scheduler': {\n        # Port on which to listed for buildbot try requests.\n        'port': 10087,\n        # Credentials accepted by buildbot try.\n        'credentials': [('try_user_name', 'try_user_pass')],\n        # Default environment/slaves used to execute the try schedulers for\n        # groups and other builders which don't have an explicit environment.\n        'environment': 'meta-director',\n        },\n    }\n\n\nGitHub Integration\n==================\n\nYou can send Buildbot's results for a specific commit to GitHub.\nTo send them you will need a GitHub token.\n\nEach project also need to define the GitHub slug. See `projects` documentation.\n\nGitHub is configured using the `github` key from root dict::\n\n    {\n    OTHER_CONFIGS: {}\n    'github': {\n        'token': 'GITHUB-TOKEN-VALUE',\n        },\n    }\n\n\nmaster.cfg integration\n======================\n\nThis is the basic content of your master.cfg file::\n\n    # You will need to import more things for here to define steps.\n    from chevah.buildbot_configuration_builder.builder import (\n        generate_configuration\n        )\n\n    config = {\n        'global': {YOUR_DATA},\n        'try_scheduler': {YOUR_DATA},\n        'github': {YOUR_DATA},\n        'web': {YOUR_DATA},\n        'email': {YOUR_DATA},\n        'slaves': {YOUR_DATA},\n        'environments': {YOUR_DATA},\n        'projects': {YOUR_DATA},\n        }\n    BuildmasterConfig = generate_configuration(config)",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/chevah/buildbot-configuration-builder",
        "keywords": null,
        "license": "MIT",
        "maintainer": null,
        "maintainer_email": null,
        "name": "chevah-buildbot-configuration-builder",
        "package_url": "https://pypi.org/project/chevah-buildbot-configuration-builder/",
        "platform": "any",
        "project_url": "https://pypi.org/project/chevah-buildbot-configuration-builder/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "https://github.com/chevah/buildbot-configuration-builder"
        },
        "release_url": "https://pypi.org/project/chevah-buildbot-configuration-builder/0.3.0/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "Configuration layer on top of Buildbot dict.",
        "version": "0.3.0"
    },
    "last_serial": 2092797,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f7d1dbfbef1cdde09512664fbd7ab813",
                    "sha256": "7f2ca6622421943d2da18266be085f4c70aad111f8edb5b2bfecc28572d1446c"
                },
                "downloads": -1,
                "filename": "chevah-buildbot-configuration-builder-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "f7d1dbfbef1cdde09512664fbd7ab813",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 28442,
                "upload_time": "2015-04-02T19:43:26",
                "url": "https://files.pythonhosted.org/packages/70/bf/65661ce7853f32aea708baa952359a1847a6acfdd6136471eba1ba318a30/chevah-buildbot-configuration-builder-0.1.0.tar.gz"
            }
        ],
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "90b5594ce0a42b31728da9b9ee09b72a",
                    "sha256": "8cf125d0d483fd933c5567cc79676257eefdefd63c8436924027f8bdca8f79e7"
                },
                "downloads": -1,
                "filename": "chevah-buildbot-configuration-builder-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "90b5594ce0a42b31728da9b9ee09b72a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 28469,
                "upload_time": "2016-03-09T14:46:42",
                "url": "https://files.pythonhosted.org/packages/3a/8f/14e58f7300ffe96b7b35a17c74a2ff32385b44f815d9308447a16eb6662a/chevah-buildbot-configuration-builder-0.2.0.tar.gz"
            }
        ],
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0e502921e1e55e488de0aba863403a2e",
                    "sha256": "d2335ac8e32dc607945aa3e843c82f18beb8653c3cd6bb6b59742812474e8c57"
                },
                "downloads": -1,
                "filename": "chevah-buildbot-configuration-builder-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "0e502921e1e55e488de0aba863403a2e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 29455,
                "upload_time": "2016-04-30T20:15:03",
                "url": "https://files.pythonhosted.org/packages/17/01/b0215c1b4f76f7d72dea63dc8481e5cbbe64e37cedf3cc7dab5a15cef09c/chevah-buildbot-configuration-builder-0.3.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "0e502921e1e55e488de0aba863403a2e",
                "sha256": "d2335ac8e32dc607945aa3e843c82f18beb8653c3cd6bb6b59742812474e8c57"
            },
            "downloads": -1,
            "filename": "chevah-buildbot-configuration-builder-0.3.0.tar.gz",
            "has_sig": false,
            "md5_digest": "0e502921e1e55e488de0aba863403a2e",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 29455,
            "upload_time": "2016-04-30T20:15:03",
            "url": "https://files.pythonhosted.org/packages/17/01/b0215c1b4f76f7d72dea63dc8481e5cbbe64e37cedf3cc7dab5a15cef09c/chevah-buildbot-configuration-builder-0.3.0.tar.gz"
        }
    ]
}