{
    "info": {
        "author": "Tom Tang",
        "author_email": "tly1980@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 2 - Pre-Alpha",
            "Environment :: Web Environment",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: BSD License",
            "Operating System :: OS Independent",
            "Programming Language :: Python",
            "Topic :: Software Development :: Build Tools",
            "Topic :: Software Development :: Code Generators"
        ],
        "description": "## Agile Conf Document (WIP)\n\n[agile_conf](https://github.com/tly1980/agile_conf) - A config files (in [YAML](http://yaml.org) format) and template engine ([Jinja2](http://jinja.pocoo.org)) based configuration compile / management tool to make DevOp tasks (or maybe 1,000 other things) easier.\n\n### Motivation\n\nA lot of work of DevOps is about configs / deployment script generation and management.\n\nOne can easily implement script using [\"sed\"](http://en.wikipedia.org/wiki/Sed) to generate the configs / deployment scripts.\nHowever, [\"sed\"](http://en.wikipedia.org/wiki/Sed) is far away from a perfect tool when you want to do some slightly complicated find / replace.\nFrom my expierence, modern [Templating processor](http://en.wikipedia.org/wiki/Template_processor) does a much better job in:\n\n**translating the variables into any forms of text-based outputs** (HTML, XML, JSON, YAML, INI, ..., etc.).\n\nPowered by ([Jinja2](http://jinja.pocoo.org)), [agile_conf](https://github.com/tly1980/agile_conf) supports all the features that is built-in with ([Jinja2](http://jinja.pocoo.org)) templating, such as:\ntemplate inhertitance, include, .etc.\n\nOn top of that, [agile_conf](https://github.com/tly1980/agile_conf) comes with some useful filters for DevOps purpose:\n\n1. jsonify\n2. aws_userdata     \n(it can translate [AWS userdata](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html) into [cloudformation stack template](http://aws.amazon.com/cloudformation/aws-cloudformation-templates/))\n3. yamlify\n\nOther than that, I believe that we should be serious about config / deployment scripts. Rather than doing the scripts / config generation with the execution altogether at run time, I prefer that we can have the compiled scripts / configuration by hands **before** executing it. \n\nSo that we can review the deployment scripts / config before running them, to gain clear idea on what is going to happen and avoid the surpise you don't want.\n\nI also believe we should manage those compiled deployment scripts / configurations in [git](http://git-scm.com) or any other SCM, so that we can treat them seriously just like we usually does with our code.\nAnd because they're managed by SCM, we will have full history, diff between changes, and permissions control.\n\n\n### Basic workfolw\n\n0. Create a project, and use [git](http://git-scm.com) or other SCM to manage it.\n1. Define your variable and templates. \n2. Compile artifacts: any text-based config / scripts.\n3. Commit your changes (variable, templates and the compiled scripts and configs) to the [git](http://git-scm.com) or other SCM repository.\n4. Use Bamboo or Jenkins to check out the compiled scripts and configs and execute them.\nOr, you can always run the scripts locally as long as you have the permissions.\n5. Retired your compiled scripts and configs if you don't need them (You should destroy the resources accordingly, using the pre-compiled destroy scripts.)\n\n### Install\n\nUse [PIP](https://pip.pypa.io/en/latest/quickstart.html) to install it.\n\n```\npip install agile-conf\n```\nIf you don't have [PIP](https://pip.pypa.io/en/latest/quickstart.html), please [install](https://pip.pypa.io/en/latest/installing.html) it first.\n\nAfter the installation, you can run ```agc --help``` to verify.\n\n### Getting started\n```agc``` is the main commandline tool of [agile_conf](https://github.com/tly1980/agile_conf).\n\n0. Clone the boilplate locally.\n```\ngit clone https://github.com/tly1980/agile_conf_boilplate.git ~/workspace/agile_conf_boilplate\n```\n\nYou don't have to use this boilplate repository, you can create your own boilplate repository by using same directory structure.\n\n\n#### 1. Create a new project by using the boilplate. \n\n```\n$ agc create single_ec2 my_ec2 --bo_repo=~/workspace/agile_conf_boilplate/\ncreating project: my_ec2\nusing boilerplate: /Users/minddriven/workspace/agile_conf_boilplate/single_ec2\n```\n\nNotes: You can specify the boilplate_repo with ```--bo_repo```, or set it in enviornment variable: ```AGC_BOIL```.\n\n#### 2. Walk thorugh the project.\n\n```my_ec2``` project (build from single_ec2 boilplate) comes with following structure.\n\nPlease read through the comments.\n\n```\nmy_ec2\n\t/_script\n\t/cfn\n\t\tmodule.yaml  # the variables specifically for cfn module\n\t\tec2.josn.tpl # template of cloudformation script\n\t\t0_userdata.sh.tpl # template of the userdata. \n\t\t\t\t\t\t  # Rendering happended alphabatically\n\t\t\t\t\t\t  # '0_' prefix makes it the first one to be render.\n\tconf_perf.yaml   # config for 'perf' performance test builds.\n\tconf_prod.yaml   # config for 'prod' production builds.\n\tconf_uat.yaml    # config for 'uat' user user acceptance builds.\n\tMakefile\n\tproject.yaml    # the common variables \n\t                # that can use across \n\t                # diffeent modules\n\tREADME.md\n```\n\nIn project folder, any sub-folders do **NOT** has \"_\" prefix is a module. Each module can have its own templates. \nInside the module, any file that has \".tpl\" postfix in the name would be rendered.\n\nThe order of rendering is alphabetical. This is a simple way to avoid circulating dependencies.\n\nCommon template variables are defined in ```project.yaml```, ```conf.yaml```.\n\nModule specific variables are defined in ```module.yaml```.\n\n\nVariables defined in ```conf.yaml```, can be referenced in ```projects.yaml``` and ```module.yaml``` and templates.\n\nIn the single_ec2 projects, it has mupltiple conf.yaml for different enviornments.\n```conf_uat.yaml```, ```conf_prod.yaml``` and ```conf_prod.yaml```. When you run the command, you should run it with ```--conf``` options.\n\n\nWith a ```conf_uat.yaml```\n```yaml\nname: uat\nnetenv: uat   # will deploy to uat subnets\nnumber: 1\n```\n\nFollowing is a line in ```project.yaml```\n```yaml\nproduct_fullname: hello-ec2-{{ conf.name }}-{{ conf.number }}\n```\n\nwould be rendered into\n\n```yaml\nproduct_fullname: hello-ec2-uat-1\n```\n\nVariables defined in ```conf.yaml``` and ```project.yaml``` can be use in ```${MODULE}/module.yaml``` and templates.\n\nIf you want to see the exact value used in the templates:\nUSE ```inspect``` command.\n\n```\n$ agc inspect --conf conf_uat.yaml\n```\n\nOutput would be:\n```\nwith [conf=conf_uat.yaml]\n[conf]\nname: uat\nnetenv: uat\nnumber: 1\n\n\n[project]\ninstance_type:\n  perf: m3.large\n  prod: m3.medium\n  uat: t2.micro\nproduct_fullname: hello-ec2-uat-1\n\n\n[cfn]\nimage_id: ami-d50773ef\ninstance_type: t2.micro\nkey_name: my-key\nnetenv: uat\nsubnet_id:\n  prod: subnet-prodsubnet\n  uat: subnet-uatsubnet\nsubnet_security_groups:\n  prod:\n  - sg-prod1\n  - sg-prod2\n  uat:\n  - sg-uat1\n  - sg-uat2\nsubnet_sg_group: front\ntags:\n- Key: Name\n  Value: hello-ec2-uat-1\n- Key: Environment\n  Value: uat\n```\n\n### 3. Create a config build.\n\nRun follow command will generate a build. \nYou must provide the conf file with ```--conf```, so that command tool knows which conf file to use.\n\n```\nagc build --conf conf_uat.yaml\n```\n\nIt will generate a new folder in ```_builds/{conf.name}/{conf.number}```.\n\nIf the content inside ```conf_uat.yaml``` is following:\n\n```yaml\nname: uat\nnetenv: uat   # will deploy to uat subnets\nnumber: 1\n```\n\nYou would have a folder ```_builds/uat/1``` with following layout:\n\n```\ncfn/             # all are from cfn/*.tpl\n\t0_userdata.sh\n\tec2.json\n\tmodule.yaml\ncreate_stack.sh  # compiled from _script/create_stack.sh.tpl\nkill_stack.sh    # compiled from _script/kill_stack.sh.tpl\n```\n\n### 4. filters\n\n[agile_conf](https://github.com/tly1980/agile_conf) built-in jinja2 filters.\n\n\nHere is the example of ```aws_userdata``` filter from the ```single_ec2``` boilplate project.\n\n```bash\necho \"hello world\"\necho \"This is [{{conf.name}}-{{conf.number}}] for project: {{project.product_fullname}}\"\n```\n\nIt would be rendered into:\n```bash\necho \"hello world\"\necho \"This is [uat-1] for project: hello-ec2-uat-1\"\n```\n\nIn ```ec2.json.tpl``` we have a following code. \n\n```\n\"UserData\": {{ [_BUILD_DST_FOLDER, \"0_userdata.sh\"] |aws_userdata }},\n```\n\nIt is using a ```aws_userdata``` filter to turn ```0_userdata.sh``` into following code.\n\n```_BUILD_DST_FOLDER``` is the output destination folder of the module, exactly where the ```0_userdata.sh``` located.\n\nAnd you can see the shell script is rendred into cloudformation json structure:\n\n```\n\"UserData\": {\n    \"Fn::Base64\": {\n        \"Fn::Join\": [\n            \"\",\n            [\n                \"echo \\\"hello world\\\"\\n\",\n                \"echo \\\"This is [uat-1] for project: hello-ec2-uat-1\\\"\\n\"\n            ]\n        ]\n    }\n},\n```\n\nAnother filter is ```jsonify```.\n\nIn ```cfn/module.yaml```, tags are defined in following value:\n\n```yaml\ntags:\n  - Key: Name\n    Value: {{ project.product_fullname }}\n  - Key: Environment\n    Value: {{ conf.netenv }}\n```\n\nIn ```cfn/ec2.json.tpl```, it is how ```tags``` being used:\n\n```\n\"Tags\": {{ tags|jsonify }}\n```\n\nIt would be rendered into following:\n\n```\n\"Tags\": [\n\t\t{\"Key\": \"Name\", \"Value\": \"hello-ec2-uat-1\"},\n\t\t{\"Key\": \"Environment\", \"Value\": \"uat\"}\n\t]\n```\n\n### Commands\n\n**Command: build**\n\nCompile the variables into \n\n\n```\nagc build --conf conf_xxx.yaml\n```\n\n**Command: inspect**\n\nPrint out all the variables, would be very useful for debugging\n\n```\nagc inspect --conf conf_xxx.yaml\n```\n\n**Command: inspect**\n\n\n```\nagc inspect --conf conf_xxx.yaml\n```\n\n**Shortcut: using**\n\nIf you put following shell script in your BASH rc file,  \n\n```bash\nusing() {\n    envcmd=\"env AGC_CONF=conf_$1.yaml\"\n    shift\n    actual_cmd=\"$@\"\n    $envcmd $actual_cmd\n}\n```\n\nyou will have a very convinient short cut to switch different conf_xxx.yaml.\n\n\n```\nusing uat agc inspect\n```\n\n```\nusing uat agc build\n```\n\nIt is particular useful to do it with Makefile.\n\nSupposed you have following a Makefile.\n\n```Makefile\nbuild_uat:\n\tagc build --conf conf_uat.yaml\n\nbuild_prod:\n\tagc build --conf conf_prod.yaml\n\nbuild_perf:\n\tagc build --conf conf_prod.yaml\n\n```\n\nWith shortcut ```using```, you could have a Makefile like following:\n\n```Makefile\nbuild_uat:\n\tagc build\n```\n\nSo you can switch between different conf_xxx.yaml by:\n\n\n1. ```using uat make build```\n2. ```using prod make build```\n3. ```using perf make build```\n\n**PS**: ```using``` can work with all the command with ```--conf``` options.\n\n**Command: create**\n\nTo create a project from boilerplate repository.\n\n```\nagc create ${bo_name} ${project}\n```\n\nBefore you run this command, you should set enviornment variable ```AGC_BOIL```,  \nor use it with ```--bo_repo``` with it.\n\n> --bo_repo or AGC_BOIL can only be set to point to a local path. \n> You cannot put it GIT/HTTP URL to it, yet ... :)\n\n\n**Command: id**\n\n```agc id --conf conf_uat.yaml``` or ```using uat agc id```\n\nWill output:\n\n```{conf_name}/{conf_number}```\n\n**Command: where**\n```agc id --conf conf_uat.yaml``` or ```using uat agc where```\n\nWill output the exact location where the build is gonna to be.\n\ne.g.:\n```\n$ using uat agc where\n/Users/minddriven/workspace/play_agile_conf/my_ec2/_builds/uat/1\n```",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/tly1980/agile_conf",
        "keywords": null,
        "license": "BSD",
        "maintainer": null,
        "maintainer_email": null,
        "name": "agile_conf",
        "package_url": "https://pypi.org/project/agile_conf/",
        "platform": "any",
        "project_url": "https://pypi.org/project/agile_conf/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "https://github.com/tly1980/agile_conf"
        },
        "release_url": "https://pypi.org/project/agile_conf/1.0.3/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "A config files (in [YAML](http://yaml.org) format) and template engine ([Jinja2](http://jinja.pocoo.org)) based configuration compile / management tool to make DevOp tasks",
        "version": "1.0.3"
    },
    "last_serial": 1453481,
    "releases": {
        "1.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2440ec1d04336bc97a79b69f22c121b1",
                    "sha256": "0b11c0f44ccd9e6c51e24ed1eaf130f481c7f640404441d186c9eae03412cd7d"
                },
                "downloads": -1,
                "filename": "agile_conf-1.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "2440ec1d04336bc97a79b69f22c121b1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9745,
                "upload_time": "2015-03-08T07:27:07",
                "url": "https://files.pythonhosted.org/packages/a4/0d/ef87f2195d42dc58219257d2887645761671af7147633ad2070e59d1c634/agile_conf-1.0.2.tar.gz"
            }
        ],
        "1.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3d61c853adf438906a7b9353cf12fbcc",
                    "sha256": "04cca977c1a265d54eb6beb94d95d9bd73be7f74524e594ced817f8ac6ae1abd"
                },
                "downloads": -1,
                "filename": "agile_conf-1.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "3d61c853adf438906a7b9353cf12fbcc",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 14653,
                "upload_time": "2015-03-09T00:09:56",
                "url": "https://files.pythonhosted.org/packages/8e/d5/b081c7b592268939c4af838898416100648c9a2d0e5474ed5912a2682a13/agile_conf-1.0.3.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "3d61c853adf438906a7b9353cf12fbcc",
                "sha256": "04cca977c1a265d54eb6beb94d95d9bd73be7f74524e594ced817f8ac6ae1abd"
            },
            "downloads": -1,
            "filename": "agile_conf-1.0.3.tar.gz",
            "has_sig": false,
            "md5_digest": "3d61c853adf438906a7b9353cf12fbcc",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 14653,
            "upload_time": "2015-03-09T00:09:56",
            "url": "https://files.pythonhosted.org/packages/8e/d5/b081c7b592268939c4af838898416100648c9a2d0e5474ed5912a2682a13/agile_conf-1.0.3.tar.gz"
        }
    ]
}