{ "info": { "author": "Nick Rushton", "author_email": "nick@tune.com", "bugtrack_url": null, "classifiers": [ "License :: OSI Approved :: MIT License", "Programming Language :: Python", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy" ], "description": "# EMR Launcher\n\nLaunches EMR clusters using config files for consistent run-time behavior when setting up a cluster.\n\n## Installing\n\n```\npip install emr_launcher\n```\n\n## Usage\n\nStarting a new cluster:\n```\nemr_launcher launch /path/to/config/.json\n```\n\nAdding steps to an existing cluster\n```\nemr_launcher launch /path/to/config/.json --job-id \n```\n\n## Creating configs\n\nthe json file maps directly to boto3's `run_job_flow` function found [here](http://boto3.readthedocs.io/en/latest/reference/services/emr.html#EMR.Client.run_job_flow), you can use the documentation as a guide to build your config or build off the [Example Config](https://github.com/tuneinc/emr_launcher/blob/master/example_config.json)\n\n## Template functions\n\nemr_launcher uses templating within the json configuration to call useful functions, for example having an anonymous output location:\n\n```\n...\n\"--conf\", \"spark.output=s3://mybucket/output/{{ emr_launcher.uuid() }}/\n...\n```\n\na full set of usable template functions can be found by running:\n\n```\nemr_launcher list-template-functions\n\nemr_launcher\n============\n emr_launcher.get_environ\n Return the environment variables dictionary,\n Example: {{ get_environ()['USER'] }}\n A parent python program can use \"os.environ[key] = value\" before calling the emr launcher.\n\n emr_launcher.get_relative_date\n Returns a formatted datetime string,\n relative to the current time,\n as ajusted by the timedelta arguments.\n Example:\n {{ emr_launcher.get_relative_date(format='%Y-%m-01 00:00:00', timedelta_args=dict(days=-2)) }}\n\n emr_launcher.millis_to_iso\n converts a given milliseconds since epoch into an iso date string\n Args:\n ms_epoch - int\n Return\n string - formatted date string\n\n emr_launcher.uuid\n returns a UUID4 hex string\n```\n\n## Plugins\n\nPlugins are discovered by the naming convention `emr_launcher_` (ex: `emr_launcher_consul`). To install a plugin simply run:\n```\npip install emr_launcher_\n```\n\nAvailable plugins:\n\n[emr_launcher_aws](https://github.com/tuneinc/emr_launcher_aws)\n\n[emr_launcher_consul](https://github.com/tuneinc/emr_launcher_consul)\n\n[emr_launcher_artifactory](https://github.com/tuneinc/emr_launcher_artifactory)\n\n\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/tuneinc/emr_launcher", "keywords": "", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "emr-launcher", "package_url": "https://pypi.org/project/emr-launcher/", "platform": "", "project_url": "https://pypi.org/project/emr-launcher/", "project_urls": { "Homepage": "https://github.com/tuneinc/emr_launcher" }, "release_url": "https://pypi.org/project/emr-launcher/1.1.1/", "requires_dist": [ "blessings (==1.6.1)", "boto3 (==1.4.4)", "jinja2", "pytz (==2017.3)" ], "requires_python": ">=2.7.0", "summary": "Launches an AWS Elastic MapReduce cluster using templated configuration files written in JSON. Meant to make deployments consistent and reproducable.", "version": "1.1.1" }, "last_serial": 4079770, "releases": { "1.0.0": [ { "comment_text": "", "digests": { "md5": "c3620ce0477454c59f8d98b06c17bf21", "sha256": "907b60159aa730fd6926c9628a5163b99dee56ee590db630399a42ca987449d6" }, "downloads": -1, "filename": "emr_launcher-1.0.0-py2.7.egg", "has_sig": false, "md5_digest": "c3620ce0477454c59f8d98b06c17bf21", "packagetype": "bdist_egg", "python_version": "2.7", "requires_python": ">=2.7.0", "size": 10124, "upload_time": "2018-06-11T20:26:00", "url": "https://files.pythonhosted.org/packages/74/01/5bab43806873e968ec532527adfd825af6d4fd28953f55b939a3465331ad/emr_launcher-1.0.0-py2.7.egg" }, { "comment_text": "", "digests": { "md5": "338cb2223c5ee2425189bf1a2eecc733", "sha256": "acf305651eac5243105ec5fc9df717dcdcfcf8b4571c881a027b339427d813ab" }, "downloads": -1, "filename": "emr_launcher-1.0.0-py2-none-any.whl", "has_sig": false, "md5_digest": "338cb2223c5ee2425189bf1a2eecc733", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": ">=2.7.0", "size": 7441, "upload_time": "2018-06-11T20:25:59", "url": "https://files.pythonhosted.org/packages/87/db/509d6c56970d7b1b6f8bad5d6f8e3520248a0044c44f967e72f5911b06f5/emr_launcher-1.0.0-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "3d0e501c2ffe005c6d3734ec64cb6e86", "sha256": "1f5cc98bfdb9ff33d1d1c7ff7c86d29c89d9e8861b3e2ad615f2a8020f4f9e5e" }, "downloads": -1, "filename": "emr_launcher-1.0.0.tar.gz", "has_sig": false, "md5_digest": "3d0e501c2ffe005c6d3734ec64cb6e86", "packagetype": "sdist", "python_version": "source", "requires_python": ">=2.7.0", "size": 5681, "upload_time": "2018-06-11T20:26:01", "url": "https://files.pythonhosted.org/packages/df/6e/8b88fcb2c8b862aab9411b031fadec43bcbf3195beb06f0ba2aee54f2fe7/emr_launcher-1.0.0.tar.gz" } ], "1.1.0": [ { "comment_text": "", "digests": { "md5": "f4f22f4963a688b4ff2e2f711c26f03e", "sha256": "efa2f318eabd84800d9cedc99ce22783fa50349fc6af3c63fe2e59bd10f235a4" }, "downloads": -1, "filename": "emr_launcher-1.1.0-py2-none-any.whl", "has_sig": false, "md5_digest": "f4f22f4963a688b4ff2e2f711c26f03e", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": ">=2.7.0", "size": 8506, "upload_time": "2018-06-11T22:35:42", "url": "https://files.pythonhosted.org/packages/95/ed/4b875c64aa0573c6a394c72dbb9c6b8c541103988eb3eead9cb103a572b4/emr_launcher-1.1.0-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c7fabe83460c46ddbd1cc8e01f1ee15c", "sha256": "e7c05d754a8ffdf39a8f68665fc1caa98883011eae1aa9fd17a978ac25bc4cda" }, "downloads": -1, "filename": "emr_launcher-1.1.0.tar.gz", "has_sig": false, "md5_digest": "c7fabe83460c46ddbd1cc8e01f1ee15c", "packagetype": "sdist", "python_version": "source", "requires_python": ">=2.7.0", "size": 5953, "upload_time": "2018-06-11T22:35:43", "url": "https://files.pythonhosted.org/packages/00/3b/44c6c28a36cde2bbcbd2432a3cb654bcc3e2ec3ba7ae033f5db55f03377f/emr_launcher-1.1.0.tar.gz" } ], "1.1.1": [ { "comment_text": "", "digests": { "md5": "dd69899c607eb8d016a1f376e92e133c", "sha256": "6a6ac1ee4b3971058cf0ef7dc06c503f5f48b4fc40bbe5677168211ec2f4dc8e" }, "downloads": -1, "filename": "emr_launcher-1.1.1-py2-none-any.whl", "has_sig": false, "md5_digest": "dd69899c607eb8d016a1f376e92e133c", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": ">=2.7.0", "size": 9587, "upload_time": "2018-06-11T23:24:28", "url": "https://files.pythonhosted.org/packages/37/e4/21f306bf7ee827d17d553ca39f54d54be73ee88da0a53b5121507cb1361a/emr_launcher-1.1.1-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ebc5f6cc975381daf8d04f65c3dfce40", "sha256": "72ea2040b3da787a790bde60fbc169be99fffb9b7f64db44038e971e9514d9ee" }, "downloads": -1, "filename": "emr_launcher-1.1.1.tar.gz", "has_sig": false, "md5_digest": "ebc5f6cc975381daf8d04f65c3dfce40", "packagetype": "sdist", "python_version": "source", "requires_python": ">=2.7.0", "size": 6447, "upload_time": "2018-06-11T23:24:29", "url": "https://files.pythonhosted.org/packages/7b/0f/f13862da2b0c2e2d7a7de3d2ddd3bb51f47018ddea6cf1a54835a8fe9955/emr_launcher-1.1.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "dd69899c607eb8d016a1f376e92e133c", "sha256": "6a6ac1ee4b3971058cf0ef7dc06c503f5f48b4fc40bbe5677168211ec2f4dc8e" }, "downloads": -1, "filename": "emr_launcher-1.1.1-py2-none-any.whl", "has_sig": false, "md5_digest": "dd69899c607eb8d016a1f376e92e133c", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": ">=2.7.0", "size": 9587, "upload_time": "2018-06-11T23:24:28", "url": "https://files.pythonhosted.org/packages/37/e4/21f306bf7ee827d17d553ca39f54d54be73ee88da0a53b5121507cb1361a/emr_launcher-1.1.1-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ebc5f6cc975381daf8d04f65c3dfce40", "sha256": "72ea2040b3da787a790bde60fbc169be99fffb9b7f64db44038e971e9514d9ee" }, "downloads": -1, "filename": "emr_launcher-1.1.1.tar.gz", "has_sig": false, "md5_digest": "ebc5f6cc975381daf8d04f65c3dfce40", "packagetype": "sdist", "python_version": "source", "requires_python": ">=2.7.0", "size": 6447, "upload_time": "2018-06-11T23:24:29", "url": "https://files.pythonhosted.org/packages/7b/0f/f13862da2b0c2e2d7a7de3d2ddd3bb51f47018ddea6cf1a54835a8fe9955/emr_launcher-1.1.1.tar.gz" } ] }