{
"info": {
"author": "amancevice",
"author_email": "smallweirdnum@gmail.com",
"bugtrack_url": null,
"classifiers": [
"Development Status :: 3 - Alpha",
"Intended Audience :: Developers",
"Intended Audience :: System Administrators",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 2.7",
"Topic :: Utilities"
],
"description": "# pline Python library\n\n![](\"https://travis-ci.org/amancevice/pline.svg?branch=master\"/)
\n\nAWS Data Pipeline Wrapper for `boto3`. Construct a Data Pipeline using Python objects.\n\nLast updated: `0.4.2`\n\n## Installation\n\n```\npip install pline\n```\n\n## Overview\n\nThe payload `boto3` requires for a pipeline definition is somewhat complex. This library \nprovides the tools to model your pipeline using Python objects and transform the payload\ninto the expected data structure.\n\n```python\nimport pline\n\nmy_activity = pline.activities.ShellCommandActivity(\n name='MyActivity', id='Activity_adbc1234')\nmy_activity.command = \"echo $1 $2\"\nmy_activity.scriptArgument = ['hello', 'world']\n\ndict(my_activity)\n{ 'id' : 'Activity_adbc1234',\n 'name' : 'MyActivity',\n 'fields' : [ {'key': 'command', 'stringValue': 'echo $1 $2'},\n {'key': 'type', 'stringValue': 'ShellCommandActivity'},\n {'key': 'scriptArgument', 'stringValue': 'hello'},\n {'key': 'scriptArgument', 'stringValue': 'world'} ]}\n ```\n\n#### Data Pipeline Objects\n\nEvery object in a pipeline is an acestor of the `DataPipelineObject` class. Each object \nowns three key attributes:\n\n* `name`\n* `id`\n* `fields`\n\nThe `name` and `id` attributes must be set at initialization time, but `fields` is \nhandled internally by the object and should not be accessed directly.\n\nSetting an object's attribute can be done via the initialization call or after the fact:\n\n```python\nnode = pline.data_nodes.S3DataNode(\n id='MyDataNode1', name='MyDataNode1', workerGroup='TestGroup')\n# => \nnode.directoryPath = 's3://bucket/pipeline/'\nprint node.workerGroup\n# => 'TestGroup'\nprint node.directoryPath\n# => 's3://bucket/pipeline/'\n```\n\n`Pipeline` instances handle the conversion of pipeline objects to a payload, but objects can\nbe viewed in `boto`-friendly format by converting them to a `dict`:\n\n```python\ndict(node)\n{ 'name' : 'MyDataNode1',\n 'id' : 'MyDataNode1',\n 'fields' : [\n { 'key' : 'type', 'stringValue' : 'S3DataNode' },\n { 'key' : 'directoryPath', 'stringValue' : 's3://bucket/pipeline/' },\n { 'key' : 'workerGroup', 'stringValue' : 'TestGroup' }, ] }\n```\n\n#### Data Pipeline Parameters\n\nAs of `0.2.0`, `pline` supports passing parameters to data pipelines. Parameters can be added to the \npipeline and passed into `DataPipelineObject` instances.\n\n```python\nmy_param = pline.parameters.String(\n id = 'MyParam1',\n value = 'Here is the value I am using',\n description = 'This value is extremely important',\n watermark = 'Choose a value between 0 and 99.')\n```\n\n#### Typed Data Pipeline Objects/Parameters\n\nMost objects in a data pipeline are typed -- that is, they are given a `type` attribute on initialization\nthat is added to the `fields` attribute. By default, the type is taken from the name of the class (which\ncorresponds to the type given by AWS' specs).\n\nCustom classes can override this behavior by defining a `TYPE_NAME` class-level attribute:\n\n```python\nclass MyCustomS3DataNode(pline.S3DataNode):\n TYPE_NAME = 'S3DataNode'\n # ...\n\nclass MyCustomParam(pline.AwsS3ObjectKey):\n TYPE_NAME = 'AwsS3ObjectKey'\n # ...\n```\n\n\n## Example Pipeline\n\n#### Create a pipeline object\n\n```python\npipeline = pline.Pipeline(\n name = 'MyPipeline',\n unique_id = 'MyPipeline1',\n desc = 'An example pipeline description',\n region = 'us-west-2' )\n```\n\n#### Connect (optional)\n\nThe pipeline will connect to AWS automatically if you have your AWS credentials set at\nthe environmental level. If you want to connect using a specific configuration:\n\n```python\npipeline.connect(\n aws_access_key_id = 'my_access_key',\n aws_secret_access_key = 'my_secret_key' )\n```\n\n#### Create a schedule object\n\n```python\nschedule = pline.Schedule(\n id = 'Schedule1',\n name = 'Schedule',\n period = '1 day',\n startAt = pline.keywords.startAt.FIRST_ACTIVATION_DATE_TIME,\n occurrences = 1 )\n```\n\n#### Create the default pipeline definition \n\nThe pipeline object has a helper-method to create this object with sensible defaults:\n\n```python\ndefinition = pipeline.definition( schedule,\n pipelineLogUri = \"s3://bucket/pipeline/log\" )\n```\n\n#### Create an EC2 resource\n\nThis will be the machine running the tasks.\n\n```python\nresource = pline.resources.Ec2Resource(\n id = 'Resource1',\n name = 'Resource',\n role = 'DataPipelineDefaultRole',\n resourceRole = 'DataPipelineDefaultResourceRole',\n schedule = schedule )\n```\n\n#### Create an activity\n\n```python\nactivity = pline.activities.ShellCommandActivity(\n id = 'MyActivity1',\n name = 'MyActivity',\n runsOn = resource,\n schedule = schedule,\n command = 'echo hello world' )\n```\n\n\n#### Create a parameterized activity and its parameter\n\n```python\nparam = pline.parameters.String(\n id = 'myShellCmd',\n value = 'grep -rc \"GET\" ${INPUT1_STAGING_DIR}/* > ${OUTPUT1_STAGING_DIR}/output.txt',\n description = 'Shell command to run' )\n\nparam_activity = pline.activities.ShellCommandActivity(\n id = 'MyParamActivity1',\n name = 'MyParamActivity1',\n runsOn = resource,\n schedule = schedule,\n command = param )\n```\n\n#### Add the objects to the pipeline\n\n```python\npipeline.add(schedule, definition, resource, activity, param_activity)\n```\n\n#### Add the parameters to the pipeline\n\n```python\npipeline.add_param(param)\n```\n\n#### View the pipeline definition payload\n\n```python\nprint pipeline.payload()\n```\n\n#### Validate the pipeline definiton\n\n```python\npipeline.validate()\n```\n\n#### Create the pipeline in AWS\n\nThis will send the request to create a pipeline through boto\n\n```python\npipeline.create()\n```\n\n#### Adding new objects to the pipeline\n\nSometimes you may want to add an object to the pipeline after it has been created\n\n```python\n# Add an alert\nsns_alarm = pline.actions.SnsAlarm(\n name = 'SnsAlarm',\n id = 'SnsAlarm1',\n topicArn = 'arn:aws:sns:us-east-1:12345678abcd:my-arn',\n role = 'DataPipelineDefaultRole' )\n\n# Associate it with the activity\nactivity.onFailure = sns_alarm\n\n# Add it to the pipeline\npipeline.add(sns_alarm)\n```\n\nUpdate the pipeline on AWS and activate it\n\n```python\npipeline.update()\npipeline.activate()\n```\n\n## ShellCommand helper\n\nThe `ShellCommand` class can be used to compose chained commands\n\n```python\ncmd = pline.utils.ShellCommand(\n 'docker start registry',\n 'sleep 3',\n 'docker pull localhost:5000/my_docker',\n 'docker stop registry' )\n# => docker start registry;\\\n# sleep 3;\\\n# docker pull localhost:5000/my_docker;\\\n# docker stop registry\n\ncmd.append('echo all done')\n# => docker start registry;\\\n# sleep 3;\\\n# docker pull localhost:5000/my_docker;\\\n# docker stop registry;\\\n# echo all done\n\nactivity.command = cmd\n```",
"description_content_type": null,
"docs_url": null,
"download_url": "",
"downloads": {
"last_day": -1,
"last_month": -1,
"last_week": -1
},
"home_page": "http://www.smallweirdnumber.com",
"keywords": "",
"license": "",
"maintainer": "",
"maintainer_email": "",
"name": "pline",
"package_url": "https://pypi.org/project/pline/",
"platform": "UNKNOWN",
"project_url": "https://pypi.org/project/pline/",
"project_urls": {
"Homepage": "http://www.smallweirdnumber.com"
},
"release_url": "https://pypi.org/project/pline/0.4.4/",
"requires_dist": null,
"requires_python": "",
"summary": "AWS Data Pipeline Wrapper",
"version": "0.4.4"
},
"last_serial": 2123871,
"releases": {
"0.0.1": [],
"0.0.2": [
{
"comment_text": "",
"digests": {
"md5": "c5e5df4dbb79654f6457bb61b1e41629",
"sha256": "72d21b1633da3d3711dd50961ad668a5986aa6fda005382655c482d9c732023f"
},
"downloads": -1,
"filename": "pline-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "c5e5df4dbb79654f6457bb61b1e41629",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 3146,
"upload_time": "2015-05-27T19:40:33",
"url": "https://files.pythonhosted.org/packages/8f/60/4e7aaf704ab47841421684e7916565029892c07307b4f14ea48d5623bb3a/pline-0.0.2.tar.gz"
}
],
"0.0.3": [
{
"comment_text": "",
"digests": {
"md5": "1daa6ba9603b47b5bf2c538fdc2a23f9",
"sha256": "d1070af37504d745a06b498bc007af55979f33714e6616b7f9968d7d277f7942"
},
"downloads": -1,
"filename": "pline-0.0.3.tar.gz",
"has_sig": false,
"md5_digest": "1daa6ba9603b47b5bf2c538fdc2a23f9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 5038,
"upload_time": "2015-05-28T17:58:53",
"url": "https://files.pythonhosted.org/packages/dd/9b/4c62d10eb12031cfc6960a99643d18422e15f5280d06b8dce550d91f1356/pline-0.0.3.tar.gz"
}
],
"0.0.4": [
{
"comment_text": "",
"digests": {
"md5": "e5c496e352b6dadfddfd46f96dc53ea3",
"sha256": "56fa0b9e2e00abbdf49c0883c31f533184e990913bf186becb97caa2c17b1f81"
},
"downloads": -1,
"filename": "pline-0.0.4.tar.gz",
"has_sig": false,
"md5_digest": "e5c496e352b6dadfddfd46f96dc53ea3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 5627,
"upload_time": "2015-05-28T18:35:10",
"url": "https://files.pythonhosted.org/packages/5d/a0/5a2e24adfd2da6fa7a6b17bd6bd8387091ff855bf57053ecefa26ea08bf5/pline-0.0.4.tar.gz"
}
],
"0.0.5": [
{
"comment_text": "",
"digests": {
"md5": "311be348c02406dd737a3c8fdd5c4f36",
"sha256": "61608deb1a82d1fae76a928a3ca3dbfb542e924b9f5966712f24020f2f146003"
},
"downloads": -1,
"filename": "pline-0.0.5.tar.gz",
"has_sig": false,
"md5_digest": "311be348c02406dd737a3c8fdd5c4f36",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 5647,
"upload_time": "2015-05-28T18:54:05",
"url": "https://files.pythonhosted.org/packages/e3/82/7e823878eb2e0d17fe26d120913d9f5e11d9c25b7f6fe20946cb2980edc4/pline-0.0.5.tar.gz"
}
],
"0.1.0": [
{
"comment_text": "",
"digests": {
"md5": "28e2e30201d65a00fb1b4ae5d811a350",
"sha256": "99d953f482fd11a8f9fe9c3d526041aac27d7f1bfe5e261f283d68d4036c04c7"
},
"downloads": -1,
"filename": "pline-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "28e2e30201d65a00fb1b4ae5d811a350",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6215,
"upload_time": "2015-05-29T14:58:16",
"url": "https://files.pythonhosted.org/packages/12/ab/b66091f47ab576ee03bc4561fa3b2d84fde01a0eb78dc04588662c07664f/pline-0.1.0.tar.gz"
}
],
"0.1.1": [
{
"comment_text": "",
"digests": {
"md5": "7423e5de37672e5d938e014f245219bc",
"sha256": "8d4390cc331e3d7b469978c166b66ad8d73184d61fd6da89a7f1cc088dbafa55"
},
"downloads": -1,
"filename": "pline-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "7423e5de37672e5d938e014f245219bc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6288,
"upload_time": "2015-05-29T16:17:06",
"url": "https://files.pythonhosted.org/packages/41/53/fc8e1c23e8aa0cc1d94e54e97a9cfdfd7dc50512b24979ebf07391e4bb41/pline-0.1.1.tar.gz"
}
],
"0.1.3": [
{
"comment_text": "",
"digests": {
"md5": "e6ab7433237aa4a9b565aaf6d34865a2",
"sha256": "055894382d6a99d42384fa6e0b715837ed9760894809d4282392eb5a9037c3ea"
},
"downloads": -1,
"filename": "pline-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "e6ab7433237aa4a9b565aaf6d34865a2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6298,
"upload_time": "2015-06-02T16:14:49",
"url": "https://files.pythonhosted.org/packages/c3/97/6378341494cc181a430ce3932c06834a60dd2b407fb0f3d32ab67737dc52/pline-0.1.3.tar.gz"
}
],
"0.1.4": [
{
"comment_text": "",
"digests": {
"md5": "577b13c31aa79e9f8a68242e070cade6",
"sha256": "ce1627b688f5d255d58461c08e4136b10e2e8668ca63655c7b5d333a50eb4e96"
},
"downloads": -1,
"filename": "pline-0.1.4.tar.gz",
"has_sig": false,
"md5_digest": "577b13c31aa79e9f8a68242e070cade6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6313,
"upload_time": "2015-06-10T15:01:05",
"url": "https://files.pythonhosted.org/packages/5d/54/591304bf45dc5917b57092fb37ee1042345f3a817c506c6d8609737f7819/pline-0.1.4.tar.gz"
}
],
"0.1.5": [
{
"comment_text": "",
"digests": {
"md5": "e0714975c7646a357d8682e133580ab8",
"sha256": "3db739515dcd9c6295ccd2a99c9c912e66b925c813afc24ac2f18ee470086029"
},
"downloads": -1,
"filename": "pline-0.1.5.tar.gz",
"has_sig": false,
"md5_digest": "e0714975c7646a357d8682e133580ab8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6326,
"upload_time": "2015-06-10T18:56:24",
"url": "https://files.pythonhosted.org/packages/7e/a3/eee76612fea4ff625a26228701a3d4f987477ae7b741a5862790d46148cb/pline-0.1.5.tar.gz"
}
],
"0.2.0": [
{
"comment_text": "",
"digests": {
"md5": "a5fe25595eb956ff869dd34249bf0ebd",
"sha256": "c770cdb818b5f3635bfd36d49f149937c3ce14d40195fa2f0156b62b8d0e0dab"
},
"downloads": -1,
"filename": "pline-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "a5fe25595eb956ff869dd34249bf0ebd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10147,
"upload_time": "2015-08-11T16:50:52",
"url": "https://files.pythonhosted.org/packages/2d/c6/558e9766ed101df3e4f1cdc52b1caf95b6e942709f948ded1c5ae95e8a60/pline-0.2.0.tar.gz"
}
],
"0.2.1": [
{
"comment_text": "",
"digests": {
"md5": "64833c89eec2e4c4c1b99dbe55883a8f",
"sha256": "40bc92bd57a83428b88ce4d2701a3041cd3f03a22779df72197c37c61827a710"
},
"downloads": -1,
"filename": "pline-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "64833c89eec2e4c4c1b99dbe55883a8f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10154,
"upload_time": "2015-08-20T14:27:15",
"url": "https://files.pythonhosted.org/packages/31/34/ac36841baf873049db74e17c17819ca3ced64bea8fce9c9208e2b4e04773/pline-0.2.1.tar.gz"
}
],
"0.2.2": [
{
"comment_text": "",
"digests": {
"md5": "9b015e70cfa9d973b4b1d4e5ded57f8f",
"sha256": "b1aa41c6e1838577a01df4f4a3ab027964d2ddda26ecdeaceb4a487838466300"
},
"downloads": -1,
"filename": "pline-0.2.2.tar.gz",
"has_sig": false,
"md5_digest": "9b015e70cfa9d973b4b1d4e5ded57f8f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10216,
"upload_time": "2015-08-20T15:32:25",
"url": "https://files.pythonhosted.org/packages/32/4f/7932323a1c89437aec715554bd3dc21e9024f325cd03e320e906c7202a5a/pline-0.2.2.tar.gz"
}
],
"0.3.0": [
{
"comment_text": "",
"digests": {
"md5": "d423e1a99e9cf3a6d4c74d735c7a4095",
"sha256": "dff868b3237d174626710ded816b3f0414ed6db3d11d99e635626ec3c569bdd0"
},
"downloads": -1,
"filename": "pline-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "d423e1a99e9cf3a6d4c74d735c7a4095",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10348,
"upload_time": "2015-08-31T20:06:39",
"url": "https://files.pythonhosted.org/packages/b8/bb/79748b435925f1796c3ecee7bd241fcd7ec86e177bce944f56dbc323da75/pline-0.3.0.tar.gz"
}
],
"0.4.0": [
{
"comment_text": "",
"digests": {
"md5": "63e4e759f2ee54f12488fc4c34d38001",
"sha256": "aeb7c54a5a67b78444c971e59ef6d12744074bded73b2045729aa4f48dce736e"
},
"downloads": -1,
"filename": "pline-0.4.0.tar.gz",
"has_sig": false,
"md5_digest": "63e4e759f2ee54f12488fc4c34d38001",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10273,
"upload_time": "2016-02-18T19:35:32",
"url": "https://files.pythonhosted.org/packages/c5/cd/778c4810968134f25941aa6b6ef571a86bde4b69e09a6a6b53f29978ace1/pline-0.4.0.tar.gz"
}
],
"0.4.1": [
{
"comment_text": "",
"digests": {
"md5": "fd7536eb432e64b2631df700ff8b631e",
"sha256": "074e0286334640f860bcffb9e617dbd8b15a81484155d56947e80499303d5fd1"
},
"downloads": -1,
"filename": "pline-0.4.1.tar.gz",
"has_sig": false,
"md5_digest": "fd7536eb432e64b2631df700ff8b631e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10382,
"upload_time": "2016-03-02T19:55:02",
"url": "https://files.pythonhosted.org/packages/4b/99/e5a73c1262b138690242dce7a89f79fa6958d3b46bc131f0396ca6a032e4/pline-0.4.1.tar.gz"
}
],
"0.4.2": [
{
"comment_text": "",
"digests": {
"md5": "4d08a30754ec72a0fa49501b8e6ba55d",
"sha256": "1d92737d4de6365843c50209353bdd734dee1961ad0df39eecfc112371044d12"
},
"downloads": -1,
"filename": "pline-0.4.2.tar.gz",
"has_sig": false,
"md5_digest": "4d08a30754ec72a0fa49501b8e6ba55d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 7947,
"upload_time": "2016-03-15T19:41:09",
"url": "https://files.pythonhosted.org/packages/d4/74/11c11804431f5c0ffd038bb1070b442bbf84c84175f63fac826e44d667c6/pline-0.4.2.tar.gz"
}
],
"0.4.3": [
{
"comment_text": "",
"digests": {
"md5": "3b3973a4e38b0d79779290d85bfeff33",
"sha256": "71f6e6d4ddb54fb4a8dea29ca59fe720b2367278b25f8785d023ebfd41d4c7da"
},
"downloads": -1,
"filename": "pline-0.4.3.tar.gz",
"has_sig": false,
"md5_digest": "3b3973a4e38b0d79779290d85bfeff33",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 8022,
"upload_time": "2016-05-18T16:45:07",
"url": "https://files.pythonhosted.org/packages/37/11/3cbb4652de7939ba861c647c9dd764268a1b4285b8566e3793d4f364d59b/pline-0.4.3.tar.gz"
}
],
"0.4.4": [
{
"comment_text": "",
"digests": {
"md5": "ffff4307da52441ff4a3fec6a7bdb71f",
"sha256": "a5ddc68dd849e2effaab16aceb6346f9dd2031dbae1d16cd8f86ce6765a5e212"
},
"downloads": -1,
"filename": "pline-0.4.4.tar.gz",
"has_sig": false,
"md5_digest": "ffff4307da52441ff4a3fec6a7bdb71f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 8025,
"upload_time": "2016-05-19T16:28:41",
"url": "https://files.pythonhosted.org/packages/20/a5/ad6d733405be2ebda4f4092cf6ee58eb26fcbdcdaa8afe6815f02760c1fc/pline-0.4.4.tar.gz"
}
]
},
"urls": [
{
"comment_text": "",
"digests": {
"md5": "ffff4307da52441ff4a3fec6a7bdb71f",
"sha256": "a5ddc68dd849e2effaab16aceb6346f9dd2031dbae1d16cd8f86ce6765a5e212"
},
"downloads": -1,
"filename": "pline-0.4.4.tar.gz",
"has_sig": false,
"md5_digest": "ffff4307da52441ff4a3fec6a7bdb71f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 8025,
"upload_time": "2016-05-19T16:28:41",
"url": "https://files.pythonhosted.org/packages/20/a5/ad6d733405be2ebda4f4092cf6ee58eb26fcbdcdaa8afe6815f02760c1fc/pline-0.4.4.tar.gz"
}
]
}