{ "info": { "author": "Brent Nash", "author_email": "brenash@amazon.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: Other/Proprietary License", "Natural Language :: English", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2.7", "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: Utilities" ], "description": "Python Kinesis Producer Library Deaggregation Module\r\n====================================================\r\n\r\nThe `Amazon Kinesis Producer Library\r\n(KPL) `__\r\ngives you the ability to write data to Amazon Kinesis with a highly\r\nefficient, asyncronous delivery model that can `improve\r\nperformance `__.\r\nWhen you write to the Producer, you can also elect to turn on\r\nAggregation, which writes multiple producer events to a single Kinesis\r\nRecord, aggregating lots of smaller events into a 1MB record. When you\r\nuse Aggregation, the KPL serialises data to the Kinesis stream using\r\n`Google Protocol\r\nBuffers `__, and\r\nconsumer applications must be able to deserialise this protobuf message.\r\nThis module gives you the ability to process KPL serialised data using\r\nany Python consumer including `AWS\r\nLambda `__.\r\n\r\nInstallation\r\n------------\r\n\r\nThe Python KPL Deaggregation module is available on the Python Package\r\nIndex (PyPI) as\r\n`aws\\_kpl\\_deagg `__. You\r\ncan install it via the ``pip`` command line tool:\r\n\r\n::\r\n\r\n pip install aws_kpl_deagg\r\n\r\nAlternately, you can simply copy the aws\\_kpl\\_deagg module from this\r\nrepository and use it directly with the caveat that the `Google protobuf\r\nmodule `__ must also be available\r\n(if you install via ``pip``, this dependency will be handled for you).\r\n\r\nUsage\r\n-----\r\n\r\nThe Python KPL Deaggregation module provides a simple interface for\r\nworking with KPL encoded data in a consumer application. The\r\naws\\_kpl\\_deagg Python module provides for both bulk and generator-based\r\nprocessing.\r\n\r\nWhen using deaggregation, you provide a Kinesis Record, and get back\r\nmultiple Kinesis User Records. If a Kinesis Record that is provided is\r\nnot a KPL encoded message, that's perfectly fine - you'll just get a\r\nsingle record output from the single record input. A Kinesis User Record\r\nwhich is returned from the kpl-deagg looks like:\r\n\r\n::\r\n\r\n {\r\n 'eventVersion' : String - The version number of the Kinesis event used\r\n 'eventID' : String - The unique ID of this Kinesis event\r\n 'kinesis' :\r\n {\r\n 'partitionKey' : String - The Partition Key provided when the record was submitted\r\n 'explicitHashKey' : String - The hash value used to explicitly determine the shard the data record is assigned to by overriding the partition key hash (or None if absent) \r\n 'data' : String - The original data transmitted by the producer (base64 encoded)\r\n 'kinesisSchemaVersion' : String - The version number of the Kinesis message schema used,\r\n 'sequenceNumber' : BigInt - The sequence number assigned to the record on submission to Kinesis\r\n 'subSequenceNumber' : Int - The sub-sequence number for the User Record in the aggregated record, if aggregation was in use by the producer\r\n 'aggregated' : Boolean - Always True for a user record extracted from a KPL aggregated record\r\n },\r\n 'invokeIdentityArn' : String - The ARN of the IAM user used to invoke this Lambda function\r\n 'eventName' : String - Always \"aws:kinesis:record\" for a Kinesis record\r\n 'eventSourceARN' : String - The ARN of the source Kinesis stream\r\n 'eventSource' : String - Always \"aws:kinesis\" for a Kinesis record\r\n 'awsRegion' : String - The name of the source region for the event (e.g. \"us-east-1\")\r\n }\r\n\r\nTo get started, include the ``aws_kpl_deagg`` module:\r\n\r\n``import aws_kpl_deagg``\r\n\r\nNext, when you receive a Kinesis Record in your consumer application,\r\nyou will extract the User Records using the deaggregations methods\r\navailable in the ``aws_kpl_deagg`` module.\r\n\r\n**IMPORTANT**: The deaggregation methods available in the\r\n``aws_kpl_deagg`` module expect input records in the same\r\ndictionary-based format they are normally received from AWS Lambda. See\r\nthe `Programming Model for Authoring Lambda Functions in\r\nPython `__\r\nsection of the AWS documentation for more details.\r\n\r\nBulk Conversion\r\n~~~~~~~~~~~~~~~\r\n\r\nThe bulk conversion method of deaggregation takes in a list of Kinesis\r\nRecords, extracts all the aggregated User Records and accumulates them\r\ninto a list. Any records that are passed in to this method that are not\r\nKPL-aggregated records will be returned unchanged. The method returns a\r\nlist of Kinesis User Records in the same format as they are normally\r\ndelivered by Lambda's Kinesis event handler.\r\n\r\n::\r\n\r\n user_records = deaggregate_records(raw_kinesis_records)\r\n\r\nGenerator-based Conversion\r\n~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n\r\nThe generator-based conversion method of deaggregation uses a Python\r\n`generator function `__ to\r\nextract User Records from a raw Kinesis Record one at a time in an\r\niterative fashion. Any records that are passed in to this method that\r\nare not KPL-aggregated records will be returned unchanged. For example,\r\nyou could use this code to iterate through each deaggregated record:\r\n\r\n::\r\n\r\n for record in iter_deaggregate_records(raw_kinesis_records): \r\n \r\n #Process each record\r\n pass \r\n\r\nExamples\r\n--------\r\n\r\nThis module includes two example AWS Lambda function in the file\r\n`lambda\\_function.py `__, which gives you the\r\nability to easily build new functions to process KPL encoded data.\r\n\r\nBulk Conversion Example\r\n~~~~~~~~~~~~~~~~~~~~~~~\r\n\r\n::\r\n\r\n from __future__ import print_function\r\n\r\n from aws_kpl_deagg.deaggregator import deaggregate_records\r\n import base64\r\n\r\n def lambda_bulk_handler(event, context):\r\n \r\n raw_kinesis_records = event['Records']\r\n \r\n #Deaggregate all records in one call\r\n user_records = deaggregate_records(raw_kinesis_records)\r\n \r\n #Iterate through deaggregated records\r\n for record in user_records: \r\n \r\n # Kinesis data in Python Lambdas is base64 encoded\r\n payload = base64.b64decode(record['kinesis']['data'])\r\n \r\n #TODO: Process each record\r\n \r\n return 'Successfully processed {} records.'.format(len(user_records))\r\n\r\nGenerator-based Conversion Example\r\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n\r\n::\r\n\r\n from __future__ import print_function\r\n\r\n from aws_kpl_deagg.deaggregator import iter_deaggregate_records\r\n import base64\r\n\r\n def lambda_generator_handler(event, context):\r\n \r\n raw_kinesis_records = event['Records']\r\n record_count = 0\r\n \r\n #Deaggregate all records using a generator function\r\n for record in iter_deaggregate_records(raw_kinesis_records): \r\n \r\n # Kinesis data in Python Lambdas is base64 encoded\r\n payload = base64.b64decode(record['kinesis']['data'])\r\n \r\n #TODO: Process each record\r\n \r\n record_count += 1\r\n \r\n return 'Successfully processed {} records.'.format(record_count)\n\r\n--------------\r\n\r\nCopyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights\r\nReserved.\r\n\r\nLicensed under the Amazon Software License (the \"License\"). You may not\r\nuse this file except in compliance with the License. A copy of the\r\nLicense is located at\r\n\r\n::\r\n\r\n http://aws.amazon.com/asl/\r\n\r\nor in the \"license\" file accompanying this file. This file is\r\ndistributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\r\nKIND, express or implied. See the License for the specific language\r\ngoverning permissions and limitations under the License.", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://github.com/awslabs/kinesis-deaggregation", "keywords": "aws,kinesis,deaggregation", "license": "SEE LICENSE IN LICENSE.TXT", "maintainer": null, "maintainer_email": null, "name": "aws_kpl_deagg", "package_url": "https://pypi.org/project/aws_kpl_deagg/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/aws_kpl_deagg/", "project_urls": { "Download": "UNKNOWN", "Homepage": "http://github.com/awslabs/kinesis-deaggregation" }, "release_url": "https://pypi.org/project/aws_kpl_deagg/1.0.10/", "requires_dist": null, "requires_python": null, "summary": "Python module to simplify processing of Amazon Kinesis Records which have been created with the Kinesis Producer Library.", "version": "1.0.10" }, "last_serial": 5508094, "releases": { "1.0.10": [ { "comment_text": "", "digests": { "md5": "054e62af9024f0a8490a0eea6c7fa394", "sha256": "440700aaad9d725f2daf97c3c665778b0d17e72e16e1b846faca6773271292e3" }, "downloads": -1, "filename": "aws_kpl_deagg-1.0.10.zip", "has_sig": false, "md5_digest": "054e62af9024f0a8490a0eea6c7fa394", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17944, "upload_time": "2016-02-18T18:15:32", "url": "https://files.pythonhosted.org/packages/2b/0e/d1c4000d0e7fda0abdaf2e9adea8f66da7f68cc88af85c34c9d9bd1e7bfd/aws_kpl_deagg-1.0.10.zip" } ], "1.0.7": [ { "comment_text": "", "digests": { "md5": "f3042906da49e740e14f8726744ee75b", "sha256": "975795c59dcb9e9a3add6ec0a0974b074dab389f79c84cf621a9ca1b54a4fae7" }, "downloads": -1, "filename": "aws_kpl_deagg-1.0.7.zip", "has_sig": false, "md5_digest": "f3042906da49e740e14f8726744ee75b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 16612, "upload_time": "2016-01-21T22:51:04", "url": "https://files.pythonhosted.org/packages/99/a4/a05fb77f5a6a3ae0c78f88db8457e683996551716ab4ef563ff780c81ff9/aws_kpl_deagg-1.0.7.zip" } ], "1.0.8": [ { "comment_text": "", "digests": { "md5": "5eabc1807dc73ed9b9f197566f520929", "sha256": "3a6d1010b2d08378bc3c2814b7240f8fc47961cef8e602689215a775d1db416e" }, "downloads": -1, "filename": "aws_kpl_deagg-1.0.8.zip", "has_sig": false, "md5_digest": "5eabc1807dc73ed9b9f197566f520929", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17896, "upload_time": "2016-01-22T00:24:42", "url": "https://files.pythonhosted.org/packages/21/8d/837299d259bc9da1afa3380728205a8a6e195418367455f67aa604253ee4/aws_kpl_deagg-1.0.8.zip" } ], "1.0.9": [ { "comment_text": "", "digests": { "md5": "fa35a7e432e4845db5b7071dca146f20", "sha256": "2bd23ce1b1e1a37abb1143172bd124aeb201c8818e328947008ffd02e83ac61b" }, "downloads": -1, "filename": "aws_kpl_deagg-1.0.9.zip", "has_sig": false, "md5_digest": "fa35a7e432e4845db5b7071dca146f20", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17907, "upload_time": "2016-02-11T23:51:55", "url": "https://files.pythonhosted.org/packages/e0/75/e22e8975781bf1f6a6a2656c66be1c0b0f36f5bdbc11828e8b9b9e3f0b2e/aws_kpl_deagg-1.0.9.zip" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "054e62af9024f0a8490a0eea6c7fa394", "sha256": "440700aaad9d725f2daf97c3c665778b0d17e72e16e1b846faca6773271292e3" }, "downloads": -1, "filename": "aws_kpl_deagg-1.0.10.zip", "has_sig": false, "md5_digest": "054e62af9024f0a8490a0eea6c7fa394", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17944, "upload_time": "2016-02-18T18:15:32", "url": "https://files.pythonhosted.org/packages/2b/0e/d1c4000d0e7fda0abdaf2e9adea8f66da7f68cc88af85c34c9d9bd1e7bfd/aws_kpl_deagg-1.0.10.zip" } ] }