{
    "info": {
        "author": "Jude D'Souza",
        "author_email": "dsouza_jude@hotmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Environment :: Console",
            "Intended Audience :: Developers",
            "Operating System :: OS Independent",
            "Programming Language :: Python",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.3",
            "Programming Language :: Python :: 3.4"
        ],
        "description": "\nxFlow\n=====\nA serverless workflow architecture using AWS Lambda functions and Kinesis.\n\n\nHow it works\n============\nThe concept behind xFlow is to define a workflow which comprises of\na number steps that gets executed to complete a task such that the output\nof one step is fed in as the input to the next step - except that instead of the\noutput being fed in as input, the preceding step publishes an event which is subscribed\nby the next step. This way each step is executed independently and can run in parallel.\n\nThese steps are AWS Lambda functions and primarily, they should do the following:\n\n- It should be subscribed to listen on certain events\n- Do work based on these events\n- Optionally emit an output event for other lambda functions to work on\n\nxFlow uses AWS Kinesis as the streaming engine to publish and subscribe events in a workflow.\nAs described above, these events would then be processed via AWS Lambda functions after they\nhave been subscribed to it. The configuration file that you provide to xFlow would be the glue\nthat combines events to the processors in a workflow.\n\nxFlow will internally create the AWS Lambda functions and subscribe them to AWS Kinesis.\nAll you need to do is write the body of the lambda functions.\n\nxFlow supports all languages AWS Lambda supports. At this moment, lambdas can only be\nuploaded from the local filesystem or s3. In the future we will support integration with\ngithub.\n\n\nCreating a sample workflow:\n==========================\n\n- Create a config that defines your workflow. For example:\n\n```yaml\n\ngeneral:\n  lambda_timeout_time: 3\n\naws:\n  region:\n  subnet_ids:\n  security_group_ids:\n  lambda_execution_role_name: lambda-execute\n\nlambdas:\n  - name: lambda_reader\n    description: Reads contents of a file that was uploaded and publishes those contents.\n    source: /xFlow/examples/wordcount/lambda_reader.py\n    handler: read\n    runtime: python2.7\n\n  - name: lambda_parser\n    description: Reads contents, parses it into an array of words.\n    source: s3://xFlow/flows/wordcount/lambda_parser.py\n    handler: parse\n    runtime: python2.7\n\n  - name: lambda_filter\n    description: Filters non-letters from words and non-words.\n    source: /xFlow/examples/wordcount/lambda_filter.py\n    handler: filter_out_non_words\n    runtime: python2.7\n\n  - name: lambda_aggregator\n    description: Groups similar words and counts them.\n    source: /xFlow/examples/wordcount/lambda_aggregator.py\n    handler: aggregate\n    runtime: python2.7\n\n  - name: lambda_summarizer\n    description: Prints unique word counts.\n    source: /xFlow/examples/wordcount/lambda_summarizer.py\n    handler: summarize\n    runtime: python2.7\n\nsubscriptions:\n  - event: FileUploaded\n    subscribers:\n      - lambda_reader\n  - event: FileDownloaded\n    subscribers:\n      - lambda_parser\n  - event: FileParsed\n    subscribers:\n      - lambda_filter\n  - event: FileFiltered\n    subscribers:\n      - lambda_aggregator\n  - event: FileAggregated\n    subscribers:\n      - lambda_summarizer\n  - event: FileSummarized\n    subscribers:\n\n# Optional - Can be used to track a complete workflow\nworkflows:\n  - id: compute_word_count\n    flow:\n      - FileUploaded\n      - FileDownloaded\n      - FileParsed\n      - FileFiltered\n      - FileAggregated\n      - FileSummarized\n```\n\n- Setup the workflow via the following command:\n\n  `xflow word_count.cfg --configure`\n\n  Behind the scenes, this will do the following:\n  - Create (or update) AWS Lambda functions.\n  - For each lambda function, subscribe them to AWS Kinesis based on the events they are listening for.\n  - Lambda functions will be executed once an event is published to AWS Kinesis.\n\n- Tracking the workflow is done via the following command:\n\n  `xflow word_count.cfg --track <WORKFLOW_ID> <EXECUTION_ID>`\n\n  An instance of a particular workflow is tracked via its `execution_id`. Therefore all events defined\n  in a workflow must contain this field. Since it is required to have the events defined in order it\n  will then be easy to determine which events were successfully processed and which were not at any\n  particular time in the last 7 days.\n\n  The goal of the tracker is to return the state of all events in the workflow given as input\n  the `workflow_id` and its `execution_id`. The state would indicate whether the event was processed or not.\n  It will then grab the last unprocessed event and fetch all logs for its subscribers (the lambdas) that failed\n  to process that event.\n\n  For this to work, there would be a separate tracker for every workflow defined. The tracker would subscribe\n  itself to every stream in that workflow and hence it would receive events published to that stream. The tracker itself would be a lambda function and its name would be `tracker_<workflow_id>`. Hence it will be invoked on every\n  successful publishing of an event to the stream.\n\n  The actual tracking is done via AWS CloudWatchLogs. For every workflow a CloudWatch `LogGroup` is created. And for every instance of a workflow execution, a `LogStream` would be created.\n\n  Hence, when an event is published to a stream, the tracker would receive the event and extract the\n  `execution_id`. It will create a new log stream from this execution_id if not already created. It will\n  then store the event in that stream. Since the lambda's name is generated from the workflow_id, it is also\n  easy to extract the log group for the log stream.\n\n  For every subsequent event published to the kinesis stream, the corresponding lambda for the workflow will be invoked and it will save the event to the log stream.\n\n- Running in server mode:\n\n  `xflow word_count.cfg --server`\n\n  This will run xflow via server mode. On startup, the server will setup the necessary streams, lambda functions and workflows. You can then publish events to a stream or track workflow executions in a RESTful way. Following are examples how you would do this.\n\n  Publishing:\n\n  `curl -XPOST localhost/publish -d '{\"stream\":\"FileUploaded\", \"event\":{\"execution_id\":\"ex1\", \"message\":\"Test with ccc\"}}'`\n\n  Tracking:\n\n  `curl -v localhost/track/workflows/compute_word_count/executions/ex1`\n\n\nInstallation:\n=============\n\n- Via pip\n\n```bash\n>> pip install xflow\n```\n\n- Via github\n\n```bash\n>> git clone git@github.com:dsouzajude/xFlow.git\n>> cd xFlow\n>> python setup.py install\n```\n\n\nxFlow Requirements (and roadmap):\n=================================\n- Integration with github (so we can download the lambda function from there given the link)\n- Monitoring around lambda functions\n- Centralized Logging (or ability to route logs) to log server\n- CI/ CD for lambda functions\n",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "https://github.com/dsouzajude/xflow/archive/0.1.tar.gz",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://github.com/dsouzajude/xFlow",
        "keywords": "serverless,lambda,kinesis,workflow,streams,eventbus",
        "license": "UNKNOWN",
        "maintainer": null,
        "maintainer_email": null,
        "name": "xFlow",
        "package_url": "https://pypi.org/project/xFlow/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/xFlow/",
        "project_urls": {
            "Download": "https://github.com/dsouzajude/xflow/archive/0.1.tar.gz",
            "Homepage": "http://github.com/dsouzajude/xFlow"
        },
        "release_url": "https://pypi.org/project/xFlow/0.1.16/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "A serverless workflow architecture using AWS Lambda functions and Kinesis",
        "version": "0.1.16"
    },
    "last_serial": 2671688,
    "releases": {
        "0.1.10": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "03b8a5aeaaed4fba532c2b2a1e25bad4",
                    "sha256": "943488384a555a0e6d57f37da1fe20bfa1f6642ca1868561f6ed3a9d73cc9279"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.10.tar.gz",
                "has_sig": false,
                "md5_digest": "03b8a5aeaaed4fba532c2b2a1e25bad4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16253,
                "upload_time": "2017-02-27T19:37:45",
                "url": "https://files.pythonhosted.org/packages/63/b1/bb23ab7611b5621eab374fbd3ce7ea7a36134210c3e58b77e07b77f48ad0/xFlow-0.1.10.tar.gz"
            }
        ],
        "0.1.12": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0eb71bd8dca11b1fd294e20834706d39",
                    "sha256": "46a19b32c398883c4bc4a2d6ad62fb67b51908bd3cbe9ba31176217fb96a9665"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.12.tar.gz",
                "has_sig": false,
                "md5_digest": "0eb71bd8dca11b1fd294e20834706d39",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16264,
                "upload_time": "2017-02-27T19:44:17",
                "url": "https://files.pythonhosted.org/packages/c5/8a/1a608c0a9e5d7d0e39b4ecf4923aa179cd53a814ae66564031cbd5507bcb/xFlow-0.1.12.tar.gz"
            }
        ],
        "0.1.13": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "56b47eec318b974b91681482cc5415d1",
                    "sha256": "acdeb3af588c1487dc3748a89a5954343b5f1ac5aefc91040366c793b8ba8302"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.13.tar.gz",
                "has_sig": false,
                "md5_digest": "56b47eec318b974b91681482cc5415d1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16256,
                "upload_time": "2017-02-27T19:50:13",
                "url": "https://files.pythonhosted.org/packages/af/ac/0499c037a021782221c34444cbfd39566208014eaf4b2181df5a9a78c80d/xFlow-0.1.13.tar.gz"
            }
        ],
        "0.1.14": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "595cd4ebb3fb599a0ce0b59cf7f2d688",
                    "sha256": "eebc853b6fb113efb9335999572904b6bee56bd42de3391c1ca6c61e6c40f9da"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.14.tar.gz",
                "has_sig": false,
                "md5_digest": "595cd4ebb3fb599a0ce0b59cf7f2d688",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16268,
                "upload_time": "2017-02-27T19:53:46",
                "url": "https://files.pythonhosted.org/packages/3b/39/e4cd55da1d796a1d953d87ecdf28a738f61d2dc4bc87adc8cbca387d8ba6/xFlow-0.1.14.tar.gz"
            }
        ],
        "0.1.15": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fcca722a662c47bd615ffe83a5976641",
                    "sha256": "b4b9c8c2eecef4e7e0a07c166b7f5a0c6d936919656ff69041a5e1bd2a9314d5"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.15.tar.gz",
                "has_sig": false,
                "md5_digest": "fcca722a662c47bd615ffe83a5976641",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16347,
                "upload_time": "2017-02-27T20:07:01",
                "url": "https://files.pythonhosted.org/packages/1c/b9/759d636a014d7106672aa2ca4298ad250126dacf0c41b513261f21668442/xFlow-0.1.15.tar.gz"
            }
        ],
        "0.1.16": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f1ede29b793eafd62c11df6c18d0aa0c",
                    "sha256": "6c680ae957e1d354a046716f991774106960ab1c0e4011b8ab428807af8fc767"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.16.tar.gz",
                "has_sig": false,
                "md5_digest": "f1ede29b793eafd62c11df6c18d0aa0c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16353,
                "upload_time": "2017-02-27T20:19:05",
                "url": "https://files.pythonhosted.org/packages/a9/7f/ec735e51def27b17e256d636c062a10bd0acc385a7c7621cb63772dc8b9f/xFlow-0.1.16.tar.gz"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "1b3a8f371ba42652911b922b2daf356c",
                    "sha256": "ddbeb6aa6c87ae0e89cc14b86b47faf21a12a189bc0de36468cb272fd158f235"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "1b3a8f371ba42652911b922b2daf356c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16133,
                "upload_time": "2017-02-27T16:11:22",
                "url": "https://files.pythonhosted.org/packages/7c/88/e567d84e9949b5cf240da81bf0c07426f8339ef5b9dcc66f0c2973a0209b/xFlow-0.1.2.tar.gz"
            }
        ],
        "0.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "da84ea1053c49fa595f414a6c336c324",
                    "sha256": "6149c3c7b853a09e686d3355f7e436956b9b2e37aa379d9b438beb267ed6df40"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.3.tar.gz",
                "has_sig": false,
                "md5_digest": "da84ea1053c49fa595f414a6c336c324",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16147,
                "upload_time": "2017-02-27T16:25:08",
                "url": "https://files.pythonhosted.org/packages/5e/5f/cdf0d47ce18b385e5e37a561becec470c0d54da8c0e239832d8eb23e7045/xFlow-0.1.3.tar.gz"
            }
        ],
        "0.1.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e68e58bafb378a3774c1a711552a1cc1",
                    "sha256": "24910518d5ad12445d6aff5bed06b0876027b1fc47820955661a5ce8a0c73d81"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.4.tar.gz",
                "has_sig": false,
                "md5_digest": "e68e58bafb378a3774c1a711552a1cc1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16223,
                "upload_time": "2017-02-27T16:28:31",
                "url": "https://files.pythonhosted.org/packages/77/80/99a8d8344dcb7bedcfbd8a059901ba4e6437bb438911cd96c5e89a0bf048/xFlow-0.1.4.tar.gz"
            }
        ],
        "0.1.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3d9c983eaa94652e87bccd5421609c69",
                    "sha256": "642b30d82e8300488219af017d083b20196ca5745cb6d8e0bb797c8b764d73fa"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.5.tar.gz",
                "has_sig": false,
                "md5_digest": "3d9c983eaa94652e87bccd5421609c69",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16215,
                "upload_time": "2017-02-27T18:17:25",
                "url": "https://files.pythonhosted.org/packages/4e/d2/1347705ef732c89f437819a2e816f9413bf3acae49dd76a23f0f2e310b9e/xFlow-0.1.5.tar.gz"
            }
        ],
        "0.1.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7f23bd8b4e69faaab459763ff9889756",
                    "sha256": "efa9b86bb04a5d32fadd5f24267bcde78112c83914e4f882a49b92dfcfee4439"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.6.tar.gz",
                "has_sig": false,
                "md5_digest": "7f23bd8b4e69faaab459763ff9889756",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16211,
                "upload_time": "2017-02-27T18:49:53",
                "url": "https://files.pythonhosted.org/packages/99/5f/018ee8faa44ed6df9782b66d996848367bbd23ef46635cb5b80c5dd583da/xFlow-0.1.6.tar.gz"
            }
        ],
        "0.1.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "477a14392bec7dccff1226490622b2a4",
                    "sha256": "5c78ab7e859946c33fe4d2df5ef3c49a7ad2afc9a508e8bad07d4a2467f50c2e"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.7.tar.gz",
                "has_sig": false,
                "md5_digest": "477a14392bec7dccff1226490622b2a4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15857,
                "upload_time": "2017-02-27T18:52:57",
                "url": "https://files.pythonhosted.org/packages/f1/fc/3f1c2e5492654b53b72bf3b0a8db7e2f6c92ec5d52884ffbe88731d82745/xFlow-0.1.7.tar.gz"
            }
        ],
        "0.1.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "481b8946be0f19652cf546a1031d6223",
                    "sha256": "663f4b246cabeefa2eb4301b8cc0b7f1a8d644c5c00146c7788e47cb94f20214"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.8.tar.gz",
                "has_sig": false,
                "md5_digest": "481b8946be0f19652cf546a1031d6223",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16131,
                "upload_time": "2017-02-27T19:10:49",
                "url": "https://files.pythonhosted.org/packages/3a/19/4542d4274b3e1e624b7c8b363285dcf682ac969b3e089dec1f1e64dbe10c/xFlow-0.1.8.tar.gz"
            }
        ],
        "0.1.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8bbd322a555f4caa76155ff9b50a7810",
                    "sha256": "1c4d8dad64e5e821cd1165ab14ad0338f1b0ef3fea25db4e9a421f05ffda015c"
                },
                "downloads": -1,
                "filename": "xFlow-0.1.9.tar.gz",
                "has_sig": false,
                "md5_digest": "8bbd322a555f4caa76155ff9b50a7810",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16220,
                "upload_time": "2017-02-27T19:21:52",
                "url": "https://files.pythonhosted.org/packages/f5/ed/3eee7d17c63c63871db7f5140bacb580e94b73d206a0fb9feede9831a7d1/xFlow-0.1.9.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "f1ede29b793eafd62c11df6c18d0aa0c",
                "sha256": "6c680ae957e1d354a046716f991774106960ab1c0e4011b8ab428807af8fc767"
            },
            "downloads": -1,
            "filename": "xFlow-0.1.16.tar.gz",
            "has_sig": false,
            "md5_digest": "f1ede29b793eafd62c11df6c18d0aa0c",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 16353,
            "upload_time": "2017-02-27T20:19:05",
            "url": "https://files.pythonhosted.org/packages/a9/7f/ec735e51def27b17e256d636c062a10bd0acc385a7c7621cb63772dc8b9f/xFlow-0.1.16.tar.gz"
        }
    ]
}