{ "info": { "author": "Shuichiro MAKIGAKI", "author_email": "shuichiro.makigaki@gmail.com", "bugtrack_url": null, "classifiers": [], "description": "# Splunk HEC Stream\n\nPython streaming logging handler for Splunk HEC\n\n---\n\nThis handler is NOT an event sender.\nThe handler itself does not involve forwarding to Splunk HEC endpoint.\n\nThis means that log sources don't have to be concerned about buffering, transforming and retrying.\nThese functions are responsible for log forwarding services (such as AWS Kinesis Firehose, Fluentd, Logstash, etc.).\n\n## Use case\n\n* Forward logs from AWS Lambda functions to Splunk \n * Send events to Splunk HEC endpoint via AWS Kinesis Firehose and CloudWarch Logs\n* Use with log collector such as Fluentd and Logstash\n * Read events from log files and process them by log collector\n\n## How to install\n\n```\npip3 install splunk-hec-stream\n```\n\n## Example\n\n```python\nimport logging\nimport json\nfrom datetime import datetime\n\nfrom splunk_hec_stream.logging import SplunkHECStreamHandler\n\n\nlogging.basicConfig(\n level=logging.INFO,\n handlers=[SplunkHECStreamHandler(\"main\", \"splunk-logger-test\", \"aws:lambda\", \"_json\")]\n)\n\nlogging.info({\"key1\": \"value1\"})\nlogging.info(\"test\")\nlogging.info('''test\nln''')\nlogging.info(json.dumps({\"key1\": \"value1\"}))\n# You can overwrite logged time by _time extra key (that must be float)\nlogging.info({\"key\": \"value\"}, extra={'_time': datetime.utcnow().timestamp()})\n```\n\nThis example codes put following logs to stdout:\n\n```json\n{\"loggingHandler\":\"SplunkHECStreamHandler\",\"time\":1557301830.617483,\"host\":\"aws:lambda\",\"index\":\"main\",\"source\":\"splunk-logger-test\",\"sourcetype\":\"_json\",\"event\":{\"key1\": \"value1\"}}\n{\"loggingHandler\":\"SplunkHECStreamHandler\",\"time\":1557301830.617758,\"host\":\"aws:lambda\",\"index\":\"main\",\"source\":\"splunk-logger-test\",\"sourcetype\":\"_json\",\"event\":{\"message\": \"test\"}}\n{\"loggingHandler\":\"SplunkHECStreamHandler\",\"time\":1557301830.617904,\"host\":\"aws:lambda\",\"index\":\"main\",\"source\":\"splunk-logger-test\",\"sourcetype\":\"_json\",\"event\":{\"message\": \"test\\nln\"}}\n{\"loggingHandler\":\"SplunkHECStreamHandler\",\"time\":1557301830.618075,\"host\":\"aws:lambda\",\"index\":\"main\",\"source\":\"splunk-logger-test\",\"sourcetype\":\"_json\",\"event\":{\"message\": \"{\\\"key1\\\": \\\"value1\\\"}\"}}\n{\"loggingHandler\":\"SplunkHECStreamHandler\",\"time\":1557269430.618213,\"host\":\"aws:lambda\",\"index\":\"main\",\"source\":\"splunk-logger-test\",\"sourcetype\":\"_json\",\"event\":{\"key\": \"value\"}}\n```\n\nBy forwarding these JSON lines to Splunk HEC endpoint, Splunk can read and store them as events.\n\n## Use with AWS Lambda\n\nThis package is useful to forward logs of AWS Lambda to Splunk.\n\n1. Lambda functions put logs to CloudWatch,\n2. Subscription Filter forwards them to Firehose,\n3. and the Firehose forwards them to Splunk.\n\n### How to\n\n1. Create a lambda layer that contains this library\n2. Configure Kinesis Firehose to send events to Splunk HEC endpoint\n * `/contrib/aws_firehose_splunk_hec_stream_processor.py` can be used for event processor lambda.\n3. Configure CloudWatch Logs subscription filter, and send the filtered events to the Firehose stream\n * `loggingHandler` key in JSON can be used to filter logs that forward to Splunk HEC endpoint.\n\n### Terraform\n\n`/contrib/terraform` directory contains Terraform modules for above forwarding system.\n\n```hcl-terraform\nprovider \"aws\" {}\n\nvariable \"python_lib_path\" {\n default = \"/usr/local/lib/python3.7/site-packages/splunk_hec_stream\"\n}\n\nmodule \"handler_layer\" {\n source = \"github.com/shuichiro-makigaki/splunk_hec_stream//contrib/terraform/aws_lambda_layer\"\n layer_name = \"splunk_hec_stream_handler\"\n lib_path = var.python_lib_path\n}\n\nmodule \"firehose_processor\" {\n source = \"github.com/shuichiro-makigaki/splunk_hec_stream//contrib/terraform/aws_firehose\"\n lib_path = var.python_lib_path\n hec_endpoint = \"https://example.com\"\n hec_token = \"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\"\n layer_arn = module.handler_layer.arn\n s3_delivery_bucket_name = \"XXXXXXXX\"\n}\n```\n\nVariable `python_lib_path` should be replaced following your side.", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/shuichiro-makigaki/splunk_hec_stream", "keywords": "", "license": "MIT License", "maintainer": "", "maintainer_email": "", "name": "splunk-hec-stream", "package_url": "https://pypi.org/project/splunk-hec-stream/", "platform": "", "project_url": "https://pypi.org/project/splunk-hec-stream/", "project_urls": { "Homepage": "https://github.com/shuichiro-makigaki/splunk_hec_stream" }, "release_url": "https://pypi.org/project/splunk-hec-stream/0.3/", "requires_dist": null, "requires_python": "", "summary": "Splunk HEC Stream", "version": "0.3" }, "last_serial": 5380653, "releases": { "0.1": [ { "comment_text": "", "digests": { "md5": "286703c45c13a8f20b61340a7d01c9aa", "sha256": "c1b5e19bcb6ff945f0f9387a4504e9e44a882f2461687792d692ea92f481cf63" }, "downloads": -1, "filename": "splunk_hec_stream-0.1.tar.gz", "has_sig": false, "md5_digest": "286703c45c13a8f20b61340a7d01c9aa", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 2354, "upload_time": "2019-04-19T04:01:45", "url": "https://files.pythonhosted.org/packages/e5/bf/d0091f1f8b9fa23d16ee488b4bd842eb3dedd57f3aac4f831ca3f98dfe17/splunk_hec_stream-0.1.tar.gz" } ], "0.2": [ { "comment_text": "", "digests": { "md5": "ef35e25a600312823ccd504f5ea86f53", "sha256": "ee57186e674b05e4466f06a22905211ac0813afc67892586b90db3860b450bce" }, "downloads": -1, "filename": "splunk_hec_stream-0.2.tar.gz", "has_sig": false, "md5_digest": "ef35e25a600312823ccd504f5ea86f53", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 2261, "upload_time": "2019-04-25T10:16:56", "url": "https://files.pythonhosted.org/packages/74/88/d78fc464b9b5e364d26113126a6b4d675d756d2891fc7d867df68e437258/splunk_hec_stream-0.2.tar.gz" } ], "0.3": [ { "comment_text": "", "digests": { "md5": "2b816ac0d177627f70b905b27f41b4ee", "sha256": "52766da4f2fa82a268eea759c47966c4abe651fb6b323804b65121641ef8c2e7" }, "downloads": -1, "filename": "splunk_hec_stream-0.3.tar.gz", "has_sig": false, "md5_digest": "2b816ac0d177627f70b905b27f41b4ee", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3628, "upload_time": "2019-06-10T10:06:36", "url": "https://files.pythonhosted.org/packages/81/d2/feb789b6884e880795d3a673684f4b4bf7400ffaa5ac03d90330381db4d8/splunk_hec_stream-0.3.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "2b816ac0d177627f70b905b27f41b4ee", "sha256": "52766da4f2fa82a268eea759c47966c4abe651fb6b323804b65121641ef8c2e7" }, "downloads": -1, "filename": "splunk_hec_stream-0.3.tar.gz", "has_sig": false, "md5_digest": "2b816ac0d177627f70b905b27f41b4ee", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3628, "upload_time": "2019-06-10T10:06:36", "url": "https://files.pythonhosted.org/packages/81/d2/feb789b6884e880795d3a673684f4b4bf7400ffaa5ac03d90330381db4d8/splunk_hec_stream-0.3.tar.gz" } ] }