{ "info": { "author": "Henrik Palmlund Wahlgren @ Palmlund Wahlgren Innovative Technology AB", "author_email": "henrik@pwit.se", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "License :: OSI Approved :: BSD License", "Natural Language :: English", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3 :: Only", "Topic :: Software Development :: Libraries" ], "description": "# Instrumentor\n\nInstrumentation library for python apps. Backed by Redis and exposes to Prometheus\n\n-- Work in progress -- \n-- Docs as design document --\n\n# About Instrumentor\n\nWhen building our multi utility AMR solution Utilitarian we wanted to add effective \nmetrics handling to expose our metrics to be scraped by Prometheus.\n\nWe could not find an existing solution that would do this in a desirable way. There is \nfor example [django-prometheus](https://github.com/korfuri/django-prometheus), but \nessentially they require you to switch out most the django stack to their own classes \nso that that has monitoring built in. It also uses the official \n[python prometheus client](https://github.com/prometheus/client_python) which seems to \ndo a good job, but it seems hard to use in web-apps that are multiprocess or \ndeployed across several servers. It also requires each process to start its own \nhttp server that Prometheus can scrape. So if you have an application that scales \nautomatically it would be troublesome to to keep track of the instances that should be \nscraped since each instance only keeps data about its on process. \n\nWe also have several little services processing messages coming in from different \nqueues and we would like to instrument those too without starting up separate \nwebservers for each little job.\n\nWe also want to instrument our celery jobs, but since we don't have control over \nexactly where they are executed we thing a centralised approach is valid.\n\nSo we want to design Instrumentor to be a metrics utility that can be used in several \ndifferent of types of applications that uses a shared store for the metrics so \nthat we can keep all our metrics in once place and give us control on how we expose them.\n\n# Design\n\nSince Prometheus data pretty much is a collection of counters over time we can model \nall our metrics by using counters that Prometheus scrapes at desired intervals. \nThe best way of keeping track of counters in a distributed environment is [Redis](https://redis.io/). \n\nSo all metrics are sent to Redis and we can choose to expose a web applications metrics \nin each web application by transforming the data in redis to and appropriate Prometheus format.\nWe could also create a separate web application which only functions as a metrics \nendpoint for instrumented applications.\n\nSince we also have a predetermined format in Redis other application in other \nprogramming languages could make use of the same structure and we have a common way to \nhandle application logging across all our services.\n\n## Namespacing\n\nEach applications metrics should live in a separate namespace. This makes it possible \nhave several applications metrics stored in the same Redis instance. It also makes it \neasy to move applications metrics to other Redis instances if you require separation.\n\nTo create a namespace we use redis hashes.\n\n## Metrics\n\nGeneral for all metric types is that we have a collection of counters, a name and a \ndescription.\n\nBy defining a specific format for the key names in the hash we can encode the data we \nas needed.\n\nWe also need to keep information about different labels on each metric. Each \ncombination of different lables should create a separate counter. Labels should be \nsorted so that each distinct combination of labels only generate one counter. Example\nmetric with labels {green, red, yellow} should increase the same counter as metric \nwith label {yellow, red, green} to keep down the number of counters and reduce parsing \nneed. \n\nTo simplify parsing and to be able to read raw data a bit easier we should also include \nhint about the metric type in the key.\n\nRules for Prometheus metrics should also apply. For example Histogram and Summary is \njust several counters with predefined label names. These label names should also be \nprotected from misuse on other types. \n\nThe library will assume that normal Prometheus best practices are used when defining \nmetrics and will not enforce any rules or checks on the user. For example, don't use \ntoo many labels. Each label creates a time series in Prometheus and also a counters \nin Redis.\n\nEach metrics should have a key that ends with _description to hold the description of \nthe metrics. There is need for some form of control to not keep sending the description \non every update.\n\n### Encoding format\n\nTo name the different values we should use a combination of values separated with the \nrecommended separation character of semicolon.\n\n{metric_name}:{extension}:{labels}\n\nlabels should be encoded with label name and label text in a comma separated list\n\n{label_name}=\"{label_value}\",{label_name}=\"{label_value}\"\n\nExample metric api_http_requests:\n\n```\napi_http_requests_total::method=\"POST\",handler=\"/messages\"\n```\n\n\n### Counter\n\nA counter is a cumulative metric that represents a single increasing counter whose \nvalue can only increase or be reset to zero on restart. \n\nExample for http_requests\n```\napi_http_requests_total:: -> 32 # counter without labels\napi_http_requests_total::method=\"POST\",handler=\"/messages\" -> 12 # counter matching the labels\napi_http_requests_total:description:: -> \"Total HTTP Requets to API\"\napi_http_requests_total:type:: -> \"counter\"\n```\n\n### Gauge\n\nA Prometheus Gauge is a value that can increase and decrease.\n\nExample for temperature\n```\ntemperature_celcius:: -> 32 # counter without labels\ntemperature_celcius::location=\"MainOffice\",sensor=\"34\" -> 12 # counter matching the labels\ntemperature_celcius:descri ption: -> \"Total HTTP Requets to API\"\ntemperature_celcius:type: -> \"gauge\"\n```\n\n### Histogram\n\nA histogram samples observations (usually things like request durations or response \nsizes) and counts them in configurable buckets. It also provides a sum of all \nobserved values. Prometheus histograms are cumulative histograms so all obervations \nthat fits in other buckets are added.\n\nYou will need to predefine your buckets sizes.\n\nExample other data format\n```\nhttp_request_duration_seconds:bucket:le=\"0.05\" -> 24054\nhttp_request_duration_seconds:bucket:le=\"0.1\" -> 33444\nhttp_request_duration_seconds:bucket:le=\"0.2\" -> 100392\nhttp_request_duration_seconds:bucket:le=\"0.5\" -> 129389\nhttp_request_duration_seconds:bucket:le=\"1\" -> 133988\nhttp_request_duration_seconds:bucket:le=\"+Inf\" -> 144320\nhttp_request_duration_seconds:sum: -> 53423\nhttp_request_duration_seconds:count: -> 144320 \nhttp_request_duration_seconds:description: -> \"Duration of HTTP requests\"\nhttp_request_duration_seconds:type: -> \"histogram\"\n```\nTo save some bytes extensions are encoded with the letter they start with. \ndescription=d, sum=s, count=c, type=t, bucket=b. The type value is also \nshortened: counter=c, gauge=g, histogram=h, summary=s\n\n### Summary\n\nIs similar to histogram but instead of using buckets you specify percentiles. \n\nYou will need to provide a max value to calculate the percentile against.\n\nValues above the max value should be set to 1 ??\n\nAdd counter that will increase for each value that is over max value.\n\n```\nrpc_duration_seconds::quantile:=\"0.01\" -> 3102\nrpc_duration_seconds::quantile=\"0.05\" -> 3272\nrpc_duration_seconds::quantile=\"0.5\" -> 4773\nrpc_duration_seconds::quantile=\"0.9\" -> 9001\nrpc_duration_seconds::quantile=\"0.99\" -> 76656\nrpc_duration_seconds:sum: -> 1.7560473e+07\nrpc_duration_seconds:count: -> 2693\nrpc_duration_seconds:description: -> \"Duration of RPC\" \nrpc_duration_seconds:type: -> \"summary\"\n```\n\n\n## Interaction with Redis\n\nBy setting the eager flag to true, all metrics updates will be sent directly instead of\nat a single point in time when `.store()`is called. To make the communication with \nredis efficient pipelining is used. \n\n## High level API\n\nWe try to follow the directions from Prometheus when \n[developing client libraries.](https://prometheus.io/docs/instrumenting/writing_clientlibs/)\n\n### CollectorRegistry\n\nMain class for managing the metrics in your application. You will register each metric \nin the `CollectorRegistry` and it will handle all communication to Redis.\n\n```\n\nreg = CollectorRegistry(redis_host='localhost', \n redis_port='6372', \n redis_db=0,\n namespace='myapp' \n eager=False)\n\n\nhttp_requests_total = Counter(.....)\n\nreg.register(http_requests_total)\nreg.unregister(http_requests_total) # incase it is needed...\n\n```\n\n\n### Metrics\n\n\n```\n\nhttp_requests_total = Counter(name='http_requests_total', \n description='Total amount of http requests', \n allowed_labels={'code', 'method', 'path'})\n\n\nhttp_requests_total.inc() # increment counter by 1\nhttp_request_total.inc(3) # increment counter by 3\n\nhttp_requests_total.inc(labels={\"code\": \"200\"})\n\n````\n\n\n```\ntemperature_celcius = Gauge(name='temperature_celcius',\n description='Temperature in celcius',\n allowed_labels={'location', 'sensor'},\n start_value=2)\n\ntemperature_celcius.inc()\ntemperature_celcius.inc(3)\ntemperature_celcius.dec()\ntemperature_celcius.dec(3)\ntermperature_celcius.set(5)\n```\n\n```\n\nresponse_time_seconds = Histogram(name='response_time_seconds', \n description='Response time in seconds',\n allowed_labels={'path', 'method'},\n buckets={0.05, 0.2, 0.3, 0.7, 0.9, 2})\n\n\nresponse_time_seconds = LinearHistogram(name='response_time_seconds', \n description='Response time in seconds',\n allowed_labels={'path', 'method'},\n start=0.5, width=20, count=10)\n\nresponse_time_seconds = ExponentialHistogram(name='response_time_seconds', \n description='Response time in seconds',\n allowed_labels={'path', 'method'},\n start=0.5, factor=2, count=10) \n\n\nresponse_time_seconds.observe(34)\n\n\n```\n\n\n```\n\nresponse_time_seconds = Summary(name='response_time_seconds', \n description='Response time in seconds',\n allowed_labels={'path', 'method'},\n quantile={0.05, 0.2, 0.3, 0.7, 0.9})\n\n\n\nresponse_time_seconds.observe(34)\n\n```\n\n\n### Instrumenting\n\n\n```python\n# In a collection module, ex mymetrics.py\nimport redis\nimport instrumentor\n\nr = redis.Redis()\nreg = instrumentor.CollectionRegistry(redis_client=r, namespace='myapp')\n\nhttp_requests_total = instrumentor.Counter(name='http_request_total', description='Total amount of http requests')\n\nreg.register(http_requests_total)\n\n#In other module:\n\nfrom mymetrics import http_requests_total\n\nhttp_requests_total.inc()\n\n```\n\nMain use cases are counting or timing.\n\nWe provide a simple decorator that can be used for counting method calls.\n\nIt is available for counters and gauges.\n\n```python\nfrom mymetrics import http_request_total\n\n@http_request_total.count\ndef my_func():\n pass\n\n```\n\nIt is also possible to use the general decorator and supply the metric as an input arg.\n\n```python\n\nfrom instrumentor import count\nfrom mymetrics import http_requests_total\n\n@count(metric=http_requests_total, labels={\"code\": \"200\"})\ndef my_func():\n pass\n\n\n```\n\nTiming is done via decorator on the metric instance or the general decorator/context manager.\n\n```python\nimport instrumentor\nfrom mymetrics import my_func_runtime_seconds\nimport time\n@my_func_runtime_seconds.time\ndef myfund():\n time.sleep(1)\n\n\n# or\n\n@instrumentor.timer(metric=my_func_runtime_seconds)\ndef myfunc():\n pass\n\n\nwith instrumentor.timer(\n metric=my_func_runtime_seconds, \n milliseconds=True,\n labels={\"my-label\": \"test\"}):\n\n time.sleep(1)\n\n\n\n\n```\n\nOther special instrumenting cases can be built using the normal functions on metrics.\n\n\n#### Instrumenting Django\n\nCollecting metrics about requests and response times are probably better to do in the \nload balancer/reverse proxy.\n\nWe mainly want to instrument aspects of out application.\n\nRegistry should be in its own module and be imported in `__init__` so it gets loaded by \ndefault (simmilar to setup of Celery)\n\nBy making a middleware class that calls `transfer()` on the register\nwhen the response is returning we can make use of pipelining and \nupdate all metrics that where affected during the request.\n\n#### Instrumenting Flask\n\nSimilar to django. \n\nRegistry probably be made as an extension so it can be saved in the global app context.\n\nhave the `transfer()` be registed to be run using Flasks `after_request` decorator.\n\n\n#### Instrumenting Celery Tasks\n\nBy making a new decorator that can wrap the normal task decorator it is possible \nto add instrumenting capabilities when running a task.\nafter task has run call `transfer()` on the registry.\nIt should also be possible to add celery specific metrics as task execution time, \nmemory consumption etc. \n\n\n\n## Exposition\n\nSince the format in redis is predefined an exposition client could be written in \nany language. Included in the library is a very simple exposition client.\nThe results from the client can then be returned in a for example a django view.\nThe client only knows about the namespace it should collect and does so with the \nHGETALL command in Redis.\n\nMetrics could be exposed in the web application that you are instrumenting or \na separate webapp just for exposition could be set up, that also could expose \nseveral namespaces (applications). This way scraping is decoupled from you application \nand can be scaled accordingly.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n# Changelog\nThe format is based on Keep a Changelog: https://keepachangelog.com/en/1.0.0/, and this project adheres to Semantic Versioning: https://semver.org/spec/v2.0.0.html\n\n### Unreleased\n### Added\n### Changed\n### Deprecated\n### Removed\n### Fixed\n### Security\n\n# v0.0.1 (2019-08-20)\nInitial implementation of Counter, Gauge and Summary\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/u9n/instrumentor", "keywords": "", "license": "BSD-3", "maintainer": "", "maintainer_email": "", "name": "instrumentor", "package_url": "https://pypi.org/project/instrumentor/", "platform": "", "project_url": "https://pypi.org/project/instrumentor/", "project_urls": { "Homepage": "https://github.com/u9n/instrumentor" }, "release_url": "https://pypi.org/project/instrumentor/0.0.1/", "requires_dist": [ "redis (==3.3.7)", "hiredis (==1.0.0)", "attrs (==19.1.0)" ], "requires_python": ">=3.6", "summary": "Instrumentation library for python apps. Backed by Redis and exposes to Prometheus", "version": "0.0.1" }, "last_serial": 5703002, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "eac51a810676521f3600737f8abb9306", "sha256": "6873a20b97ccbcb4c688fe8ac19c50a9011f6142b9d278e43e9b813c7ce5c502" }, "downloads": -1, "filename": "instrumentor-0.0.1-py3-none-any.whl", "has_sig": false, "md5_digest": "eac51a810676521f3600737f8abb9306", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6", "size": 14418, "upload_time": "2019-08-20T12:17:20", "url": "https://files.pythonhosted.org/packages/ae/4b/5206fe58c56a215b2e0944ef9fa5bf9e09dcb046ce6217c440a93c568e2a/instrumentor-0.0.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c6380915a75010f93a59fdb3c79ae11a", "sha256": "4353567c6ddfc040b9338fb8d8e7ff3b53a67314aa94ccaf870db738c08f2a36" }, "downloads": -1, "filename": "instrumentor-0.0.1.tar.gz", "has_sig": false, "md5_digest": "c6380915a75010f93a59fdb3c79ae11a", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 14668, "upload_time": "2019-08-20T12:17:21", "url": "https://files.pythonhosted.org/packages/70/38/420bac7eae24dc8cc99119a0f1215d1651616a3ba395cff98b1b773b7ab3/instrumentor-0.0.1.tar.gz" } ], "0.0.1.dev0": [ { "comment_text": "", "digests": { "md5": "df3a906bca9ce4caeb6604b200d1691e", "sha256": "d9af6b43e5e0224a2564d6acc07e6cbce5bfe019e58f00783f92195fbbca3688" }, "downloads": -1, "filename": "instrumentor-0.0.1.dev0-py3-none-any.whl", "has_sig": false, "md5_digest": "df3a906bca9ce4caeb6604b200d1691e", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6", "size": 12369, "upload_time": "2019-08-19T16:17:01", "url": "https://files.pythonhosted.org/packages/aa/83/a448ae404e95ed546d0620308d0676c33d14e22152c6ef548eaf36887a3b/instrumentor-0.0.1.dev0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "2a3045f4f6b632909c795d20a15aabca", "sha256": "1fb00138214ae44e8b8d31000e87e5f91d5375280fc723713e18a3577bcaab68" }, "downloads": -1, "filename": "instrumentor-0.0.1.dev0.tar.gz", "has_sig": false, "md5_digest": "2a3045f4f6b632909c795d20a15aabca", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 12566, "upload_time": "2019-08-19T16:17:03", "url": "https://files.pythonhosted.org/packages/8a/f3/76689a16d5169f3b31d8c23a56272d14942613b9124faa0de0eb1bb93c46/instrumentor-0.0.1.dev0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "eac51a810676521f3600737f8abb9306", "sha256": "6873a20b97ccbcb4c688fe8ac19c50a9011f6142b9d278e43e9b813c7ce5c502" }, "downloads": -1, "filename": "instrumentor-0.0.1-py3-none-any.whl", "has_sig": false, "md5_digest": "eac51a810676521f3600737f8abb9306", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6", "size": 14418, "upload_time": "2019-08-20T12:17:20", "url": "https://files.pythonhosted.org/packages/ae/4b/5206fe58c56a215b2e0944ef9fa5bf9e09dcb046ce6217c440a93c568e2a/instrumentor-0.0.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c6380915a75010f93a59fdb3c79ae11a", "sha256": "4353567c6ddfc040b9338fb8d8e7ff3b53a67314aa94ccaf870db738c08f2a36" }, "downloads": -1, "filename": "instrumentor-0.0.1.tar.gz", "has_sig": false, "md5_digest": "c6380915a75010f93a59fdb3c79ae11a", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 14668, "upload_time": "2019-08-20T12:17:21", "url": "https://files.pythonhosted.org/packages/70/38/420bac7eae24dc8cc99119a0f1215d1651616a3ba395cff98b1b773b7ab3/instrumentor-0.0.1.tar.gz" } ] }