{
    "info": {
        "author": "Pavlos Parissis",
        "author_email": "pavlos.parissis@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Environment :: Console",
            "Intended Audience :: Information Technology",
            "Intended Audience :: System Administrators",
            "Natural Language :: English",
            "Operating System :: POSIX",
            "Programming Language :: Python :: 3.4",
            "Topic :: Utilities"
        ],
        "description": ".. README.rst\n\n============\nhaproxystats\n============\n\n    *A HAProxy statistics collection program*\n\n.. contents::\n\nIntroduction\n------------\n\n**haproxystats** is a statistics collector for `HAProxy`_ load balancer which\nprocesses various statistics and pushes them to graphing systems (Graphite).\nIt is designed to satisfy the following requirements:\n\n#. Fast and configurable processing of HAProxy statistics\n#. Perform aggregation when HAProxy runs in multiprocess (nbproc > 1)\n#. Pull statistics at very low intervals (10secs)\n#. Flexible dispatching of statistics to different systems (Graphite,  kafka)\n\nThe main design characteristic is the split between pulling the statistics and\nprocessing them. This provides the ability to pull data as frequently\nas possible without worrying about the impact on processing time. It also\nreduces the risk of losing data in case of trouble during the processing phase.\n\nIt runs locally on each load balancer node, offering a decentralized setup for\nthe processing phase, but it can be easily extended in the future to have a\ncentralized setup for the processing phase. In that centralized setup it will\nbe possible to perform aggregation on a cluster level as well.\nUntil then users can deploy `carbon-c-relay`_ for aggregation.\n\nBecause of this design haproxystats comes with two programs:\n**haproxystats-pull** and **haproxystats-process**. The former pulls\nstatistics from HAProxy via `stats socket`_ and it uses the `asyncio`_ framework\nfrom Python to achieve high concurrency and low footprint. The latter\nprocesses the statistics and pushes them to various destinations. It utilizes\n`Pandas`_ for data analysis and the multiprocess framework from Python.\n\nhaproxystats requires Python 3.4, docopt and Pandas to be available in the\nsystem.\n\nHow haproxystats works\n----------------------\n\n\n.. image:: haproxystats-architecture.png\n\n\nhaproxystats-pull sends `info`_ and `stat`_ commands to all haproxy processes\nin order to collect statistics for the daemon and for all\nfrontends/backends/servers. Data returned from each process and for each\ncommand is stored in individual files which are saved under one directory. The\ntime (seconds since the epoch) of retrieval is used to name that directory.\nhaproxystats-process watches for changes on the parent directory and when a\ndirectory is created it adds its full path to the queue. Multiple workers pick\nup items (directories) from the queue and process statistics from those\ndirectories.\n\nhaproxystats-pull\n#################\n\nhaproxystats-pull leverages the `asyncio`_ framework from Python by utilizing\ncoroutines to multiplex I/O access over several `stats socket`_, which are\nsimple UNIX and TCP sockets.\n\nThe actual task of storing the data to the file system is off-loaded to a very\nlight `pool of threads`_ in order to avoid blocking the coroutines during the\ndisk IO phase.\n\nhaproxystats-pull manages the *incoming* directory and makes sure directories\nare created with correct names. It also suspends the collection when the number\nof directories under the *incoming* directory exceeds a threshold. This avoids\nfilling up the disk when haproxystats-process is unavailable for sometime.\nThis an example of directory structure:\n\n.. code-block:: bash\n\n    incoming\n    \u251c\u2500\u2500 1457298067\n    \u2502\u00a0\u00a0 \u251c\u2500\u2500 admin1.sock_info\n    \u2502\u00a0\u00a0 \u251c\u2500\u2500 admin1.sock_stat\n    \u2502\u00a0\u00a0 \u251c\u2500\u2500 admin2.sock_info\n    \u2502\u00a0\u00a0 \u251c\u2500\u2500 admin2.sock_stat\n    \u2502\u00a0\u00a0 \u251c\u2500\u2500 admin3.sock_info\n    \u2502\u00a0\u00a0 \u251c\u2500\u2500 admin3.sock_stat\n    \u2502\u00a0\u00a0 \u251c\u2500\u2500 admin4.sock_info\n    \u2502\u00a0\u00a0 \u2514\u2500\u2500 admin4.sock_stat\n    \u2514\u2500\u2500 1457298072\n        \u251c\u2500\u2500 admin1.sock_info\n        \u251c\u2500\u2500 admin1.sock_stat\n        \u251c\u2500\u2500 admin2.sock_info\n        \u251c\u2500\u2500 admin2.sock_stat\n        \u251c\u2500\u2500 admin3.sock_info\n        \u251c\u2500\u2500 admin3.sock_stat\n        \u251c\u2500\u2500 admin4.sock_info\n        \u2514\u2500\u2500 admin4.sock_stat\n\nhaproxystats-process\n####################\n\nhaproxystats-process is a multiprocess program. The parent process uses the\nLinux kernel's `inotify`_ API to watch for changes in *incoming* directory.\n\nIt receives an event when a directory is either created or moved in *incoming*\ndirectory. The event contains the absolute path name of that directory. It\nmaintains an internal queue in which it puts directory names. Multiple child\nprocesses pick directory names from the queue and process the data.\n\nIts worker dispatches statistics to various destinations. The directories are\nremoved from *incoming* directory when all statistics are successfully\nprocessed.\n\nWhen haproxystats-process starts it scans the *incoming* directory\nfor new directories and processes them instantly, so you don't lose statistics\nif haproxystats-process is unavailable for sometime.\n\nDispatchers\n###########\n\nhaproxystats-process currently supports 2 different dispatchers.\n\n1. **Graphite**\n\nPushes statistics to a Graphite system via a local or remote carbon-relay.\nThe recommended method is to use `carbon-c-relay`_. It is very fast and capable\nof handling millions of metrics per second. This dispatcher utilizes an internal\nqueue to store metrics which are failed to be sent to Graphite.\n\nAn example of graphite namespace::\n\n    <loadbalancers>.<lb-01>.haproxy.frontend.<frontendname>.\n    <loadbalancers>.<lb-01>.haproxy.backend.<backendname>.\n    <loadbalancers>.<lb-01>.haproxy.backend.<backendname>.server.<servername>\n    <loadbalancers>.<lb-01>.haproxy.server.<servername>.\n    <loadbalancers>.<lb-01>.haproxy.daemon.\n    <loadbalancers>.<lb-01>.haproxy.haproxystats.<metric names>.\n\n2. **local-store**\n\nStores statistics in the local disk. Use it only for debugging purposes.\n\nStatistics for HAProxy\n######################\n\nIn addition the statistics that are exposed by HAProxy, haproxystats provides\nthe following statistics.\n\nHAProxy process\n~~~~~~~~~~~~~~~\n\nHAProxy exposes Idle_pct and haproxystats-process converts it to CPU\nutilization without removing Idle_pct metric. This avoids the usage of\nscale(-1) and offset(100) functions on graphite::\n\n    CpuUsagePct  CPU utilization in percentage\n\nThe following metrics are calculated only when HAProxy is configured with more\nthan 1 processes (nbproc > 1)::\n\n    25PercentileCpuUsagePct 25th percentile of CpuUsagePct across all processes\n    50PercentileCpuUsagePct 50th percentile              -//-\n    75PercentileCpuUsagePct 75th percentile              -//-\n    95PercentileCpuUsagePct 95th percentile              -//-\n    99PercentileCpuUsagePct 99th percentile              -//-\n    StdCpuUsagePct          standard deviation           -//-\n\nQueuing system\n##############\n\nThe *incoming* directory together with the inotify API provides a simple\nqueueing system which is used as a communication channel between\nhaproxystats-pull and haproxystats-process programs.\n\nThere isn't any feedback mechanism in place, thus haproxystats-pull monitors\nthe number of directories before it pulls data from HAProxy and suspends its\njob when the number of directories exceeds a threshold.\n\nSee **queue-size** parameter of **pull** section.\n\nStatistics for haproxystats\n###########################\n\n**haproxystats** provides statistics for the time it takes to process,\ncalculate and send HAProxy metrics. By default provides the following list\nof metric names with values in seconds::\n\n    loadbalancers.lb-01.haproxy.haproxystats.WallClockTimeHAProxy\n    loadbalancers.lb-01.haproxy.haproxystats.WallClockTimeFrontends\n    loadbalancers.lb-01.haproxy.haproxystats.WallClockTimeBackends\n    loadbalancers.lb-01.haproxy.haproxystats.WallClockTimeServers\n    loadbalancers.lb-01.haproxy.haproxystats.WallClockTimeAllStats\n\nIt also provides the number of metrics which are send to graphite::\n\n    loadbalancers.lb-01.haproxy.haproxystats.MetricsHAProxy\n    loadbalancers.lb-01.haproxy.haproxystats.MetricsFrontend\n    loadbalancers.lb-01.haproxy.haproxystats.MetricsBackend\n    loadbalancers.lb-01.haproxy.haproxystats.MetricsServer\n\nConfiguration\n-------------\n\nhaproxystats uses the popular `INI`_ format for its configuration file.\nThis is an example configuration file (/etc/haproxystats.conf)::\n\n\n    [DEFAULT]\n    loglevel = info\n    retries  = 2\n    timeout  = 1\n    interval = 2\n\n    [paths]\n    base-dir = /var/lib/haproxystats\n\n    [pull]\n    loglevel        = info\n    socket-dir      = /run/haproxy\n    retries         = 1\n    timeout         = 0.1\n    interval        = 0.5\n    pull-timeout    = 2\n    pull-interval   = 10\n    dst-dir         = ${paths:base-dir}/incoming\n    tmp-dst-dir     = ${paths:base-dir}/incoming.tmp\n    workers         = 8\n    queue-size      = 360\n\n    [process]\n    src-dir             = ${paths:base-dir}/incoming\n    workers             = 4\n    per-process-metrics = false\n\n    [graphite]\n    server          = 127.0.0.1\n    port            = 3002\n    retries         = 3\n    interval        = 1.8\n    connect-timeout = 1.0\n    write-timeout   = 1.0\n    delay           = 10\n    backoff         = 2\n    namespace       = loadbalancers\n    prefix-hostname = true\n    fqdn            = true\n    queue-size      = 1000000\n\n    #[local-store]\n    #dir = ${paths:base-dir}/local-store\n\nAll the above settings are optional as haproxystats comes with default values\nfor all of them. Thus, both programs can be started without supplying any\nconfiguration.\n\nDEFAULT section\n###############\n\nSettings in this section can be overwritten in other sections.\n\n* **loglevel** Defaults to **info**\n\nLog level to use, possible values are: debug, info, warning, error, critical\n\n* **retries** Defaults to **2**\n\nNumber of times to retry a connection after a failure. Used by haproxystats-pull\nand haproxystats-process when they open a connection to a UNIX/TCP socket and\nGraphite respectively.\n\n* **timeout** Defaults to **1** (seconds)\n\nTime to wait for establishing a connection. Used by haproxystats-pull and\nhaproxystats-process when they open a connection to a UNIX/TCP socket and Graphite\nrespectively.\n\n* **interval** Defaults to **2**\n\nTime to wait before trying to open a connection. Used by haproxystats-pull and\nhaproxystats-process when they retry a connection to a UNIX/TCP socket and Graphite\nrespectively.\n\npaths section\n#############\n\n* **base-dir** Defaults to **/var/lib/haproxystats**\n\nThe directory to use as the base of the directory structure.\n\npull section\n############\n\n* **socket-dir** Unset by default\n\nA directory with HAProxy socket files.\n\n* **servers** Unset by default\n\nA list of servers to pull statistics from. You define a server by passing a URL,\nhere some examples::\n\n    tcp://127.0.0.1:5555\n    tcp://foo.bar.com:4444\n    tcp://[fe80::3f2f:46b3:ef0c:a420]:4444\n    unix:///run/haproxy.sock\n\nOnly TCP and UNIX schemes are supported and the port for TCP servers **must**\nbe set. For UNIX scheme you can only pass a file and not a directory, but\n**socket-dir** option can be set as well, so you can use a directory and UNIX\nsocket files at the same time. You can use comma as separator to pass multiple\nservers::\n\n    servers = unix:///run/haproxy.sock,tcp://127.0.0.1:555,tcp://127.0.0.1:556\n\n* **buffer-limit** Defaults to **6291456** (bytes)\n\nAt most size bytes are read and returned from the sockets. Setting too low and\nit will slow down the retrieval of statistics.\nOnly values greater than or equal to 1 are accepted.\n\n* **retries** Defaults to **1**\n\nNumber of times to reconnect to UNIX/TCP socket after a failure.\n\n* **timeout** Defaults to **0.1** (seconds)\n\nTime to wait for establishing a connection to UNIX/TCP socket. There is no need to\nset it higher than few ms as haproxy accepts a connection within 1-2ms.\n\n* **interval** Defaults to **0.5** (seconds)\n\nTime to wait before trying to reconnect to UNIX/TCP socket after a failure. Tune it\nbased on the duration of the reload process of haproxy. haproxy reloads within\nfew ms but in some environments with hundreds different SSL certificates it can\ntake a bit more.\n\n* **pull-interval** Defaults to **10** (seconds)\n\nHow often to pull statistics from HAProxy. A value of *1* second can overload\nthe haproxy processes in environments with thousands backends/servers.\n\n* **pull-timeout** Defaults to **2** (seconds)\n\nTotal time to wait for the pull process to finish. Should be always less than\n**pull-interval**.\n\n* **dst-dir** Defaults **/var/lib/haproxystats/incoming**\n\nA directory to store statistics retrieved by HAProxy.\n\n* **tmp-dst-dir** Defaults **/var/lib/haproxystats/incoming.tmp**\n\nA directory to use as temporary storage location before directories are moved\nto **dst-dir**.  haproxystats-pull stores statistics for each process under\nthat directory and only when data from all haproxy processes are successfully\nretrieved they are moved to **dst-dir**. Make sure **dst-dir** and\n**tmp-dst-dir** are on the same file system, so the move of the directories\nbecome a rename which is a quick and atomic operation.\n\n* **workers**  Defaults to **8**\n\nNumber of threads to use for writing statistics to disk. These are very\nlight threads and don't consume a lot of resources. Shouldn't be set higher\nthan the number of haproxy processes.\n\n* **queue-size** Defaults to **360**\n\nSuspend the pulling of statistics when the number of directories in **dst-dir**\nexceeds this limit.\n\nprocess section\n###############\n\n* **src-dir** Defaults **/var/lib/haproxystats/incoming**\n\n\nA directory to watch for changes. It should point to the same directory as\nthe **dst-dir** option from *pull* section.\n\n* **workers** Defaults to **4**\n\nNumber of workers to use for processing statistics. These are real processes\nwhich can consume a fair bit of CPU.\n\n* **frontend-metrics** Unset by default\n\nA list of frontend metric names separated by space to process. By default all\nstatistics are processed and this overwrites the default selection.\n\nhaproxystats-process emits an error and refuses to start if metrics aren't\nvalid HAProxy metrics. Check the list of valid metrics in Chapter 9.1 of\n`management`_ documentation of HAProxy.\n\n* **backend-metrics** Unset by default\n\nA list of backend metric names separated by space to process. By default all\nstatistics are processed and this overwrites the default selection.\n\nhaproxystats-process emits an error and refuses to start if metrics aren't\nvalid HAProxy metrics. Check the list of valid metrics in Chapter 9.1 of\n`management`_ documentation of HAProxy.\n\n* **server-metrics** Unset by default\n\nA list of server metric names separated by space to process. By default all\nstatistics are processed and this overwrites the default selection.\n\nhaproxystats-process emits an error and refuses to start if metrics aren't\nvalid HAProxy metrics. Check the list of valid metrics in Chapter 9.1 of\n`management`_ documentation of HAProxy.\n\n* **aggr-server-metrics** Defaults to **false**\n\nAggregates server's statistics across all backends.\n\n* **exclude-frontends** Unset by default\n\nA file which contains one frontend name per line for which processing is\nskipped.\n\n* **exclude-backends** Unset by default\n\nA file which contains one backend name per line for which processing is\nskipped.\n\n* **per-process-metrics** Defaults to **false**\n\nHAProxy daemon provides statistics and by default **haproxystat-process**\naggregates those statistics when HAProxy runs in multiprocess mode\n(nbproc > 1).\n\nSet this to **true** to get those statistics also per process as well.\nThis is quite useful for monitoring purposes where someone wants to monitor\nsessions per process in order to see if traffic is evenly distributed to all\nprocesses by the kernel.\n\nIt is also useful in setups where configuration for frontends and backends is\nunevenly spread across all processes, for instance processes 1-4 manage SSL\nfrontends and processes 5-7 manage noSSL frontends.\n\nThis adds another path in Graphite under haproxy space::\n\n    loadbalancers.lb-01.haproxy.daemon.process.<process_num>.<metric>\n\n* **calculate-percentages** Defaults to **false**\n\nCalculates percentages for a selection of metrics for HAProxy daemon. When\n**per-process-metrics** is set to **true** the calculation happens also per\nHAProxy process. This adds the following metric names::\n\n    ConnPercentage\n    ConnRatePercentage\n    SslRatePercentage\n    SslConnPercentage\n\nThose metrics can be used for alerting when the current usage on connections\nis very close the configured limit.\n\n* **liveness-check-interval** Defaults to **10** (seconds)\n\nHow often to check if all workers are alive and trigger a termination if at\nleast one is dead.\n\ngraphite section\n################\n\nThis dispatcher **is enabled** by default and it can't be disabled.\n\n* **server** Defaults to **127.0.0.1**\n\nGraphite server to connect to.\n\n* **port**  Defaults to **3002**\n\nGraphite port to connect to.\n\n* **retries** Defaults to **3**\n\nNumber of times to reconnect to Graphite after a failure.\n\n* **interval** Defaults to **1.8** (seconds)\n\nTime to wait before trying to reconnect to Graphite after a failure.\n\n* **connect-timeout** Defaults to **1** (seconds)\n\nTime to wait for establishing a connection to Graphite relay.\n\n* **write-timeout** Defaults to **1** (seconds)\n\nTime to wait on sending data to Graphite relay.\n\n* **delay** Defaults to **10** (seconds)\n\nHow long to wait before trying to connect again after number of retries has\nexceeded the threshold set in **retries**. During the delay period metrics are\nstored in the queue of the dispatcher, see **queue-size**.\n\n* **backoff** Defaults to **2**\n\nA simple exponential backoff to apply for each retry.\n\n* **namespace** Defaults to **loadbalancers**\n\nA top level graphite namespace.\n\n* **prefix-hostname** Defaults to **true**\n\nInsert the hostname of the load balancer in the Graphite namespace, example::\n\n    loadbalancers.lb-01.haproxy.\n\n* **fqdn** Defaults to **true**\n\nUse FQDN or short name in the graphite namespace\n\n* **queue-size**  Defaults to **1000000**\n\nhaproxystats-process uses a queue to store metrics which failed to be sent due\nto a connection error/timeout. This is a First In First Out queueing system.\nWhen the queue reaches the limit, the oldest items are removed to free space.\n\n* **group-namespace** Unset by default.\n\ngroup graphite metrics by patterns. When a frontend, backend or server matches a\ngiven pattern, the metric will be prefixed by this namespace, plus a\nconfigurable group name which must be specified in the **frontend-groups**,\n**backend-groups** or **server-groups** sections. These sections consist of\ngroup names and their corresponding regular expression that will be matched\nagainst frontend, backend or server names (depending on the section).\n\nFor example:\n\nLet's assume our metrics look something like::\n\n    loadbalancers.lb-01.haproxy.frontend.foo-001.<metric>\n    loadbalancers.lb-01.haproxy.frontend.foo-002.<metric>\n    ...\n    loadbalancers.lb-01.haproxy.frontend.bar-001.<metric>\n    loadbalancers.lb-01.haproxy.frontend.bar-002.<metric>\n    ...\n\nAnd we want them to be grouped to like this::\n\n    loadbalancers.lb-01.haproxy.flavor.abc.frontend.foo-001.<metric>\n    loadbalancers.lb-01.haproxy.flavor.abc.frontend.foo-002.<metric>\n    ...\n    loadbalancers.lb-01.haproxy.flavor.xyz.frontend.bar-001.<metric>\n    loadbalancers.lb-01.haproxy.flavor.xyz.frontend.bar-002.<metric>\n    ...\n\nThe configuration should contain these settings::\n\n    [graphite]\n    group-namespace = flavor\n\n    [frontend-groups]\n    abc = ^foo-\n    xyz = ^bar-\n\nNote that if the **group-namespace** setting is specified, then at least one of\n**frontend-groups**, **backend-groups** or **server-groups** sections must be\nspecified as well.\n\nAlso note that if frontend, backend or server names contain dots, these will be\nconverted to underscores for graphite -- because dots are graphite's namespace\nseparator. The patterns will have to take this into account.\n\n* **group-namespace-double-writes** Unset by default.\n\nBoolean; required only if **group-namespace** is specified. If True, send to\ngraphite the original metric as well as the grouped metrics. If False, send\nonly the grouped metrics. (See **group-namespace**.)\n\nfrontend-groups, backend-groups, and server-groups sections\n###########################################################\n\nSpecify the patterns to match against frontend, backend and/or server names, to\ngroup graphite metrics and give them a variable prefix. See **group-namespace**.\n\nThese sections are optional, unless **group-namespace** is set.\n\nlocal-store section\n###################\n\nThis dispatcher **isn't** enabled by default.\n\nThe primarily use of local-store dispatcher is to debug/troubleshoot possible\nproblems with the processing or/and with Graphite. There isn't any clean-up\nprocess in place, thus you need remove the files after they are created.\nDon't leave it enabled for more than 1 hour as it can easily fill up the disk\nin environments with hundreds frontends/backends and thousands servers.\n\n* **dir** Defaults to **/var/lib/haproxystats/local-store**\n\nA directory to stores statistics after they have been processed. The current\nformat is compatible with Graphite.\n\nSystemd integration\n-------------------\n\nhaproxystats-pull and haproxystats-process are simple programs which are not\ndaemonized and they output logging messages to stdout. This is by design as it\nsimplifies the code. The daemonization and logging is off-loaded to systemd\nwhich has everything we need for that job.\n\nUnder contrib/systend directory there are service files for both programs.\nThese are functional systemd Unit files which are used in production.\n\nThe order in which these 2 programs start doesn't matter and there isn't any\nsoft or hard dependency between them.\n\nFurthermore, these programs don't need to run as root. It highly recommended to\ncreate a dedicated user to run them. You need to add that user to the group of\n*haproxy* and adjust socket configuration of haproxy to allow write for the\ngroup, see below an example configuration::\n\n    stats socket /run/haproxy/sock1 user haproxy group haproxy mode 660 level admin process 1\n    stats socket /run/haproxy/sock2 user haproxy group haproxy mode 660 level admin process 2\n    stats socket /run/haproxy/sock3 user haproxy group haproxy mode 660 level admin process 3\n\nsystemd Unit files use haproxystats user which has to be created prior running\nhaproxystats programs.\n\nGraceful shutdown\n-----------------\n\nIn an effort to reduce the loss of statistics both programs support graceful\nshutdown. When *SIGHUP* or *SIGTERM* signals are sent they perform a clean exit.\nWhen a signal is sent to haproxystats-process it may take some time for the\nprogram to exit, as it waits for all workers to empty the queue.\n\nPuppet module\n-------------\n\nA puppet module is available under contrib directory which provides classes for\nconfiguring both programs.\n\nBecause haproxystats-process is CPU bound program, CPU Affinity is configured\nusing systemd. By default it pins the workers to the last CPUs.\n\nYou should take care of pinning haproxy processes to other CPUs in order to\navoid haproxystats-process *stealing* CPU cycles from haproxy. In production\nservers you usually pin the first 80% of CPUs to haproxy processes and you\nleave the rest of CPUs for other processes. The default template of puppet\nmodule enforces this logic.\n\nhaproxystats-pull is a single threaded program which doesn't use a lot of CPU\ncycles and by default is assigned to the last CPU.\n\nAnsible Playbook\n----------------\n\nA Ansible playbook is available under contrib directory. For installation\ninstruction of the playbook please read Installation chapter of this document.\n\nNagios checks\n-------------\n\nSeveral nagios checks are provided for monitoring purposes, they can be found\nunder contrib/nagios directory.\n\n* check_haproxystats_process_number_of_procs.sh\n\nMonitor the number of processes of haproxystats-process program. Systemd\nmonitors only the parent process and this check helps to detect cases where\nsome worker(s) die unexpectedly\n\n* check_haproxystats_process.sh\n\nA wrapper around systemctl tool to detect a dead parent process.\n\n* check_haproxystats_pull.sh\n\nA wrapper around systemctl tool to a check if haproxystats-pull is running.\n\n* check_haproxystats_queue_size.py\n\nChecks the size of the *incoming* directory queue which is consumed by\nhaproxystats-process and alert when exceeds a threshold.\n\n\nStarting the programs\n---------------------\n\n::\n\n    haproxystats-pull -f ./haproxystats.conf\n\n::\n\n    haproxystats-process -f ./haproxystats.conf\n\nUsage::\n\n    % haproxystats-pull -h\n    Pulls statistics from HAProxy daemon over UNIX socket(s)\n\n    Usage:\n        haproxystats-pull [-f <file>] [-p | -P]\n\n    Options:\n        -f, --file <file>  configuration file with settings\n                           [default: /etc/haproxystats.conf]\n        -p, --print        show default settings\n        -P, --print-conf   show configuration\n        -h, --help         show this screen\n        -v, --version      show version\n\n\n    % haproxystats-process -h\n    Processes statistics from HAProxy and pushes them to Graphite\n\n    Usage:\n        haproxystats-process [-f <file>] [-d <dir>] [-p | -P]\n\n    Options:\n        -f, --file <file>  configuration file with settings\n                           [default: /etc/haproxystats.conf]\n        -d, --dir <dir>    directory with additional configuration files\n        -p, --print        show default settings\n        -P, --print-conf   show configuration\n        -h, --help         show this screen\n        -v, --version      show version\n\n\nDevelopment\n-----------\nI would love to hear what other people think about **haproxystats** and provide\nfeedback. Please post your comments, bug reports and wishes on my `issues page\n<https://github.com/unixsurfer/haproxystats/issues>`_.\n\nHow to setup a development environment\n######################################\n\nInstall HAProxy::\n\n    % sudo apt-get install haproxy\n\nUse a basic HAProxy configuration in multiprocess mode::\n\n    global\n        log 127.0.0.1 len 2048 local2\n        chroot /var/lib/haproxy\n        stats socket /run/haproxy/admin1.sock mode 666 level admin process 1\n        stats socket /run/haproxy/admin2.sock mode 666 level admin process 2\n        stats socket /run/haproxy/admin3.sock mode 666 level admin process 3\n        stats socket /run/haproxy/admin4.sock mode 666 level admin process 4\n        # allow read/write access to anyone----------^\n        stats timeout 30s\n        user haproxy\n        group haproxy\n        daemon\n        nbproc 4\n        cpu-map 1 0\n        cpu-map 2 1\n        cpu-map 3 1\n        cpu-map 4 0\n\n    defaults\n        log global\n        mode    http\n        timeout connect 5000\n        timeout client  50000\n        timeout server  50000\n\n    frontend frontend_proc1\n        bind 0.0.0.0:81 process 1\n        default_backend backend_proc1\n\n    frontend frontend_proc2\n        bind 0.0.0.0:82 process 2\n        default_backend backend_proc1\n\n    frontend frontend1_proc34\n        bind :83 process 3\n        bind :83 process 4\n        default_backend backend1_proc34\n\n    backend backend_proc1\n        bind-process 1\n        default-server inter 1000s\n        option httpchk GET / HTTP/1.1\\r\\nHost:\\ .com\\r\\nUser-Agent:\\ HAProxy\n        server member1_proc1 10.189.224.169:80 weight 100 check fall 2 rise 3\n        server member2_proc1 10.196.70.109:80 weight 100 check fall 2 rise 3\n        server bck_all_srv1 10.196.70.109:88 weight 100 check fall 2 rise 3\n\n    backend backend1_proc34\n        bind-process 3,4\n        default-server inter 1000s\n        option httpchk GET / HTTP/1.1\\r\\nHost:\\ .com\\r\\nUser-Agent:\\ HAProxy\n        server bck1_proc34_srv1 10.196.70.109:80 check fall 2 inter 5s rise 3\n        server bck1_proc34_srv2 10.196.70.109:80 check fall 2 inter 5s rise 3\n        server bck_all_srv1 10.196.70.109:80 check fall 2 inter 5s rise 3\n\n    backend backend_proc2\n        bind-process 2\n        default-server inter 1000s\n        option httpchk GET / HTTP/1.1\\r\\nHost:\\ .com\\r\\nUser-Agent:\\ HAProxy\n        server bck_proc2_srv1_proc2 127.0.0.1:8001 check fall 2 inter 5s rise 3\n        server bck_proc2_srv2_proc2 127.0.0.1:8002 check fall 2 inter 5s rise 3\n        server bck_proc2_srv3_proc2 127.0.0.1:8003 check fall 2 inter 5s rise 3\n        server bck_proc2_srv4_proc2 127.0.0.1:8004 check fall 2 inter 5s rise 3\n\nStart HAProxy and check it is up::\n\n    sudo systemctl start haproxy.service;systemctl status -l haproxy.service\n\nCreate a python virtual environment using virtualenvwrapper tool::\n\n    mkvirtualenv --python=`which python3` haproxystats-dev\n\n**Do not** exit the *haproxystats-dev* virtual environment.\n\nClone the project, if you are planning to contribute then you should fork it on\nGitHub and clone that project instead::\n\n    mkdir ~/repo;cd ~/repo\n    git clone https://github.com/unixsurfer/haproxystats\n\nInstall necessary libraries::\n\n    cd haproxystats\n    pip install -U pbr setuptools\n    pip install -r ./requirements.txt\n\nStart a TCP server which acts a Graphite relay and listens on 127.0.0.1:39991::\n\n    python3 ./contrib/tcp_server.py\n\nInstall haproxystats::\n\n    python setup.py install\n\nCreate necessary directory structure::\n\n    mkdir -p ./var/var/lib/haproxystats\n    mkdir -p ./var/etc\n    mkdir -p ./var/etc/haproxystats.d\n\nAdjust the following configuration and save it in ./var/etc/haproxystats.conf::\n\n    [DEFAULT]\n    loglevel = debug\n    retries  = 2\n    timeout  = 1\n    interval = 2\n\n    [paths]\n    base-dir = /home/<username>/repo/haproxystats/var/var/lib/haproxystats\n\n    [pull]\n    socket-dir    = /run/haproxy\n    retries       = 1\n    timeout       = 0.1\n    interval      = 0.5\n    pull-timeout  = 10\n    pull-interval = 10\n    dst-dir       = ${paths:base-dir}/incoming\n    tmp-dst-dir   = ${paths:base-dir}/incoming.tmp\n    workers       = 8\n\n    [process]\n    src-dir               = ${paths:base-dir}/incoming\n    workers               = 2\n    calculate-percentages = true\n    per-process-metrics   = true\n\n    [graphite]\n    server          = 127.0.0.1\n    port            = 39991\n    retries         = 3\n    interval        = 0.8\n    timeout         = 0.9\n    delay           = 10\n    backoff         = 2\n    namespace       = loadbalancers\n    prefix_hostname = true\n    fqdn            = true\n    queue-size      = 1000\n\n    [local-store]\n    dir = ${paths:base-dir}/local-store\n\nStart haproxystats-pull and haproxystats-process on 2 different terminals::\n\n    haproxystats-pull -f var/etc/haproxystats.conf\n    haproxystats-process -f var/etc/haproxystats.conf\n\nExit from *haproxystats-dev* virtual environment::\n\n    deactivate\n\n**Start hacking and don't forget to make a Pull Request**\n\nInstallation\n------------\n\nUse pip::\n\n    pip install haproxystats\n\nFrom Source::\n\n   sudo python setup.py install\n\nBuild (source) RPMs::\n\n   python setup.py clean --all; python setup.py bdist_rpm\n\nBuild a source archive for manual installation::\n\n   python setup.py sdist\n\nUse Ansible Playbook:\n\nTo deploy haproxystats By Ansible Playbook go to contrib/ansible-playbook\ndirectory::\n\n   cd contrib/ansible-playbook\n\nThen enter your haproxy server IP address in hosts file::\n\n   vi hosts\n\nAfter that set information of your environment in variable file::\n\n   vi group_vars/all\n\nNow for run Ansible Playbook use this command::\n\n   ansible-playbook -i hosts main-playbook.yml\n\nWhen Ansible Playbook run successful completely, you can take control haproxystats-pull and haproxystats-process by systemd::\n\n   systemctl start haproxystats-pull.service\n\n   systemctl start haproxystats-process.service\n\n\nHow to make a release\n---------------------\n\n#. Bump version in haproxystats/__init__.py\n\n#. Commit above change with::\n\n      git commit -av -m'RELEASE 0.1.3 version'\n\n#. Create a signed tag, pbr will use this for the version number::\n\n      git tag -s 0.1.3 -m 'bump release'\n\n#. Create the source distribution archive (the archive will be placed in the\n   **dist** directory)::\n\n      python setup.py sdist\n\n#. pbr updates ChangeLog file and we want to squeeze this change to the\n   previous commit, thus run::\n\n      git commit -av --amend\n\n#. Move current tag to the last commit::\n\n      git tag -fs 0.1.3 -m 'bump release'\n\n#. Push changes::\n\n      git push;git push --tags\n\n#. Upload to Python Package Index::\n\n      twine upload -s -p  dist/*\n\n\nContributors\n------------\n\nThe following people have contributed to project with feedback and code reviews\n\n- K\u00e1roly Nagy https://github.com/charlesnagy\n\n- Dan Achim https://github.com/danakim\n\nLicensing\n---------\n\nApache 2.0\n\nAcknowledgement\n---------------\nThis program was originally developed for Booking.com.  With approval\nfrom Booking.com, the code was generalised and published as Open Source\non github, for which the author would like to express his gratitude.\n\nContacts\n--------\n\n**Project website**: https://github.com/unixsurfer/haproxystats\n\n**Author**: Pavlos Parissis <pavlos.parissis@gmail.com>\n\n.. _HAProxy: http://www.haproxy.org/\n.. _stats socket: http://cbonte.github.io/haproxy-dconv/1.6/management.html#9.2\n.. _carbon-c-relay: https://github.com/grobian/carbon-c-relay\n.. _Pandas: http://pandas.pydata.org/\n.. _asyncio: https://docs.python.org/3/library/asyncio.html\n.. _inotify: http://linux.die.net/man/7/inotify\n.. _stat: http://cbonte.github.io/haproxy-dconv/1.6/management.html#show%20stat\n.. _info: http://cbonte.github.io/haproxy-dconv/1.6/management.html#show%20info\n.. _pool of threads: https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor\n.. _INI: https://en.wikipedia.org/wiki/INI_file\n.. _carbon-c-relay: https://github.com/grobian/carbon-c-relay\n.. _management: http://cbonte.github.io/haproxy-dconv/1.6/management.html#9.1",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/unixsurfer/haproxystats",
        "keywords": "haproxystats haproxy stats collector statistics",
        "license": "Apache 2.0",
        "maintainer": "Pavlos Parissis",
        "maintainer_email": "pavlos.parissis@gmail.com",
        "name": "haproxystats",
        "package_url": "https://pypi.org/project/haproxystats/",
        "platform": "",
        "project_url": "https://pypi.org/project/haproxystats/",
        "project_urls": {
            "Homepage": "https://github.com/unixsurfer/haproxystats"
        },
        "release_url": "https://pypi.org/project/haproxystats/0.5.1/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "A HAProxy statistics collection program",
        "version": "0.5.1"
    },
    "last_serial": 5419714,
    "releases": {
        "0.3.10": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "572022bb9fb8b3f68907b05819a27529",
                    "sha256": "540dc00cb587612c3e1e435ad1f12e2a346060dffd789c4d73140f306a4febe3"
                },
                "downloads": -1,
                "filename": "haproxystats-0.3.10.tar.gz",
                "has_sig": true,
                "md5_digest": "572022bb9fb8b3f68907b05819a27529",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 120007,
                "upload_time": "2016-07-17T00:12:50",
                "url": "https://files.pythonhosted.org/packages/03/0a/cbad508bb80322a82d6857f55b8fc8ee387208540505fab54b8f855fd729/haproxystats-0.3.10.tar.gz"
            }
        ],
        "0.3.12": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "332067b2daacaa1b97553892cf28a1dd",
                    "sha256": "d37c96eab1ac5c16b6171cde011e57f3b4dc5497afa2fc1b66c0accd9ae335ba"
                },
                "downloads": -1,
                "filename": "haproxystats-0.3.12.tar.gz",
                "has_sig": true,
                "md5_digest": "332067b2daacaa1b97553892cf28a1dd",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 121393,
                "upload_time": "2016-09-28T23:21:50",
                "url": "https://files.pythonhosted.org/packages/c1/1f/d3e3fa0fed928da50c81c0189d9115c9b8f28a91909b6d70bbc018fa384a/haproxystats-0.3.12.tar.gz"
            }
        ],
        "0.3.14": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "89fb8db186beef226837f9c5281e428e",
                    "sha256": "6a06612a595f035391e835ccd9d26e47c86df573e994293a0b5021cd8bbf02f2"
                },
                "downloads": -1,
                "filename": "haproxystats-0.3.14.tar.gz",
                "has_sig": true,
                "md5_digest": "89fb8db186beef226837f9c5281e428e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 121753,
                "upload_time": "2016-10-30T14:48:24",
                "url": "https://files.pythonhosted.org/packages/ab/02/02b62e38ea02f1623183f0d7953941e6526ae94b6ac62feebd4f1f55405d/haproxystats-0.3.14.tar.gz"
            }
        ],
        "0.3.15": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d1c31cf58da7feb7a99934f77934918a",
                    "sha256": "7398ab86d644c505faf6b9133e2a432450114697c96e006ed0554df4e4f5e8ae"
                },
                "downloads": -1,
                "filename": "haproxystats-0.3.15.tar.gz",
                "has_sig": true,
                "md5_digest": "d1c31cf58da7feb7a99934f77934918a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 125352,
                "upload_time": "2017-10-04T20:48:39",
                "url": "https://files.pythonhosted.org/packages/16/35/9c9f3184064a490acea83971df91e109cfc56c009a1806257c428bf6b9a2/haproxystats-0.3.15.tar.gz"
            }
        ],
        "0.3.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bd127dffe899417e05b297228624234f",
                    "sha256": "87e067cf3dcda6adaccbee8deabe7f9c7e92f902946b112f4956ad49cb7ccdb8"
                },
                "downloads": -1,
                "filename": "haproxystats-0.3.8.tar.gz",
                "has_sig": true,
                "md5_digest": "bd127dffe899417e05b297228624234f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 119874,
                "upload_time": "2016-05-07T18:21:44",
                "url": "https://files.pythonhosted.org/packages/bf/82/ae644e92f9d0bb51858238c0ef8150e112f63940485cd1e9233fc32925db/haproxystats-0.3.8.tar.gz"
            }
        ],
        "0.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a5e4354491ee927ad11dc0effc047bc3",
                    "sha256": "fbfe5566516af1e07e20cd5f95aa5550fb32d2114540ed83cd049d341d04b807"
                },
                "downloads": -1,
                "filename": "haproxystats-0.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "a5e4354491ee927ad11dc0effc047bc3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 127046,
                "upload_time": "2018-10-05T09:31:07",
                "url": "https://files.pythonhosted.org/packages/3a/40/9e1b86191ea61c77419154f337e4c30d79b880dafe6fab2956f9b55327d7/haproxystats-0.4.0.tar.gz"
            }
        ],
        "0.4.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "58a6c17a60627dc3b23f18626601185b",
                    "sha256": "7956fa2bb5990fed544d4c3d0ce804e7272c6fcf291bfb0dba07d22e7f5bae27"
                },
                "downloads": -1,
                "filename": "haproxystats-0.4.1.tar.gz",
                "has_sig": false,
                "md5_digest": "58a6c17a60627dc3b23f18626601185b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 127294,
                "upload_time": "2018-11-20T21:23:39",
                "url": "https://files.pythonhosted.org/packages/50/38/5783584d3f885d3bc935374de0677cb10ccc8eb3da590fc070cfb4667397/haproxystats-0.4.1.tar.gz"
            }
        ],
        "0.4.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0e84d635d685de1e094e92d47c385923",
                    "sha256": "5df25b6994207e6e7927a4e384439328589a52c026f3f11dfd4e67a26fb5a961"
                },
                "downloads": -1,
                "filename": "haproxystats-0.4.2.tar.gz",
                "has_sig": false,
                "md5_digest": "0e84d635d685de1e094e92d47c385923",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 127307,
                "upload_time": "2018-11-22T21:37:57",
                "url": "https://files.pythonhosted.org/packages/31/3b/dcb34a27968a255424bf87d1dd8433c52cdabd25792a5362fd58d5f2524a/haproxystats-0.4.2.tar.gz"
            }
        ],
        "0.5.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b291c60e46d2cf1ce3919d7804df38a0",
                    "sha256": "17e9db6d6b6321360bde7f9c700f00746ece929010699fe66ee4de884fbc389c"
                },
                "downloads": -1,
                "filename": "haproxystats-0.5.1.tar.gz",
                "has_sig": false,
                "md5_digest": "b291c60e46d2cf1ce3919d7804df38a0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 131273,
                "upload_time": "2019-06-19T10:52:37",
                "url": "https://files.pythonhosted.org/packages/19/5d/2960fd0df575e2c131f6d3b16a9997d39ba5c1a3fd183a1ac51af8ad6299/haproxystats-0.5.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "b291c60e46d2cf1ce3919d7804df38a0",
                "sha256": "17e9db6d6b6321360bde7f9c700f00746ece929010699fe66ee4de884fbc389c"
            },
            "downloads": -1,
            "filename": "haproxystats-0.5.1.tar.gz",
            "has_sig": false,
            "md5_digest": "b291c60e46d2cf1ce3919d7804df38a0",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 131273,
            "upload_time": "2019-06-19T10:52:37",
            "url": "https://files.pythonhosted.org/packages/19/5d/2960fd0df575e2c131f6d3b16a9997d39ba5c1a3fd183a1ac51af8ad6299/haproxystats-0.5.1.tar.gz"
        }
    ]
}