{
    "info": {
        "author": "Monte M. Goode",
        "author_email": "MMGoode@lbl.gov",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Environment :: Console",
            "Intended Audience :: Developers",
            "Intended Audience :: Telecommunications Industry",
            "License :: OSI Approved :: BSD License",
            "Operating System :: POSIX",
            "Programming Language :: Python :: 2",
            "Programming Language :: Python :: 3",
            "Topic :: Internet",
            "Topic :: Software Development :: Libraries",
            "Topic :: System :: Networking"
        ],
        "description": "Tstat (TCP STatistic and Analysis Tool) Transport Tools\n=======================================================\n\nClient programs to read a tstat file hierarchy, parse the logs, convert\nthem to JSON objects, and send them to a remote archive (like a RabbitMQ\nserver). This tool was originally developed to support the `National\nScience Foundation NetSage Project <http://www.netsage.global/home>`__.\n\nOverview\n--------\n\ntstat\\_send\n~~~~~~~~~~~\n\nWhen invoked, it crawls a tstat file hierarchy laid out like this:\n\n::\n\n    tstat/2016_02_17_12_53.out:\n    total 808\n    -rw-r--r--@ root users  135103 Feb 17 13:54 log_tcp_complete\n    -rw-r--r--@ root users  277154 Feb 17 13:54 log_udp_complete\n\n    tstat/2016_02_17_13_54.out:\n    total 792\n    -rw-r--r--@ root users  123801 Feb 17 14:54 log_tcp_complete\n    -rw-r--r--@ root users  278301 Feb 17 14:54 log_udp_complete\n\nSelected data are extracted from the logs, formatted into JSON objects,\nand lists of the objects are sent to a remote server for archiving. The\nobjects from a single set of logs log are broken into a series of\nsmaller lists (up to 100 objects per list). Each \"sub-list\" gets sent to\nthe remote server so no one single send operation swamps the remote\nserver.\n\nWhen the logs in each directory have been successfully processed (the\ndata have been sent, delivery confirmations received, etc), a dotfile\nnamed ``.processed`` will be dropped in that directory. That marks that\ndirectory as processed, and those logs will be ignored on subsequent\nruns. The ``tstat_cull`` utility similarly uses the .processed dotfiles\nto prune old logs.\n\nIt is not a persistent process and would be run periodically from cron\n(for example) to periodically process logs on a \"live\" machine.\n\nCurrently, the only \"transport\" that is supported is sending the JSON to\na RabbitMQ server, but it would be relatively straightforward to\nimplement other transports like using HTTP to send to a REST API.\n\nUsage\n-----\n\ntstat\\_send arguments\n~~~~~~~~~~~~~~~~~~~~~\n\nRequired\n^^^^^^^^\n\n--directory\n'''''''''''\n\nPath to the \"root\" of the directory structure where tstat writes the\ntimestamped directories and logfiles. No default.\n\n--config\n''''''''\n\nPath to the .ini style config file used to pass configuration directives\nto the underlying transport code.\n\nDefault: ``./config.ini``\n\nOptional\n^^^^^^^^\n\n--threshold\n'''''''''''\n\nThe transfer threshold in megabytes. Any transfer below this threshold\nbelow will be ignored.\n\nDefault: ``1000``\n\n--transport\n'''''''''''\n\nSpecify the underlying transport to send the JSON over. Currently, only\nRabbitMQ is supported.\n\nDefault: ``rabbit``\n\n--sensor\n''''''''\n\nThe sensor\\_id element of the message metadata defaults to\n``socket.gethostname()`` - using this flag will set that value manually.\n\n--instance\n''''''''''\n\nThe instance\\_id element of the message metadata defaults to 0 - using\nthis flag will set that value manually. This makes it possible to\ndifferentiate between instances on sensors that have multiple tstat\nstreams.\n\n--single\n''''''''\n\nProcess a single \"timestamped directory\" of files, send JSON and exit.\nThis is primarily for development or debugging.\n\n--no-transport\n''''''''''''''\n\nSkips sending the messages to the selected transport and dumps them to\nstandard out instead. Use standard shell redirection\n``... --no-transport > file.json`` to save output to a file.\n\n--verbose and --debug\n'''''''''''''''''''''\n\n``--verbose`` triggers additional log output. ``--debug`` changes the\nlog level to ``logging.DEBUG`` in the transport module. This is\nprimarily for debugging connection problems with RabbitMQ, or to get\ndetailed output on the transactions with the remote server.\n\nConfig file\n-----------\n\n``tstat_send`` uses an .ini style configuration file to pass options to\nthe underlying transport code. Doing this with command-line args would\nbe too ponderous. Example config file:\n\n::\n\n    [rabbit]\n    # host/port are required for all transport variants\n    host = localhost\n    # this is the rabbit ssl port, if not, use default 5672 or custom port\n    port = 5671\n    # these are required for some transports\n    username = esnet\n    password = some_mysterious_password\n    use_ssl = True\n    # these are rabbit specific - the exchange key is required\n    # even if you don't set the value to anything/use the default \"\".\n    vhost = netsage\n    queue = netsage_tstat\n    routing_key = netsage_tstat\n    exchange =\n\n    # This is an optional stanza. The key/value pairs\n    # will be passed to channel.queue_declare() as kwargs\n    # (ie: durable, exclusive, auto_delete, etc).\n    [rabbit_queue_options]\n\n    # This is an optional stanza. The key/value pairs\n    # will generate a dict to be passed as kwargs to ssl.wrap_socket()\n    # https://docs.python.org/2/library/ssl.html#ssl.wrap_socket\n    [ssl_options]\n\n-  The values ``host`` and ``port`` will be required for all transport\n   variants. If they are not supplied, a configuration error occur.\n-  The rabbit transport requires the ``username`` and ``password``\n   config values. They may also be enabled in other transport variants.\n-  ``vhost, queue, routing_key and exchange`` should be self-explanatory\n   RabbitMQ directives.\n-  The ``rabbit_queue_options`` stanza is optional and can be used to\n   pass additional kwargs to ``queue_declare()`` if need be. By default\n   the code only passes the ``queue`` argument with the name of the\n   queue.\n-  The ``ssl_options`` stanza is optional too. Only necessary if\n   additional args (paths to keyfiles, etc) need to be passed to the\n   underlying ``ssl`` library.\n\nMessage format\n--------------\n\nEvery log line may generate zero, one or two JSON objects. This depends\non the threshold set with the ``--bits`` flag and what kind of transfer\nit is. The generated objects will be sub-divided into a series of lists\nof up to 100 objects each. That way, each send operation is of a\nmanageable size rather than sending one huge list.\n\nAll numeric values are being converted to \"actual\" numeric types. All\nfloating point values are being rounded to three decimal places to avoid\nsmall values being rendered in scientific notation. Timestamps logged in\nfloating point ms are being converted to integer epoch seconds.\n\nUDP logs\n~~~~~~~~\n\nThis is the most basic/common format.\n\n::\n\n    {\n        \"interval\": 600,\n        \"values\": {\n            \"duration\": 0.0,\n            \"num_bits\": 544,\n            \"num_packets\": 1\n        },\n        \"meta\": {\n            \"src_ip\": \"198.129.77.102\",\n            \"src_port\": 123,\n            \"dst_ip\": \"198.124.252.130\",\n            \"dst_port\": 123,\n            \"protocol\": \"udp\"\n        },\n        \"start\": 1455745857,\n        \"end\": 1455745857\n    },\n\nTCP logs\n~~~~~~~~\n\nThe TCP logs are identical, but have additional values in the ``values``\nstanza:\n\n::\n\n    {\n        \"interval\": 600,\n        \"values\": {\n            \"duration\": 191.796,\n            \"num_bits\": 22120,\n            \"num_packets\": 14,\n            \"tcp_rexmit_bytes\": 0,\n            \"tcp_rexmit_pkts\": 0,\n            \"tcp_rtt_avg\": 4.442,\n            \"tcp_rtt_min\": 0.007,\n            \"tcp_rtt_max\": 39.094,\n            \"tcp_rtt_std\": 10.648,\n            \"tcp_pkts_rto\": 0,\n            \"tcp_pkts_fs\": 0,\n            \"tcp_pkts_reor\": 0,\n            \"tcp_pkts_dup\": 0,\n            \"tcp_pkts_unk\": 0,\n            \"tcp_pkts_fc\": 0,\n            \"tcp_pkts_unrto\": 0,\n            \"tcp_pkts_unfs\": 0,\n            \"tcp_cwin_min\": 16,\n            \"tcp_cwin_max\": 960,\n            \"tcp_out_seq_pkts\": 0,\n            \"tcp_window_scale\": 7,\n            \"tcp_mss\": 1460,\n            \"tcp_max_seg_size\": 960,\n            \"tcp_min_seg_size\": 16,\n            \"tcp_win_max\": 960,\n            \"tcp_win_min\": 16,\n            \"tcp_initial_cwin\": 21\n        },\n        \"meta\": {\n            \"src_ip\": \"198.128.14.246\",\n            \"src_port\": 58635,\n            \"dst_ip\": \"198.129.77.102\",\n            \"dst_port\": 22,\n            \"protocol\": \"tcp\"\n        },\n        \"start\": 1455698490,\n        \"end\": 1455698490\n    },\n\nUtility programs\n----------------\n\ntstat\\_cull\n~~~~~~~~~~~\n\nCrawls a tstat directory to cull processed logs from disk. Tstat can\ngenerate a lot of output, this cleans up processed files, while leaving\na configurable \"buffer\" of time to keep processed logs. The default is\ntoo keep 2 days worth of processed logs on disk, rather than deleting\nthem right after they have been processed.\n\nThis script checks the ``mtime`` of the ``.processed`` state file in a\ndirectory of processed logs. If it is older than the ``--ttl`` time to\nlive in hours (default: 48), the directory and logs are removed.\n\nRequired args\n^^^^^^^^^^^^^\n\n--directory\n'''''''''''\n\nThis is the path to the \"root\" of the directory structure where tstat\nwrites the timestamped directories and logfiles. No default.\n\nOptional args\n^^^^^^^^^^^^^\n\n--ttl\n'''''\n\nThe time to live in hours. Set this if you don't want to use the default\nof 48 hours.\n\n--dry-run\n'''''''''\n\nDo a dry run. Just log the directories that will be deleted but don't\ndelete them.\n\nExtending tstat\\_send with additional transports\n------------------------------------------------\n\nAdding additional transports is fairly straightforward.\n\nSee doc/extending.md in the source distribution.",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/esnet/tstat-transport",
        "keywords": null,
        "license": "UNKNOWN",
        "maintainer": null,
        "maintainer_email": null,
        "name": "tstat_transport",
        "package_url": "https://pypi.org/project/tstat_transport/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/tstat_transport/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "https://github.com/esnet/tstat-transport"
        },
        "release_url": "https://pypi.org/project/tstat_transport/0.6.5/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "Tools to send Tstat (TCP STatistic and Analysis Tool) log data to archive servers.",
        "version": "0.6.5"
    },
    "last_serial": 3050739,
    "releases": {
        "0.6.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "08f8ae07fd73af8671ae602287d1fa50",
                    "sha256": "2f76d8a9cd188a6913fd4afe11a3f2c483a43323ef7203d2aa002291fbe4d577"
                },
                "downloads": -1,
                "filename": "tstat_transport-0.6.3.tar.gz",
                "has_sig": false,
                "md5_digest": "08f8ae07fd73af8671ae602287d1fa50",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 20998,
                "upload_time": "2017-01-17T15:09:29",
                "url": "https://files.pythonhosted.org/packages/6a/83/7b342a44b833148f380eeb56c8de3befa1a1d4b2af205f7543c90a89b81c/tstat_transport-0.6.3.tar.gz"
            }
        ],
        "0.6.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6dd6978e98709d739366c30b083cc4fd",
                    "sha256": "645da56b0d6bbf12846760acb24d25aa4b264820c0e04c398b40b808ddba2942"
                },
                "downloads": -1,
                "filename": "tstat_transport-0.6.4.tar.gz",
                "has_sig": false,
                "md5_digest": "6dd6978e98709d739366c30b083cc4fd",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 21009,
                "upload_time": "2017-04-25T20:51:24",
                "url": "https://files.pythonhosted.org/packages/e6/dc/837232ae7dadc0b69a6dfd767b96089f03580dee5b5ca10d8f3f79bff1b6/tstat_transport-0.6.4.tar.gz"
            }
        ],
        "0.6.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "eeae6e5047377fc35baac971509b8687",
                    "sha256": "fc9e000c24b8239fa902a78072ce32f0f0769b362575c76f8bfa9c5260c01228"
                },
                "downloads": -1,
                "filename": "tstat_transport-0.6.5.tar.gz",
                "has_sig": false,
                "md5_digest": "eeae6e5047377fc35baac971509b8687",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 21305,
                "upload_time": "2017-06-22T13:54:31",
                "url": "https://files.pythonhosted.org/packages/e2/d1/2c36d5f4b1b572c015019673067f1f7f1360e8d19de5bf645d3c0b6b739b/tstat_transport-0.6.5.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "eeae6e5047377fc35baac971509b8687",
                "sha256": "fc9e000c24b8239fa902a78072ce32f0f0769b362575c76f8bfa9c5260c01228"
            },
            "downloads": -1,
            "filename": "tstat_transport-0.6.5.tar.gz",
            "has_sig": false,
            "md5_digest": "eeae6e5047377fc35baac971509b8687",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 21305,
            "upload_time": "2017-06-22T13:54:31",
            "url": "https://files.pythonhosted.org/packages/e2/d1/2c36d5f4b1b572c015019673067f1f7f1360e8d19de5bf645d3c0b6b739b/tstat_transport-0.6.5.tar.gz"
        }
    ]
}