{ "info": { "author": "UNKNOWN", "author_email": "UNKNOWN", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 2.7", "Topic :: System :: Monitoring" ], "description": "Qasino\n======\n\nQasino is a stats server that supports stats stored as tables in a\ndatabase for querying with SQL. Unlike a conventional database, stats\ndo not accumulate over time (which would make querying the stats more\ndifficult). Instead Qasino keeps only the most recent set of stats\nreported. The full flexiblity of SQL including joining, filtering and\naggregation are availble to analyze the current state of your network.\nStats reported from different systems but using the same table name\nare merged automatically into a single table. In addition schemas are\nautomatically created and updated based on the incoming updates\nrequiring zero maintanence.\n\nMany stats systems provide history for stats but lack an effective way\nto join, correlate or connect stats together. Qasino does not\n(directly) keep a history of stats but does make it really easy to\ncorrelate and cross reference stats together.\n\nSometimes you want to know the current state of your systems and that\nmight include more then just numerical data. Qasino excels at giving\nyou that information. You can report a richer set of data such as\ntext data, datetimes or multi-row relationships. For example, cpu\nusage and ops per second, or configuration files with the md5sum of\neach and current timestamp.\n\nQasino primarily is a server process. It exposes interfaces to\npublish tables to it using a JSON API via HTTP or ZeroMQ. There is a\nsimple CLI client to connect to Qasino and run queries. There is a\nsimple command line table publisher that can read input tables as CSV\nor JSON. And there is a more advanced publisher client meant to run\nas an agent that dynamically publishes CSV files.\n\nIn the future there will be integration with stat collectors like\nStatsd, Diamond and Graphite.\n\nCurrently Qasino is implemented in Python using the Twisted framework\nand HTTP and ZeroMQ transports with Sqlite for the backend data store.\n\nQasino was inspired by the monitoring system used at Akamai\nTechnologies called Query. More information can be found \n[here](http://www.akamai.com/dl/technical_publications/lisa_2010.pdf \"Keeping Track of 70,000+ Servers: The Akamai Query System\") \nand [here](http://www.akamai.com/dl/technical_publications/query_osr.pdf \"Scaling a Monitoring Infrastructure for the Akamai Network\").\nQasino provides similar functionality but for much smaller scale environments.\n\n##Installation\n\n###Git clone\n\nTo run qasino, you can clone this repo. You'll need to have the following\nPython libraries installed:\n\n- python-twisted\n- python-zmq\n- python-apsw\n- python-yaml\n- python-requests\n- python-txzmq\n- python-jinja2\n\nThe server and the client can be run right from the main directory.\nThey will find the libraries they need in the lib directory. To run the server:\n\n python bin/qasino_server.py\n\nConnect with the SQL client:\n\n python bin/qasino_sqlclient.py -Hqasino-server-hostname\n\nTo run the CSV publisher:\n\n python bin/qasino_csvpublisher.py --identity 1.2.3.4 --index-file index.csv\n\n###Docker\n\nAlternately, you can build qasino using [Docker](https://www.docker.com/).\nThis will let you deploy qasino in a Docker container. Simply call the Docker\nbuild command on the Dockerfile included in this repo:\n\n docker build -t=\"my-container-name\" /path/to/qasino/\n\nTo run the qasino server using Docker, call `docker run` on the container you\nbuilt. You need to use the `-P` flag (or set port mappings manually) in order\nto send requests to the qasino server. For example:\n\n docker run -P my-container-name /opt/qasino/bin/qasino_server.py\n\nYou can find the ports that Docker assigned to your qasino container using\n`docker ps`.\n\n###Pip (client only)\n\nYou can also install the qasino SQL client using `pip`. This will let you query \na qasino server from the command line (like running the \n`bin/qasino_sqlclient.py` program). This should work on most Linux / OS X \ncomputers:\n\n pip install qasino_client\n\nAfterwards, the command `qasinosql` will start a command line interface to a \nqasino server. `qasinosql` takes the same flags as `qasino_sqlclient.py`; in \nfact, the former is simply a wrapper around the latter. For example:\n\n qasinosql -H 1.2.3.4 -p 15598\n\n\n##Overview\n\nThe qasino server receives requests from clients to publish tabular\ndata. The data is added to a backend sqlite database. The data is\ncollected for a fixed period (default 30 seconds) after which it is\nsnapshotted. The snapshotted database becomes the data source for all\nincoming SQL requests until the next snapshot. For this reason all\npublishers need to publish updated stats every snapshot period (with\nthe exception of static tables or persistent tables which are\ndescribed below).\n\nTo orchestrate this process better the server publishes on a ZeroMQ\npub-sub channel the snapshot (aka generation) signal. This should\ntrigger all publishers to send table data. qasino_cvspublisher works\nthis way. It is by no means required though. In fact a simpler\napproach is just to have all publishers publish their data on an\ninterval that matches the generation interval.\n\n##Querying (SQL)\n\nQasino has a SQL interface. Since Qasino uses SQLite on the backend\nyou can refer to the [SQLite SQL documentation](http://www.sqlite.org/lang.html) \nfor SQL syntax details. Qasino can be queried with SQL using four different methods:\n\n###Web UI\n\nPoint your browser at Qasino and you'll get a basic web interface.\nThe default page shows all the tables that Qasino knows about. There\nare also tabs for describing tables and inputing custom SQL\nstatements.\n\n###Line receiver\n\nConnect to a qasino server on port 15000 for line based text only queries. You can \nsimply connect using telnet and send your query.\n\n $ telnet 1.2.3.4 15000\n Trying 1.2.3.4...\n Connected to 1.2.3.4.\n Escape character is '^]'.\n select * from qasino_server_info;\n generation_number generation_duration_s generation_start_epoch\n ================= ===================== ======================\n 1382105093 30 1382105123.1\n 1 rows returned\n\n###Python Client\n\nConnect using bin/qasino_sqlclient.py. This client uses ZeroMQ to\nsend JSON formated messages to the server. (It can also connect using\nHTTPS given the --use-https option but that will require the right\ncredentials to work).\n\n $ bin/qasino_sqlclient.py -H1.2.3.4\n Connecting to 1.2.3.4:15598.\n qasino> select * from qasino_server_info;\n generation_number generation_duration_s generation_start_epoch\n ================= ===================== ======================\n 1382119193 30 1382119223.1\n 1 rows returned\n qasino> \n\nIt uses a json message with the following simple format:\n\n {\n \"op\" : \"query\",\n \"sql\" : \"select * from qasino_server_info;\" \n }\n\n\n###HTTP Interface\n\nLastly you can connect with a simple HTTP request. The default HTTP\nport is 15597. These requests can also go to the SSL port 443 but\nwill require basic auth. There are a couple variations.\n\nFirst you can POST a JSON request:\n\n $ curl -X POST 'http://1.2.3.4:15597/request?op=query' -d '{ \"sql\" : \"select * from qasino_server_info;\" }'\n {\"table\": {\"rows\": [[\"1382119553\", \"30\", \"1382119583.1\"]], \"column_names\": [\"generation_number\", \"generation_duration_s\", \"generation_start_epoch\"]}, \"max_widths\": {\"1\": 21, \"0\": 17, \"2\": 22}, \"response_op\": \"result_table\", \"identity\": \"1.2.3.4\"}\n $\n\nOr you can make a GET request with 'sql' as a query string param (be sure to url-encode it):\n\n $ curl 'http://localhost:15597/request?op=query' --get --data-urlencode 'sql=select * from qasino_server_info;'\n {\"table\": {\"rows\": [[\"1382131545\", \"30\", \"1382131575.89\"]], \"column_names\": [\"generation_number\", \"generation_duration_s\", \"generation_start_epoch\"]}, \"max_widths\": {\"1\": 21, \"0\": 17, \"2\": 22}, \"response_op\": \"result_table\", \"identity\": \"1.2.3.4\"}\n\nOr make a GET request with the 'format=text' query string parameter to get a human readable rendering of the table. Note you can also put this url right in a browser.\n\n $ curl 'http://1.2.3.4:15597/request?op=query&format=text' --get --data-urlencode 'sql=select * from qasino_server_info;'\n generation_number generation_duration_s generation_start_epoch\n ================= ===================== ======================\n 1382131635 30 1382131665.89\n 1 rows returned\n $\n\n###Internal Tables\n\nQasino server automatically publishes the following internal tables:\n- qasino_server_info\n- qasino_server_tables\n- qasino_server_connections\n- qasino_server_views\n\nThe following commands are shortcuts to looking at these tables:\n- SHOW info;\n- SHOW tables;\n- SHOW connections;\n- SHOW views;\n\nThe schema for a table can be found with 'DESC ;' and the\ndefinition of a view can be found with 'DESC VIEW ;'\n\n\n##Publishing\n\nCurrently the only publishing clients officially implemented are a CSV\nfile publisher that is meant to run as an agent remotely and publishes\nCSV files as tables (qasino_csvpublisher.py) and a command line\nutility to publish one-offs from JSON or CSV input files (qasino_publish.py).\n\nAll publishers must specify an \"identity\" which is a unique identifier\nthat tells the server where the input table is coming from. A given\ntablename can only be reported from the same identity once per\nreporting cycle but different identities can report rows for the same\ntablename - the results will be merged together. \n\nA common paradigm is to make the identity the hostname or IP address\nof the machine reporting the rows. In addition it is suggested to\ninclude a column in the table that indicates the same thing (identity,\nhostname or IP address). In this way, two nifty things happen:\n\n1. All machines reporting rows for a common table (lets say 'table_foo') will be merged together.\n2. You can make queries that select by machine e.g. \"SELECT * FROM table_foo WHERE identity IN ('1.2.3.4', '5.6.7.8');\"\n\n###Schema Merging\n\nTypically all your publishers have the same schema for a given table\nbut if its different (perhaps if you are rolling out a new release\nthat adds columns to a table) the server will always add columns that\ndon't already exist in the table. The schema you will end up with\nwill be a union of all the columns. \n\nChanging types of an existing column is not recommended (there may be\nsome undefined behavior). Just add a new column with the different\ntype.\n\n###Types\n\nCSV input tables (see below) support the following types (and are converted into the types in the JSON list below):\n- string (also str)\n- ip (alias for string)\n- float\n- integer (also int)\n- ll (alias for integer)\n- time (alias for integer)\n\nJSON input tables (see below) support the following types (which are sqlite types):\n- integer\n- real \n- text\n\n###Qasino_csvpublisher.py\n\nqasino_csvpublisher.py takes an index file with a list of CSV files in\nit to publish and/or a index list file with a list of index files to\nprocess in the same way. It runs until killed and monitors the\nindexes and tables they refer to for changes. The data is continually\npublished to the server every cycle so that the CSV content is always\nreflected in tables on the server. The intent is so that applications\nor processes can simply drop properly formated CSV files into\nlocations and they will automatically get loaded and published to the\nserver.\n\n####Index File Format\n\nAn index file starts with a version number followed by one or more CSV tables to publish.\n\nEach line specifying a table to publish contains either a filename and tablename or just a tablename where the filename is inferred.\n\nSo either:\n\n ,\n\nor just\n\n \n\nIn the latter case the filename is inferred to be `.csv`.\n\nSo for example you might have an index file `myindex.csv` like the following:\n\n 1\n myapplication_table1\n myapplication_table2\n\n####CSV File Format\n\nThe CSV files contain the following format:\n\n \n \n \n \n \n \n \n ...\n\nSo for example you might create a file `myapplication_table1.csv`:\n\n 1\n myapplication_table1\n ipaddr,datacenter,stat1,stat2\n ip,string,integer,integer\n IP Address,Datacenter,Num Frobs,Num Foobs\n 1.2.3.4,BOS,123,456\n\nAnd then you might run the csv publisher like this:\n\n python bin/qasino_csvpublisher.py --index myindex.csv --identity 1.2.3.4\n\n###Qasino_publish.py\n\nThe command line utility qasino_publish.py reads from file or stdin an\ninput table (CSV or JSON) and sends it via HTTP, HTTPS or ZeroMQ to a\nserver. Its largely meant as an example client but could be used in a\ncron or script. See --help for more information. See above for CSV file format.\n\n###HTTP Publishing\n\nYou can publish by sending a properly formatted JSON request via an HTTP connection.\n\nFor example to publish the same \"myapplication_table1\" table you could put the following in a file `myapplication_table1.json`:\n\n { \"op\" : \"add_table_data\",\n \"identity\" : \"1.2.3.4\",\n \"table\" : { \"tablename\" : \"myapplication_table1\",\n \"column_names\" : [ \"ipaddr\", \"datacenter\", \"stat1\", \"stat2\" ],\n \"column_types\" : [ \"ip\", \"string\", \"int\", \"int\" ],\n \"rows\" : [ [ \"1.2.3.4\", \"BOS\", 123, 456 ] ]\n }\n }\n\nAnd then send the following curl:\n\n $ curl -d @myapplication_table1.json -X POST 'http://1.2.3.4:15597/request?op=add_table_data'\n {\"identity\": \"1.2.3.4\", \"response_op\": \"ok\"}\n $\n\nThe table should appear in Qasino. Publishing would have to happen regularly for it to persist.\n\n $ bin/qasino_sqlclient.py\n Connecting to 1.2.3.4:15598.\n qasino> select * from qasino_server_tables where tablename = 'myapplication_table1';\n tablename\n ====================\n myapplication_table1\n 1 rows returned\n qasino> select * from myapplication_table1;\n ipaddr datacenter stat1 stat2\n ======= ========== ===== =====\n 1.2.3.4 BOS 123 456\n 1 rows returned\n qasino>\n\n###Persistent tables\n\nA node can publish a table with the option 'persist' to indicate that\nthe table should be carried through each generation. An option is\ngiven in the top level dict of the JSON object. For example:\n\n { \"op\" : \"add_table_data\",\n \"persist\" : 1,\n \"identity\" : \"1.2.3.4\",\n \"table\" : { \"tablename\" : \"myapplication_table1\",\n \"column_names\" : [ \"ipaddr\", \"datacenter\", \"stat1\", \"stat2\" ],\n \"column_types\" : [ \"ip\", \"string\", \"int\", \"int\" ],\n \"rows\" : [ [ \"1.2.3.4\", \"BOS\", 123, 456 ] ]\n }\n }\n\nSome things to note.\n\n- The stats are carried forward for each successive generation so that means the server has to hold onto an extra copy of the stats which can consume additional memory.\n- If the server is restarted the persistent stats will go away until they are resent.\n- The tables are tracked per tablename (at the moment), so multiple updates for the same table overwrite.\n\n###Static tables\n\nStatic tables are set using the \"static\" option similar to persist\nabove but get loaded into a special persistent Sqlite DB that is\nconnected to the ephemeral databases. Tables that are static persist\nbetween Qasino server restarts. They are also stored by tablename so\nmultiple updates to the same table overwrite.\n\n###Views\n\nViews are supported by using a view configuration file. The Qasino\nserver will look by default (change it with the --views-file option)\nfor a file in the current directory called 'views.conf' that is a YAML\nfile that has the following format:\n\n ---\n \n - viewname: testview\n view: | \n create view testview as select \n * from qasino_server_info;\n\n - viewname: anotherview\n view: | \n create view anotherview as select \n * from qasino_server_tables;\n\nIt is an array of items with 'viewname' and 'view' properties. The\n'view' property specifies the actual view to create. \n\nThe views file is monitored for changes and automatically reloaded.\n\nYou can get the definition of a view from the qasino_server_views\ntable or the 'DESC VIEW' command.", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/MediaMath/qasino", "keywords": "qasino stats monitoring mediamath", "license": "Apache 2.0", "maintainer": null, "maintainer_email": null, "name": "qasino_client", "package_url": "https://pypi.org/project/qasino_client/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/qasino_client/", "project_urls": { "Download": "UNKNOWN", "Homepage": "https://github.com/MediaMath/qasino" }, "release_url": "https://pypi.org/project/qasino_client/1.6.2/", "requires_dist": null, "requires_python": null, "summary": "A command line client for the qasino project.", "version": "1.6.2" }, "last_serial": 1393700, "releases": { "1.6.0": [ { "comment_text": "", "digests": { "md5": "dd13008ffa332e2dc00250113371064e", "sha256": "f4cbc9ea20c7c7d38fdef696d9e2d23b487dd9ebbbc8b15bc026fafcc2e92a23" }, "downloads": -1, "filename": "qasino_client-1.6.0.tar.gz", "has_sig": false, "md5_digest": "dd13008ffa332e2dc00250113371064e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 37554, "upload_time": "2015-01-22T18:23:20", "url": "https://files.pythonhosted.org/packages/36/d2/edc2e570a19c779db99aab70faa48c65a1a0f2fb48593df2fa93643c6530/qasino_client-1.6.0.tar.gz" } ], "1.6.1": [ { "comment_text": "", "digests": { "md5": "19b3addb9b5b736fbef249910a15c35b", "sha256": "dec55e022430a4f7fc78ffbe449c5a1ed60c02fa3464ce33ad52a013648c8d16" }, "downloads": -1, "filename": "qasino_client-1.6.1.tar.gz", "has_sig": false, "md5_digest": "19b3addb9b5b736fbef249910a15c35b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 37583, "upload_time": "2015-01-22T21:48:44", "url": "https://files.pythonhosted.org/packages/36/70/4ab73740347a71254b33b574c9291e59b69fb0c441b299639c87489d4b75/qasino_client-1.6.1.tar.gz" } ], "1.6.2": [ { "comment_text": "", "digests": { "md5": "43347dae224f9bd700a52cafb90647f5", "sha256": "22e0a2a79ad23a29b50e0ffb0efca2a189c5f6ffee880e39407ade1d9430bd0a" }, "downloads": -1, "filename": "qasino_client-1.6.2.tar.gz", "has_sig": false, "md5_digest": "43347dae224f9bd700a52cafb90647f5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 43120, "upload_time": "2015-01-23T17:38:08", "url": "https://files.pythonhosted.org/packages/96/96/7fab3e763bbf6074d306a2252bbb39f5c10414784f7ffacad26f6752f459/qasino_client-1.6.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "43347dae224f9bd700a52cafb90647f5", "sha256": "22e0a2a79ad23a29b50e0ffb0efca2a189c5f6ffee880e39407ade1d9430bd0a" }, "downloads": -1, "filename": "qasino_client-1.6.2.tar.gz", "has_sig": false, "md5_digest": "43347dae224f9bd700a52cafb90647f5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 43120, "upload_time": "2015-01-23T17:38:08", "url": "https://files.pythonhosted.org/packages/96/96/7fab3e763bbf6074d306a2252bbb39f5c10414784f7ffacad26f6752f459/qasino_client-1.6.2.tar.gz" } ] }