{ "info": { "author": "Gavin M. Roy", "author_email": "gavinmroy@gmail.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "License :: OSI Approved :: BSD License", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy" ], "description": "Mikkoo\n======\nA `PgQ `_ to\n`RabbitMQ `_ relay. Mikkoo is a PgQ consumer that\nthat publishes to RabbitMQ. In addition, it includes a built in auditing\nsystem that can be used to confirm that all PgQ events are received by\nRabbitMQ.\n\nMikkoo is named for the rabbit in the \"Clever Rabbit and the Elephant\" fable.\n\n|Version| |Downloads| |Status|\n\nInstallation\n------------\nMikkoo is available on the `Python Package Index `_\nand can be installed via `pip`:\n\n.. code:: bash\n\n pip install mikkoo\n\nOnce you've setup `Skytools `_ you may want to\ninstall the optional included utility functions in `mikkoo.sql `_ to make usage\neasier.\n\nYou can do this with a combination of ``curl`` and ``psql``:\n\n.. code:: bash\n\n curl -L https://github.com/gmr/mikkoo/blob/master/mikkoo.sql | psql\n\nThis will install multiple stored procedures and an audit table in a mikkoo schema.\nTake a look at the DDL to get a good idea of what each funciton is and how it can\nbe used.\n\nPgQ Setup\n---------\n\n1. Install ``pgq`` into your database and create the queue:\n\n .. code:: sql\n\n # CREATE EXTENSION pgq;\n CREATE EXTENSION\n # SELECT pgq.create_queue('test');\n create_queue\n --------------\n 1\n (1 row)\n\n2. Ensure that `pgqd `_\n is running.\n\nPgQ Event to AMQP Mapping\n-------------------------\nWhen inserting events into a PgQ queue, the ``pgq.insert_event/7`` function\nshould be used with the following field mappings:\n\n+---------------+-----------------------+\n| PgQ Event | AMQP |\n+===============+=======================+\n| ``ev_type`` | Routing Key |\n+---------------+-----------------------+\n| ``ev_data`` | Message body |\n+---------------+-----------------------+\n| ``ev_extra1`` | Exchange |\n+---------------+-----------------------+\n| ``ev_extra2`` | Content-Type Property |\n+---------------+-----------------------+\n| ``ev_extra3`` | AMQP Properties [1]_ |\n+---------------+-----------------------+\n| ``ev_extra4`` | Headers [2]_ |\n+---------------+-----------------------+\n| ``ev_time`` | Headers.timetamp |\n+---------------+-----------------------+\n| ``ev_txid`` | Headers.txid |\n+---------------+-----------------------+\n\n.. [1] AMQP properties should be set as a JSON blob. Values set in the ``ev_extra3``\n field will overwrite the automatically created properties ``app_id``,\n ``content_type``, ``correlation_id``, ``headers``, and ``timestamp``.\n.. [2] If ``ev_extra4`` is specified and is a JSON key/value dictionary, it will\n be assigned to the ``headers`` AMQP property.\n\nThere is a convenience schema in the `mikkoo.sql `_ file that adds\nstored procedures for creating properly formatted mikkoo events in PgQ. In\naddition, there is are auditing functions that allow for the creation of an\naudit-log of events that were sent to PgQ.\n\n\nAMQP Message Properties\n^^^^^^^^^^^^^^^^^^^^^^^\nThe following table defines the available fields that can be set in a JSON blob\nin the ``ev_extra3`` field when inserting an event.\n\n+----------------------+----------------+\n| Property | PgSQL Type |\n+======================+================+\n| ``app_id`` | text |\n+----------------------+----------------+\n| ``content_encoding`` | text |\n+----------------------+----------------+\n| ``content_type`` | text |\n+----------------------+----------------+\n| ``correlation_id`` | text |\n+----------------------+----------------+\n| ``delivery_mode`` | int2 |\n+----------------------+----------------+\n| ``expiration`` | text |\n+----------------------+----------------+\n| ``message_id`` | text |\n+----------------------+----------------+\n| ``headers`` | text/json [3]_ |\n+----------------------+----------------+\n| ``timestamp`` | int4 |\n+----------------------+----------------+\n| ``type`` | text |\n+----------------------+----------------+\n| ``priority`` | int4 |\n+----------------------+----------------+\n| ``user_id`` | text |\n+----------------------+----------------+\n\n.. [3] ``headers`` should be sent to a key/value JSON blob if specified\n\nValues assigned in the JSON blob provided to ``ev_extra3`` take precedence over\nthe automatically assigned ``app_id``, ``content_type``, ``correlation_id``,\n``headers``, and ``timestamp`` values created by Mikkoo at processing time.\n\nAs of 1.0, Mikkoo will automatically add four AMQP headers property values. These\nvalues will not overwrite any values with the same name specified in ``ev_extra4``.\noverwriting values with the same name, even if they are specified in the\nThe ``sequence`` value is a dynamically generated ID that attempts to provide\nfuzzy distributed ordering information. The ``timestamp`` value is the ISO-8601\nrepresentation of the ``ev_time`` field, which is created when an event is added\nto PgQ. The ``txid`` value is the ``ev_txid`` value, the PgQ transaction ID for\nthe event. These values are added to help provide some level of deterministic\nordering. The ``origin`` value is the hostname of the server that Mikkoo is running\non.\n\nEvent Insertion Example\n-----------------------\n\nThe following example inserts a JSON blob message body of ``{\"foo\": \"bar\"}`` that\nwill be published to the ``postgres`` exchange in RabbitMQ using the ``test.routing-key``\nrouting key. The content type is specified in ``ev_extra2`` and the AMQP ``type``\nmessage property is specified in ``ev_extra3``.\n\n.. code:: sql\n\n # SELECT pgq.insert_event('test', 'test.routing-key', '{\"foo\": \"bar\"}', 'postgres', 'application/json', '{\"type\": \"example\"}', '');\n insert_event\n --------------\n 4\n (1 row)\n\nWhen this message is received by RabbitMQ it will have a message body of:\n\n.. code:: json\n\n {\"foo\": \"bar\"}\n\nAnd it will have message properties similar to the following:\n\n+----------------------+-------------------------------------------------------+\n| Property | Example Value |\n+======================+=======================================================+\n| ``app_id`` | ``mikkoo`` |\n+----------------------+-------------------------------------------------------+\n| ``content_type`` | ``application/json`` |\n+----------------------+-------------------------------------------------------+\n| ``correlation_id`` | ``0ad6b212-4c84-4eb0-8782-9a44bdfe949f`` |\n+----------------------+-------------------------------------------------------+\n| ``headers`` | +---------------+-----------------------------------+ |\n| | | Key | Example Value | |\n| | +===============+===================================+ |\n| | | ``origin`` | ``mikkoo.domain.com`` | |\n| | +---------------+-----------------------------------+ |\n| | | ``sequence`` | ``4539586784185129828`` | |\n| | +---------------+-----------------------------------+ |\n| | | ``timestamp`` | ``2017-02-23 06:17:14.471682-00`` | |\n| | +---------------+-----------------------------------+ |\n| | | ``txid`` | ``41356335556`` | |\n| | +---------------+-----------------------------------+ |\n+----------------------+-------------------------------------------------------+\n| ``timestamp`` | ``1449600290`` |\n+----------------------+-------------------------------------------------------+\n| ``type`` | ``example`` |\n+----------------------+-------------------------------------------------------+\n\nConfiguration\n-------------\nThe Mikkoo configuration file uses `YAML `_ for markup and allows\nfor one or more PgQ queue to be processed.\n\nIf you have a Sentry or a Sentry account, the ``Application/sentry_dsn`` setting\nwill turn on sentry exception logging, if the\n`raven `_ client library is installed.\n\nQueues are configured by name under the ``Application/workers`` stanza. The\nfollowing example configures two workers for the processing of a queue named\n``invoices``. Each worker process connects to a local PostgreSQL and RabbitMQ\ninstance using default credentials.\n\n.. code:: yaml\n\n Application:\n workers:\n invoices:\n postgres_url: postgresql://localhost:5432/postgres\n rabbitmq_url: amqp://localhost:5672/%2f\n confirm: False\n\nQueue Configuration Options\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\nThe following table details the configuration options available per queue:\n\n+--------------------+---------------------------------------------------------------------+\n| Key | Description |\n+====================+=====================================================================+\n| ``confirm`` | Enable/Disable RabbitMQ Publisher Confirmations. Default: ``True`` |\n+--------------------+---------------------------------------------------------------------+\n| ``consumer_name`` | Overwrite the default PgQ consumer name. Default: ``mikkoo`` |\n+--------------------+---------------------------------------------------------------------+\n| ``max_failures`` | Maximum failures before discarding an event. Default: ``10`` |\n+--------------------+---------------------------------------------------------------------+\n| ``postgresql_url`` | The url for connecting to PostgreSQL |\n+--------------------+---------------------------------------------------------------------+\n| ``rabbitmq_url`` | The AMQP url for connecting to RabbitMQ |\n+--------------------+---------------------------------------------------------------------+\n| ``retry_delay`` | How long in seconds until PgQ emits failed events. Default: ``10`` |\n+--------------------+---------------------------------------------------------------------+\n| ``unregister`` | Unregister a consumer with PgQ on shutdown. Default: ``True`` |\n+--------------------+---------------------------------------------------------------------+\n| ``wait_duration`` | How long to wait before checking the queue after the last empty |\n| | result. Default: ``1`` |\n+--------------------+---------------------------------------------------------------------+\n\nExample Configuration\n^^^^^^^^^^^^^^^^^^^^^\n\nThe following is an example of a full configuration file:\n\n.. code:: yaml\n\n Application:\n\n poll_interval: 10\n sentry_dsn: [YOUR SENTRY DSN]\n\n statsd:\n enabled: true\n host: localhost\n port: 8125\n\n workers:\n test:\n confirm: False\n consumer_name: my_consumer\n max_failures: 5\n postgres_url: postgresql://localhost:5432/postgres\n rabbitmq_url: amqp://localhost:5672/%2f\n retry_delay: 5\n unregister: False\n wait_duration: 5\n\n Daemon:\n user: mikkoo\n pidfile: /var/run/mikkoo\n\n Logging:\n version: 1\n formatters:\n verbose:\n format: '%(levelname) -10s %(asctime)s %(process)-6d %(processName) -20s %(name) -18s: %(message)s'\n datefmt: '%Y-%m-%d %H:%M:%S'\n handlers:\n console:\n class: logging.StreamHandler\n formatter: verbose\n debug_only: True\n loggers:\n helper:\n handlers: [console]\n level: INFO\n propagate: true\n mikkoo:\n handlers: [console]\n level: INFO\n propagate: true\n pika:\n handlers: [console]\n level: ERROR\n propagate: true\n queries:\n handlers: [console]\n level: ERROR\n propagate: true\n tornado:\n handlers: [console]\n level: ERROR\n propagate: true\n root:\n handlers: [console]\n level: CRITICAL\n propagate: true\n disable_existing_loggers: true\n incremental: false\n\nRunning Mikkoo\n--------------\nAfter creating a configuration file for Mikkoo like the one above, simply run the mikkoo application providing the path to the configuration file:\n\n.. code:: bash\n\n mikkoo -c mikkoo.yml\n\nThe application will attempt to daemonize unless you use the ``-f`` foreground CLI switch.\n\nMikkoo's CLI help can be invoked with ``--help`` and yields the following output:\n\n.. code:: bash\n\n $ mikkoo -h\n usage: mikkoo [-h] [-c CONFIG] [-f]\n\n Mikkoo is a PgQ to RabbitMQ Relay\n\n optional arguments:\n -h, --help show this help message and exit\n -c CONFIG, --config CONFIG\n Path to the configuration file\n -f, --foreground Run the application interactively\n\n\n.. |Version| image:: https://img.shields.io/pypi/v/mikkoo.svg?\n :target: http://badge.fury.io/py/mikkoo\n\n.. |Status| image:: https://img.shields.io/travis/gmr/mikkoo.svg?\n :target: https://travis-ci.org/gmr/mikkoo\n\n.. |Downloads| image:: https://img.shields.io/pypi/dm/mikkoo.svg?\n :target: https://pypi.python.org/pypi/mikkoo", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/gmr/mikkoo", "keywords": "amqp rabbitmq postgresql", "license": "BSD", "maintainer": "", "maintainer_email": "", "name": "mikkoo", "package_url": "https://pypi.org/project/mikkoo/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/mikkoo/", "project_urls": { "Homepage": "https://github.com/gmr/mikkoo" }, "release_url": "https://pypi.org/project/mikkoo/1.0.0/", "requires_dist": null, "requires_python": "", "summary": "Mikkoo is a PgQ to RabbitMQ Relay", "version": "1.0.0" }, "last_serial": 2663963, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "bcbf02df84822ffebac135fc027836c1", "sha256": "5f821e8f8c1736a4506da87c57ddbe58a139a43a98f49bcba8a613da7ca5028c" }, "downloads": -1, "filename": "mikkoo-0.1.0.tar.gz", "has_sig": false, "md5_digest": "bcbf02df84822ffebac135fc027836c1", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17479, "upload_time": "2015-12-10T02:54:39", "url": "https://files.pythonhosted.org/packages/c1/f4/d7ff8b083369ec0f62d30d10e211346341925c99db1e9c53fcfe8b955c09/mikkoo-0.1.0.tar.gz" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "04042d3b7fb9c6677933f0e73fbebed1", "sha256": "998319677caae2845e394b3814e55eeeb847aefd5edb5362e8768c0a09929804" }, "downloads": -1, "filename": "mikkoo-0.2.0.tar.gz", "has_sig": false, "md5_digest": "04042d3b7fb9c6677933f0e73fbebed1", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17537, "upload_time": "2015-12-23T21:05:59", "url": "https://files.pythonhosted.org/packages/42/da/ac8ebe0b74aefbc906ab0aad001263cbda80c4e5142da3bb246516db2a8b/mikkoo-0.2.0.tar.gz" } ], "0.3.0": [ { "comment_text": "", "digests": { "md5": "97dc1150988b2490592ab35f50b6bd3f", "sha256": "695f8dc517ba9c776587a8f5ba1b5a7832a5b2bd7827f1a5eea6153dac5ae806" }, "downloads": -1, "filename": "mikkoo-0.3.0.tar.gz", "has_sig": false, "md5_digest": "97dc1150988b2490592ab35f50b6bd3f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17610, "upload_time": "2015-12-23T22:10:28", "url": "https://files.pythonhosted.org/packages/2c/4e/21e2c7a496d30cb85f341db25652c54ff523173f88d5156d896dad12a2ff/mikkoo-0.3.0.tar.gz" } ], "0.4.0": [ { "comment_text": "", "digests": { "md5": "e1ba06114f3f165a39ad4ebddf35b3d5", "sha256": "0aa6711dbfd28cbfcfac6d50373e3a74d79c2e582e1a8ddd2a0d65a88cf249ae" }, "downloads": -1, "filename": "mikkoo-0.4.0.tar.gz", "has_sig": false, "md5_digest": "e1ba06114f3f165a39ad4ebddf35b3d5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18972, "upload_time": "2015-12-24T01:21:00", "url": "https://files.pythonhosted.org/packages/97/d6/3014cec13499f4ea312dba69cc1cfa437a141de51dfa74685eaeebd47343/mikkoo-0.4.0.tar.gz" } ], "0.4.1": [ { "comment_text": "", "digests": { "md5": "c885dbea0ddfe9aa07cb1aac9dbb488d", "sha256": "d1a726459462b2156c4459ddee94a33416961436673da2f1ab06f931d6bee566" }, "downloads": -1, "filename": "mikkoo-0.4.1.tar.gz", "has_sig": false, "md5_digest": "c885dbea0ddfe9aa07cb1aac9dbb488d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18990, "upload_time": "2015-12-24T01:58:09", "url": "https://files.pythonhosted.org/packages/f1/51/06a2208fabbec3a4a751f7a66189a1c378f9e9434673ba0b26a99493e014/mikkoo-0.4.1.tar.gz" } ], "0.4.2": [ { "comment_text": "", "digests": { "md5": "3418d866528ef8863018919b3ee3be9e", "sha256": "af4530916729ba6fa4ece9ab54ad7e3eab2bfae675f81e2cc86bfcee8118c3e1" }, "downloads": -1, "filename": "mikkoo-0.4.2.tar.gz", "has_sig": false, "md5_digest": "3418d866528ef8863018919b3ee3be9e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18882, "upload_time": "2015-12-31T21:44:05", "url": "https://files.pythonhosted.org/packages/7c/a2/a469b10f307d9c6435fc177a71c0cc6f656decb7765b2e54a4773131311b/mikkoo-0.4.2.tar.gz" } ], "0.4.3": [ { "comment_text": "", "digests": { "md5": "9c88cbd119c84a3fa874500f98ac7d81", "sha256": "6a0bd1d67a1ba68ed7802ededa14dcc68f0a0077d86d5f219beed6c329142239" }, "downloads": -1, "filename": "mikkoo-0.4.3.tar.gz", "has_sig": false, "md5_digest": "9c88cbd119c84a3fa874500f98ac7d81", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18929, "upload_time": "2015-12-31T21:47:19", "url": "https://files.pythonhosted.org/packages/65/90/d0551c6eb2f132f0e50d69d999feaf6083aa8e2e6f0856594dc92ba27b63/mikkoo-0.4.3.tar.gz" } ], "0.4.4": [ { "comment_text": "", "digests": { "md5": "a28adabcea23d53e9fc93d4ff4be6787", "sha256": "2dc28b37087f334b1a76967332497a30ec1f42ba6486d82cd2d214e4acbcd7d1" }, "downloads": -1, "filename": "mikkoo-0.4.4.tar.gz", "has_sig": false, "md5_digest": "a28adabcea23d53e9fc93d4ff4be6787", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 19838, "upload_time": "2016-01-22T18:35:10", "url": "https://files.pythonhosted.org/packages/59/e7/ca891ca85fa5517d4d34330b31ca614a1f6fabd58baae22da2da6ea8e686/mikkoo-0.4.4.tar.gz" } ], "0.4.5": [ { "comment_text": "", "digests": { "md5": "aaa04ee192c68395f522565b338b1aea", "sha256": "22a068bbdec41a5a516d76302f1b61aa79e9ff768a60f113dca23007817659cf" }, "downloads": -1, "filename": "mikkoo-0.4.5.tar.gz", "has_sig": false, "md5_digest": "aaa04ee192c68395f522565b338b1aea", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 19836, "upload_time": "2016-10-06T16:49:12", "url": "https://files.pythonhosted.org/packages/43/6d/e0fa91ac9027103af56269f04299ea851586c2585e0c51519357d858862b/mikkoo-0.4.5.tar.gz" } ], "1.0.0": [ { "comment_text": "", "digests": { "md5": "8f13a1a38fc9b6533dff9d23a3def899", "sha256": "759c867272494938ac502a724bcf676e8a8641e5acab42c1a31d937cb3bcf6ff" }, "downloads": -1, "filename": "mikkoo-1.0.0.tar.gz", "has_sig": false, "md5_digest": "8f13a1a38fc9b6533dff9d23a3def899", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 20501, "upload_time": "2017-02-23T18:13:26", "url": "https://files.pythonhosted.org/packages/6a/17/7384492be98fc4692604674395e0a6bbbb81368d7b8d3a895550ba21a893/mikkoo-1.0.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "8f13a1a38fc9b6533dff9d23a3def899", "sha256": "759c867272494938ac502a724bcf676e8a8641e5acab42c1a31d937cb3bcf6ff" }, "downloads": -1, "filename": "mikkoo-1.0.0.tar.gz", "has_sig": false, "md5_digest": "8f13a1a38fc9b6533dff9d23a3def899", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 20501, "upload_time": "2017-02-23T18:13:26", "url": "https://files.pythonhosted.org/packages/6a/17/7384492be98fc4692604674395e0a6bbbb81368d7b8d3a895550ba21a893/mikkoo-1.0.0.tar.gz" } ] }