{ "info": { "author": "TradeForecaster Global Markets, LLC", "author_email": "github@tradeforecaster.com", "bugtrack_url": null, "classifiers": [ "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.6" ], "description": "[![Build Status](https://travis-ci.org/tfgm/sbedecoder.svg?branch=master)](https://travis-ci.org/tfgm/sbedecoder)\n\nPython based Simple Binary Encoding (SBE) decoder\n=================================================\n\nOverview\n--------\n\nsbedecoder is a simple python package for parsing SBE encoded data. \n\nsbedecoder dynamically generates an SBE parser from an xml description of the format. This is accomplished by\ncreating an instance of `SBESchema()` and calling it's `parse()` method with a file name:\n\n from sbedecoder import SBESchema\n schema = SBESchema()\n schema.parse('path/to/schema.xml')\n\nThe `SBESchema()` can be initialized with `include_message_size_header=True` if the messages being parsed\nrequire an extra 2 byte (unit16) framing message_size_header field (i.e. for CME MDP 3.0).\n\nBy default, message names are derived from the \"name\" field of the message definition in the schema.\nIn some cases (i.e. CME MDP 3.0), the message \"description\" field of the message definition provides a\nmore friendly name for the message. To use message descriptions as the name of the message,\ninitialize your SBESchema with `use_description_as_message_name=True`.\n\nFor convenience, an `MDPSchema()` subclass of `SBESchema()` is provided with `include_message_size_header=True`\nand `use_description_as_message_name=True` specifically to handle CME Group MDP 3.0 schema's.\n\nMessages are parsed from any structure that looks like a buffer containing the raw binary\ndata (buffer, str, bytearay, etc). To parse SBE encoded data into a message based on a\nschema instance, just call `SBEMessage.parse_message()`:\n\n from sbedecoder import SBEMessage\n message = SBEMessage.parse_message(schema, msg_buffer, offset=0)\n\n`offset` is an optional parameter that indicates where within the msg_buffer the message\nstarts (including the size header if the schema has `include_message_size_header` set).\n\nA parsed message is represented as an instance of the `SBEMessage()` class. `SBEMessages()` are\ncomprised of zero or more `sbedecoder.message.SBEField()` instances and zero or more\n`sbedecoder.message.SBERepeatingGroup()` instances. An `SBEField()` object can be one of a primitive\n`TypeMessageField()`, a `SetMessageField()` or an `EnumMessageField()`\n\n**Note:** Unless using code generation, you cannot store the messages for later processing.\nYou must process the messages on each iteration, because the messages re-use instances of\nfield objects, wrapping them around new values.\n\nThe CME Group sends MDP 3.0 messages in packets that include a 4 byte sequence number\nand a 8 byte timestamp. In addition, there can be multiple messages in a single packet\nand each message is framed the with a 2 byte (unit16) message size field as mentioned above.\n\nTo parse these messages, you can create a `MDPSchema()`, use that to create a\n`MDPMessageFactory()` and then create a `SBEParser()` which can then iterate over the messages in\na packet like this:\n\n from sbedecoder import MDPSchema\n from sbedecoder import MDPMessageFactory\n from sbedecoder import SBEParser\n\n schema = SBESchema()\n schema.parse('path/to/schema.xml')\n message_factory = MDPMessageFactory(schema)\n message_parser = SBEParser(message_factory)\n\n for packet in SOME_DATASOURCE:\n for message in message_parser.parse(packet, offset=12):\n process(message)\n\n\nThis \"Message Factory\" concept could easily be extended to new framing schemes by creating a new sub class of `SBEMessageFactory()`\n\nFor more information on SBE, see: http://www.fixtradingcommunity.org/pg/structure/tech-specs/simple-binary-encoding.\n\nInstall\n-------\n\nThe sbedecoder project is available on PyPI:\n\n pip install sbedecoder\n\nIf you are installing from source:\n\n python setup.py install\n\n**Note**: The SBE decoder has only been tested with python 2.7 and 3.6. On Windows, we typically use the \nAnaconda python distribution. Anaconda does not distribute python's test code. If you have \nissues with dpkt (ImportError: No module named test), you can either install the latest dpkt \nfrom source (https://github.com/kbandla/dpkt) or just comment out the import (from test import \npystone) in ..\\\\Anaconda\\\\lib\\\\site-packages\\\\dpkt\\\\decorators.py. Newer versions of dpkt no \nlonger have this dependency.\n\n\nmdp_decoder.py\n--------------\n\nmdp_decoder.py serves as an example of using the sbedecoder package. It is a full decoder for processing CME Group\nMDP 3.0 (MDP3) messages from a pcap file. For help with using mdp_decoder.py:\n\n mdp_decoder.py --help\n\nAn SBE template for CME Group MDP 3.0 market data can be found at \nftp://ftp.cmegroup.com/SBEFix/Production/Templates/templates_FixBinary.xml\n\nExample output:\n\n :packet - timestamp: 2015-06-25 09:45:01.924492 sequence_number: 93696727 sending_time: 1435243501924423666 \n ::MDIncrementalRefreshVolume - transact_time: 1435243501923350056 match_event_indicator: LastVolumeMsg (2)\n :::no_md_entries - num_groups: 1\n ::::md_entry_size: 4483 security_id: 559884 rpt_seq: 2666379 md_update_action: New (0) md_entry_type: e \n ::MDIncrementalRefreshBook - transact_time: 1435243501923350056 match_event_indicator: LastQuoteMsg (4)\n :::no_md_entries - num_groups: 2\n ::::md_entry_px: 18792.0 ({'mantissa': 187920000000, 'exponent': -7}) md_entry_size: 1 security_id: 559884 rpt_seq: 2666380 number_of_orders: 1 md_price_level: 1 md_update_action: Delete (2) md_entry_type: Bid (0) \n ::::md_entry_px: 18746.0 ({'mantissa': 187460000000, 'exponent': -7}) md_entry_size: 6 security_id: 559884 rpt_seq: 2666381 number_of_orders: 1 md_price_level: 10 md_update_action: New (0) md_entry_type: Bid (0) \n\nExample output (with `--pretty`):\n\n\n```\npacket - timestamp: 2016-03-10 15:33:21.301819 sequence_number: 76643046 sending_time: 1454679022595400091\n Message 1 of 2: TID 32 (MDIncrementalRefreshBook) v6\n TransactTime (60): 02/05/2016 07:30:22.595256135 (1454679022595256135)\n MatchEventIndicator (5799): LastQuoteMsg\n NoMDEntries (268): 1\n Entry 1\n MDEntryPx (270): 98890000000 (9889.0)\n MDEntrySize (271): 296\n SecurityID (48): 807004\n RptSeq (83): 14273794\n NumberOfOrders (346): 16\n MDPriceLevel (1023): 2\n MDUpdateAction (279): Change\n MDEntryType (269): Offer\n Message 2 of 2: TID 32 (MDIncrementalRefreshBook) v6\n TransactTime (60): 02/05/2016 07:30:22.595256135 (1454679022595256135)\n MatchEventIndicator (5799): LastImpliedMsg, EndOfEvent\n NoMDEntries (268): 8\n Entry 1\n MDEntryPx (270): 475000000 (47.5)\n MDEntrySize (271): 296\n SecurityID (48): 817777\n RptSeq (83): 1573080\n NumberOfOrders (346): Null\n MDPriceLevel (1023): 2\n MDUpdateAction (279): Change\n MDEntryType (269): ImpliedBid\n Entry 2...\n```\n\nmdp_book_builder.py\n-------------------\n\nmdp_book_builder.py serves as an example of using the sbedecoder package to build limit orderbooks for a given contract.\n\nFor help with using mdp_book_builder.py:\n\n mdp_book_builder.py --help\n\nVersioning\n----------\n\nsbedecoder supports the `sinceVersion` attribute of fields, enumerants, groups, ..., etc, and so it can\ndecode older (e.g. archived) binary data so long as the schema has evolved correctly to maintain support\nfor the old format\n\nPerformance\n-----------\n\nsbedecoder itself isn't optimized for performance however it can be adequate for simple backtesting scenarios amd \npost trade analytics. Due to the amount of printing done by mdp_decoder.py, it can be quite slow to parse large \npcap files.\n\nPyPy\n----\n\nFor improved performance (4 to 5x), sbedecoder will run under PyPy. Assuming your pypy install is in /opt:\n\n /opt/pypy/bin/pip install lxml\n /opt/pypy/bin/pip install dpkt\n /opt/pypy/bin/pypy setup.py install\n\nCode Generation\n---------------\n\nA SBE class generator script is available to generate a python file that contains the class definitions that match those\nthat are created dynamically via the SBESchema.parse method.\n\nFor help with using sbe_class_generator.py:\n\n sbe_class_generator.py --help\n\nAn usage would be (from the generator directory):\n\n/sbe_class_generator.py --schema schema.xml --output generated.py --template ./sbe_message.tmpl\n\nThis command will output a file called generated.py containing the class definitions that were dynamically created\nwhile parsing the 'schema.xml' file. The template file used to generated the classes is contained in sbe_message.tmpl.\n\nThe generated.py file can simply be used for examining the class construction, or it can replace the contents of the\ngenerated.py file in the sbedecoder core project. By replacing the generated.py file in the sbedecoder package, a\ndeveloper will get access to the class definitions in the IDE.\n\nIn order to make use of the standard parser functionality using the generated code one should use the SBESchema.load\nmethod instead of the parse method.\n\nAn example of how to do this is below and is contained in the mdp_book_builder.py script:\n\n try:\n from sbedecoder.generated import __messages__ as generated_messages\n mdp_schema.load(generated_messages)\n except:\n mdp_schema.parse(args.schema)\n\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/tfgm/sbedecoder", "keywords": "sbe mdp3 orderbook message decoder", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "sbedecoder", "package_url": "https://pypi.org/project/sbedecoder/", "platform": "", "project_url": "https://pypi.org/project/sbedecoder/", "project_urls": { "Homepage": "https://github.com/tfgm/sbedecoder" }, "release_url": "https://pypi.org/project/sbedecoder/0.1.10/", "requires_dist": [ "dpkt", "lxml", "six" ], "requires_python": "", "summary": "Simple Binary Encoding (SBE) decoder (handles CME MDP3 messages)", "version": "0.1.10" }, "last_serial": 4796747, "releases": { "0.1.10": [ { "comment_text": "", "digests": { "md5": "7aaa6ea0462bc19221c25944ae63916d", "sha256": "d509f8bded6cf25b0b550e8db0e9d4ab5a19be478f820f6a7c764dd9c27fc1ec" }, "downloads": -1, "filename": "sbedecoder-0.1.10-py2-none-any.whl", "has_sig": false, "md5_digest": "7aaa6ea0462bc19221c25944ae63916d", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 23514, "upload_time": "2019-02-08T18:08:06", "url": "https://files.pythonhosted.org/packages/fe/0c/7c2a6a6c6ac80d1f324631fb4994020b63c2cebaf9e55e1d6ed81239af43/sbedecoder-0.1.10-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c1dd2dd0ffaf45aae9d5d4098aa4d54e", "sha256": "789faf6624819509dbdccbdf3dc851b541178924fa7abce183046d759b18aa5a" }, "downloads": -1, "filename": "sbedecoder-0.1.10-py3-none-any.whl", "has_sig": false, "md5_digest": "c1dd2dd0ffaf45aae9d5d4098aa4d54e", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 23512, "upload_time": "2019-02-08T18:08:08", "url": "https://files.pythonhosted.org/packages/4a/09/3d3a304e6f5f0c214b462471c88e00abbe91dc37bfbc14cd721540d58bd8/sbedecoder-0.1.10-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "bc817b34d43e40e549135dc62c3925de", "sha256": "14c4fe942360eea23153df95ca0ef092b01d71a283e0ef5596effb7f78ecc9f4" }, "downloads": -1, "filename": "sbedecoder-0.1.10.tar.gz", "has_sig": false, "md5_digest": "bc817b34d43e40e549135dc62c3925de", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18765, "upload_time": "2019-02-08T18:08:09", "url": "https://files.pythonhosted.org/packages/78/92/dbee408aa9434d56d74e2d2384b7e0a829fbdc539b8146d6e37f84987d47/sbedecoder-0.1.10.tar.gz" } ], "0.1.5": [ { "comment_text": "", "digests": { "md5": "3ae416dabef3aa4bb9a222c9a57cc88f", "sha256": "f1f5417ccd52b8bc237ed956efd500a488e20c2f6e21860c88c72a31644f302e" }, "downloads": -1, "filename": "sbedecoder-0.1.5-py2-none-any.whl", "has_sig": false, "md5_digest": "3ae416dabef3aa4bb9a222c9a57cc88f", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 25550, "upload_time": "2018-04-27T14:48:45", "url": "https://files.pythonhosted.org/packages/f8/b9/5c8dde55ab0e8d329eee55d153c7154491a145360757483f2c4a8567c889/sbedecoder-0.1.5-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "33ffe8065e0a7d489f832b2fe72d6054", "sha256": "d28fab64d74195845cfa8531c0bed1e2875e86cbc6b9208d34523469948c39bc" }, "downloads": -1, "filename": "sbedecoder-0.1.5.tar.gz", "has_sig": false, "md5_digest": "33ffe8065e0a7d489f832b2fe72d6054", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 21648, "upload_time": "2018-04-27T14:48:47", "url": "https://files.pythonhosted.org/packages/f0/3d/67a0b545568c2b2c09a386ca5ddf27b0d364dda4c9d00d7af7a9de9c874c/sbedecoder-0.1.5.tar.gz" } ], "0.1.7": [ { "comment_text": "", "digests": { "md5": "6a6278abbbc83e7e7ce8bac833bef796", "sha256": "7d39f07b923897da45b2bbf47acb88dbb31d23ea8b7b9cbf2cad2afa29e66fda" }, "downloads": -1, "filename": "sbedecoder-0.1.7-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "6a6278abbbc83e7e7ce8bac833bef796", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 25775, "upload_time": "2018-07-31T19:47:30", "url": "https://files.pythonhosted.org/packages/85/0e/257640a2e49f1a4c55f7852187e9e1c3b82ffe1262df1bd7affbaa6a5b54/sbedecoder-0.1.7-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "f5043df1180614c6650f04afdc9d6104", "sha256": "81c99e0e570fa192c491e22bf267399c0dc53d40c3a429c524d2393f30197d21" }, "downloads": -1, "filename": "sbedecoder-0.1.7.tar.gz", "has_sig": false, "md5_digest": "f5043df1180614c6650f04afdc9d6104", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 21790, "upload_time": "2018-07-31T19:47:31", "url": "https://files.pythonhosted.org/packages/1d/28/5b1e1fcfd0dc4716d8874088828950ddba60477135a172d8e468348989cd/sbedecoder-0.1.7.tar.gz" } ], "0.1.8": [ { "comment_text": "", "digests": { "md5": "4e2be537d04452c4dd78a0356cc1cf57", "sha256": "faa16135dd59bb10504db86e7b1d5b37d91104eaee40dcacd6635c1b4603e68a" }, "downloads": -1, "filename": "sbedecoder-0.1.8-py3-none-any.whl", "has_sig": false, "md5_digest": "4e2be537d04452c4dd78a0356cc1cf57", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 25770, "upload_time": "2018-08-09T19:36:36", "url": "https://files.pythonhosted.org/packages/a8/77/60116922055d15c96479b351217ebbc1d2a5f611bca50f787353b3e010a4/sbedecoder-0.1.8-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "05302d630535468d660a7c689f3b3284", "sha256": "7c2ccb9bd665f414b0a630d43058704b94fdd63980f88f841691cdc63e7ac4c3" }, "downloads": -1, "filename": "sbedecoder-0.1.8.tar.gz", "has_sig": false, "md5_digest": "05302d630535468d660a7c689f3b3284", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 21807, "upload_time": "2018-08-09T19:36:37", "url": "https://files.pythonhosted.org/packages/8e/e5/fe181a61c8e0fa4696b36f46879bea1518cdede08c84c007c73e35a9ee7c/sbedecoder-0.1.8.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "7aaa6ea0462bc19221c25944ae63916d", "sha256": "d509f8bded6cf25b0b550e8db0e9d4ab5a19be478f820f6a7c764dd9c27fc1ec" }, "downloads": -1, "filename": "sbedecoder-0.1.10-py2-none-any.whl", "has_sig": false, "md5_digest": "7aaa6ea0462bc19221c25944ae63916d", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 23514, "upload_time": "2019-02-08T18:08:06", "url": "https://files.pythonhosted.org/packages/fe/0c/7c2a6a6c6ac80d1f324631fb4994020b63c2cebaf9e55e1d6ed81239af43/sbedecoder-0.1.10-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c1dd2dd0ffaf45aae9d5d4098aa4d54e", "sha256": "789faf6624819509dbdccbdf3dc851b541178924fa7abce183046d759b18aa5a" }, "downloads": -1, "filename": "sbedecoder-0.1.10-py3-none-any.whl", "has_sig": false, "md5_digest": "c1dd2dd0ffaf45aae9d5d4098aa4d54e", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 23512, "upload_time": "2019-02-08T18:08:08", "url": "https://files.pythonhosted.org/packages/4a/09/3d3a304e6f5f0c214b462471c88e00abbe91dc37bfbc14cd721540d58bd8/sbedecoder-0.1.10-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "bc817b34d43e40e549135dc62c3925de", "sha256": "14c4fe942360eea23153df95ca0ef092b01d71a283e0ef5596effb7f78ecc9f4" }, "downloads": -1, "filename": "sbedecoder-0.1.10.tar.gz", "has_sig": false, "md5_digest": "bc817b34d43e40e549135dc62c3925de", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18765, "upload_time": "2019-02-08T18:08:09", "url": "https://files.pythonhosted.org/packages/78/92/dbee408aa9434d56d74e2d2384b7e0a829fbdc539b8146d6e37f84987d47/sbedecoder-0.1.10.tar.gz" } ] }