{ "info": { "author": "Leo Noordergraaf", "author_email": "leo@noordergraaf.net", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Environment :: Other Environment", "Intended Audience :: Developers", "License :: OSI Approved :: GNU General Public License v3 (GPLv3)", "Operating System :: OS Independent", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3.4", "Topic :: System :: Logging", "Topic :: Utilities" ], "description": "Distlog\n=======\n\nIs a Python library and a set of applications to facilitate logging in a\ndistributed application.\n\nThe reason to create yet another logging tool arose while developing an\napplication based on (micro) services. Even though each component produces a\nliberal amount of logging data making sense of it is challenging at times.\n\nThis is because each log file only tells part of the story.\nThe challenge is to find those parts of the log entries that are relevant to a\nparticular problem request. I tried to solve this by using correlation IDs.\nThese IDs are assigned when a request first enters the system and is passed on\nto each service that is invoked to satisfy that request. It sort of works but it\ntakes a lot of time piecing all the bits together.\n\nHence the desire to let the system sort it out itself.\n\nRequirements\n============\n\nMy preferred solution should:\n\nInteroperate\n------------\n\nInteroperate with Python's native logging library.\n This will automatically include logging activity from incorporated libraries\n which nearly always use Python's native logging support.\n\nDistributed\n-----------\n\nAllow multiple clients to produce log entries.\n It is after all meant as a solution for a distributed application.\n\nCentralize all logs in a single location.\n This will make it possible to combine entries from different sources\n relating to the same request.\n\nAllow multiple clients to process the log entries.\n Some will store the log entries on permanent storage (file, database).\n Others may filter the entries and display them as if tailing a logfile.\n\nStructured\n----------\n\nOffer additional structure to the log entries.\n Normal logs are a sequential but otherwise unstructured list of hopefully\n significant facts. What I'd like to see is that the log entries tell me what\n is being achieved and wheter that succeeded or not. Each of these goals may\n itself consist of subgoals creating a tree of goals which together reach a\n conclusion.\n\n For many years now we modularize and structure our code, yet none of that is\n visible in the log entries that tell us how the program is progressing.\n\nThe structure in the log entries should transcent process bounderies.\n When one process invokes the services of another process the structure of\n the log entries should reflect this. A process switch should not break the\n tree structure described above.\n\nDataset oriented\n----------------\n\nSupport record logging.\n Traditionally log entries are supposed to be a single text string. I want to\n see all of Python's LogRecord fields in the log entry. And probably quite a\n few more fields as well. We can assume the LogRecord fields as a stable\n basis set of properties that are expanded by custom properties depending on\n the application and function.\n\nOh and the whole thing should be modular off course.\n - Structured logging\n - Multi process logging\n - Record logging\n\n Should be separate functional modules that can be combined to provide all of\n the above.\n\nThere is just one little thing: it doesn't exist. Yet.\n\nThe distlog package\n====================\nThe distlog package exports a couple classes and functions:\n\n* :class:`ZmqHandler`\n This is a handler for the standard Python logging library that transports\n :class:`LogRecords`. For it to work it needs a serializer that prepares the\n :class:`LogRecord` instances for transportation. This serializer should be\n derived from the :class:`~distlog.formatters.Serializer` class. Choose one\n of the predefined serializers :class:`JSONFormatter` or\n :class:`PickleFormatter` and set it as the formatter using\n :method:`ZmqHandler.setFormatter`.\n\n* :class:`JSONFormatter`\n This is one of the predefined serializers, it converts a :class:`LogRecord`\n instances to JSON strings that can be sent via 0MQ.\n\n* :class:`PickleFormatter`\n This is the other predefined serializer. it converts a :class:`LogRecord` to\n the binary Python specific pickle format. Note that the serialization method\n is part of the message so that the receiver knows how to deserialize the\n message.\n\n* :function:`task`\n This function returns a :class:`Task` object that implements as a context\n handler and acts as the top level logging context for Pythons normal logger\n functions.\n\n* :function:`to`\n This function returns a :class:`Task` instance that is a child of another\n logging context thus constructing a hierarchy of logging contexts.\n\n* :function:`import_task`\n Creates a top level logging context that is subordinate to the logging\n context in a different process. Using :method:`Task.get_foreign_task` to\n obtain a task/context id, transferring this id from one process to another\n and passing it as parameter to :function:`import_task` it is possible to\n relate the activities across processes as belonging to one and the same\n activity.\n\nLogging infrastructure\n======================\nThe process(es) forming the application that is being logged send out their log\nmessages using :class:`ZmqHandler`. This class sends the message out using a\n0MQ PUSH socket that binds to the well-known address of the corresponding PULL\nsocket of the `distlogd` process.\n\nThe `distlogd` process receives all log messages and filters them using a\nconfigurable rule set. Based on the rule(s) which triggered, the message is\nforwarded to one or more processors.\n\nThe configuration file, probably a YAML file, may instruct `distlogd` to load\nprocessors as plugins. A processor will apply one or more filters on a message\nand uses the outcome to decide to handle a message or not.", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/lnoor/distlog", "keywords": "scoped structured distributed logging", "license": "GPLv3", "maintainer": "", "maintainer_email": "", "name": "distlog", "package_url": "https://pypi.org/project/distlog/", "platform": "any", "project_url": "https://pypi.org/project/distlog/", "project_urls": { "Homepage": "https://github.com/lnoor/distlog" }, "release_url": "https://pypi.org/project/distlog/0.1.dev0/", "requires_dist": null, "requires_python": "", "summary": "Logging for distributed systems.", "version": "0.1.dev0" }, "last_serial": 3766325, "releases": { "0.1.dev0": [ { "comment_text": "", "digests": { "md5": "c572d61a93afaef453c2fbe23968195d", "sha256": "a4039c48b68b565fb4145f1022017670553e5bb8866c0e42f7f07fec8a49122b" }, "downloads": -1, "filename": "distlog-0.1.dev0.tar.gz", "has_sig": false, "md5_digest": "c572d61a93afaef453c2fbe23968195d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11665, "upload_time": "2018-04-15T12:45:00", "url": "https://files.pythonhosted.org/packages/17/0b/fc436f38d8191bc21f42cea7f60b7ca7a9ffdbef7fa632a3448619ce7c0b/distlog-0.1.dev0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "c572d61a93afaef453c2fbe23968195d", "sha256": "a4039c48b68b565fb4145f1022017670553e5bb8866c0e42f7f07fec8a49122b" }, "downloads": -1, "filename": "distlog-0.1.dev0.tar.gz", "has_sig": false, "md5_digest": "c572d61a93afaef453c2fbe23968195d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11665, "upload_time": "2018-04-15T12:45:00", "url": "https://files.pythonhosted.org/packages/17/0b/fc436f38d8191bc21f42cea7f60b7ca7a9ffdbef7fa632a3448619ce7c0b/distlog-0.1.dev0.tar.gz" } ] }