{ "info": { "author": "David N. Mashburn", "author_email": "david.n.mashburn@gmail.com", "bugtrack_url": null, "classifiers": [], "description": "Simple Metrics Manager facilitates managing metrics.\n\nA \"metric\" as defined here consists of:\n\n * A string name (and an associated python constant)\n * A function that takes no arguments and returns some data\n * The data returned from the metric function\n\n### StorageInterfaces\n\nA `StorageInterface` is a class that supports storing metric data using some\npersistent backend. The current default is the fairly generic '.npz' format\nused to store numpy arrays and simple python objects `NpzStorageInterface`.\n\nTo prevent data corruption, it always saves data to a temp file and then\nmoves it into the real file (potentially replacing any older copies).\n\nIt has a simple save/load/exists API.\n\n#### Pre-defined StorageInterface classes:\n\n`JsonStorageInterface` uses `json.load` and `json.dump` and\nautomatically coerces numpy arrays to lists.\n\n`NpyStorageInterface` just uses `np.save` and `np.load` directly.\n\n`NpzStorageInterface` uses numpy's `np.load` and (hidden) `_savez` method.\nIt and does some simple checks to allow saving arrays as well as lists\nof arrays and dictionaries of arrays.\n\nThis is very fast, efficient, reliable, and gives quite a bit of flexibility.\n\nThe option to `allow_pickle=False` means that saving object arrays will fail,\nso change it to `True` if you need it, but pickle is considered volatile,\nso changes to python or relevant package versions might mean your data\nis unreadable. This is super convenient though, so it hasn't stopped big\npackages like scikit-learn from relying on it.\n\nYou could also easily add a pure pickle-based storage interface yourself\nwith `pickle.load` and `pickle.dump`, but I don't include it here for\nthe reasons listed above :)\n\n\n\n### Cache Managers\n\nA `DatedCacheManager` uses a `StorageInterface` and supports the following:\n\n * Automatic caching to both memory and persistent storage,\n * Automatic cache utilization, falling back on memory cache and then\n persistent cache (override with `force=False`)\n * Dating of all metrics using \"side-car\" metrics (*_date)\n * Printing of all major actions (disable with `verbose=False`)\n\n\nIt has the following API:\n\n * `set_functions_dict`\n - Set the core data for the manager, a dictionary that actually defines the metrics.\n Keys are metric names and values are metric functions (no arguments).\n By calling this after `\\__init__`, the manager itself can be used within metric functions.\n See \"Usage\" below.\n * `exists`\n - Boolean, whether the metric is in the memory cache.\n * `clear_cache`\n - Remove all metrics from the cache.\n * `compute`\n - Call a metric function.\n Caches and returns the data.\n * `save`\n - Compute a metric and save the result with the StorageInterface.\n Caches to memory and disk and returns the data.\n * `load`\n - Load a metric (assumed to exist)\n Tries the cache, then the StorageInterface, fails otherwise.\n Returns the data\n * `get`\n - Call save and then load.\n Caches to disk and the `StorageInterface`.\n Returns the cached data.\n Overloaded with `[]` (aka `__getitem__`).\n\nIn addition, `ParameterizedDatedCacheManager` adds a powerful decorator\ncalled `collect` that collects metrics automatically.\n\n#### Basic usage:\n\n CM = DataCacheManager(NpyStorageInterface(SOME_DIRECTORY))\n\n @CM.collect\n def metric_1():\n stuff = do_something()\n return stuff\n\n @CM.collect\n def metric_2():\n stuff = do_something_else(CM.metric_1()) # <- use of the cached version here\n other_stuff = convert(stuff)\n return other_stuff\n\nNow calls to `CM.metric_1()` and `CM.metric_2()` seamlessly use the\ncache (both in-memory and on disk).\n\nThis allows for complex dependencies to be handled automatically\nand efficiently.\n\nIf you prefer, `metric_1(cache=True)` is equivalent to `CM.metric_1()`.\nBoth will invoke `CM['metric_1']` which in turn will invoke the\n\"undecorated\" `metric_1()` as it appears above when it does not exist.\n\nPassing `use_stored=False` on any of these will invalidate the\ncache and overwrite it with the new value.\n\nThe original function (called without `cache=True`) will be unchanged.\n\nExample: `metric_1()` will intentionally _skip_ the cache and just run\nlike normal.\n\n#### Parameterized usage:\n CM = ParameterizedDataCacheManager(NpyStorageInterface(SOME_DIRECTORY))\n\n @CM.collect(params_list=[[1], [4]])\n def poly_X(a):\n return do_something_hard(a)\n\n @CM.collect(params_list=[1, 2], [1, 3], [4, 5])\n def poly_Y(a, b):\n return CM.poly_X(a) ** 2 + do_something_else(b)\n \n`CM.poly_Y(1, 2)` will run `poly_Y(1, 2)` which will run `CM.poly_X(1)`\nwhich will run `poly_X(1)`\n\nThen another call to `CM.poly_Y(1, 2)` will load `CM['poly_Y_1_2']` and return it\n\nA call to `poly_Y(1, 3)` will run `poly_Y(1, 3)` which will run `CM.poly_X(1)`\nwhich will load `CM['poly_X_1]`\n\nThe point is that if `poly_X` or `poly_Y` take a long time, subsequent calls\nwill be fast.\n\nPre-running everything can be done like so:\n\n CM.poly_X.get_all()\n CM.poly_Y.get_all()\n\nwhich is handy, for instance, if you need to run a large set of\ncomputations overnight.\n\nThe above will fail if you pass undefinied parameters (which is good if\nyou want to ensure you have pre-cached everything you might invoke).\n\nIf you _really_ want to call the functions in the cache manager with any\nparameters, just add this option to the code above:\n\n CM = ParameterizedDataCacheManager(NpyStorageInterface(SOME_DIRECTORY),\n dynamic_metric_creation=True)\n\nWith this change, things like `CM.poly_Y(12, 13)` will also work.\n\nFinally, to reiterate, the original functions `poly_X` and `poly_Y` will\nstill work with any parameters regardless of this setting because they\nare _unchanged_ unless you specify `cache=True`.\n\n#### Manual usage of DatedCacheManager:\n\n METRIC_1 = 'metric_1'\n METRIC_2 = 'metric_2'\n\n CM = DataCacheManager(NpyStorageInterface(SOME_DIRECTORY))\n\n def metric_1_function():\n stuff = do_something()\n return stuff\n\n def metric_2_function():\n stuff = do_something_else(CM[METRIC_1])\n other_stuff = convert(stuff)\n return other_stuff\n\n FUNCTIONS_DICT = {\n METRIC_1: metric_1_function,\n METRIC_2: metric_2_function,\n ...\n }\n\n CM.set_functions_dict(FUNCTIONS_DICT)\n\nThen in some other place invoke: `CM[METRIC_1]` or `CM[METRIC_2]`\n\nThis has the same effect as the \"Basic usage\" example above but with a\nlot more boilerplate. You can still use this if you despise decorator\nmagic though ;)", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://pypi.python.org/pypi/simple_metrics_manager/", "keywords": "", "license": "LICENSE.txt", "maintainer": "", "maintainer_email": "", "name": "simple_metrics_manager", "package_url": "https://pypi.org/project/simple_metrics_manager/", "platform": "", "project_url": "https://pypi.org/project/simple_metrics_manager/", "project_urls": { "Homepage": "http://pypi.python.org/pypi/simple_metrics_manager/" }, "release_url": "https://pypi.org/project/simple_metrics_manager/0.2.1.1/", "requires_dist": null, "requires_python": "", "summary": "Just a simple system to manage a set of metrics (string name / function / returned data) that supports caching (memory and disk)", "version": "0.2.1.1" }, "last_serial": 4572759, "releases": { "0.1.0.0": [ { "comment_text": "", "digests": { "md5": "a85ffaabcf60e793dddbef1690f99985", "sha256": "65233ddf8b664245216f5036b6fe2be8bf5a9443d9325ef572ed1f20c82c16e8" }, "downloads": -1, "filename": "simple_metrics_manager-0.1.0.0.tar.gz", "has_sig": false, "md5_digest": "a85ffaabcf60e793dddbef1690f99985", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4153, "upload_time": "2017-02-01T03:26:23", "url": "https://files.pythonhosted.org/packages/23/74/fc8add928c4c5a8924c8b0ff215b52ba0ce91aa803781e3d31ae783e1d0a/simple_metrics_manager-0.1.0.0.tar.gz" } ], "0.1.1.0": [ { "comment_text": "", "digests": { "md5": "b193736b3f4dc29d48bae35b6c130e5d", "sha256": "380fef75fc5966334d39d93c85ba6c4796ee1492c0702e3044ad486384b63061" }, "downloads": -1, "filename": "simple_metrics_manager-0.1.1.0.tar.gz", "has_sig": false, "md5_digest": "b193736b3f4dc29d48bae35b6c130e5d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4552, "upload_time": "2017-05-04T20:50:08", "url": "https://files.pythonhosted.org/packages/1d/45/cdd0e09fc59e9558d2bc079da1a4d8e03c4d951b3ace6320c0f2da37126d/simple_metrics_manager-0.1.1.0.tar.gz" } ], "0.1.1.2": [ { "comment_text": "", "digests": { "md5": "fa28dec4584097764f35d3601600873d", "sha256": "e609e5240af2f666126ef30f45d772d87b6be52762aa0875803942b10c9bef11" }, "downloads": -1, "filename": "simple_metrics_manager-0.1.1.2.tar.gz", "has_sig": false, "md5_digest": "fa28dec4584097764f35d3601600873d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4673, "upload_time": "2017-11-20T18:13:53", "url": "https://files.pythonhosted.org/packages/09/a0/9d293e3950c36f3fdc415aec633160a986591d83d56e185b52af7655738c/simple_metrics_manager-0.1.1.2.tar.gz" } ], "0.1.1.3": [ { "comment_text": "", "digests": { "md5": "78187b769ac7de810ca3d7627433a5a2", "sha256": "acd56565b67ef135f94418a7eedcc0518261e9143a3666d7dfa511de2d9465c6" }, "downloads": -1, "filename": "simple_metrics_manager-0.1.1.3.tar.gz", "has_sig": false, "md5_digest": "78187b769ac7de810ca3d7627433a5a2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4690, "upload_time": "2018-04-03T18:19:12", "url": "https://files.pythonhosted.org/packages/8c/66/95ceef73b73bc95aa907c2c61d1914252ee93ac20ee2958f1c67628d0486/simple_metrics_manager-0.1.1.3.tar.gz" } ], "0.1.1.4": [ { "comment_text": "", "digests": { "md5": "f7982e45fc58ed77e6f8ee5f3e8ab220", "sha256": "7963b9802b48d5dec7d227ef95c8e14ff94d981b855632f87c428a2b9159d603" }, "downloads": -1, "filename": "simple_metrics_manager-0.1.1.4.tar.gz", "has_sig": false, "md5_digest": "f7982e45fc58ed77e6f8ee5f3e8ab220", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4693, "upload_time": "2018-04-03T19:39:45", "url": "https://files.pythonhosted.org/packages/78/bd/87fec28dd57a57bf6f511dc0c4eb3e2560d8d5ea91c8d93b55081592f793/simple_metrics_manager-0.1.1.4.tar.gz" } ], "0.2.0.0": [ { "comment_text": "", "digests": { "md5": "bac4e86cf35676d0a5d20f0cd903d3f7", "sha256": "cccb53de2fccd49084e1122231178d3b459410994b295a46207c94fc22aaebec" }, "downloads": -1, "filename": "simple_metrics_manager-0.2.0.0.tar.gz", "has_sig": false, "md5_digest": "bac4e86cf35676d0a5d20f0cd903d3f7", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6206, "upload_time": "2018-12-03T19:45:17", "url": "https://files.pythonhosted.org/packages/62/56/810cefd0c3bc5b6d50a3b15fc655e89387aa03150cb1347e48674aaff60c/simple_metrics_manager-0.2.0.0.tar.gz" } ], "0.2.1.0": [ { "comment_text": "", "digests": { "md5": "f9510aecb0eb4e966d64fabb0a1b10fe", "sha256": "46e12dc1ee2dc8c8bfd7c1298bcbbaff174aae2aeee04c644a050863455dbff4" }, "downloads": -1, "filename": "simple_metrics_manager-0.2.1.0.tar.gz", "has_sig": false, "md5_digest": "f9510aecb0eb4e966d64fabb0a1b10fe", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6228, "upload_time": "2018-12-04T16:25:21", "url": "https://files.pythonhosted.org/packages/50/9b/aefc326d9aae7705607626969fce7822c672ea6cb6b09ee700d0382dc971/simple_metrics_manager-0.2.1.0.tar.gz" } ], "0.2.1.1": [ { "comment_text": "", "digests": { "md5": "690e9eafeae0ed3eace8c487da335395", "sha256": "5e7153b84c875bf73dc7ecbc9d59af7f59bb25d117956fa8a3d49c3996074462" }, "downloads": -1, "filename": "simple_metrics_manager-0.2.1.1.tar.gz", "has_sig": false, "md5_digest": "690e9eafeae0ed3eace8c487da335395", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8034, "upload_time": "2018-12-07T17:32:56", "url": "https://files.pythonhosted.org/packages/7b/ee/5815940fde6608a6a7d750c9c66d09139d39aef7c4471a231259f62c285d/simple_metrics_manager-0.2.1.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "690e9eafeae0ed3eace8c487da335395", "sha256": "5e7153b84c875bf73dc7ecbc9d59af7f59bb25d117956fa8a3d49c3996074462" }, "downloads": -1, "filename": "simple_metrics_manager-0.2.1.1.tar.gz", "has_sig": false, "md5_digest": "690e9eafeae0ed3eace8c487da335395", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8034, "upload_time": "2018-12-07T17:32:56", "url": "https://files.pythonhosted.org/packages/7b/ee/5815940fde6608a6a7d750c9c66d09139d39aef7c4471a231259f62c285d/simple_metrics_manager-0.2.1.1.tar.gz" } ] }