{
    "info": {
        "author": "Alex Barcelo",
        "author_email": "alex@betarho.net",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Environment :: Console",
            "Intended Audience :: Information Technology",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
            "Operating System :: POSIX",
            "Programming Language :: Python :: 2",
            "Programming Language :: Python :: 3",
            "Topic :: Database",
            "Topic :: Scientific/Engineering :: Information Analysis",
            "Topic :: System :: Logging"
        ],
        "description": "# Tabular Annotations of Data for Batch Jobs\n\nDo you use a queue system (slurm, pbs) and are tired of manually tracking result of your executions?\n\nSo was I. So I decided to start this small project. This project is suited for you if you...\n\n - ... execute the same application with different parameters\n - ... want to track the execution parameters and/or its outputs\n - ... the data of interest can easily be displayed in a tabular fashion\n - ... don't want to track the data yourself by going into the log files generated in the application\n\n**You don't need to install any additional dependency or package**. You will only need the system python interpreter (either Python 2.7 or 3.x will work). YAML is supported if you desire to use it, but it is entirely optional. The main design decision is to be cluster-friendly, and those environments can be very restricted, but I have always found Python 2.7 in them by default. Hopefully, `tad4bj` will keep working regardless of being inside a virtual environment or changing versions.\n\nThe application `tad4bj` can be used both through the native python bindings or through the shell. There are extra features if you use it through its bindings --e.g. arbitrary pickled structures and mutable types.\n\nInternally, the data is stored into a SQLite database. You can browse it yourself, given that the \"schema overhead\" is kept at its minimum.\n\n## Quickstart\n\nKeep reading for quick examples and steps to have `tad4bj` working for you in your environment.\n\n### Installation (with `pip`)\n\nIf you are fortunate enough to have `pip` --lucky you-- then you can simply:\n\n    pip install tad4bj\n\nI recommend doing that inside a Python virtual environment. This is useful for local testing, or internet enabled clusters.\n\nIf you cannot use `pip`, then keep reading...\n\n### Installation\n\n 1. Download and extract the zip file into your cluster's home (or execution environment). The files will remain in the path you choose now.\n 2. Execute the `./local_install.py ~/bin` and don't use _sudo_! If you have your user scripts elsewhere, change the path.\n 3. Logout and login.\n\nIf you don't use `bash`, then you may want to manually add some exports to your shell rc file. If you use `bash`, the `local_install`\nscript automatically updates it to make `tad4bj` work.\n\n### Basic shell usage and examples\n\nCommand line help is more or less useful:\n\n`tad4bj -h`\n\nFirst of all you will need a schema for your table. Look into the examples folder or go to [Schema](#schema) for more detailed documentation. Then you can prepare the database with:\n\n`tad4bj --table <mytablename> init <schemafile.json>`\n\nA file will have just appeared in your home called `tad4bj.db`. You can tune that if you don't like this path. This is a SQLite database, so feel free to browse it or extract information from there.\n\nYou can use it `tad4bj` cli, e.g.:\n\n```\n$ tad4bj --table <mytablename> set --jobid 123 description \"Hey look this is a string description of this\"\n$ tad4bj --table <mytablename> get --jobid 123 description\nHey look this is a string description of this\n$ tad4bj --table <mytablename> setnow --jobid 124 start\n$ tad4bj --table <mytablename> set --jobid 124 description \"Another execution, another description\"\n$ tad4bj --table <mytablename> setdict --jobid 123 filename.json'\n$ command_generating_yaml | tad4bj --table <mytablename> setdict --jobid 124 --dialect yaml -\n```\n\n#### What happens with `--table` and `--jobid`\n\nThe previous examples have explicit table name and job identifier. By default, if you are using slurm or pbs, those parameters are not needed:\n\n - The job name will be used as table name\n - The job identifier will be used as row identifier --aka `--jobid`\n\nThose values from the scheduler are taken from their default environment variables. You can change the behaviour simply by keep using explicit `--table` and `--jobid` flags. Remember while creating the table (`tad4bj init`) to match the job name that you are using --or the other way around, use the same job name to ensure that the annotations are correctly stored in the created table. Otherwise you will have SQL errors of table not exists.\n\nBy using the job scheduler autodetection, using `tad4bj` from inside submitted jobs is easy:\n\n```\ntad4bj set description \"Submitted jobs are easy to annotate\"\n```\n\n### Using python bindings\n\nAssuming that you are using either slurm or pbs and the python code is being executed inside a submitted job, then it's easy! Just use it like this:\n\n```python\nfrom tad4bj.slurm import handler as tadh\nfrom datetime import datetime\n\n...\n\ntadh[\"start\"] = datetime.now()\n\nelements = [1, 2, 3]\n\ntadh[\"pickled_item\"] = elements\n\nelements.append(4)  # note that pickled_item column will be (eventually) updated!\n\ntadh[\"pickled_item\"].append(5)  # another way of updating information\n\n# and you can also get data\ndescr = tadh[\"description\"]\n```\n\nSome relevant notes regarding the python bindings:\n\n - The import is working because `PYTHONPATH` is updated in your `.bashrc` (see [Installation](#installation)).\n - The job id and the table name are working because `tad4bj.slurm` gets them from the environment. If you are using pbs, just use `tad4bj.pbs`. If you are using another job scheduler, pull requests are welcome. You can prepare the handler manually, look the documentation for more details.\n - Mutable types are useful but have certain quirks, see [Mutable types](#mutable-types) for some additional notes.\n\n## Processing the tabular data\n\nNow you have all your execution information. How to process it? Well, it's a SQLite database so feel free to use whatever exporters and tools you are used to.\n\nBut if you are used to Python --maybe with Jupyter, maybe with other tools, maybe you want to add a custom CSV exporter-- then you can take advantage of the `Mapping` interface of the `DataStorage` class.\n\nAs an example, you could do things like:\n\n```python\nfrom tad4bj import DataStorage\nfrom datetime import datetime\n\nd = DataStorage(\"mytable\")\n\nd.keys()  # Those are job ids\nd.values() # Job ID also appear here\n\nfilter(lambda x: x.start > datetime(2018, 1, 1), d.values())\n\nd[123].description\n```\n\nThis interface is meant to be a read-only friendly layer for data processing.\n\n## Schema\n\n**ToDo**\n\n### Adding columns to the table\n\nTypically you will initialize a table with the fields you feel you need. You will execute stuff. And after that, you will realize that you did not consider some fields that now you need to track, leaving you with an incomplete table. This has happened to everyone, and `tad4bj` includes an easy way to add columns to the database --but don't expect a \"smart\" migration system. If you need to fine-tune things then you can open the database manually, although you should check the documentation before attempting that.\n\nSo, when you need to add new columns to the database, then you can update the schema file and use the following CLI call:\n\n```\ntad4bj --table <mytablename> update <schemafile.json>\n```\n\nAlready-existing fields will be ignored, and new fields will be added to the database. No sanity tests are done, so double check that you are not changing the wrong table. No undo mechanisms are included.\n\n## Mutable types\n\nWhile using the python bindings, you can use structured fields which can contain mutable types --e.g. a JSON field with a lists. The main design decision is that the binding tracks the objects that have been assigned to (or read from) the database.\n\nWhen the application ends (clean shutdown), all objects in memory are written. That means that if a mutable object has been assigned to the database and then modified, the updated version will be written to the database.\n\nIf there is a dirty shutdown (for instance, a job scheduler time limit) the database may not be updated or even some assignments may be lost. You may want to manually call to the `commit` method in the handler to ensure that the database is updated:\n\n```python\ntadh.commit()\n```\n\nThis ensures that all the mutable objects are updated and the database file is updated. It is a good idea to call this `commit` method before time-consuming or crash-prone blocks. \n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/alexbarcelo/tad4bj",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "tad4bj",
        "package_url": "https://pypi.org/project/tad4bj/",
        "platform": "",
        "project_url": "https://pypi.org/project/tad4bj/",
        "project_urls": {
            "Homepage": "https://github.com/alexbarcelo/tad4bj"
        },
        "release_url": "https://pypi.org/project/tad4bj/1.2/",
        "requires_dist": [
            "pyyaml ; extra == 'yaml'"
        ],
        "requires_python": "",
        "summary": "Tabular Annotations of Data for Batch Jobs",
        "version": "1.2"
    },
    "last_serial": 5399366,
    "releases": {
        "1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ea007d3cd0672a19012743385fd996ec",
                    "sha256": "dba1fbe129a46d47a5650f4034183cf3806ab414d2627bec0c275ae9667dd4bc"
                },
                "downloads": -1,
                "filename": "tad4bj-1.0-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "ea007d3cd0672a19012743385fd996ec",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 16374,
                "upload_time": "2019-01-15T10:26:37",
                "url": "https://files.pythonhosted.org/packages/c9/58/c0d80b22164842a46d93c1c34b51368a8961c57eb6176df2b20822d21c19/tad4bj-1.0-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "5b5fdf61cf708089dfb30586926b876b",
                    "sha256": "ae4079c78985a8f63a855cd14d3dc848105398a1846efdfdb685d57401c7dc6a"
                },
                "downloads": -1,
                "filename": "tad4bj-1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "5b5fdf61cf708089dfb30586926b876b",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 16375,
                "upload_time": "2019-01-15T10:26:40",
                "url": "https://files.pythonhosted.org/packages/b5/20/cfcc10ab556c09d0a485d15373a4aa0fc1bf48c0a96a39a449e51d96705d/tad4bj-1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "6f86db2a3bb26b304bba210016640e4a",
                    "sha256": "9cd2193d140962b0fb5095484d7d534a02f45fa3f6481284b137bce81f6b07c2"
                },
                "downloads": -1,
                "filename": "tad4bj-1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "6f86db2a3bb26b304bba210016640e4a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13198,
                "upload_time": "2019-01-15T10:26:42",
                "url": "https://files.pythonhosted.org/packages/9f/05/cea5cb1f51d17bfa0e7f136ec416089abcb2e228b61948c1773413e6f39e/tad4bj-1.0.tar.gz"
            }
        ],
        "1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "dff086eabc6e4f6b85fe6bda636a7e1c",
                    "sha256": "c01b966ba78c56e0bb25c4593b27051f8d8c3d600b83df689010a4fd343dc612"
                },
                "downloads": -1,
                "filename": "tad4bj-1.1-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "dff086eabc6e4f6b85fe6bda636a7e1c",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 24325,
                "upload_time": "2019-01-15T13:38:43",
                "url": "https://files.pythonhosted.org/packages/47/bc/d73a53d9d754c56480d4f1c77a81c4e6c6cbc3ff77dd6c79258706e5b670/tad4bj-1.1-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "5ed03d4cbc62ecb6ceffffd0f55c33bb",
                    "sha256": "bec5c2e566169e5e2fdf231815e7d5300358282cd7786635ae1b64dd81b2238c"
                },
                "downloads": -1,
                "filename": "tad4bj-1.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "5ed03d4cbc62ecb6ceffffd0f55c33bb",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 24324,
                "upload_time": "2019-01-15T13:38:44",
                "url": "https://files.pythonhosted.org/packages/53/bc/f41b69f4245dba4d3d210636489eba0411e71738ff666ab4d3c25bb8c141/tad4bj-1.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "8d43afae2c8dbca1523da13ee3cc9609",
                    "sha256": "be37c66a7a1d2fe3b6f5c8d1e5ef1c36aef9c2f559726d1ce91143b995ac89ff"
                },
                "downloads": -1,
                "filename": "tad4bj-1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "8d43afae2c8dbca1523da13ee3cc9609",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13202,
                "upload_time": "2019-01-15T13:38:45",
                "url": "https://files.pythonhosted.org/packages/0f/30/2450439d35c9e36e355a84d13800bc4975508b066794ce8c2710d645376b/tad4bj-1.1.tar.gz"
            }
        ],
        "1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ad5bacc75ba946ddf5ba9de3dc3eac21",
                    "sha256": "b3a7062639576ada6260c9d0a34a31d6fd69ec3e510e606bacdcf59791dd017f"
                },
                "downloads": -1,
                "filename": "tad4bj-1.2-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "ad5bacc75ba946ddf5ba9de3dc3eac21",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 24536,
                "upload_time": "2019-05-29T10:06:01",
                "url": "https://files.pythonhosted.org/packages/c5/3c/ff692b7c9accac8b3136b57fd8f10b5ff2a8ec0c2dd7f87eac4a2b31aca1/tad4bj-1.2-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "af9dfdbb018eec4d10758ad9679e73cc",
                    "sha256": "b15ecbf17799b9ee1f2118831a7c68acfbbbeb4f5d4b2941cbd92349473d862c"
                },
                "downloads": -1,
                "filename": "tad4bj-1.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "af9dfdbb018eec4d10758ad9679e73cc",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 24535,
                "upload_time": "2019-05-29T10:04:42",
                "url": "https://files.pythonhosted.org/packages/e8/26/a0e2d2e56397c9529ba630bd22efef316c455331c7c477ed4b121979d6aa/tad4bj-1.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "6547f79b1165333b998a5e73518d7d02",
                    "sha256": "b7d790438e9b7acb62b5a4daf920e4f1bddb611479849343c77a5659a51fd072"
                },
                "downloads": -1,
                "filename": "tad4bj-1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "6547f79b1165333b998a5e73518d7d02",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13192,
                "upload_time": "2019-05-29T10:04:44",
                "url": "https://files.pythonhosted.org/packages/12/f2/915e183cdc347bacdf0090d6204b61c1eec0325ad348441077b177d43446/tad4bj-1.2.tar.gz"
            }
        ],
        "2.0.dev1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bedd9f9977ccdf7c5b1018d7b7ec760d",
                    "sha256": "29970364f15e503aa485b8cd9b59042c2a72c1a16cae46b8483219834156bd41"
                },
                "downloads": -1,
                "filename": "tad4bj-2.0.dev1-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "bedd9f9977ccdf7c5b1018d7b7ec760d",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 25198,
                "upload_time": "2019-06-14T07:55:19",
                "url": "https://files.pythonhosted.org/packages/92/dd/31c851bdbf541119fc92583930020849c38a3f52fc37105a2fe63a7b51ab/tad4bj-2.0.dev1-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "85844a4572488a1f46748fbfdb1d8233",
                    "sha256": "eb894da0c50a2c0e5e4fc8ca34208cfbdc6e3c44ea623579f1b961c2aaddc375"
                },
                "downloads": -1,
                "filename": "tad4bj-2.0.dev1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "85844a4572488a1f46748fbfdb1d8233",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 25198,
                "upload_time": "2019-06-14T07:55:21",
                "url": "https://files.pythonhosted.org/packages/2c/30/cdd7241d6d9f428b2f73c8dbf3ebc965c81308b4ae60410d8e499f37b834/tad4bj-2.0.dev1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "cbb266cbf3e22bd4170ca9ebb26553d3",
                    "sha256": "bfa881dc25c41523de9cc2ce85e12567ce865d97532296f284d3323485b0b46a"
                },
                "downloads": -1,
                "filename": "tad4bj-2.0.dev1.tar.gz",
                "has_sig": false,
                "md5_digest": "cbb266cbf3e22bd4170ca9ebb26553d3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13992,
                "upload_time": "2019-06-14T07:55:24",
                "url": "https://files.pythonhosted.org/packages/e5/d6/3368562c9f6e7e096b093a945a6a252f5a3a0284e277171f3e088148e323/tad4bj-2.0.dev1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "ad5bacc75ba946ddf5ba9de3dc3eac21",
                "sha256": "b3a7062639576ada6260c9d0a34a31d6fd69ec3e510e606bacdcf59791dd017f"
            },
            "downloads": -1,
            "filename": "tad4bj-1.2-py2-none-any.whl",
            "has_sig": false,
            "md5_digest": "ad5bacc75ba946ddf5ba9de3dc3eac21",
            "packagetype": "bdist_wheel",
            "python_version": "py2",
            "requires_python": null,
            "size": 24536,
            "upload_time": "2019-05-29T10:06:01",
            "url": "https://files.pythonhosted.org/packages/c5/3c/ff692b7c9accac8b3136b57fd8f10b5ff2a8ec0c2dd7f87eac4a2b31aca1/tad4bj-1.2-py2-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "af9dfdbb018eec4d10758ad9679e73cc",
                "sha256": "b15ecbf17799b9ee1f2118831a7c68acfbbbeb4f5d4b2941cbd92349473d862c"
            },
            "downloads": -1,
            "filename": "tad4bj-1.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "af9dfdbb018eec4d10758ad9679e73cc",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 24535,
            "upload_time": "2019-05-29T10:04:42",
            "url": "https://files.pythonhosted.org/packages/e8/26/a0e2d2e56397c9529ba630bd22efef316c455331c7c477ed4b121979d6aa/tad4bj-1.2-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "6547f79b1165333b998a5e73518d7d02",
                "sha256": "b7d790438e9b7acb62b5a4daf920e4f1bddb611479849343c77a5659a51fd072"
            },
            "downloads": -1,
            "filename": "tad4bj-1.2.tar.gz",
            "has_sig": false,
            "md5_digest": "6547f79b1165333b998a5e73518d7d02",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 13192,
            "upload_time": "2019-05-29T10:04:44",
            "url": "https://files.pythonhosted.org/packages/12/f2/915e183cdc347bacdf0090d6204b61c1eec0325ad348441077b177d43446/tad4bj-1.2.tar.gz"
        }
    ]
}