{ "info": { "author": "Christian Bourjau", "author_email": "christian.bourjau@cern.ch", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Intended Audience :: Developers", "Intended Audience :: Science/Research", "License :: OSI Approved :: GNU General Public License v3 (GPLv3)", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2.7", "Topic :: Scientific/Engineering :: Physics", "Topic :: Software Development :: Libraries :: Python Modules" ], "description": ".. image:: https://badge.fury.io/py/nittygriddy.svg\n :target: https://badge.fury.io/py/nittygriddy\n\n\t \n============\nNittygriddy\n============\n\nNitty griddy aims to make the deployment of analyses using the Aliroot/Aliphysics framework easy, transparent, and reproducible.\nThe primary goal is to completely eliminate the ungodly practice of copy-pasting together ``run.C`` files from all over the place.\nA nittygriddy \"project\" requires the user to only provide the settings for the specific analysis they want to run - nothing related to how or where you would like to run your analysis.\nThe way these settings are supplied is analogous to how one configures a LEGO train.\nThe deployment in ``local`` (sequential), ``proof lite`` (local parallel) or ``grid`` is then completely transparent and does not require the editing of any files.\nNittygriddy also makes it easy to download parts of a desired dataset for local *offline* development - no need for running grid test mode.\nOn top of that, nittygriddy makes it easy to trigger the merging process once a grid analysis is finished.\n\nAnd last but not least, nittygriddy does not do a \"vendor lock-in\": nittygriddy produces straight forward ``run.C`` files, stores them in a tidy folder structure and enables you to always rerun your analysis without nittygriddy in the future with a simple ``root run.C``.\n\n.. image:: https://imgs.xkcd.com/comics/standards.png\n\nDisclaimer\n==========\nI am using nittygriddy daily for quite a while now and find it to be extremely helpful. I think/hope that it could also be a great tool for others. I will try hard to keep the current API stable, but if needed, I might have to make some breaking changes, which might need minor actions from the users. But as I said earlier, if that were to happen and you don't want to use nittygriddy anymore, you can always use the generated ``run.C`` files independently. If you have issues with nittygriddy, would love to see datasets added, or have any other comments and suggestions it is best to open an issue here on github or create a pull request straight away. See more on how to contribute at the bottom of this document.\n\n\nInstalling nittygriddy\n======================\n\nNittygriddy is available on `pypi`. Thus installation is easily done with `pip`, but first you should make sure that the stuff you install with `pip` will later be avialble on the command line. Add this to your `.bashrc` or similar: ::\n\n # .bashrc or similar\n export PYTHONUSERBASE=$HOME/.local_pip\n export PATH=$PYTHONUSERBASE/bin:$PATH\n\nThis will make everything installed as unprivileged user with `pip --user` go\nunder `PYTHONUSERBASE`. Now you can run: ::\n\n\n pip install --user nittygriddy\n\n\nWhich installes nittygriddy into your home directory. The command ``nitty`` should now be avialable on the command line, along with its hopefully helpful ``--help``.\n\n\nUsing nittygriddy for the impatient\n===================================\nDon't wanna read instructions? This is how you start an analysis on local data::\n\n $ alien-token-init \n $ nitty datasets --download LHC15o_pass1_HIR 1 # Download 1GB of the specified dataset\n $ nitty new my-analysis # Create an example analysis folder (runs AliMultSelectionTask)\n $ cd my-analysis\n $ # Run the analysis of this folder on the LHC15o dataset locally\n $ nitty run local LHC15o_pass1_HIR\n $ ls ./latest/ # marvel at the resulting AnalysisResults.root file\n \n\nSetting up a \"train\" folder\n===========================\n\nWhat I call a \"train\" in the following is really nothing more than a folder where you store the settings for a specific analysis. An analysis usually consists of several task (e.g. ``AliMultSelectionTask`` and your own task). Currently, all the settings are in one single file (``ConfigureTrain.C``) which make any folder a \"train folder\". Also, note that in order to use nittygriddy **your task must be present in your local aliphysics installation** (which it really should anyways)!\n\n`ConfigureTrain.C`\n------------------\nThe only ``.C`` file needed. It reflects setting up the options for your task analog to what is done in the lego trains. Note that this file will be compiled and therefore has to be valid cpp (thus all the ``#includes`` in the example below). The reason for this is that using the interpreter is just plain evil and will lead to undefined behavior and sad users eventually. The shortest possible version, would look something like this:\n\n.. code-block:: cpp\n\n #include \"TROOT.h\"\n #include \"AliVEvent.h\"\n #include \"AliBasedNdetaTask.h\"\n\n void ConfigureTrain() {\n // Load you AddTask macro\n gROOT->LoadMacro(\"$ALICE_PHYSICS/PWGLF/FORWARD/analysis2/AddTaskdNdeta.C\");\n \n // Execute your AddTask macro. You can pass options in the function call if necessary\n AliBasedNdetaTask* task =\n reinterpret_cast< AliBasedNdetaTask* > (gROOT->ProcessLine(\"AddTaskdNdeta()\"));\n \n /*\n Set your task's options here:\n task->SelectCollisionCandidates(AliVEvent::kMB);\n */\n }\n\n\n..\n nittygriddy.json *(Not used, yet)*\n ----------------------------------\n This file contains some default options as well as depedencies which need to be loaded for execution. Again, this is analogus to the lego train interface. An example file might look like: ::\n\n [\n {\n\t \"Dependencies\":\"libOADB.so libSTEERBase.so libAOD.so libANALYSISalice.so libPWGCFCorrelationsC2.so\"\n }\n ];\n..\n\n`nitty_datasets.yml`\n--------------------\nThis file contains information about custom datasets. If the standard ones are used this is not necessary. The file should be located in your home directory at `~/nitty_datasets.yml`. The file has to be in `Yaml` formating. A typical entry looks like this:\n\n.. code-block:: yaml\n\n LHC15o_pass1_HIR_FMD:\n data_pattern: pass1/AOD/*/AliAOD.root\n datadir: /alice/data/2015/LHC15o/\n datatype: aod\n is_mc: 'false'\n notes: 5.02 TeV, Good runs with FMD, ITS, and V0. This is not the full run list\n since it requires the FMD!\n run_list: 246980, 246865, 246809, 246808, 246766, 246765, 246763, 246759, 246676,\n 246675, 246495, 246493, 246276, 246275, 246225, 246185, 246153, 246089, 246052,\n 245963, 245954, 245833, 245705, 245683\n run_number_prefix: '000'\n system: PbPb\n\n LHC15f_AOD171_FMD:\n ...\n \nNote that `Yaml`, just like Python, is indention based. But don't worry, if the syntax is faulty, nittygriddy will complain. This includes missing fields in each entry. Datasets which are defined in `nitty_datasets.yml` will be merged with the default ones which come with nittygriddy.\n\n\nUsing Nittygriddy\n=================\n\nI'm lost. Note that ``--help`` is also available for each subcommand::\n\n $ nitty --help\n\nCreate a new train folder. I.e. create a new folder with a minimal working ``ConfigureTrain.C``::\n\n $ nitty new a_new_train\n \nList all avialable datasets::\n\n $ nitty datasets -l\n\nSearch all datasets for something (e.g. the collsions system)::\n\n $ nitty datasets -s PbPb\n\nShow details about a dataset::\n\n $ nitty datasets --show LHC10h_AOD160\n\nDownload 5GB of data from the given dataset for offline developing.\nThe files are saved in ``~/lhc_data/`` following the same folder structure as on the grid::\n\n $ nitty datasets --download LHC10h_AOD160 5\n\nRun your analysis in proof lite locally::\n\n $ nitty run lite LHC10h_AOD160\n\nOr submit it to the grid using a PARfile (see below)::\n \n $ nitty run grid LHC10h_AOD160 --par_files=\"PWGCFCorrelationsC2.par\"\n\nOnce your analysis is finished on the grid, change to the output dir (``latest`` is a link pointing to the latest thing nittygriddy started) and trigger the merging::\n \n $ cd latest\n $ nitty merge online\n\nOnce all the final merging stages are reached, you can merge individual runs offline on your own computer::\n \n $ nitty merge offline\n\nThere are many more things you can do. Just check ``--help``.\n\n\nTips and Tricks\n===============\n\nPARfiles\n--------\nPar files can be used if you latest changes to your task are not yet in the latest aliphysics tag.\nSee Dario's `page `_ for a bit more background. Long story short, if your task is properly set up in AliPhysics, you should be able to do::\n\n $ cd $ALICE_PHYSICS/../build\n $ make PWGCFCorrelationsC2.par # you can use TAB completion to find the right par file\n $ make -j$MJ install\n\nThis should create the .par file for your analysis in ``$ALICE_PHYSICS/PARfiles``. If you get an error instead, you might not have your analysis set up properly in cmake. Dario's post should have you covered.\n\n\nMy analysis crashes miserably when running in Proof lite\n--------------------------------------------------------\nProof lite is quite picky about initializing your task's members in the constructors. This makes it a great test for running on the grid, but the error message is rather cryptic. Check if you initialized all your members in the constructor.\n\nI get a crash if I do something grid related\n--------------------------------------------\nDo you have a valid alien-token? Its on the todo-list to ask for it more gracefully if its not present.\n\n\nMigrate to LEGO trains\n----------------------\nOnce your analysis works, you should be able to almost seamlessly use your ``ConfigureTrain.C`` content in the LEGO wagon setup. Please use LEGO-trains whenever possible and reasonable to save resources!\n\n\nDebug your code like a boss (with GDB)\n--------------------------------------\nThere was a talk at one of the ALICE weeks about using GDB for debugging `(link) `_.\nUnfortunately, the talk did not cover how to use GDB with your task in aliphysics.\nNittygriddy makes this quite easy now with the ``--wait_for_gdb`` option::\n\n $ nitty run local LHC10h_AOD160 --wait_for_gdb\n\nThe above sets up your analysis, prints out its process id (eg. 27575) and then waits for you to attach gdb. In principle it should be as easy as::\n\n $ gdb -p 27575\n\nBut there might be a few caveats. I wrote a small blog post about how to use gdb `here `_.\n\nProfile your code\n-----------------\n\nNittygriddy makes it easy to use ``gdb`` as a stochastic profiler. This means that the running analysis is interuped many times, and a statistic is made where the analysis spends most of its time. This kind of profiling can be very visualized in so called \"Flame Graphs\". Nittygriddy packages some of the files from the original [FlameGraph project](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#links) to make this process as easy as possible and this is how.\n\nFirst, you have to start a local analysis. You probably want to start it with the ``--wait_for_gdb`` flag to get the ``pid``, but you can also just find the ``pid`` any other way, if you prefer::\n\n $ nitty run local LHC10h_AOD160 --wait_for_gdb\n\nNow, you have to open a second terminal where you attach the profiler to the ``pid`` of the running analysis (e.g. 27575)::\n\n $ nitty profile 27575 --nsamples=100\n\nRemember to resume the analysis in the first terminal! Now you can use your browser to check out the flamegraph. For the above ``pid`` it would be at ``/tmp/27575.svg``. The ``svg`` is updated every 5 samples, so give it some time!\n\nAn example of a flamegraph might then look something like this:\n\n.. image:: examples/flame_graph.png\n\n\nWhat is happening behind the scene?\n===================================\n\nWhen running your analysis nitty griddy create a new folder in your train folder.\nIt then generates a ``run.C`` file from your options and copies it into that folder.\nThis ``run.C`` can be run on independently and should be easy to read.\nThis has the advantage that you can always just stop using ``nittygriddy`` and drop back to modifying the macros yourself - no vendor lockin!\nHowever, if you would like to continue using ``nittygriddy``, you should not edit those macros directly since they might get overwritten and it defeats the purpose of this program in the first place.\n\nContributing\n============\nContributions of any kind (issues, pull requests, general comments...) are always welcome! If you would like to hack on nittygriddy (for example to add new datasets to the default ones) you should check out nitty griddy as an editable python package: ::\n\n $ git clone https://github.com/cbourjau/nittygriddy.git\n $ cd nittygriddy\n $ pip install -e .\n\nThis installs nittygriddy in ``editable`` mode, meaning that any changes to the files in the repository clone are immediately available to the command line tool without re-installation. This means that updates can be raked in with a simple ``git pull origin/master``.", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/cbourjau/nittygriddy", "keywords": "alice,cern,grid,proof", "license": "", "maintainer": "", "maintainer_email": "", "name": "nittygriddy", "package_url": "https://pypi.org/project/nittygriddy/", "platform": "", "project_url": "https://pypi.org/project/nittygriddy/", "project_urls": { "Homepage": "https://github.com/cbourjau/nittygriddy" }, "release_url": "https://pypi.org/project/nittygriddy/1.0.6/", "requires_dist": null, "requires_python": "", "summary": "Convinient way to deploy your ALICE analysis locally (sequential and proof lite) or on the grid", "version": "1.0.6" }, "last_serial": 5476622, "releases": { "0.9.15": [ { "comment_text": "", "digests": { "md5": "00c2c30305186fab6e84e36c70738939", "sha256": "7b8425258b21e3e3ca009aa0a52b0df32ffd3ab4f982ef07cb456797b29b66de" }, "downloads": -1, "filename": "nittygriddy-0.9.15.tar.gz", "has_sig": false, "md5_digest": "00c2c30305186fab6e84e36c70738939", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 287737, "upload_time": "2017-10-31T14:35:29", "url": "https://files.pythonhosted.org/packages/c3/ab/1242f81712f046c26393834a5ca9ef4d3e2f399c54b41b7478cde2cacbaf/nittygriddy-0.9.15.tar.gz" } ], "0.9.16": [ { "comment_text": "", "digests": { "md5": "9eb4efe5f3f4bb8fdefb0da283d74a27", "sha256": "074dc3defc992fa2f232464ce754c24da265b9f9beb58819d3d077175c89c1ef" }, "downloads": -1, "filename": "nittygriddy-0.9.16.tar.gz", "has_sig": false, "md5_digest": "9eb4efe5f3f4bb8fdefb0da283d74a27", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 287729, "upload_time": "2017-10-31T15:01:50", "url": "https://files.pythonhosted.org/packages/93/2b/c99e81ceb07b021e4bd065ca3a5b9e20c311e3dc3e1a515afbf4c981b3e8/nittygriddy-0.9.16.tar.gz" } ], "0.9.17": [ { "comment_text": "", "digests": { "md5": "5d29db925d4ba3576791a0403d308d68", "sha256": "fc7784ce9079d9c9c66eb62fc01063a5c4c343122ad7f5000a77d75a5bf19010" }, "downloads": -1, "filename": "nittygriddy-0.9.17.tar.gz", "has_sig": false, "md5_digest": "5d29db925d4ba3576791a0403d308d68", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 287707, "upload_time": "2017-10-31T15:08:46", "url": "https://files.pythonhosted.org/packages/af/a3/b41f7b173c66c5fc0493f90c2d6d0a7132d8a8e08442b3a66d1a530a27b0/nittygriddy-0.9.17.tar.gz" } ], "1.0.1": [ { "comment_text": "", "digests": { "md5": "57e1b0b53428506a44f7c58339181d28", "sha256": "57c3ca80635bfb575734d2dc3153ac3e31bcff9731452adb7d8ae718068e7698" }, "downloads": -1, "filename": "nittygriddy-1.0.1.tar.gz", "has_sig": false, "md5_digest": "57e1b0b53428506a44f7c58339181d28", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 288250, "upload_time": "2017-11-16T13:48:21", "url": "https://files.pythonhosted.org/packages/b6/d4/ea287cf41d2acffe55d6c5031c51992748c8a76440d0eb451f47a50a8890/nittygriddy-1.0.1.tar.gz" } ], "1.0.2": [ { "comment_text": "", "digests": { "md5": "b0ecaf8d7619e5fd6f1af4f7277b06fb", "sha256": "70eb9478b5a88d20426630cf8505c260ce988cb9ffb5f3d8320af9a4a1525a88" }, "downloads": -1, "filename": "nittygriddy-1.0.2.tar.gz", "has_sig": false, "md5_digest": "b0ecaf8d7619e5fd6f1af4f7277b06fb", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 288487, "upload_time": "2017-12-04T13:34:38", "url": "https://files.pythonhosted.org/packages/ad/77/3a24e9802d2b7fa8873857f91dc612ebd18717de1aed054a3647fcb2e4d6/nittygriddy-1.0.2.tar.gz" } ], "1.0.3": [ { "comment_text": "", "digests": { "md5": "e5cb3f6dafe089bf9b6931ff45ebeeb8", "sha256": "9874760dbcb94a482cbebd8b2f2050a2261f59877e40f39654ae207a90863e99" }, "downloads": -1, "filename": "nittygriddy-1.0.3.tar.gz", "has_sig": false, "md5_digest": "e5cb3f6dafe089bf9b6931ff45ebeeb8", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 288808, "upload_time": "2017-12-07T14:03:33", "url": "https://files.pythonhosted.org/packages/19/d9/8d5a84d6bf89b82c53ebc86e037925725bcd9f6d85d5186f2f0509e64b0d/nittygriddy-1.0.3.tar.gz" } ], "1.0.4": [ { "comment_text": "", "digests": { "md5": "b645815d77340b5dd5cd2b767a5d80bd", "sha256": "cc2a35eca58540115cdfad3fb776d7b77c6da3792b6a7116e1a37b7b80d5b001" }, "downloads": -1, "filename": "nittygriddy-1.0.4.tar.gz", "has_sig": false, "md5_digest": "b645815d77340b5dd5cd2b767a5d80bd", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 288824, "upload_time": "2017-12-07T14:40:51", "url": "https://files.pythonhosted.org/packages/f2/72/cd339cf2d3c6639712b33cf30647551044d28bc536e37e37ac2f0a5d2921/nittygriddy-1.0.4.tar.gz" } ], "1.0.5": [ { "comment_text": "", "digests": { "md5": "970b885d00c334a58aaacc90bec9236c", "sha256": "7909d690e6e56c35581e3fc9a74433b69da99df7e73d256568397f276d68e263" }, "downloads": -1, "filename": "nittygriddy-1.0.5.tar.gz", "has_sig": false, "md5_digest": "970b885d00c334a58aaacc90bec9236c", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 288868, "upload_time": "2017-12-08T11:24:39", "url": "https://files.pythonhosted.org/packages/3f/88/773a854846c81247201e9ebf0d181c8787b3f41461ffe0bd04814506b946/nittygriddy-1.0.5.tar.gz" } ], "1.0.6": [ { "comment_text": "", "digests": { "md5": "618bdab09d333b380c7d305724ce5cd6", "sha256": "c39e701e8284019a18078e15e5e11185423f888d9c740ac09bff30a71f842539" }, "downloads": -1, "filename": "nittygriddy-1.0.6.tar.gz", "has_sig": false, "md5_digest": "618bdab09d333b380c7d305724ce5cd6", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 289057, "upload_time": "2018-02-14T11:45:36", "url": "https://files.pythonhosted.org/packages/df/96/3f6afc177726fd366b5dda4d75930f0008098323fc72356cf8b36aaff31b/nittygriddy-1.0.6.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "618bdab09d333b380c7d305724ce5cd6", "sha256": "c39e701e8284019a18078e15e5e11185423f888d9c740ac09bff30a71f842539" }, "downloads": -1, "filename": "nittygriddy-1.0.6.tar.gz", "has_sig": false, "md5_digest": "618bdab09d333b380c7d305724ce5cd6", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 289057, "upload_time": "2018-02-14T11:45:36", "url": "https://files.pythonhosted.org/packages/df/96/3f6afc177726fd366b5dda4d75930f0008098323fc72356cf8b36aaff31b/nittygriddy-1.0.6.tar.gz" } ] }