{ "info": { "author": "Matteo Concas and Dario Berzano", "author_email": "matteo.concas@cern.ch", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Intended Audience :: Education", "License :: OSI Approved :: GNU General Public License v3 (GPLv3)", "Programming Language :: Python :: 2.7", "Topic :: Scientific/Engineering :: Physics" ], "description": "jdl2makeflow\n============\n\n\nRun `AliEn `__ JDLs on multiple platforms\nusing `Makeflow `__.\n\nRequirements\n------------\n\n`Makeflow `__ needs to be\ninstalled on your system. Makeflow is part of the Cooperative Computing\ntools (cctools). To install it, `download the latest version of\ncctools `__ then\nunpack, compile and install it (we are assuming 6.1.1 is the latest,\ncheck the download page first):\n\n::\n\n cd /tmp\n curl -L http://ccl.cse.nd.edu/software/files/cctools-6.1.1-source.tar.gz | tar xzf -\n cd cctools-*-source/\n ./configure && make -j10\n sudo make install\n\nRun the last command (``make install``) *as root* to install it\nsystem-wide. Adjust ``-j10`` to the number of parallel cores you want to\nuse during the build. If you do not have root privileges:\n\n::\n\n cd /tmp\n curl -L http://ccl.cse.nd.edu/software/files/cctools-6.1.1-source.tar.gz | tar xzf -\n cd cctools-*-source/\n ./configure --prefix=$HOME/cctools && make -j10 && make install\n echo 'export PATH=$HOME/cctools/bin:$PATH' >> ~/.bashrc\n echo 'export LD_LIBRARY_PATH=$HOME/cctools/bin:$LD_LIBRARY_PATH' >> ~/.bashrc\n\nWe are installing under ``~/cctools`` but you can use the directory you\nwant. Also, we are assuming your shell configuration file is\n``~/.bashrc``, adjust it according to your shell.\n\nGet jdl2makeflow\n----------------\n\nAs easy as:\n\n::\n\n sudo pip install alien-jdl2makeflow\n\nIf you cannot install it as root, you will probably have to export some\nPython variables to make it work. If you have a user installation of\nsome Python distribution like\n`Anaconda `__ this is probably\nalready done for you.\n\nBasic usage\n-----------\n\nGet and configure the JDL used to run a Monte Carlo on AliEn, and all\nthe required files. Normally you would only require the ``Custom.cfg``\nfile. The JDL can contain variable overrides in a way that AliEn will\nignore them and they will only be considered by the Makeflow workflow:\nthis allows you to keep a single JDL that works both locally and on the\nGrid.\n\nRun:\n\n::\n\n jdl2makeflow /path/to/job.jdl\n\nBy default, it will print a summary and create all necessary files under\na working directory called ``work`` (override with ``-w``). You then\nneed to move to the working directory and run:\n\n::\n\n cd work\n makeflow\n\nRelevant JDL variables\n----------------------\n\nNot all JDL variables are used for local use, and some of them are\ninterpreted in a different way.\n\n- ``Executable``: the AliEn path of the executable to run. For local\n use, we will only consider the program name and strip the path, and\n search for it in the following directories in order:\n\n 1. full path first\n 2. basename (must be in ``$PATH``)\n 3. ``$ALIDPG_ROOT/bin``\n 4. current working directory\n\n- ``SplitArguments``: arguments to pass to the executable.\n- ``InputFile``: list of files that need to be made available in the\n job's working directory for the job to run. Only the basename of the\n file will be considered and it will be searched for in the current\n directory. No AliEn access will be performed.\n- ``Output``: a list telling what of the files produced by the job need\n to be copied to the destination. Files can also be placed in zip\n archives. See the example JDL for more information.\n- ``OutputDir``: the output directory for each job. This is normally an\n AliEn path, but locally we can either specify a XRootD path\n (``root://...``) or a local path. Note that XRootD paths might\n require authentication information to be available to the job.\n- ``Packages``: packages to be loaded in the Grid environment. They are\n expected from CVMFS, and CVMFS must be available locally. You can\n also use local installations for testing, see ``SourceEnvScript``\n below.\n- ``JDLVariables``: list of arbitrary variables from the JDL that will\n be exported in the job's environment. The variable name will be\n altered. If, for instance, you want to export the JDL variable\n ``ArbitraryVar``, this will be available to your jobs as\n ``ALIEN_JDL_ARBITRARYVAR``. The same convention is used by AliEn for\n Grid jobs.\n- ``Split``: determine how many jobs have to be run of the kind\n specified by the JDL. The syntax ``production:123-456`` tells the\n script to run jobs with a different ID from 123 to 456 included\n (that's 334 jobs). You will probably want to change it for local use\n as the range is very large on the Grid. The job index is made\n available to some other variables through the ``#alien_counter#``\n variable, see below.\n\nSince the same JDL will be used for running many jobs, it is in some\ncases useful to distinguish between output directories, and to tell the\njob what is its index. You can use in variable values:\n\n- ``#alien_counter#``: will be replaced with the job index\n- ``#alien_counter_05i#``: will be replaced with the job index\n zero-padded to 5 digits (any format supported by ``printf`` can be\n specified of course)\n\nSee the examples for more information.\n\nExtra JDL variables\n-------------------\n\nThe following JDL variables are interpreted only by ``jdl2makeflow`` and\nwill be ignored by AliEn.\n\n- ``SourceEnvScript``: path to an environment script to be sourced in\n order to load the required packages. If this script is defined, it\n will be automatically loaded by each job, and ``Packages`` will be\n ignored. This is useful for local development when one wants to test\n changes with a local AliRoot build. If ``SourceEnvScript`` is\n specified, CVMFS is not required for packages.\n- ``ExtraVariables``: same as ``JDLVariables``, but the variables\n listed (which must be defined in the JDL) will be exported in the job\n environment as-is, with their name not manipulated. So, the variable\n ``ArbitraryVar`` will be exported as ``ArbitraryVar``.\n- ``NextStages``: the toplevel JDL runs several jobs of the same kind,\n but other tiers of processing will follow (merging stages mostly).\n This list tells the local workflow what are the next stages (order\n does not matter). For the moment, only the values ``FinalQA`` and\n ``SpacePointCalibration`` are supported. This variable allows you to\n run those stages without supplying their respective JDLs (parameters\n will be deduced from the current one and modified accordingly).\n- ``QADetectorInclude``: string with a list of space-separated detector\n names to be included when generating the QA plots. Leave empty for\n including all detectors. In order to expose it to the job you must\n add it to ``JDLVariables``.\n- ``QADetectorExclude``: exclude detectors from QA plots. Same format\n as ``QADetectorInclude``. Has to be added to ``JDLVariables`` too.\n- ``DontArchive``: set it to ``1`` to store output files, as specified\n in ``Output``, as they are, without compressing them. Useful for\n debug.\n- ``SaveAll``: set it to ``1`` to save all files produced by jobs,\n ignoring ``Output`` completely. Files will not be compressed in zip\n files. Useful for debug.\n\nOverriding JDL variables\n------------------------\n\nAliEn JDL files have variables of type \"string\" and \"list\":\n\n::\n\n StringVar = \"this is a string\";\n ListVar = { \"this\", \"is\", \"a\", \"list\" };\n\nIn the most common case you need to override some variables for the\nlocal use. For instance, the variable ``OutputDir`` represents the AliEn\noutput directory and does not have any sense locally. You can override\nevery variable by defining a new variable with the same name and\n``_override`` appended:\n\n::\n\n OutputDir = \"/alien/path/not/making/sense/locally\";\n OutputDir_override = \"/home/myuser/joboutput\";\n\nThe latter will be considered by Makeflow. You can also append to\nstrings and lists. For instance, the ``SplitArguments`` variable is a\nstring representing the arguments to pass to the executable, but in the\nlocal scenario you might want to pass more arguments. Appends work the\nsame as overrides, but you will use the ``_append`` name at the end:\n\n::\n\n SplitArguments = \"--run 244411 --mode full --uid #alien_counter# --nevents 200 --generator Pythia8_Monash2013 --trigger Custom.cfg\";\n SplitArguments_append = \" --ocdb $OCDB_PATH\";\n\nor, you need to provide the input directory with credential\ninformations:\n\n::\n\n InputFile = { \"LF:/alice/cern.ch/user/a/aliprod/LHC16h8a/Custom.cfg\" };\n InputFile_append = { \"my-proxy\" };\n\nBugs and issues\n---------------\n\nThis project was originally conceived to run ALICE Monte Carlos locally,\nor on local batch farms (including\n`Mesos `__!) with Makeflow,\nusing the exact same JDL files one would use on the AliEn Grid.\n\nIts support is therefore very limited to the ALICE Monte Carlo use\ncases, but we are extending it to support more use cases more flexibly.\n\nIn case of problems please `open an\nissue `__.\n\n :target: https://badge.fury.io/py/alien-jdl2makeflow", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/alisw/release-validation", "keywords": "HEP Computing", "license": "GPL", "maintainer": "", "maintainer_email": "", "name": "alien-jdl2makeflow", "package_url": "https://pypi.org/project/alien-jdl2makeflow/", "platform": "", "project_url": "https://pypi.org/project/alien-jdl2makeflow/", "project_urls": { "Homepage": "https://github.com/alisw/release-validation" }, "release_url": "https://pypi.org/project/alien-jdl2makeflow/0.0.5/", "requires_dist": null, "requires_python": ">=2.7", "summary": "Run AliEn JDLs with Makeflow", "version": "0.0.5" }, "last_serial": 3050305, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "383d92ac04401daaaa209afe2f0f5d14", "sha256": "3523dde5d7d78fa0e464fc560c1cee89023fa7729835e6f384a34ab916cd2677" }, "downloads": -1, "filename": "alien_jdl2makeflow-0.0.1.tar.gz", "has_sig": false, "md5_digest": "383d92ac04401daaaa209afe2f0f5d14", "packagetype": "sdist", "python_version": "source", "requires_python": ">=2.7", "size": 9915, "upload_time": "2017-07-14T13:26:24", "url": "https://files.pythonhosted.org/packages/a5/69/b070d5da5ab3b584de22e5714939eef1900e0d1370ea812aa7d016378037/alien_jdl2makeflow-0.0.1.tar.gz" } ], "0.0.2": [ { "comment_text": "", "digests": { "md5": "28234e7d9515f9d14ec60b9235096cca", "sha256": "9b203323c7846308f7d31c918388fe330b7e6b1bc4852a3fd615cfc41867e0bf" }, "downloads": -1, "filename": "alien_jdl2makeflow-0.0.2.tar.gz", "has_sig": false, "md5_digest": "28234e7d9515f9d14ec60b9235096cca", "packagetype": "sdist", "python_version": "source", "requires_python": ">=2.7", "size": 9882, "upload_time": "2017-07-14T13:39:11", "url": "https://files.pythonhosted.org/packages/d4/e5/7db3516464baa4206dbfa99d1774c3817012bfc773ab37e777fd6f713654/alien_jdl2makeflow-0.0.2.tar.gz" } ], "0.0.3": [ { "comment_text": "", "digests": { "md5": "2e73661930ea277315e7159f200dd12f", "sha256": "43b77a3f3023f1d56a9a96cd23c5113146b010d182446b59af4bb999729c9066" }, "downloads": -1, "filename": "alien-jdl2makeflow-0.0.3.tar.gz", "has_sig": false, "md5_digest": "2e73661930ea277315e7159f200dd12f", "packagetype": "sdist", "python_version": "source", "requires_python": ">=2.7", "size": 13650, "upload_time": "2017-07-14T14:52:01", "url": "https://files.pythonhosted.org/packages/ec/07/4b885fedcf3d4d9f4e86b64e6780c79f09875bb9ec3ca4498e5f737fb717/alien-jdl2makeflow-0.0.3.tar.gz" } ], "0.0.4": [ { "comment_text": "", "digests": { "md5": "a6513f3206f06ff1c2538fc9a36eb84f", "sha256": "26b1af8b4d91d6913e4d11cf527acd058e6a29b702a62015116ad86fc9bc6493" }, "downloads": -1, "filename": "alien-jdl2makeflow-0.0.4.tar.gz", "has_sig": false, "md5_digest": "a6513f3206f06ff1c2538fc9a36eb84f", "packagetype": "sdist", "python_version": "source", "requires_python": ">=2.7", "size": 12507, "upload_time": "2017-07-17T08:20:57", "url": "https://files.pythonhosted.org/packages/3a/0c/08398003c3d9c2a3aa1c40f3ada82fe239c2725b48e2a6b52d1905649cad/alien-jdl2makeflow-0.0.4.tar.gz" } ], "0.0.5": [ { "comment_text": "", "digests": { "md5": "e053653fab78bab23a4b0b44923d4f50", "sha256": "fe6f6662c3a29c400a81ae72e4e1ca27f85400ff43d59e1d431981d2004f43d9" }, "downloads": -1, "filename": "alien-jdl2makeflow-0.0.5.tar.gz", "has_sig": false, "md5_digest": "e053653fab78bab23a4b0b44923d4f50", "packagetype": "sdist", "python_version": "source", "requires_python": ">=2.7", "size": 13506, "upload_time": "2017-07-26T15:24:29", "url": "https://files.pythonhosted.org/packages/66/df/e14b3f3ac897ba7800c31fd2f79dc397fd4ab7906fa35830af7925d0cf33/alien-jdl2makeflow-0.0.5.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "e053653fab78bab23a4b0b44923d4f50", "sha256": "fe6f6662c3a29c400a81ae72e4e1ca27f85400ff43d59e1d431981d2004f43d9" }, "downloads": -1, "filename": "alien-jdl2makeflow-0.0.5.tar.gz", "has_sig": false, "md5_digest": "e053653fab78bab23a4b0b44923d4f50", "packagetype": "sdist", "python_version": "source", "requires_python": ">=2.7", "size": 13506, "upload_time": "2017-07-26T15:24:29", "url": "https://files.pythonhosted.org/packages/66/df/e14b3f3ac897ba7800c31fd2f79dc397fd4ab7906fa35830af7925d0cf33/alien-jdl2makeflow-0.0.5.tar.gz" } ] }