{ "info": { "author": "B-Open Solutions srl", "author_email": "oss@bopen.eu", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Environment :: Console", "Environment :: Web Environment", "Intended Audience :: Developers", "Intended Audience :: System Administrators", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Topic :: System :: Monitoring" ], "description": "=========\nMarioBros\n=========\n\nMarioBros is a Python module to configure `Spotify-Luigi `_ in a makefile-like manner.\n\nSpotify-Luigi is a Python module that helps you build complex pipelines of batch jobs.\nIt handles dependency resolution, workflow management, visualization etc.\nThe configuration files will called MarioFiles.\n\nMaster branch status:\n\n.. image:: https://travis-ci.org/bopen/mariobros.svg?branch=master\n :target: https://travis-ci.org/bopen/mariobros\n :alt: Build Status on Travis CI\n\n.. image:: https://coveralls.io/repos/bopen/mariobros/badge.svg?branch=master&service=github\n :target: https://coveralls.io/github/bopen/mariobros\n :alt: Coverage Status on Coveralls\n\nWriting MarioFile\n-----------------\n\nThe MarioFiles are structured in several section representing the various tasks.\n\nTask definition\n+++++++++++++++\n\nAn example of task definition is::\n\n [task_name]\n target: source1 source2 source3\n task_command\n\nThe task name is defined in the square brackets. Target and sources are divided by two points.\nIn the line below there is the task command.\nIn the task_command you can call the target name and the sources name with ``${TARGET}`` and ``${SOURCES[i]}`` where ``i``\nis an index or a python slice.\nIn the curly brackets you can execute python code with variables defined upper.\nYou can match parts of the target name with parts of the sources name with python regular expression syntax.\nFor example::\n\n [task_name]\n (*.)-(*.)-(*.).txt: \\1.txt \\2.txt \\3.txt\n task_command --output ${TARGET} --first-input ${SOURCES[0]} --other-inputs ${SOURCES[1:]}\n\nThis task match a target like ``first-second-third.txt`` with 3 source files named ``first.txt``, ``second.txt`` and ``third.txt``.\n``\\1`` represent the first match, ``\\2`` represent the second match and ``\\3`` the third one.\n\nYou can also define task variables and use them below. You have to define them below the task name as follow::\n\n [task_name]\n variable1 = value1\n variable2 = value2\n target: source\n task_command -o ${TARGET} -i ${SOURCES} --par1 ${variable1} --par2 ${variable2}\n\n\nDefault section\n+++++++++++++++\n\nThe default section is the one defined, without specifying the name, at the top of the file. In this section there are definitions of the global variables\nand the definition of the default task. The default task is executed when no other task is requested::\n\n global_variable1 = value1\n global_variable2 = value2\n\n default_target: source1 source2\n default_task_command\n\nAt the top of the file you can include other MarioFiles with the path in this way::\n\n include mariofile_path.ini\n\nThe global variables of the included MarioFile will be added at the top of the MarioFile.\nThe tasks of the included MarioFiles will be added at the end of the MarioFile.\nIf there are variables or tasks with the same names, only the top level one will be included.\n\nSumming up, a typical MarioFile will look like this::\n\n include included_mariofile.ini\n\n global_var1 = value1\n global_var2 = value2\n\n (*.)-main_target.out: \\1-\\1-task1\n default_task -o ${SOURCES} -i ${TARGET}\n\n [task_1]\n RESOURCES_CPU = 4\n (*.)-(*.)-task1: source1 source2\n task_1 -o ${TARGET} -i ${SOURCES} -j ${RESOURCES_CPU}\n\nExecuting Mario\n---------------\n\nThe Mario command line is:\n\n.. code-block:: console\n\n $ mario --help\n Usage: mario [OPTIONS] [TARGETS]...\n\n Options:\n -f, --file, --mariofile PATH Main configuration file\n --workers INTEGER Set the number of workers\n --local-scheduler Run local scheduler.\n --print-ns Print namespaces: Print the MarioFile with the\n included tasks and variables\n -n, --dry-run Don't actually run any commands; just print\n them.\n --help Show this message and exit.\n\nWith:\n\n 1. All the mario options and arguments are optional.\n 2. The default task request is ``[DEFAULT]``.\n 3. The default mariofile is ``mario.ini``.\n 4. With ``--print-ns`` flag it print the whole MarioFile with the included tasks and variables.\n 5. With ``--dry-run`` flag it doesn't actually run any commands; just print them.\n\nWith external scheduler\n+++++++++++++++++++++++\n\nFirst of all you have to run ``luigid``::\n\n $ luigid\n\nThen you can run ``mario`` script with the command line described above::\n\n $ mario\n\nIn this way mario will execute the default target with mario.ini as MarioFile.\nYou can also request a specific target with a MarioFile different from mario.ini as follow::\n\n $ mario -f my_mariofile.ini target.out\n\nYou can visualize a scheduler in `localhost:8082 `_ address.\nThere will be a list of the tasks and a tree diagram of the processing.\n\nWith local scheduler\n++++++++++++++++++++\n\nIf you don't need to visualize the scheduler you can run ``mario`` without ``luigid`` running and with the local scheduler as follow::\n\n $ mario --local-scheduler\n\nLuigi configuration file\n------------------------\n\nLuigi can store the statistic of the processing and can manage the resources.\nYou have to configure luigi writing *client.cfg* file and then launch ``luigid`` from the *client.cfg* directory.\n\nWriting statistics in database\n++++++++++++++++++++++++++++++\n\nIn *client.cfg* file you can configure the db path for the statistics. An example of *client.cfg* is::\n\n [scheduler]\n record_task_history = True\n state_path = /path/to/luigi-state.pickle\n\n [task_history]\n db_connection = sqlite:////path/to/db/luigi-task-hist.db\n\nResources management\n++++++++++++++++++++\n\nYou can also specify the required resources for the single tasks.\n\nYou have to define the available resources in the *client.cfg* file defining the ``[resources]`` section as follow::\n\n [resources]\n cpus = 64\n\nYou can specify required resources for the single tasks setting the local task variables ``RESOURCES_RESOURCE_NAME``\nwhere ``RESOURCE_NAME`` is the name of the resource defined in the *client.cfg*, for example::\n\n [task_name]\n RESOURCES_cpus = 4\n target: sources\n task_command -j ${RESOURCES_cpus}\n\nThe request resource is ``4`` for ``cpus``.\n\nInstall\n-------\n\ninstall in the current python environment::\n\n pip install mariobros", "description_content_type": null, "docs_url": null, "download_url": "https://github.com/bopen/mariobros/archive/0.4.2.tar.gz", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/bopen/mariobros", "keywords": "", "license": "Apache License 2.0", "maintainer": "", "maintainer_email": "", "name": "mariobros", "package_url": "https://pypi.org/project/mariobros/", "platform": "", "project_url": "https://pypi.org/project/mariobros/", "project_urls": { "Download": "https://github.com/bopen/mariobros/archive/0.4.2.tar.gz", "Homepage": "https://github.com/bopen/mariobros" }, "release_url": "https://pypi.org/project/mariobros/0.4.2/", "requires_dist": null, "requires_python": "", "summary": "Simple configuration for Spotify Luigi.", "version": "0.4.2" }, "last_serial": 2676044, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "898058a0ef97fd687238597dd6ea5136", "sha256": "1a322614cf55d624cdc8120242e97f418e9b079dc202fa525ada1338d26f6701" }, "downloads": -1, "filename": "mariobros-0.1.0.tar.gz", "has_sig": false, "md5_digest": "898058a0ef97fd687238597dd6ea5136", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18421, "upload_time": "2016-01-13T10:08:47", "url": "https://files.pythonhosted.org/packages/c9/3c/a79c28ee8f92f7b6936f9de2c04e39e06aa3e5c91e2147f9cfe7d96dcbf1/mariobros-0.1.0.tar.gz" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "c8f19c1f85d3cd2fcc6755d322091c96", "sha256": "6d9e822d5da3a1a5d4b420a6786e86e4e7678d63aab2ff8116de514b45c67255" }, "downloads": -1, "filename": "mariobros-0.2.0.tar.gz", "has_sig": false, "md5_digest": "c8f19c1f85d3cd2fcc6755d322091c96", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 18969, "upload_time": "2016-01-19T09:48:56", "url": "https://files.pythonhosted.org/packages/05/7b/a94a563958339189a5dbbca1ca191389378da3cd047e8a91bc8376afc6ba/mariobros-0.2.0.tar.gz" } ], "0.3.0": [ { "comment_text": "", "digests": { "md5": "8d1b021a8683fe6f65d7f04d9281a285", "sha256": "213685d8d571751986196c01a5cd727a95c09127cf3196d078b1ee073ba52c2f" }, "downloads": -1, "filename": "mariobros-0.3.0.tar.gz", "has_sig": false, "md5_digest": "8d1b021a8683fe6f65d7f04d9281a285", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 19617, "upload_time": "2016-02-10T16:51:22", "url": "https://files.pythonhosted.org/packages/7e/a2/4f20529dc0bbebf5379f03a19e366af6e5a861b7aa11adbc32ca70a1f379/mariobros-0.3.0.tar.gz" } ], "0.4.0": [ { "comment_text": "", "digests": { "md5": "fa3cb7c8e5e38cfb17b544a8b18054ef", "sha256": "fb5a96949be949707be7065abba35bb3d22c0276231531b30b0140b691a9e8d6" }, "downloads": -1, "filename": "mariobros-0.4.0.tar.gz", "has_sig": false, "md5_digest": "fa3cb7c8e5e38cfb17b544a8b18054ef", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 22319, "upload_time": "2016-08-02T10:50:15", "url": "https://files.pythonhosted.org/packages/59/87/3061b6fab0008fd9e8c68012fdaa0d1614b6230e62fea49de15dec763368/mariobros-0.4.0.tar.gz" } ], "0.4.1": [ { "comment_text": "", "digests": { "md5": "db6ae4f47b413df69904f26777a6f5dc", "sha256": "bc7f68f38fdf1d2859d60fd70e8e6897f12794f6b0694037d71f4f3aa40efcc5" }, "downloads": -1, "filename": "mariobros-0.4.1.tar.gz", "has_sig": false, "md5_digest": "db6ae4f47b413df69904f26777a6f5dc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 22478, "upload_time": "2017-02-14T16:05:40", "url": "https://files.pythonhosted.org/packages/bf/c1/20e7462e6125c08e67218aa6ac604d77e96b8de7f5161c42fd53be4f7713/mariobros-0.4.1.tar.gz" } ], "0.4.2": [ { "comment_text": "", "digests": { "md5": "cd394813dcc651e2e09e3b1ef3f5ac96", "sha256": "5756cce18752b29ed751f82e56f2cfaf747bf9cb80f3f64717542ffddf9f70c5" }, "downloads": -1, "filename": "mariobros-0.4.2.tar.gz", "has_sig": false, "md5_digest": "cd394813dcc651e2e09e3b1ef3f5ac96", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 22551, "upload_time": "2017-03-01T16:28:03", "url": "https://files.pythonhosted.org/packages/34/6c/56e87b9a3c5959c0796aea364df50c65c14573e66def6848f55e7f6d454e/mariobros-0.4.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "cd394813dcc651e2e09e3b1ef3f5ac96", "sha256": "5756cce18752b29ed751f82e56f2cfaf747bf9cb80f3f64717542ffddf9f70c5" }, "downloads": -1, "filename": "mariobros-0.4.2.tar.gz", "has_sig": false, "md5_digest": "cd394813dcc651e2e09e3b1ef3f5ac96", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 22551, "upload_time": "2017-03-01T16:28:03", "url": "https://files.pythonhosted.org/packages/34/6c/56e87b9a3c5959c0796aea364df50c65c14573e66def6848f55e7f6d454e/mariobros-0.4.2.tar.gz" } ] }