{
    "info": {
        "author": "Samuel Lampa",
        "author_email": "samuel.lampa@farmbio.uu.se",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Environment :: Console",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: MIT License",
            "Natural Language :: English",
            "Operating System :: POSIX :: Linux",
            "Programming Language :: Python",
            "Programming Language :: Python :: 2",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.4",
            "Topic :: Scientific/Engineering",
            "Topic :: Scientific/Engineering :: Bio-Informatics",
            "Topic :: Scientific/Engineering :: Chemistry"
        ],
        "description": ".. note::\n\n   For the latest source, issues and discussion, etc, please visit the\n   `GitHub repository <https://github.com/samuell/sciluigi>`_\n\n\n.. figure:: http://i.imgur.com/2aMT04J.png\n   :alt: SciLuigi Logo\n\n   SciLuigi Logo\n\n-  ***UPDATE, Nov, 2016: A paper with the motivation and design\n   decisions behind SciLuigi `now\n   available <http://dx.doi.org/10.1186/s13321-016-0179-6>`__***\n-  If you use SciLuigi in your research, please cite it like this: Lampa\n   S, Alvarsson J, Spjuth O. Towards agile large-scale predictive\n   modelling in drug discovery with flow-based programming design\n   principles. *J Cheminform*. 2016.\n   doi:\\ `10.1186/s13321-016-0179-6 <http://dx.doi.org/10.1186/s13321-016-0179-6>`__.\n-  ***A Virtual Machine with a realistic, runnable, example workflow in\n   a Jupyter Notebook, is available\n   `here <https://github.com/pharmbio/bioimg-sciluigi-casestudy>`__***\n-  ***Watch a 10 minute screencast going through the basics of using\n   SciLuigi `here <https://www.youtube.com/watch?v=gkKUWskRbjw>`__***\n-  ***See a poster describing the motivations behind SciLuigi\n   `here <http://dx.doi.org/10.13140/RG.2.1.1143.6246>`__***\n\nScientific Luigi (SciLuigi for short) is a light-weight wrapper library\naround `Spotify <http://spotify.com>`__'s\n`Luigi <http://github.com/spotify/luigi>`__ workflow system that aims to\nmake writing scientific workflows more fluent, flexible and modular.\n\nLuigi is a flexile and fun-to-use library. It has turned out though that\nits default way of defining dependencies by hard coding them in each\ntask's requires() function is not optimal for some type of workflows\ncommon e.g. in bioinformatics where multiple inputs and outputs, complex\ndependencies, and the need to quickly try different workflow\nconnectivity in an explorative fashion is central to the way of working.\n\nSciLuigi was designed to solve some of these problems, by providing the\nfollowing \"features\" over vanilla Luigi:\n\n-  Separation of dependency definitions from the tasks themselves, for\n   improved modularity and composability.\n-  Inputs and outputs implemented as separate fields, a.k.a. \"ports\", to\n   allow specifying dependencies between specific input and\n   output-targets rather than just between tasks. This is again to let\n   such details of the network definition reside outside the tasks.\n-  The fact that inputs and outputs are object fields, also allows\n   auto-completion support to ease the network connection work (Works\n   great e.g. with\n   `jedi-vim <https://github.com/davidhalter/jedi-vim>`__).\n-  Inputs and outputs are connected with an intuitive \"single-assignment\n   syntax\".\n-  \"Good default\" high-level logging of workflow tasks and execution\n   times.\n-  Produces an easy to read audit-report with high level information per\n   task.\n-  Integration with some HPC workload managers. (So far only\n   `SLURM <http://slurm.schedmd.com/>`__ though).\n\nBecause of Luigi's easy-to-use API these changes have been implemented\nas a very thin layer on top of luigi's own API with no changes at all to\nthe luigi core, which means that you can continue leveraging the work\nalready being put into maintaining and further developing luigi by the\nteam at Spotify and others.\n\nWorkflow code quick demo\n------------------------\n\n***For a brief 10 minute screencast going through the basics below, see\n`this link <https://www.youtube.com/watch?v=gkKUWskRbjw>`__***\n\nJust to give a quick feel for how a workflow definition might look like\nin SciLuigi, check this code example (implementation of tasks hidden\nhere for brevity. See Usage section further below for more details):\n\n.. code:: python\n\n    import sciluigi as sl\n\n    class MyWorkflow(sl.WorkflowTask):\n        def workflow(self):\n            # Initialize tasks:\n            foowrt = self.new_task('foowriter', MyFooWriter)\n            foorpl = self.new_task('fooreplacer', MyFooReplacer,\n                replacement='bar')\n\n            # Here we do the *magic*: Connecting outputs to inputs:\n            foorpl.in_foo = foowrt.out_foo\n\n            # Return the last task(s) in the workflow chain.\n            return foorpl\n\nThat's it! And again, see the \"usage\" section just below for a more\ndetailed description of getting to this!\n\nSupport: Getting help\n---------------------\n\nPlease use the `issue\nqueue <https://github.com/pharmbio/sciluigi/issues>`__ for any support\nquestions, rather than mailing the author(s) directly, as the solutions\ncan then help others who face similar issues (we are a very small team\nwith very limited time, so this is important).\n\nPrerequisites\n-------------\n\n-  Python 2.7 - 3.4\n-  Luigi 1.3.x - 2.0.1\n\nInstall\n-------\n\n1. Install SciLuigi, including its dependencies (luigi etc), through\n   PyPI:\n\n   .. code:: bash\n\n       pip install sciluigi\n\n2. Now you can use the library by just importing it in your python\n   script, like so:\n\n   .. code:: python\n\n       import sciluigi\n\n   Note that you can aliase it to a shorter name, for brevity, and to\n   save keystrokes:\n\n   .. code:: python\n\n       import sciluigi as sl\n\nUsage\n-----\n\nCreating workflows in SciLuigi differs slightly from how it is done in\nvanilla Luigi. Very briefly, it is done in these main steps:\n\n1. Create a workflow tasks class\n2. Create task classes\n3. Add the workflow definition in the workflow class's ``workflow()``\n   method.\n4. Add a run method at the end of the script\n5. Run the script\n\nCreate a Workflow task\n~~~~~~~~~~~~~~~~~~~~~~\n\nThe first thing to do when creating a workflow, is to define a workflow\ntask.\n\nYou do this by:\n\n1. Creating a subclass of ``sciluigi.WorkflowTask``\n2. Implementing the ``workflow()`` method.\n\nExample:\n^^^^^^^^\n\n.. code:: python\n\n    import sciluigi\n\n    class MyWorkflow(sciluigi.WorkflowTask):\n        def workflow(self):\n            pass # TODO: Implement workflow here later!\n\nCreate tasks\n~~~~~~~~~~~~\n\nThen, you need to define some tasks that can be done in this workflow.\n\nThis is done by:\n\n1. Creating a subclass of ``sciluigi.Task`` (or ``sciluigi.SlurmTask``\n   if you want Slurm support)\n2. Adding fields named ``in_<yournamehere>`` for each input, in the new\n   task class\n3. Define methods named ``out_<yournamehere>()`` for each output, that\n   return ``sciluigi.TargetInfo`` objects. (sciluigi.TargetInfo is\n   initialized with a reference to the task object itself - typically\n   ``self`` - and a path name, where upstream tasks paths can be used).\n4. Define luigi parameters to the task.\n5. Implement the ``run()`` method of the task.\n\nExample:\n^^^^^^^^\n\nLet's define a simple task that just writes \"foo\" to a file named\n``foo.txt``:\n\n.. code:: python\n\n    class MyFooWriter(sciluigi.Task):\n        # We have no inputs here\n        # Define outputs:\n        def out_foo(self):\n            return sciluigi.TargetInfo(self, 'foo.txt')\n        def run(self):\n            with self.out_foo().open('w') as foofile:\n                foofile.write('foo\\n')\n\nThen, let's create a task that replaces \"foo\" with \"bar\":\n\n.. code:: python\n\n    class MyFooReplacer(sciluigi.Task):\n        replacement = sciluigi.Parameter() # Here, we take as a parameter\n                                      # what to replace foo with.\n        # Here we have one input, a \"foo file\":\n        in_foo = None\n        # ... and an output, a \"bar file\":\n        def out_replaced(self):\n            # As the path to the returned target(info), we\n            # use the path of the foo file:\n            return sciluigi.TargetInfo(self, self.in_foo().path + '.bar.txt')\n        def run(self):\n            with self.in_foo().open() as in_f:\n                with self.out_replaced().open('w') as out_f:\n                    # Here we see that we use the parameter self.replacement:\n                    out_f.write(in_f.read().replace('foo', self.replacement))\n\nThe last lines, we could have instead written using the command-line\n``sed`` utility, available in linux, by calling it on the commandline,\nwith the built-in ``ex()`` method:\n\n.. code:: python\n\n        def run(self):\n            # Here, we use the in-built self.ex() method, to execute commands:\n            self.ex(\"sed 's/foo/{repl}/g' {inpath} > {outpath}\".format(\n                repl=self.replacement,\n                inpath=self.in_foo().path,\n                outpath=self.out_replaced().path))\n\nWrite the workflow definition\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNow, we can use these two tasks we created, to create a simple workflow,\nin our workflow class, that we also created above.\n\nWe do this by:\n\n1. Instantiating the tasks, using the\n   ``self.new_task(<unique_taskname>, <task_class>, *args, **kwargs)``\n   method, of the workflow task.\n2. Connect the tasks together, by pointing the right ``out_*`` method to\n   the right ``in_*`` field.\n3. Returning the last task in the chain, from the workflow method.\n\nExample:\n^^^^^^^^\n\n.. code:: python\n\n    import sciluigi\n    class MyWorkflow(sciluigi.WorkflowTask):\n        def workflow(self):\n            foowriter = self.new_task('foowriter', MyFooWriter)\n            fooreplacer = self.new_task('fooreplacer', MyFooReplacer,\n                replacement='bar')\n\n            # Here we do the *magic*: Connecting outputs to inputs:\n            fooreplacer.in_foo = foowriter.out_foo\n\n            # Return the last task(s) in the workflow chain.\n            return fooreplacer\n\nAdd a run method to the end of the script\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNow, the only thing that remains, is adding a run method to the end of\nthe script.\n\nYou can use luigi's own ``luigi.run()``, or our own two methods:\n\n1. ``sciluigi.run()``\n2. ``sciluigi.run_local()``\n\nThe ``run_local()`` one, is handy if you don't want to run a central\nscheduler daemon, but just want to run the workflow as a script.\n\nBoth of the above take the same options as ``luigi.run()``, so you can\nfor example set the main class to use (our workflow task):\n\n::\n\n    # End of script ....\n    if __name__ == '__main__':\n        sciluigi.run_local(main_task_cls=MyWorkflow)\n\nRun the workflow\n~~~~~~~~~~~~~~~~\n\nNow, you should be able to run the workflow as simple as:\n\n.. code:: bash\n\n    python myworkflow.py\n\n... provided of course, that the workflow is saved in a file named\nmyworkflow.py.\n\nMore Examples\n~~~~~~~~~~~~~\n\nSee the `examples\nfolder <https://github.com/samuell/sciluigi/tree/master/examples>`__ for\nmore detailed examples!\n\nMore links, background info etc.\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe basic idea behind SciLuigi, and a preceding solution to it, was\npresented in workshop (e-Infra MPS 2015) talk: -\n`Slides <http://www.slideshare.net/SamuelLampa/building-workflows-with-spotifys-luigi>`__\n- `Video <https://www.youtube.com/watch?v=f26PqSXZdWM>`__\n\nSee also `this collection of\nlinks <http://bionics.it/posts/our-experiences-using-spotifys-luigi-for-bioinformatics-workflows>`__,\nto more of our reported experiences using Luigi, which lead up to the\ncreation of SciLuigi.\n\nKnown Limitations\n-----------------\n\n-  Changing the workflow scheduling based on data sent as parameters, is\n   not possible.\n-  If you have an unknown number of outputs from a task, for which you\n   want to start a full branch of the workflow, this is not possible\n   either.\n\nBoth of the limitations are due to the fact that Luigi does scheduling\nand execution separately (with the exception of Luigi's `dynamic\ndependencies <http://luigi.readthedocs.io/en/stable/tasks.html#dynamic-dependencies>`__,\nbut they work only for upstream tasks, not downstream tasks, which we\nwould need).\n\nIf you run into any of these problems, you might be interested in an\nexperimental workflow engine we develop to overcome these limitations:\n`SciPipe <http://scipipe.org/>`__.\n\nChangelog\n---------\n\n-  0.9.3b4\n-  Support for Python 3 (Thanks to @jeffcjohnson for contributing\n   this!).\n-  Bug fixes.\n\nContributors\n------------\n\n-  `See\n   here <https://github.com/pharmbio/sciluigi/graphs/contributors>`__\n\nAcknowledgements\n----------------\n\nThis work is funded by: - `Faculty grants of the dept. of Pharmaceutical\nBiosciences, Uppsala University <http://www.farmbio.uu.se>`__ -\n`Bioinformatics Infrastructure for Life Sciences,\nBILS <https://bils.se>`__\n\nMany ideas and inspiration for the API is taken from: - `John Paul\nMorrison's invention and works on Flow-Based\nProgramming <jpaulmorrison.com/fbp>`__\n",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/pharmbio/sciluigi",
        "keywords": "workflows workflow pipeline luigi",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "sciluigi",
        "package_url": "https://pypi.org/project/sciluigi/",
        "platform": "",
        "project_url": "https://pypi.org/project/sciluigi/",
        "project_urls": {
            "Homepage": "https://github.com/pharmbio/sciluigi"
        },
        "release_url": "https://pypi.org/project/sciluigi/0.9.6b7/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Helper library for writing dynamic, flexible workflows in luigi",
        "version": "0.9.6b7"
    },
    "last_serial": 3191895,
    "releases": {
        "0.9.2b3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2f19671023c7d48444237f3a89a173ba",
                    "sha256": "c90c917c69b2d27749f9f7986942005291fad58449d23742862ed6e3fe5a1f7d"
                },
                "downloads": -1,
                "filename": "sciluigi-0.9.2b3-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "2f19671023c7d48444237f3a89a173ba",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 20226,
                "upload_time": "2015-10-05T14:50:55",
                "url": "https://files.pythonhosted.org/packages/5f/af/154ae9b35f2136dd243fab5ec5de1ad63b8c0efd874d370f04418f523903/sciluigi-0.9.2b3-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d5be46e7d38e9070ed1f09153ff65275",
                    "sha256": "e69e950c32670b19894a53957d1fc3ed2a167c6c3bcacd3c4042121aa6d84673"
                },
                "downloads": -1,
                "filename": "sciluigi-0.9.2b3.tar.gz",
                "has_sig": false,
                "md5_digest": "d5be46e7d38e9070ed1f09153ff65275",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16479,
                "upload_time": "2015-10-05T14:51:00",
                "url": "https://files.pythonhosted.org/packages/ed/82/f2dcbf1c6b26f92001fae1a1bc0becea7e1712fd22345f20cd671dfe934c/sciluigi-0.9.2b3.tar.gz"
            }
        ],
        "0.9.3b4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "040d4221604ade6116da3cd099715317",
                    "sha256": "17c086720db1a5269f506d93ffd6503991300ead8a4970aa014e9ddd46abc175"
                },
                "downloads": -1,
                "filename": "sciluigi-0.9.3b4.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "040d4221604ade6116da3cd099715317",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 21750,
                "upload_time": "2015-10-26T15:43:47",
                "url": "https://files.pythonhosted.org/packages/02/51/f68a01287c7390580a2c7c67bdc2b4bc6a6fddebb0cd01608ef46c27f719/sciluigi-0.9.3b4.linux-x86_64.tar.gz"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "a44c1e91921110f52c5b3080f33cd289",
                    "sha256": "4b6a4748c96dbd01a9e1e3d63a38cd537deea4e3acb1d00d2e0a4b99e1a43475"
                },
                "downloads": -1,
                "filename": "sciluigi-0.9.3b4-py2.7.egg",
                "has_sig": false,
                "md5_digest": "a44c1e91921110f52c5b3080f33cd289",
                "packagetype": "bdist_egg",
                "python_version": "2.7",
                "requires_python": null,
                "size": 28848,
                "upload_time": "2015-10-26T15:43:52",
                "url": "https://files.pythonhosted.org/packages/e9/cb/3f5683381ba5d608272ead74f45eb45e5320218c4160b07049a53f309d5c/sciluigi-0.9.3b4-py2.7.egg"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "28f3db0d3cef27fb14e3bcb4a0ca7181",
                    "sha256": "0626d71116474e7068fcc635b6d84681fc553f2f876f478472464e87f757629d"
                },
                "downloads": -1,
                "filename": "sciluigi-0.9.3b4-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "28f3db0d3cef27fb14e3bcb4a0ca7181",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 21206,
                "upload_time": "2015-10-26T15:43:43",
                "url": "https://files.pythonhosted.org/packages/77/7c/21e2bc591d6c61378f4155aa354aa4ccd534e025488458aea47c7eab6b47/sciluigi-0.9.3b4-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "163bbd59fce06d20ab95176700ad064d",
                    "sha256": "dd29e21cd144de6267a05fcc63c48c1c83081633cd03e94c7db7575c3f37e57b"
                },
                "downloads": -1,
                "filename": "sciluigi-0.9.3b4.tar.gz",
                "has_sig": false,
                "md5_digest": "163bbd59fce06d20ab95176700ad064d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16618,
                "upload_time": "2015-10-26T15:43:56",
                "url": "https://files.pythonhosted.org/packages/be/d9/1ee1296c90a27f790c551e909ae8d91f1d9db7903cb9e4264f450498e6cf/sciluigi-0.9.3b4.tar.gz"
            }
        ],
        "0.9.4b5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "85027e90babaa216622fa50bbfa0e0d8",
                    "sha256": "ecbedb219270e2dcd02b2bfbdf9906dce44c26e6b7a636ebef5b7f43f3234ce0"
                },
                "downloads": -1,
                "filename": "sciluigi-0.9.4b5-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "85027e90babaa216622fa50bbfa0e0d8",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 21455,
                "upload_time": "2015-10-26T17:40:00",
                "url": "https://files.pythonhosted.org/packages/6c/dc/54668ef176077edf4edbd8ffeb789941e6c3dcf1b0a67c701e8f84e97601/sciluigi-0.9.4b5-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "8defc8f17a2069ab2399381b9417577f",
                    "sha256": "201f360d48352d0ba582824e25a03e0dd026a613764ecc7cab576541dafc66bb"
                },
                "downloads": -1,
                "filename": "sciluigi-0.9.4b5.tar.gz",
                "has_sig": false,
                "md5_digest": "8defc8f17a2069ab2399381b9417577f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16712,
                "upload_time": "2015-10-26T17:40:13",
                "url": "https://files.pythonhosted.org/packages/56/f1/616d4c3d4175ab45ee65b3c2642858964a33be0a26c64ea57f91e446b3e5/sciluigi-0.9.4b5.tar.gz"
            }
        ],
        "0.9.5b6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4bcf74f39e2b046f4dce9a1b5ceedda3",
                    "sha256": "7bf783ae5d57c477af86adf196f23a04de4b486471d11218b114a98756e3419c"
                },
                "downloads": -1,
                "filename": "sciluigi-0.9.5b6.tar.gz",
                "has_sig": false,
                "md5_digest": "4bcf74f39e2b046f4dce9a1b5ceedda3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17554,
                "upload_time": "2017-04-04T15:20:38",
                "url": "https://files.pythonhosted.org/packages/38/de/d28c29f0526b5fd3e7744b49aa9c328ce20d48326dd9fdc38f3c1382028d/sciluigi-0.9.5b6.tar.gz"
            }
        ],
        "0.9.6b7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4c7f0dbabdd80f7f8dee401bcc901a6a",
                    "sha256": "201ba29b0260981955e94542dc66aa4321b14cff3132281209839e6c6486ff5a"
                },
                "downloads": -1,
                "filename": "sciluigi-0.9.6b7.tar.gz",
                "has_sig": false,
                "md5_digest": "4c7f0dbabdd80f7f8dee401bcc901a6a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17600,
                "upload_time": "2017-09-21T14:44:14",
                "url": "https://files.pythonhosted.org/packages/4b/c6/2c6b90e880906daface7da68957bb4d50b2c53584017480427d170f3c964/sciluigi-0.9.6b7.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "4c7f0dbabdd80f7f8dee401bcc901a6a",
                "sha256": "201ba29b0260981955e94542dc66aa4321b14cff3132281209839e6c6486ff5a"
            },
            "downloads": -1,
            "filename": "sciluigi-0.9.6b7.tar.gz",
            "has_sig": false,
            "md5_digest": "4c7f0dbabdd80f7f8dee401bcc901a6a",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 17600,
            "upload_time": "2017-09-21T14:44:14",
            "url": "https://files.pythonhosted.org/packages/4b/c6/2c6b90e880906daface7da68957bb4d50b2c53584017480427d170f3c964/sciluigi-0.9.6b7.tar.gz"
        }
    ]
}