{
    "info": {
        "author": "Damien \"dee\" Coureau",
        "author_email": "dee909@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 1 - Planning",
            "Intended Audience :: End Users/Desktop",
            "Intended Audience :: Information Technology",
            "License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3.6",
            "Topic :: Desktop Environment :: File Managers"
        ],
        "description": "pipl\n====\n\n**Package Index PipeLine**\n\nExperimental pip/devpi based pipeline dependency platform\n\n\nWhy \n------\n\nI have so many time heard that \"CG Production is hard to industrialise because\nevery production is almost a prototype\".\nAs a developer this excuse bothered me a lot because if one thinks that CG\nproductions are prototypes, how could he handle developing - the art of creating\nsomething that solves a problem that has never been (correctly) solved before :D\nThat's as prototype as it gets !\n\nStill, software developement has found its path to industrialization, and so\nshould CG Production.\n\nSo I've started to compare the CG project classical bits to the Dev ones, and\nit hit me: There is very little difference between the two, let's use the DevOp\ntools to build a CG Pipeline.\n\n`pipl` is an attempt at making this idea real. (read \"proof of concept / prototype')\n\nHow\n------\n\nOne of the fundamental blocks at the core of a CG pipeline is the dependency\nmanagement. The idea behind `pipl` is to use python's packaging system as the\ncentral dependency resolver.\n\nOnce every bit of the pipeline can be represented as a python package,\nall the dependencies are automaticaly handled by a documented, robust and versatil\neco-system (Thanks the the Python Packaging Authority: www.pypa.io)\n\nOfficial PyPA tools like setuptools, pip, virtalenv, pipenv... can be used\nto resolve dependencies in isolated environments, manage updates, support\nrc verions, lock dependencies, etc...\n\nThis idea also opens a great opportunity: use CI/CD system to react to package\npublication. QA with tox is a obvious idea, but one can imagine anything if\nthe CI/CD system is used to execute 'pipeline' packages that process data from\nupstream packages and generate/update+publish packages.\n\nThat would be leveraging another great advantage of treating everything as a python package: it is inherently procedural and executable, so everything *can* become\nprocedural and executable.\n\n\"What version of Maya should I use to open this scene ?\" is a question no more:\nImagine that the task \"Shot 303-909 Animation\" is a python package with console_script entry_points like \"OpenScene\". This command would be available by simply activating a virtualenv, completely configured by adding dependencies to the task package.\n\n`pipl` is not intended solely for shell usage. The CLI is a first citizen but care is given to have a nice python API two so that one can easily add a GUI/Workflow system on top (our plans include using www.kabaretstudio.com for that)\n\nLast but not least, another nice feature gained by the usage of python packages is the entry_point system. By using it as a plugin system, pipl_pipe defines a extremely customisable framework: almost every aspect can be overridden by simply installing a package. The system is also very extensible, your are free to implement any pipeline and workflow.\n\n\nOverview\n------------\nYou will create and configure a `project`, then install or develop your `pipeline` inside a `workspace`, then publish them to the project index.\nWith the pipeline as dependency, you will create `Task` packages that will give the users the tools to create and edit `Assets`. \nWhen the users publish the `Task` package (to a stage index), a CI/CD handler will test them (QA) and push them to another stage index where another CI/CD handler will process them to create `Product` packages that store references to their associated `Assets`. Those `Products` will in turn be used as dependencies by some other tasks, and so on...\n\nThe `pipl` CLI workflow looks like this:\n\n- **Create a project**\nIt's a folder with a special structure inside it.\n- **Start the project's server**\nIt's a devpi server where all packages will be stored.\nIt's like our own personal PyPI...\n- **Configure the project**\nThere are a few important steps now that the server is up:\n\t- You should change the root password, the default is \"\".\n\t- You **must** create the default package index\n\t- You may want to add some \"users\" (index owners)\n\t- You may want to add some staging indexes.\n\t- You **must** upload pipl-pipe packl to the defaut package index\n\t(nb: this is not implemented yet - *l o l* )\n- **Create a Workarea** (optional, there's a default one)\nIt's a folder that knows which pipl project it belong to.\n- **Create a Workspace**\nIt's a folder inside a Workarea that contains a virtualenv and can edit\npackages.\n- **Create a Packl**\nIt's a python package with special stuffing to make it simpler and programmatically controllable.\nA packl can contain arbitrary python code, or represent a `Taks` or a `Product`.\n- **Add Dependencies**\nThis install some packls from the project index and \"links\" them to your packl.\n- **Edit the Packl**\n\t- If it's a code package, like the one configuring/implementing your pipeline (envset, configs, editors, asset types, asset store, product types, etc...), you should be able to use your favourite IDE.\n\t- If it's a *Task* package, it should provide some tools to auto-edit them. Just start the workspace-shell to use those tools.\n\t- If it's a *Product* package, the *Task* generating it will handle edition two.\n- **Publish the Packl**\nDepending on your worflow, you will publish directly to the project index or to a staging index where a CI/CD handler can run QA or processing tools.\n\nPipeline Implementation\n------------------------------\nSo how do I \"define my pipeline\" within this system ?\n\n\tNote: This is not functionnal yet and very likely to change A LOT :p\n\nThe core functionalities of the pipeline, a.k.a the \"pipeline framework\", are provided by the pipl_pipe package. That's where the concepts of Assets, Products, Tasks, etc... are defined. And that's what gives you the tools to define your owns.\n\nThe pipl_pipe packl defines the `pipl_pack.services` object.\nAs every packl in the project will have a direct or indirect dependency to the pipl_pipe packl, this object is available everywhere.\n\nIt contains \"*services*\" like `config` which provide named configuration dicts, and `asset_store` which resolves asset paths based on a Product, an Asset type name and an Asset revision.\n\nThe first step to define your pipeline is to declare your configs, your Asset types, etc... Do let you do so, `pipl_pipe` uses the entry_point system.\n\n**Config**\n\nAn `pipl_pip.config` entry_point will be automatically loaded as a config named by the entry name.\nIn your setup.py (or more likely, your \\_\\_setupl__.py), you just need to have:\n\n\tentry_points = {\n\t    'pipl_pipe.config': [\n\t\t# This is your asset_store config:\n\t        'asset_store = my_config:ASSET_STORE_CONFIG',\n\t        # You can define whatever name you want:\n\t        'kabaret = my_config:KABARET_CONFIG',\n\t    ]\n\t}\n\n**Asset types**\n\nNote that the term `Asset` may not be used here like you are used to.\nBy `Asset` we mean \"*A file or a folder that is generated by a user or a process and that exists with a different version/revision for each generation*\".\nMore pragmatically, Assets are file that cannot be embedded in a python package, most likely because they are to big.\n\n`pipl` take on those files is inspired by git-lfs: just hold a reference to the asset inside the package and delegate their storage (in our case, to the `asset_store` service)\n\nThe reason why you will want to define your own Asset subclasses is that Asset are responsible for their managed file *basenames*. So implementing a **naming convention** is done by defining your Asset types (and also overriding the `asset_store` service which is responsible for Asset managed file *paths*, see the \"*Custom Services*\" section below)\n\n\tNote: This bothers me and we'll probably find a proper way to have all the naming convention implemented is Asset only, or in the asset_store only.\n\nDefining Asset types is easy, just inherit `pipl_pipe.asset_store.Asset`  and register your public Asset types with the `pipl_pipe.asset_store.register()` decorator:\n\n\tfrom pipl_pipe.asset_store import Asset, registered\n\n\tclass _MayaScene(Asset):\n\t    pass\n\n\t@registered\n\tclass MayaAscii(_MayaScene):\n\t    def __init__(self, storage, product, asset_name):\n\t        super(MayaAscii, self).__init__(\n\t            storage, product, asset_name, 'ma'\n\t        )\nThis means that in order to use a specific Asset type, you will need a dependency to the package defining it **AND** you will need to import it (or the registration won't be triggered).\n\nThis is so to avoid polluting your python with unused Asset. As you are likely to implement all your assets in a single package, using entry_points would load them all. With the classical import method you can split them into submodules and only load the ones you need.\n\n\tNote: this is an obvious case of early optimisation and is likely to get replaced by entry_points :p)\n\n**Custom Services**\n     \nServices are loaded on the entry_point group `pipl_pipe.services`, with the name of the service. Just provide a factory function (or a class) that accept a config as argument. If a config of the same name exists, it will automatically be provided.\n\nThis is for example how you would provide your own `asset_store` service. (nb: this is not true in current imp. but it will be soon ^^)\n\nYou can even add your own extra services, you don't have overwrite an existing one.\n\n**Product**\n\nThe Products are the part you will work on most. They are defining all the procedures and all the code to apply them.\n\nA Product is a packl containing code that instantiates a Product class.\n\nProducts packls have enty_points for each of their Asset, giving managing tools the ability\nto act on all Assets of a given workspace (i.e. to copy all assets in workspace's asset storage and be able to work w/o access to the project)\n\nProducts have (typed) entry_points for them, so that Products and Tasks depending on them can access them. The Product base class has functions to get dependencies based on their type.\n\nThe Product types are available in the `tasks` service. You install yours with the entry_point `pipl_pipe.task_type`. \n\nEach Product has a dependency to the package declaring its Product type.\n\n\tNB: using a service to get a base class feels weird :/ \n\n\tTODO: polish this section (how to declare product types ?)\n\t\n**Task**\n                                           \nTasks are a special types of Product.\n\nIf you know why we must use abstract dependencies when developing a library and concrete dependencies when developing an application, know that a Product is a library and a Task is an application.\n\nIf you don't, just keep on not caring ;)\n\nA Task is a packl containing code that instantiates a Task class.\n\nA `Task` defines (thru entry_points) executable commands (build a scene using upstream packls, open maya with a particular scene, view a sequence of images, ...) and can edit/create/publish packages (Products).\n\nThis is where you build a scene, let the user edit it, then export/render Products from it (in a CI/CD handler)\n\nThere should be only one task per workspace (I don't know why yet but it feels important and I'll probably find why later ;) )\n\nTasks creation is done by creating a new packl and executing the command `create_task <packl_name> <task_type>`in the workspace shell.\nThis command is declared by pipl_pipe and will edit the packl \\_\\_init__.py and \\_\\_setupl__.py to turn it into the desired task.\n\nThe Task types are available in the `tasks` service. You install yours with the entry_point `pipl_pipe.task_type`.\n \n\tTODO: polish this section (Task type declaration)\n\n**EnvSet**\n\nEnvSet are packl that instantiate an EnvSet class.\n\nTheir purpose is close to what you do with rez: resolve environment variable based on dependencies and specified requirements.\n\nThere is an EnvSet package per dependency+requirement since we handle both with package dependencies.\n\n\tNote: EnvSet are not yet implemented (nor well defined)\n\n**Tools/Editors**\n                         \nTasks provide commands, but those commands don't need to be defined in the Tasks. Tools/Editor packages would be used as 'edit' extra_requires and would use EnvSet to define commands.\n\nEditor dependencies would be installed only in user workspace (not in CI/CD ones) thanks to the extra_requires machinery.\n\n\tNote: This is not well defined yet.\n\n\nInstallation\n------------\n\nCreate a virtual env, install and update using pip:\n\n.. code-block:: text\n\n    $ pip install -U pipl\n\n\nUsage\n-----\n\nThe package defines the `pipl` command line tool.\n\nUse `pipl --help` to list available commands:\n\n.. code-block:: text\n\n    (venv) $ pipl --help\n    Usage: pipl [OPTIONS] COMMAND [ARGS]...\n\n    Options:\n      --help  Show this message and exit.\n\n    Commands:\n      packl\n      proj\n      work\n\nUse `pipl COMMAND --help` to list sub commands:\n\n.. code-block:: text\n\n    (venv) $ pipl proj --help\n    Usage: pipl proj [OPTIONS] COMMAND [ARGS]...\n\n    Options:\n      --help  Show this message and exit.\n\n    Commands:\n      create  Creates a new project at path\n      index   Project indexes administration.\n      start   Start project's server in **foreground**.\n      user    Project users administration.\n\n\nWhen appropriate, commands will guess the *current* project/workspace/package\nfrom current path. Use an extra argument to override current path.\n\n\nProject Init\n------------\n\nCreate a project and navigate into it:\n\n.. code-block:: text\n    \n    (venv):~ $ pipl proj create /path/to/MyProject\n\nOpen a new shell within the same virtualenv and start the project's server:\n\n.. code-block:: text\n    \n    (venv):~ $ cd /path/to/MyProject\n    (venv):/path/to/MyProject $ pipl proj start\n\nNow in your first shell, you can list users:\n\n.. code-block:: text\n    \n    (venv):~ $ cd /path/to/MyProject\n    (venv):/path/to/MyProject $ pipl proj user list\n    Resolving Project path from .\n    Found Project at /path/to/MyProject\n    Connecting to http://localhost:3141 (user:None, pass:None)\n    Found 1 user(s) in project /path/to/MyProject:\n          root\n    Disconnected.\n\n\nRoot's default password is an empty string.\nAdmin commands will need it, be sure to use \"\" and not '' in command lines.\n\nThe best is to change it asap:\n\n.. code-block:: text\n     \n    (venv):/path/to/MyProject $ pipl proj user set \"\" root --password 123\n    Resolving Project path from .\n    Found Project at /path/to/MyProject\n    Connecting to http://localhost:3141 (user:root, pass:None)\n    Disconnected.\n\n\nNow you can create some users using the new root password:\n\n.. code-block:: text\n    \n    (venv):/path/to/MyProject $ pipl proj user add 123 bob b0b bob@pipl.io\n    Resolving Project path from .\n    Found Project at /path/to/MyProject\n    Connecting to http://localhost:3141 (user:root, pass:***)\n    Disconnected.\n\n\nYou will need some index to store your package. The default one can be \ncreated with:\n\n.. code-block:: text\n    \n    (venv):/path/to/MyProject $ pipl proj index create-default 123\n    Resolving Project path from .\n    Found Project at /path/to/MyProject\n    Creating default project index for \".\"\n    Connecting to http://localhost:3141 (user:root, pass:***)\n    Disconnected.\n\n    (venv):/path/to/MyProject $ pipl proj index list-indexes\n    Resolving Project path from .\n    Found Project at /path/to/MyProject\n    Connecting to http://localhost:3141 (user:None, pass:None)\n    Found 2 Index(es) in project /path/to/MyProject:\n               root/PROJ\n    Disconnected.\n\n\nWorkspaces Creation\n-------------------\n\t\n\tremoved outdated doc\n\n\nPackage Creation\n----------------\n\n\tremoved outdated doc\n\nPackage Setup\n-------------\n\n\tremoved outdated doc\n\nPackage Edition\n---------------\n\n\tremoved outdated doc\n\nPackage Upload\n--------------\n\n\tremoved outdated doc",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://gitlab.com/open-minds-group/pipl",
        "keywords": "pipeline dataflow dependency asset",
        "license": "LGPLv3+",
        "maintainer": "",
        "maintainer_email": "",
        "name": "pipl",
        "package_url": "https://pypi.org/project/pipl/",
        "platform": "",
        "project_url": "https://pypi.org/project/pipl/",
        "project_urls": {
            "Homepage": "https://gitlab.com/open-minds-group/pipl"
        },
        "release_url": "https://pypi.org/project/pipl/0.2.0.dev4/",
        "requires_dist": null,
        "requires_python": ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*",
        "summary": "Package Index PipeLine",
        "version": "0.2.0.dev4"
    },
    "last_serial": 5273522,
    "releases": {
        "0.0.1.dev13": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a4e20a12dddbb54f84731dd6db9c3bc6",
                    "sha256": "520be3c0d806f3f96b9d959227294d9986699306f470061909b749960cac2359"
                },
                "downloads": -1,
                "filename": "pipl-0.0.1.dev13.tar.gz",
                "has_sig": false,
                "md5_digest": "a4e20a12dddbb54f84731dd6db9c3bc6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*",
                "size": 18796,
                "upload_time": "2019-04-22T15:28:30",
                "url": "https://files.pythonhosted.org/packages/af/dd/2020229bd7a410c783e50ef3c610f4e66719277dac38497860f2111a35c6/pipl-0.0.1.dev13.tar.gz"
            }
        ],
        "0.2.0.dev1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3e786254a5fe4d131ae06c8691143277",
                    "sha256": "cbd96a13fb86ea0658e3786dd111decb2d31fec2e6ac956d42ebb07e6043483e"
                },
                "downloads": -1,
                "filename": "pipl-0.2.0.dev1.tar.gz",
                "has_sig": false,
                "md5_digest": "3e786254a5fe4d131ae06c8691143277",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*",
                "size": 31297,
                "upload_time": "2019-05-07T23:35:40",
                "url": "https://files.pythonhosted.org/packages/02/8b/372422d111ca76cda48d3d7d46e7e97d1d9ed41546ab261baf8a2229a5e0/pipl-0.2.0.dev1.tar.gz"
            }
        ],
        "0.2.0.dev2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f62c93f0572e8ff1496ca7fdf748e761",
                    "sha256": "c199b591acdc1eec299c405e40ff3b10a9a7752e1c07b14857a1f7e3ed17c447"
                },
                "downloads": -1,
                "filename": "pipl-0.2.0.dev2.tar.gz",
                "has_sig": false,
                "md5_digest": "f62c93f0572e8ff1496ca7fdf748e761",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*",
                "size": 31305,
                "upload_time": "2019-05-08T00:12:19",
                "url": "https://files.pythonhosted.org/packages/e7/de/0757826c00b406ebd48d4d06446b2a722b779c206f4cfc2cf388ba49138f/pipl-0.2.0.dev2.tar.gz"
            }
        ],
        "0.2.0.dev3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d964cbba7cfd4b094857ef252d175a23",
                    "sha256": "4de9c235800cee3c0334fcb97e7c7a022f3f06826b9249cfde01dfa3318b4de6"
                },
                "downloads": -1,
                "filename": "pipl-0.2.0.dev3.tar.gz",
                "has_sig": false,
                "md5_digest": "d964cbba7cfd4b094857ef252d175a23",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*",
                "size": 31321,
                "upload_time": "2019-05-15T15:44:17",
                "url": "https://files.pythonhosted.org/packages/1b/28/6c6745e08c56618eb2e10e77d89628ac1bf60a16082caf594261f17764df/pipl-0.2.0.dev3.tar.gz"
            }
        ],
        "0.2.0.dev4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b84a21a1652e861de945c7fd34a57df3",
                    "sha256": "c5fc4ba034fc8efc791c14dac26977e93b4da919d6bc0eff218cf107a4ddef7a"
                },
                "downloads": -1,
                "filename": "pipl-0.2.0.dev4.tar.gz",
                "has_sig": false,
                "md5_digest": "b84a21a1652e861de945c7fd34a57df3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*",
                "size": 31355,
                "upload_time": "2019-05-15T17:20:16",
                "url": "https://files.pythonhosted.org/packages/41/de/be2f10d75bf5dfb7068673a350fc07399fa0b1cb7712a2d657789ed3e4d0/pipl-0.2.0.dev4.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "b84a21a1652e861de945c7fd34a57df3",
                "sha256": "c5fc4ba034fc8efc791c14dac26977e93b4da919d6bc0eff218cf107a4ddef7a"
            },
            "downloads": -1,
            "filename": "pipl-0.2.0.dev4.tar.gz",
            "has_sig": false,
            "md5_digest": "b84a21a1652e861de945c7fd34a57df3",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*",
            "size": 31355,
            "upload_time": "2019-05-15T17:20:16",
            "url": "https://files.pythonhosted.org/packages/41/de/be2f10d75bf5dfb7068673a350fc07399fa0b1cb7712a2d657789ed3e4d0/pipl-0.2.0.dev4.tar.gz"
        }
    ]
}