{
    "info": {
        "author": "Marko Ristin",
        "author_email": "marko.ristin@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6"
        ],
        "description": "pypackagery\n===========\n\n.. image:: https://api.travis-ci.com/Parquery/pypackagery.svg?branch=master\n    :target: https://api.travis-ci.com/Parquery/pypackagery.svg?branch=master\n    :alt: Build Status\n\n.. image:: https://coveralls.io/repos/github/Parquery/pypackagery/badge.svg?branch=master\n    :target: https://coveralls.io/github/Parquery/pypackagery?branch=master\n    :alt: Coverage\n\n.. image:: https://badge.fury.io/py/pypackagery.svg\n    :target: https://pypi.org/project/pypackagery/\n    :alt: PyPi\n\n.. image:: https://img.shields.io/pypi/pyversions/pypackagery.svg\n    :alt: PyPI - Python Version\n\n.. image:: https://readthedocs.org/projects/pypackagery/badge/?version=latest\n    :target: https://pypackagery.readthedocs.io/en/latest/?badge=latest\n    :alt: Documentation Status\n\nPypackagery packages a subset of a monorepo and determine the dependent packages.\n\nGiven a root directory of a Python code base, a list of Python files (from that code base) and a target directory,\npypackagery determines the dependent modules of the specified files. The scripts and the *local* dependencies are copied\nto the given target directory. The *external* dependencies (such as pypi packages) are not copied\n(and need not be installed), but the list containing a subset of external dependencies is generated instead.\n\nThe external dependencies of the monorepo need to be specified in\n``<root directory>/requirements.txt`` and ``<root directory>/module_to_requirement.tsv``.\n\nThe ``requirements.txt`` follows the Pip format\n(see `pip documentation <https://pip.pypa.io/en/stable/user_guide/#id1>`_). The file defines which external packages are\nneeded by the *whole* of the code base. Pypackagery will read this list and extract the subset needed by the specified\nPython files.\n\n``module_to_requirement.tsv`` defines the correspondence between Python modules and requirements as defined in\n``requirements.txt``. This correspondence needs to be manually defined since there is no way to automatically map\nPython modules to pip packages. The correspondance in ``module_to_requirement.tsv`` is given as\nlines of two tab-separated values. The first column is the full model name (such as ``PIL.Image``) and the second is\nthe name of the package in ``requirements.txt`` (such as ``pillow``). The version of the package should be omitted and\nshould be specified *only* in the ``requirements.txt``.\n\nPlease do not forget to add the ``#egg`` fragment to URLs and files in ``requirements.txt`` so that the name of the\npackage can be uniquely resolved when joining ``module_to_requirement.tsv`` and ``requirements.txt``.\n\nRelated Projects\n================\n\n* https://github.com/pantsbuild/pex -- a library tool for generating PEX (Python EXecutable) files. It packages all the\n  files in the virtual environment including all the requirements. This works for small code bases and light\n  requirements which are not frequently re-used among the components in the code base. However, when the requirements\n  are heavy (such as OpenCV or numpy) and frequently re-used in the code base, it is a waste of bandwidth and disk space\n  to package them repeatedly for each executable independently.\n\n* https://www.pantsbuild.org/, https://buckbuild.com/ and https://github.com/linkedin/pygradle are build systems that\n  can produce PEX files (with all the problems mentioned above). We considered using them and writing a plug-in to\n  achieve the same goal as pypackagery. Finally, we decided for a separate tool since these build systems are not\n  yet supported natively by IDEs (such as Pycharm) and actually break the expected Python development work flow.\n\n* https://blog.shazam.com/python-microlibs-5be9461ad979 presents a microlib approach where a monorepo is packaged in\n  separate pip packages. While we find the idea interesting, it adds an administration overhead since every library\n  needs to live in a separate package thus making bigger refactorings tedious and error-prone (*e.g.* are the\n  requirements updated correctly and are dependency conflicts reported early?). We found it easiest to have a global\n  list of the requirements (with modules mapped to requirements), so that a sole ``pip3 install -r requirements.txt``\n  would notify us of the conflicts.\n\n  If you wanted to work only on part of the code base, and do no want to install all the requirements, you can use\n  pypackagery to determine the required subset and install only those requirements that you need.\n\n  Since we deploy often on third-party sites, we also found it difficult to secure our deployments. Namely, packaging\n  the code base into microlibs practically implies that we need to give the remote machine access to our private pypi\n  repository. In case that we only want to deploy the subset of the code base, granting access to all packages would\n  unnecessarily open up a potential security hole. With pypackagery, we deploy only the files that are actually\n  used while the third-party dependencies are separately installed on the remote instance from a subset of requirements\n  ``subrequirements.txt`` with ``pip3 install -r subrequirements.txt``.\n\n\nUsage\n=====\nRequirement Specification\n-------------------------\nAs already mentioned, the requirements are expected to follow Pip format\n(see `pip documentation <https://pip.pypa.io/en/stable/user_guide/#id1>`_) and live in ``requirements.txt`` at the root\nof the code base. The mapping from modules to requirements is expected in ``module_to_requirement.tsv`` also at the root\nof the code base.\n\nAssume that the code base lives in ``~/workspace/some-project``.\n\nHere is an excerpt from ``~/workspace/some-project/requirements.txt``:\n\n.. code-block::\n\n    pillow==5.2.0\n    pytz==2018.5\n    pyzmq==17.1.2\n\nAnd here is an excerpt from ``~/workspace/some-project/module_to_requirement.tsv``\n(mind that it's tab separated):\n\n.. code-block::\n\n    PIL\tpillow\n    PIL.Image\tpillow\n    PIL.ImageFile\tpillow\n    PIL.ImageOps\tpillow\n    PIL.ImageStat\tpillow\n    PIL.ImageTk\tpillow\n    cv2\topencv-python\n\nDirectory\n---------\nAssume that the code base lives in ``~/workspace/some-project`` and we are interested to bundle everything\nin ``pipeline/out`` directory.\n\nTo determine the subset of the files and requirements, run the following command line:\n\n\n.. code-block:: bash\n\n    pypackagery \\\n        --root_dir ~/workspace/some-project \\\n        --initial_set ~/workspace/some-project/pipeline/out\n\nThis gives us a verbose, human-readable output like:\n\n.. code-block::\n\n    External dependencies:\n    Package name | Requirement spec\n    -------------+---------------------\n    pyzmq        | 'pyzmq==17.1.2'\n    temppathlib  | 'temppathlib==1.0.3'\n\n    Local dependencies:\n    pipeline/out/__init__.py\n    common/__init__.py\n    common/logging.py\n    common/proc.py\n\nIf we want to get the same output in JSON, we need to call:\n\n.. code-block:: bash\n\n    pypackagery \\\n        --root_dir ~/workspace/some-project \\\n        --initial_set ~/workspace/some-project/pipeline/out \\\n        --format json\n\nwhich gives us a JSON-encoded dependency graph:\n\n.. code-block:: json\n\n    {\n      \"requirements\": {\n        \"pyzmq\": {\n          \"name\": \"pyzmq\",\n          \"line\": \"pyzmq==17.1.2\\n\"\n        },\n        \"temppathlib\": {\n          \"name\": \"temppathlib\",\n          \"line\": \"temppathlib==1.0.3\\n\"\n        }\n      },\n      \"rel_paths\": [\n        \"pipeline/out/__init__.py\",\n        \"common/__init__.py\",\n        \"common/logging.py\",\n        \"common/proc.py\"\n      ],\n      \"unresolved_modules\": []\n    }\n\nFiles\n-----\nAssume again that the code base lives in ``~/workspace/some-project``. We would like to get a subset of the\ncode base required by a list of scripts. We need to specify the initial set as a list of files:\n\n.. code-block:: bash\n\n    pypackagery \\\n        --root_dir ~/workspace/some-project \\\n        --initial_set \\\n            ~/workspace/some-project/pipeline/input/receivery.py \\\n            ~/workspace/some-project/pipeline/input/snapshotry.py\n\nwhich gives us:\n\n.. code-block::\n\n    External dependencies:\n    Package name | Requirement spec\n    -------------+-------------------\n    icontract    | 'icontract==1.5.1'\n    pillow       | 'pillow==5.2.0'\n    protobuf     | 'protobuf==3.5.1'\n    pytz         | 'pytz==2018.5'\n    pyzmq        | 'pyzmq==17.1.2'\n    requests     | 'requests==2.19.1'\n\n    Local dependencies:\n    pipeline/__init__.py\n    pipeline/input/receivery.py\n    pipeline/input/snapshotry.py\n    common/__init__.py\n    common/img.py\n    common/logging.py\n    protoed/__init__.py\n    protoed/pipeline_pb2.py\n\nUnresolved Modules\n------------------\nIf there is a module which could not be resolved (neither in built-ins, nor specified in the requirements nor\nliving in the code base), the pypackagery will return a non-zero return code.\n\nIf you specify ``--dont_panic``, the return code will be 0 even if there are unresolved modules.\n\nModule ``packagery``\n--------------------\nPypackagery provides a module ``packagery`` which can be used to programmatically determine the dependencies of the\nsubset of the code base. For example, this is particularly useful for deployments to a remote machine where you\nwant to deploy only a part of the code base depending on some given configuration.\n\nHere is an example:\n\n.. code-block:: python\n\n    import pathlib\n\n    import packagery\n\n    root_dir = pathlib.Path('/some/codebase')\n\n    rel_pths = [\n        pathlib.Path(\"some/dir/file1.py\"),\n        pathlib.Path(\"some/other/dir/file2.py\")]\n\n    requirements_txt = root_dir / \"requirements.txt\"\n    module_to_requirement_tsv = root_dir / \"module_to_requirement.tsv\"\n\n    requirements = packagery.parse_requirements(\n        text=requirements_txt.read_text())\n\n    module_to_requirement = packagery.parse_module_to_requirement(\n        text=module_to_requirement_tsv.read_text(),\n        filename=module_to_requirement_tsv.as_posix())\n\n    pkg = packagery.collect_dependency_graph(\n        root_dir=root_dir,\n        rel_paths=rel_pths,\n        requirements=requirements,\n        module_to_requirement=module_to_requirement)\n\n    # do something with pkg ...\n\nMind that relative paths (given as ``rel_paths`` argument) all need to be files, not directories.\n\nDocumentation\n=============\nThe documentation is available on `readthedocs <https://pypackagery.readthedocs.io/en/latest/>`_.\n\nInstallation\n============\n\n* Create a virtual environment:\n\n.. code-block:: bash\n\n    python3 -m venv venv3\n\n* Activate it:\n\n.. code-block:: bash\n\n    source venv3/bin/activate\n\n* Install pypackagery with pip:\n\n.. code-block:: bash\n\n    pip3 install pypackagery\n\nDevelopment\n===========\n\n* Check out the repository.\n\n* In the repository root, create the virtual environment:\n\n.. code-block:: bash\n\n    python3 -m venv venv3\n\n* Activate the virtual environment:\n\n.. code-block:: bash\n\n    source venv3/bin/activate\n\n* Install the development dependencies:\n\n.. code-block:: bash\n\n    pip3 install -e .[dev]\n\nWe use tox for testing and packaging the distribution:\n\n.. code-block:: bash\n\n    tox\n\nPre-commit Checks\n-----------------\nWe provide a set of pre-commit checks that lint and check code for formatting.\n\nNamely, we use:\n\n* `yapf <https://github.com/google/yapf>`_ to check the formatting.\n* The style of the docstrings is checked with `pydocstyle <https://github.com/PyCQA/pydocstyle>`_.\n* Static type analysis is performed with `mypy <http://mypy-lang.org/>`_.\n* Various linter checks are done with `pylint <https://www.pylint.org/>`_.\n* Doctests are executed using the Python `doctest module <https://docs.python.org/3.5/library/doctest.html>`_.\n\nRun the pre-commit checks locally from an activated virtual environment with development dependencies:\n\n.. code-block:: bash\n\n    ./precommit.py\n\n* The pre-commit script can also automatically format the code:\n\n.. code-block:: bash\n\n    ./precommit.py  --overwrite\n\n\nVersioning\n==========\nWe follow `Semantic Versioning <http://semver.org/spec/v1.0.0.html>`_. The version X.Y.Z indicates:\n\n* X is the major version (backward-incompatible),\n* Y is the minor version (backward-compatible), and\n* Z is the patch version (backward-compatible bug fix).",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/Parquery/pypackagery",
        "keywords": "package monorepo requirements",
        "license": "License :: OSI Approved :: MIT License",
        "maintainer": "",
        "maintainer_email": "",
        "name": "pypackagery",
        "package_url": "https://pypi.org/project/pypackagery/",
        "platform": "",
        "project_url": "https://pypi.org/project/pypackagery/",
        "project_urls": {
            "Homepage": "https://github.com/Parquery/pypackagery"
        },
        "release_url": "https://pypi.org/project/pypackagery/1.0.4/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Package a subset of a monorepo and determine the dependent packages.",
        "version": "1.0.4"
    },
    "last_serial": 5410381,
    "releases": {
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7c67a2afbec4e0ba65cd7a6b2baea6c4",
                    "sha256": "0b81e4ce4a7e4313610d9e00cbaa379dc116f5ae574aa72ff23115cbbeed35e0"
                },
                "downloads": -1,
                "filename": "pypackagery-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "7c67a2afbec4e0ba65cd7a6b2baea6c4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10398,
                "upload_time": "2018-09-18T07:22:57",
                "url": "https://files.pythonhosted.org/packages/dd/27/b01060d7a724e96571d8056abd273f059203b48a7e279d0f0b2c305c9632/pypackagery-1.0.0.tar.gz"
            }
        ],
        "1.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "23715a44ad1d17ceff4eaf075abaa299",
                    "sha256": "0359821a77663630a9060f7d3f6b8649afbb1136ee96e6cc5177bbea87641c30"
                },
                "downloads": -1,
                "filename": "pypackagery-1.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "23715a44ad1d17ceff4eaf075abaa299",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11646,
                "upload_time": "2018-09-18T08:13:09",
                "url": "https://files.pythonhosted.org/packages/fd/e8/05b4fcf5998e892b1b697b1640d12fe6cd82e45a134bd8dc1d6c055a7076/pypackagery-1.0.1.tar.gz"
            }
        ],
        "1.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bbf88990ca2a9df60b045f1323359621",
                    "sha256": "2401ed459c2150e7847ab1213de003b38b97b96e7fa47e607ae31b59905a995d"
                },
                "downloads": -1,
                "filename": "pypackagery-1.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "bbf88990ca2a9df60b045f1323359621",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13227,
                "upload_time": "2018-10-22T14:57:55",
                "url": "https://files.pythonhosted.org/packages/76/44/d1e6689d383a2316427039c655114b01d8e39bfa3686822e46fa45dcd1ce/pypackagery-1.0.2.tar.gz"
            }
        ],
        "1.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "80696853c7927ee8c28f50b6200ef412",
                    "sha256": "a52f7618e9bcffbc191c5f4630b844de3800459dc5181f2df15f6775b80c4479"
                },
                "downloads": -1,
                "filename": "pypackagery-1.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "80696853c7927ee8c28f50b6200ef412",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13234,
                "upload_time": "2018-10-23T14:21:53",
                "url": "https://files.pythonhosted.org/packages/0b/c4/1eae626d0083774c41add34070d4a39cf37ea0710aca2fdee911490f6129/pypackagery-1.0.3.tar.gz"
            }
        ],
        "1.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "67b08e2553baee32851622da5c34a9c0",
                    "sha256": "35873d40d120e231f056f29fb7983ee9651302b2a6e0dbf6102ce9b5a8817a0a"
                },
                "downloads": -1,
                "filename": "pypackagery-1.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "67b08e2553baee32851622da5c34a9c0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11737,
                "upload_time": "2018-10-29T06:36:42",
                "url": "https://files.pythonhosted.org/packages/e7/2e/a08c2ea422ac148bcb2b6a1353daa3aab3d24a79051c4dc7e1ac37c45bb4/pypackagery-1.0.4.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "67b08e2553baee32851622da5c34a9c0",
                "sha256": "35873d40d120e231f056f29fb7983ee9651302b2a6e0dbf6102ce9b5a8817a0a"
            },
            "downloads": -1,
            "filename": "pypackagery-1.0.4.tar.gz",
            "has_sig": false,
            "md5_digest": "67b08e2553baee32851622da5c34a9c0",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 11737,
            "upload_time": "2018-10-29T06:36:42",
            "url": "https://files.pythonhosted.org/packages/e7/2e/a08c2ea422ac148bcb2b6a1353daa3aab3d24a79051c4dc7e1ac37c45bb4/pypackagery-1.0.4.tar.gz"
        }
    ]
}