{ "info": { "author": "Ben Frederickson", "author_email": "ben@benfrederickson.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Topic :: Software Development :: Libraries", "Topic :: Utilities" ], "description": "parsesetup: parses information from untrusted setup.py files\n============================================================\n\nThis package returns information from a python packages `setup.py `_ file,\nwithout installing the package or trusting the code inside the\nsetup.py.\n\nWhat's this for?\n----------------\n\nThis is really an experiment in large scale processing of setup.py files.\nThe information provided in these files lists the project description, author and\nother metadata like package dependencies.\n\nUnfortunately there isn't any easy way of programmatically accessing this information\nfor an arbitrary setup.py script. Each setup.py script is a full python program,\nand the data can be arbitrary python objects passed as arguments to a function.\nThis package aims to provide a way for python programs to retrieve all this metadata\nprogrammatically, without doing things like installing the program or trusting\nthe setup.py file to not have malicious side effects.\n\nI've verified it by running on every Python repository on GitHub that has more\nthan 500 stars (about 2500 repos). Every package that can be\ninstalled on a clean python installation can be parsed by this code, and additionally a bunch of\npackages that can't be installed for different reasons can also have their setup contents\nretrieved here.\n\nHow does this work?\n-------------------\n\nSince setup.py files are Python files, the arguments to the setup call can be arbitrary\nPython expressions. This means that the only reliable way of getting these arguments is by\nevaluating the setup.py file using a Python interpreter.\n\nTo return the arguments passed to setuptools.setup call from a setup.py file, I'm temporarily\nmonkey patching the setuptools.setup function to collect its arguments - and then\nusing exec to execute the setup.py file:\n\n.. code-block:: python\n\n setup_args = [None]\n\n # patch setup functions to just keep track of arguments passed to them\n def patched_setup(**kwargs):\n setup_args[0] = kwargs\n\n setuptools.setup = distutils.core = patched_setup\n\n exec(open(setup_py_filename).read(), {\n \"__name__\": \"__main__\",\n \"__builtins__\": __builtins__,\n \"__file__\": setup_py_filename})\n\nGlobals are explicitly set up to match what most scripts expect, including ``__name__ == \"__main__\"``\nguards. Likewise there are some special cases with setting up the python path, current directory\netc that are taken care of by the full code.\n\nSandboxing with Docker\n----------------------\n\nRunning exec on an untrusted python file is a bad idea. As an example, some `setup.py scripts\n`_ do interesting things like mess around\nwith your root git config - but the potential harm could be much much worse than that.\n\nTo prevent harmful side-effects, this package runs by default in a sandboxed Docker container. In addition to security benefits,\nthis also lets us to cleanly fall back to using a Python2.7 Docker image in the case where the syntax is invalid for Python3.\n\nRunning in a docker container can be disabled by setting a ``trusted=True`` flag when calling. Also note that its probably worth\nconfiguring Docker to use `gVisor `_ to provide some extra piece of mind when parsing untrusted code.\n\nHandling Missing Dependencies\n-----------------------------\n\nA common pattern for setup.py files is to import the uninstalled module to look up things like version strings.\nWhile this works, it can have the side effect of importing the modules dependencies before they are installed.\n\nAs an example, `tensorlayer `_\nimports some metadata from its root module - which in turn imports tensorflow, which hasn't been installed yet in the docker\nimage.\n\nTo hack around this problem, this code has the option of hooking into Python's import handling to prevent ImportError's\nfrom surfacing when running.\n\nThe idea is to provide a module importer to ``sys.meta_path`` that always finds a module if the existing\nresolution fails:\n\n.. code-block:: python\n\n class MockModuleImporter(object):\n def find_module(self, fullname, path=None):\n return self\n\n def load_module(self, name):\n mock = MockModule(name)\n sys.modules[name] = mock\n return mock\n\n # This hooks into Pythons' import mechanism, meaning that any\n # module that fails to import will be replaced with a MockModule\n # object\n sys.meta_path.append([MockModuleImporter()])\n\n\nThe ``MockModule`` inherits from ``types.Modules`` and just returns a Mock object object\nwith the common magic methods defined\n\n.. code-block:: python\n\n class MockModule(types.ModuleType):\n def __getattr__(self, name, *args, **kwargs):\n return Mock()\n\n def __call__(self, *args, **kwargs):\n return Mock()\n\n\n class Mock(object):\n def __getattr__(self, *args, **kwargs):\n return self\n __call__ = __getitem__ = __setitem__ = __add__ = __getattr__\n\n ... etc ...\n\n\nThis prevents a sizeable number of errors, and doesn't seem to affect the output noticeably.\nThis behaviour can be disabled by setting ``mock_imports=False``.\n\nUsage\n-----\n\nThis code can be installed via pip:\n\n.. code-block:: shell\n\n pip install parsesetup\n\nTo run\n\n.. code-block:: python\n\n import parsesetup\n\n # parses the setup.py file, returning arguments as a dict\n setup_args = parsesetup.parse(path_to_setup_py)\n\n # Parses a single package without using docker (dangerous!)\n setup_args = parsesetup.parse(path_to_setup_py, trusted=True)\n\n # Parses multiple packages in a single docker container. All packages\n # need to share a common directory root for this to work\n with parsesetup.DockerSetupParser(ROOT_PATH) as parser:\n setup_a = parser.parse(path_to_setup_py_a)\n setup_b = parser.parse(path_to_setup_py_a)\n\nFeatures\n--------\n\n - Programmatically lets you inspect information contained in setup.py files\n - Handles both python2.7 and 3.6 scripts\n - Hooks into setuptools, distutils.core and numpy.distutils.core setups\n - Runs untrusted setup.py files in a docker container\n - Reads files with a __name__ == \"__main__\" guard\n\nReleased under the MIT License\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://github.com/benfred/parsesetup/", "keywords": "", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "parsesetup", "package_url": "https://pypi.org/project/parsesetup/", "platform": "", "project_url": "https://pypi.org/project/parsesetup/", "project_urls": { "Homepage": "http://github.com/benfred/parsesetup/" }, "release_url": "https://pypi.org/project/parsesetup/0.0.1/", "requires_dist": null, "requires_python": "", "summary": "Parses setup.py files", "version": "0.0.1" }, "last_serial": 3833116, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "1b21babf3dd5edb69747ecec5d227353", "sha256": "7a9e93d28972af069b85de16729f5b09d103df6bd8ee8edc3cfb9bccd4d712f0" }, "downloads": -1, "filename": "parsesetup-0.0.1.tar.gz", "has_sig": false, "md5_digest": "1b21babf3dd5edb69747ecec5d227353", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6600, "upload_time": "2018-05-04T04:00:42", "url": "https://files.pythonhosted.org/packages/56/da/97b71f667af643b0cc6bf2f3b63eb5a437999a5e67580e4a17030c96b51b/parsesetup-0.0.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "1b21babf3dd5edb69747ecec5d227353", "sha256": "7a9e93d28972af069b85de16729f5b09d103df6bd8ee8edc3cfb9bccd4d712f0" }, "downloads": -1, "filename": "parsesetup-0.0.1.tar.gz", "has_sig": false, "md5_digest": "1b21babf3dd5edb69747ecec5d227353", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6600, "upload_time": "2018-05-04T04:00:42", "url": "https://files.pythonhosted.org/packages/56/da/97b71f667af643b0cc6bf2f3b63eb5a437999a5e67580e4a17030c96b51b/parsesetup-0.0.1.tar.gz" } ] }