{ "info": { "author": "Nathaniel J. Smith", "author_email": "njs@pobox.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Intended Audience :: Developers", "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)", "Operating System :: MacOS :: MacOS X", "Operating System :: Microsoft", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Topic :: Software Development :: Build Tools", "Topic :: Software Development :: Compilers" ], "description": "Mach-O Mach-O Mangler\n=====================\n\n.. image:: https://travis-ci.org/njsmith/machomachomangler.svg?branch=master\n :target: https://travis-ci.org/njsmith/machomachomangler\n :alt: Automated test status (Travis)\n\n.. image:: https://ci.appveyor.com/api/projects/status/9p8cuhx8vwn2i2jp?svg=true\n :target: https://ci.appveyor.com/project/njsmith/machomachomangler\n :alt: Automated test status (Appveyor)\n\n.. image:: https://codecov.io/gh/njsmith/machomachomangler/branch/master/graph/badge.svg\n :target: https://codecov.io/gh/njsmith/machomachomangler\n :alt: Test coverage\n\n\nThis is a little library for mangling Mach-O and PE files in various\nways. These are the formats used for executables and shared libraries\non MacOS and Windows, respectively. (If you want the equivalent for\nfor Linux, then check out `patchelf\n`__.)\n\n\nMacho-O features\n----------------\n\nSome rather specialized (and complex) Mach-O mangling tools designed\nto support `the pynativelib proposal\n`__\nto allow native libraries to be distributed as standalone `wheel files\n`__. Specifically this includes:\n\n* For pynativelib libraries: a tool that takes a dylib, and a mangling\n rule, and applies the mangling rule to all the exported\n symbols. E.g., it can convert a library that exports ``SSL_new``\n into one that exports ``pynativelib_openssl__SSL_new``. It also\n changes the library id while it's at it, e.g. from ``ssl.dylib`` ->\n ``pynativelib_openssl__ssl.dylib`` (like ``install_name_tool -id``)\n\n Additionally: a tool that creates a \"placeholder\" library, which\n imports the mangled library described above, and then re-exports the\n symbols under their original names.\n\n* For code that wants to use a pynativelib library: a tool that\n takes a dylib/bundle/executable, a list of \"original\" dylibs, and\n for each \"original\" dylib, a newname for that dylib, and a\n mangling rule. It then (a) replaces the import of the original\n dylib with an absolute import of the new dylib name from a\n non-existent directory, (b) marks this as a \"weak\" import, (c)\n applies the mangling rule to all symbols imported from this dylib,\n (d) marks these symbols for lookup in the flat namespace.\n\nIt turns out that this *exact* combination of things is the only way\nprovided for by the MacOS linker/loader to have dylib/bundle A linked\nagainst dylib B where the relative on-disk location of A and B is not\nknown until after the executable starts, while preserving the usual\ntwo-level namespace rules for avoiding symbol collisions. I promise it\nwill all make sense once I have a chance to write it up properly...\n\nSome known limitations of the Mach-O mangling code:\n\n- Unsurprisingly, this kind of patching does not play well with code\n signing. The code doesn't take any special case with signatures;\n they'll probably just get messed up. If you want to sign your\n binaries, then do your mangling first before signing.\n\n- We currently only rewrite the new-style DYLD_INFO symbol table\n (introduced in 10.5), not the (almost?) totally redundant\n SYMTAB/DYSYMTAB symbol table. (Interesting fact: all Mach-O binaries\n include two completely different representations of their symbols\n tables. The new one is more compact, to save space, but then they\n keep the old one around for compatibility, so... anyway.) As far as\n I can tell, the only thing in in modern MacOS that still uses\n SYMTAB/DYSYMTAB is ``dladdr``, and I don't think anyone is relying\n on ``dladdr`` output for, well... anything? I think worst case, you\n might end up seeing the original symbol names inside a debugger or\n profiler? But this wouldn't be *too* hard to fix if it becomes a\n problem.\n\n- It doesn't do any special handling of the DYLD_INFO weak_bind table,\n or weak exports. (NB these have nothing to do\n ``__attribute__((weak))`` or ``__attribute__((weak_import))`` or any\n of the mentions of the word ``weak`` in the ``ld`` man page \u00e2\u20ac\u201c I\n think they're for implementing `vague linkage\n `__.) This is *probably* not a\n disastrous option, but I'm not 100% sure whether it's actually\n correct \u00e2\u20ac\u201c it's an incredibly obscure part of the Mach-O format, and\n Mach-O is pretty obscure to start with. Fortunately this feature is\n only used by C++ libraries, so we can get started without it.\n\n- When mangling imports, we convert any lazy imports (that need\n mangling) into eager imports. This is required because the lazy\n import stubs hard-code the memory layout of the import table into\n immediate constants inside the stub assembly itself, and I do not\n feel like trying to automatically rewrite x86-64 opcodes. Instead,\n we leave the lazy import table alone (so all the unmangled lazy\n imports can continue to use it), and eagerly bind all the mangled\n imports, so the unmangled stubs never get called.\n\n- I noticed some new code dyld in MacOS 10.12 that imposes some\n annoying arbitrary restrictions on which order the different bits of\n DYLD_INFO appear in the file. This should only affect libraries that\n are built with 10.12 as their minimum required version, so for folks\n trying to build stuff for general distribution this shouldn't matter\n for a while. This also isn't hard to fix, it just means that we'll\n probably have to start making some pointless redundant copies of\n bits of the file that we *didn't* change, just so that the second\n copy can be placed after the bit of the file that we did change,\n which is tiresome and I haven't gotten around to it yet.\n\n- When mangling imports, we don't check for re-exports, which are also\n a kind of import. Should probably fix this...\n\n\nPE features\n-----------\n\nA tool that can read in a PE file (``.exe`` or ``.dll``) that is\ncurrently linked to ``foo.dll``, and rewrite it so that it becomes\nlinked to ``bar.dll`` instead (similar to ``patchelf --replace`` on\nLinux, or ``install_name_tool -change`` on OS X). This is useful for\navoiding naming collisions between different versions of the same\nlibrary.\n\nFor example, suppose you have two Python extensions ``A.dll`` and\n``B.dll``, that are distributed separately by different people. They\nboth contain some fortran code linked to to ``libgfortran-3.dll``, so\nboth packages ship a copy of ``libgfortran-3.dll``. Because of the way\nWindows DLL loading works, what will happen is that if I load\n``A.dll`` first, then *both* ``A.dll`` and ``B.dll`` will end up using\nA's copy of ``libgfortran-3.dll``, while B's copy will be ignored. (Or\nvice-versa if I import B first.) This will happen even if I arrange\nthings so that A's copy is not on the DLL search path at the time that\nB is loaded -- Windows always checks for already-loaded DLL's with a\ngiven basename before it actually checks the DLL search path (modulo\nsome complications around SxS assemblies, but you don't really want to\ngo there).\n\nThis is bad, because there's no guarantee that ``B.dll`` will work\nwith A's version of ``libgfortran-3.dll`` (e.g., A's copy might be too\nold for B). Welcome to `DLL hell\n`_!\n\nWe could avoid all this by renaming the colliding libraries to have\ndifferent names, e.g. ``libgfortran-3-for-A.dll`` and\n``libgfortran-3-for-B.dll``. But if we just rename the files, then\neverything will break, because ``A.dll`` is looking for\n``libgfortran-3.dll``, not ``libgfortran-3-for-A.dll``.\n\nThis is where ``machomachomangler`` comes in: it lets you patch\n``A.dll`` so that it's linked to ``libgfortran-3-for-A.dll``. And then\neverything works. Hooray.\n\nThis basically solves the same problem as private SxS assemblies,\nexcept better in all ways: it's simpler (no XML manifests), more\nflexible (no finicky requirements for the filesystem layout), and\ndoesn't require reading the awful SxS assembly documentation.\n\nExample usage::\n\n $ python3 -m machomachomangler.cmd.redll A.dll A-patched.dll libgfortran-3.dll libgfortan-3-for-A.dll\n\nThere's an example in ``example/`` then you can play with. E.g. on\nDebian with a mingw-w64 cross-compiler and wine installed::\n\n $ cd pe-example/\n\n $ ./build.sh\n + i686-w64-mingw32-gcc -shared test_dll.c -o test_dll.dll\n + i686-w64-mingw32-gcc test.c -o test.exe -L. -ltest_dll\n + i686-w64-mingw32-strip test.exe\n\n $ wine test.exe\n dll_function says: test_dll\n\n $ mv test_dll.dll test_dll_renamed.dll\n\n # Apparently wine's way of signalling a missing DLL is to fail silently.\n $ wine test.exe || echo \"failed -- test_dll.dll is missing\"\n failed -- test_dll.dll is missing\n\n $ PYTHONPATH=.. python3 -m machomachomangler.cmd.redll test.exe test-patched.exe test_dll.dll test_dll_renamed.dll\n\n # Now it works again:\n $ wine test-patched.exe\n dll_function says: test_dll\n\nSome known limitations of the PE dll-import-switcheroo code:\n\n- The command line tool could be less minimalist.\n\n- GNU ``objdump`` has a bug where it can't read the import tables of\n our patched PE files -- it just shows all of the import table until\n it hits the patched entry, and then it stops displaying\n anything. (The issue is that ``binutils`` wants all the data\n involved in the import tables to come from a single PE section.)\n However, I've tried giving the patched files to Dependency Walker,\n ``wine``, and Windows itself, and they all handle them fine -- so\n the files are okay, it's just a bug in ``objdump``. Just be warned\n that if you're trying to use ``objdump`` to check if the patching\n worked, then it's almost certainly going to tell you a confusing\n lie.\n\n- Unsurprisingly, this kind of patching does not play well with code\n signing. We try to at least clear any existing signatures (so that\n the binary becomes unsigned, rather than signed with an invalid\n signature), but this hasn't been tested.\n\n- We don't try to handle files with trailing data after the end of the\n PE file proper. This commonly occurs with e.g. self-extracting\n archives and installers. Shouldn't be a big deal in theory, but I\n did find that when compiling a simple ``.exe`` with mingw-w64 the\n tool refused to work until I had run ``strip`` on the binary, even\n though in theory this should work fine -- so probably there's some\n improvements possible.\n\n [Note to self: it looks like this is a GNU extension for putting\n long section names into PE files, which I guess are they use for\n their debug format -- this is `documented here\n `__, search for\n \"Coff long section names\". It's probably not hard to handle this\n better, e.g. by stripping it ourself or even fixing it up.]\n\n- We don't try to update the PE header checksum, since the algorithm\n for doing this is (nominally) a secret, and I'm informed that for\n regular user-space code there's nothing that actually cares about\n whether it's correct. But my information could be wrong. (Note: it\n looks like binutils might know how to compute this checksum? I'm not\n sure.)\n\n [Update: Stefan Kanthak informs me that this algorithm is well\n known, and in fact it looks `pefile has an MIT-licensed Python\n implementation\n `_\n so I guess it might be good to fix this at some point.]\n\n\nGeneral limitations\n-------------------\n\nOnly tested on **Python 3.4 and 3.5**. Probably any Python 3 will\nwork, and Python 2 definitely won't without some fixes. (There's lots\nof fiddly byte-string handling.)\n\nI'm lazy, so I just load the whole binary files into memory -- maybe\nseveral copies of it. This actually wouldn't be too hard to fix (using\nmemory mapping etc.) but I guess it doesn't matter that much because\n`who has multi-gigabyte Mach-O/PE images?\n`_?\n\n\nContact\n-------\n\nwheel-builders@python.org\n\n\nLicense\n-------\n\nIt's Saturday afternoon, I've got the flu or something, and I'm\nspending my free time writing software to make some proprietary\noperating systems -- ones that are backed by one of the world's larger\ncorporations -- better able to compete for developers with other,\nbetter-designed operating systems. I mean, I'm not saying that poring\nover the PE/COFF specification isn't fun! But it's not *that*\nfun. (And honestly the Mach-O docs are absolutely terrible, to the\nextent they exist at all.)\n\nTo assuage my annoyance, this software is licensed under the *GNU\nAffero General Public License as published by the Free Software\nFoundation, either version 3 of the License or (at your option)\nany later version*. See ``LICENSE.txt`` for details.\n\nThis **shouldn't have any effect** on most uses, since it only affects\npeople who are redistributing this software or running it on behalf of\nother people; you can *use* this software to manipulate your\nBSD-licensed DLLs, your proprietary-licensed DLLs, or whatever you\nlike, and that's fine. The license affects the code for\nmachomachomangler itself; not the code you run it on.\n\nHowever, if for some reason you or your company have some kind of\nallergy to this license, send me `an email\n`_ and we'll work out an appropriate tithe.\n\nAlso, to preserve our options in case I get over this fit of pique,\nplease **license all contributions under the MIT license**. (I\ndefinitely will not switch to any proprietary license, but might\nswitch to a permissive OSS license.) Thanks!\n\n\nCode of conduct\n---------------\n\nContributors are requested to follow our `code of conduct\n`_\nin all project spaces.\n\n\n", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/njsmith/machomachomangler", "keywords": "", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "machomachomangler", "package_url": "https://pypi.org/project/machomachomangler/", "platform": "", "project_url": "https://pypi.org/project/machomachomangler/", "project_urls": { "Homepage": "https://github.com/njsmith/machomachomangler" }, "release_url": "https://pypi.org/project/machomachomangler/0.0.1/", "requires_dist": [ "attrs" ], "requires_python": "", "summary": "Tools for mangling Mach-O and PE binaries", "version": "0.0.1" }, "last_serial": 3482442, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "6a9efa0be78389fb20604fe09e281f34", "sha256": "4fa66b8b65959399874f05a4d93abe1e49db46f71bef66a68a93e060d9ba0cb3" }, "downloads": -1, "filename": "machomachomangler-0.0.1-py3-none-any.whl", "has_sig": false, "md5_digest": "6a9efa0be78389fb20604fe09e281f34", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 70901, "upload_time": "2018-01-11T23:32:27", "url": "https://files.pythonhosted.org/packages/71/df/fba669db568cc25eb44a726bbce1a0a40e3a1c5b10a1a57492877b79c00b/machomachomangler-0.0.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "2d78acc236d6c9469787191c5dbfcee2", "sha256": "2acbad5bd465c5f386e5ceac8a8a1939a9789f503dd84f491e046292cf6e54a6" }, "downloads": -1, "filename": "machomachomangler-0.0.1.tar.gz", "has_sig": false, "md5_digest": "2d78acc236d6c9469787191c5dbfcee2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 64271, "upload_time": "2018-01-11T23:32:28", "url": "https://files.pythonhosted.org/packages/d0/29/d9f07b12f19d0fe64c1e88bbbebe49faf710841144c5a67dba5b98f42ab6/machomachomangler-0.0.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "6a9efa0be78389fb20604fe09e281f34", "sha256": "4fa66b8b65959399874f05a4d93abe1e49db46f71bef66a68a93e060d9ba0cb3" }, "downloads": -1, "filename": "machomachomangler-0.0.1-py3-none-any.whl", "has_sig": false, "md5_digest": "6a9efa0be78389fb20604fe09e281f34", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 70901, "upload_time": "2018-01-11T23:32:27", "url": "https://files.pythonhosted.org/packages/71/df/fba669db568cc25eb44a726bbce1a0a40e3a1c5b10a1a57492877b79c00b/machomachomangler-0.0.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "2d78acc236d6c9469787191c5dbfcee2", "sha256": "2acbad5bd465c5f386e5ceac8a8a1939a9789f503dd84f491e046292cf6e54a6" }, "downloads": -1, "filename": "machomachomangler-0.0.1.tar.gz", "has_sig": false, "md5_digest": "2d78acc236d6c9469787191c5dbfcee2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 64271, "upload_time": "2018-01-11T23:32:28", "url": "https://files.pythonhosted.org/packages/d0/29/d9f07b12f19d0fe64c1e88bbbebe49faf710841144c5a67dba5b98f42ab6/machomachomangler-0.0.1.tar.gz" } ] }