{ "info": { "author": "Jeff Hammel", "author_email": "jhammel@mozilla.com", "bugtrack_url": null, "classifiers": [], "description": "fetch\n=====\n\nfetch stuff from the interwebs\n\n`fetch.py `_ is\na single-file python module bundled as a\n`package `_ for easy installation\nand python importing. The purpose of fetch is to mirror remote\nresources (URLs) to a local filesystem in order to synchronize and\nupdate dependencies that are desired to be mirrored in this way.\n\n\nFormat\n------\n\n``fetch`` fetches from a manifest of the format::\n\n [URL] [Destination] [Type]\n\nA URL can contain a hash tag (e.g. http://example.com/foo#bar/fleem)\nwhich is used to extract the subdirectories from a multi-directory\nresource.\n\nThe ``Type`` of the resource is used to dispatch to the included\nFetchers that take care of fetching the object.\n\nManifests are used so that a number of resources may be fetched from a\nparticular ``fetch`` run.\n\n\nExample\n-------\n\nAfter you checkout the `repository `_\nand run ``python setup.py develop``, you should be able to run\n``fetch`` on the example manifest::\n\n fetch example.txt\n\nThis will create a ``tmp`` directory relative to the manifest and pull\ndown several resources to it.\n\n\nFetchers\n--------\n\n``fetch`` includes several objects for fetching resources::\n\n file : fetch a single file\n tar : fetch and extract a tarball\n hg : checkout a mercurial repository\n git : checkout a git repository\n\nThe ``file`` fetcher cannot have a hash tag subpath since it is a single\nresource.\n\nThough ``fetch`` has a set of fetchers included, you can pass an\narbitrary list into ``fetch.Fetch``'s constructor.\n\n\nVersion Control\n---------------\n\nThe ``hg`` and the ``git`` fetchers fetch from version control systems and\nhave additional options. The only current option to the constructor\nis ``export``, which is by default True. If ``export`` is True, then\nthe repository will be exported into a non-versioned structure. If a\nsubpath is specified with a ``#`` in the URL, the repository will also\nbe exported.\n\n\nTODO\n----\n\n * use `manifestparser `_\n ``.ini`` files versus another manifest\n format: when I started work on ``fetch``, I thought a\n domain-specific manifest would be a big win. But, now, maybe a\n ``.ini`` type manifest looks about the same, and is something that\n is already used. The switch internally wouldn't be that bad, but\n if ``fetch.py`` is used as a single file, it cannot have\n \"traditional\" python dependencies. Since ``manifestparser.py`` is\n also a single file, and ``fetch`` is only usable with internet\n access anyway, maybe the\n `require `_\n pattern could be used for this purpose\n\n * clobber: generally, you will want the existing resource to be\n clobbered, avoiding renames regarding upstream dependencies\n\n * outputting only subpaths: often, you will not to fetch from the\n whole manifest, only from certain subpaths of the manifest. You\n should be able to output a subset of what is to be mirrored based\n on destination subpaths. The CLI option ``--dest`` is intended for\n this purpose but currently unused.\n\n * fetcher options: currently ``read_manifests`` reads an unused\n column into ``options`` when present in the form of a string like\n ``foo=one,bar=two`` into a dict like\n ``{'foo': 'one', 'bar': 'two'}``. This hasn't been needed yet and\n is unused. If we want to have resource-specific options, we should\n use this and make it work. Otherwise it can be deleted.\n\n * python package fetcher: often you will want to fetch a python\n package as a resource. This would essentially fetch the object\n (using another fetcher) and take the (untarred) result of\n ``python setup.py sdist`` as a resource. This will strip out files\n that aren't part of the python package. Unknowns include how to\n specify the sub-fetcher. You could also include other\n domain-specific fetchers as needed.\n\n * note python 2.5+ specifics: ``fetch`` currently uses at least\n ``os.path.relpath`` from python 2.5 and perhaps other 2.5+isms as\n well. These should at least be documented and checked for if not\n obviated. One such reimplementation is at\n https://github.com/mozilla/mozbase/blob/master/manifestdestiny/manifestparser.py#L66\n\n\nUnsolved Problems\n-----------------\n\nA common story for ``fetch`` is mirroring files into a VCS repository\nbecause the remote resources are needed as part of the repository and\nthere is no better way to retrieve and/or update them. However, what\ndo you do if these remote resources are altered? In an ideal\necosystem, the fixes would be automatically triaged and triggered for\nupstream inclusion, or the diffs from the upstream are kept in local\nmodifications (although vendor branches, etc, are more suitable for\nthe latter class of problems, and in general discouraged when a less\nintrusive system of consuming upstream dependencies are available).\n\n----\n\nJeff Hammel\nhttp://k0s.org/mozilla/hg/fetch", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://k0s.org/mozilla/fetch", "keywords": null, "license": "MPL", "maintainer": null, "maintainer_email": null, "name": "fetch", "package_url": "https://pypi.org/project/fetch/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/fetch/", "project_urls": { "Download": "UNKNOWN", "Homepage": "http://k0s.org/mozilla/fetch" }, "release_url": "https://pypi.org/project/fetch/0.0/", "requires_dist": null, "requires_python": null, "summary": "fetch stuff from the interwebs", "version": "0.0" }, "last_serial": 791933, "releases": { "0.0": [ { "comment_text": "", "digests": { "md5": "4ff8ca389dfa14531019a698ba4652b3", "sha256": "6f5f6e00d853671c4dc27023c19b2ed462334e757ce1f3dcfdcf18325ae27490" }, "downloads": -1, "filename": "fetch-0.0.tar.gz", "has_sig": false, "md5_digest": "4ff8ca389dfa14531019a698ba4652b3", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7346, "upload_time": "2011-11-16T01:41:24", "url": "https://files.pythonhosted.org/packages/95/e5/9e5775c180fce1da40c04468f78e1305b49895fb3abefbaa85f0a4de6e63/fetch-0.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "4ff8ca389dfa14531019a698ba4652b3", "sha256": "6f5f6e00d853671c4dc27023c19b2ed462334e757ce1f3dcfdcf18325ae27490" }, "downloads": -1, "filename": "fetch-0.0.tar.gz", "has_sig": false, "md5_digest": "4ff8ca389dfa14531019a698ba4652b3", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7346, "upload_time": "2011-11-16T01:41:24", "url": "https://files.pythonhosted.org/packages/95/e5/9e5775c180fce1da40c04468f78e1305b49895fb3abefbaa85f0a4de6e63/fetch-0.0.tar.gz" } ] }