PKC"EdSyy release.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. product = "pyerector" number = "1.2.5" PK:"E 6"6" CHANGES.txtv1.1.1, 2013-03-31 -- Create proper Iterator classes. v1.1.2 2013-04-04 -- Add version control modules as 'VCS'. v1.1.3 2013-05-06 -- Issue 4 - Uptodate unit tests are skipped -- Issue 6 - Build property stack of Target/Task calls... -- Issue 7 - Unittests fail on Python 2.6 -- Issue 8 - Spawn keyword argument processor error and... -- Issue 10 - Shebang task does not follow docs -- Issue 14 - Basedir is set for current directory, not... New tasks -- HashGen to create md5/sha1 hash files -- Egg to create python archive files Other fixes/enhancements: -- Add --quiet and --DEBUG options and handling output processing better; -- Add timing information after targets and the program finish. -- Add new display object, a Verbose object that should (almost) always be on. -- Replaced multiple exclusion predicates with a class to handle it. -- Use mappers instead of Uptodate instances. -- Remove 'noglob' keyword from Initer.get_files() method. -- Ignore cases where python3 is not installed. -- Add a LOT more unit tests (from 15 tests to 91). -- Prevent setup.py running if pyerect is present. -- Use setuptools instead of distutils. v1.2.0 2013-10-13 -- Issue 11 - Better use of variables; including showing variables on verbose output of "help" target. -- Issue 12 - Add ability to specify variables on the command-line: If a target has an equal sign ("="), then it is taken as a variable assignment. -- Issue 13 - Update Version model. -- Issue 16 - Convert from Config to Variable; Initer.config is kept for backward compatibility, but just references Variable -- Issue 18 - Replace Verbose instances with Python's logging module. -- Add a "DISPLAY" log level above ERROR; keep the display() routine -- Use new State class instead for noop and noTimer. -- Issue 20 - Make the product thread-safe New targets -- InitVCS, initializes the version control support New tasks -- SubPyErector, call other PyErector projects Other fixes/enhancements: -- Two new constructs: Sequential and Parallel; Parallel is a subclass of Sequential; "uptodates", "dependencies" and "tasks" members of implicitly instances of Sequential; "uptodates" cannot be Parallel, but the other two can be. Parallel targets/tasks are executed in separate threads but may access Variables without concern (see Issue 20). -- Add execution stacks for Target and Task instances; this stack is displayed instead of Python's on errors. -- Add default variables on startup: -- basedir (set by PyErector and Initer instantiation) -- pyerector.vcs.version -- pyerector.vcs.branch -- pyerector.vcs.tags -- pyerector.release.product -- pyerector.release.number -- Handle PYERECTOR_PREFIX envvar better. -- Remove extraneous "hasformat" functionality. -- Fixes to unit tests. -- Better help strings. v1.2.1 2013-10-11 -- Issue 12 - Add ability to specify variables on the command-line (update) -- Issue 15 - Redesign iterators/mappers -- Issue 19 - Dirlist does not work as might be expected -- Issue 20 - Thread-safe operations (update) -- Issue 21 - Backward compatibility for Verbose New tasks -- Echo, displays message Other fixes/enhancements: -- Add pyerector.pool.size variable, default to 10, to limit how many PyThread instances can be created (not including the PyErector instance). -- Update pyerect to better use iterators. -- Handle that PyErector always returns an exception, mainly SystemExit -- Mark as "Failed." instead of "Done." if an error occurs; have PyErector handle a proper exit status on failure. -- Update inheritance for LogFormatter - on py2.6, super() throws a TypeError -- Use of Verbose and Config classes now raise DeprecationWarnings -- Move noop and noTimer to config module. v1.2.2 2013-10-17 Other fixes/enhancements: -- Add (initial) regression tests. -- Fix BaseIterators to properly handle empty directories with noglob=True; have Clean Target use DirList. -- Fix bug when iterator has an empty sequence. -- Have the Help target depend on InitVCS. -- Unittest fix for Python3. -- Fixes found during analysis from PyCharm and pylint. -- Use short-cut class instead of function-with-mutables for initialization. -- Some SCM cleanup. v1.2.3 2013-10-25 -- Issue 9 - Propagate iterator parameters in Initer.get_files. -- Issue 23 - Add an 'api' module to aid programming New tasks -- Scp, perform remote copies (via scp) -- Ssh, perform a remote command (via ssh) New iterator/mapper -- IdentityMapper, destination file is always the destdir value. Other fixes/enhancements: -- Allow a Variable instance in a dependencies or tasks sequence. -- Code cleanup using PyCharm and pylint. -- Collapse FileSet into BaseIterator, make FileSet same as FileIterator. -- Greatly simplified the calling system, now allows for nested calls of Sequential/Parallel. -- Refactor the "allow" condition in BaseIterator.next to be more straightforward. -- Fix the RealClean target, including a fix to Exclusions. -- Fix to Help target for Targets with no docstring. -- Fix how the regression module sets the proper sys.path value. -- Fix condition where exception was shown on Abort. -- Fixes to Subcommand calls. -- Change Regression to a Target from a SubPyErector subclass so it can be called by itself. -- Create symlinks under regression/ to the proper (local) distribution, thus overriding sys.path. -- Move unittest driver script, named "unit-test.py" (to prevent imports) outside of tasks.py; pass the parameters as a string instead of a filename. v1.2.4 2014-04-08 -- Issue 25 - Move BaseIterator and FileMapper logic to Iterator and Mapper respectively; remove BaseIterator. -- Issue 26 - Add directory attribute to pyerector.vcs.base.Base subclasses; move vcs_check under Base as a classmethod. -- Issue 27 - lazy-evaluating Path works, but not properly integrated with the rest of the modules. -- Issue 28 - Allow a cons as a valid return type of Iterator.next; handle that specially in Mapper.next. -- Issue 29 - The InitVCS target has been deprecated. The VCS() function is called just before the package finishes being impored. -- Issue 30 - pyerector.version.Version is now exported from the package; it is callable with release number strings to compare with the the current release number. If greater than the current release, terminate with an error message. New target -- Testonly performs the Unittest task and the Test target now has the dependencies of "Build, Testonly", with no tasks. This allows calling the "testonly" target without rebuilding everything. New task -- Touch which is similar to the touch(1) command, creating files if they do not exist. New type -- FileVariable allows some deferred resolution; file is loaded when accessed, not when created. Other fixes/enhancements: -- The Exclusions object now accepts three values: True, False and None; False will still include VCS().directory, and None will not include any extra values. -- Change vcs modules to a plugin type setup, using same registration as being used by Initer; plugins loaded automatically when pyerector.vcs.VCS is called. -- Get the proper information back from Mercurial; previously only the tip was retrieve. -- Allow for mappers in Tokenize. -- Typo in Symlinki task. -- Add *.user and *.date variables from VCS.current_info. -- Fix the template string for git-log; add an assertion. -- Tokenize now allows for multiple Iterators/Mappers to be passed. -- Various fixes to Iterators and Mappers; add uptodates entries in pyerect. -- Do not use assertions for type checking. -- Initer.asserttype returns TypeError now. -- Blank values for version strings. -- Get a closer timestamp for svn.date comparison in unittests. -- Allow an Iterator to be passed to Iterator.adjust. -- Use th proper version variables. -- Remove unused u() function. -- Fix unit tests after making Base.args into a tuple. v1.2.4.1 2014-07-25 -- Issue 33 - Traverse up the directory to find the right VCS. v1.2.5 2014-09-02 -- Issue 34 - Remove Unittest from the tasks of Testonly Other fixes/enhancements: -- Exclusions handle usedefaults better; usedefaults=False, add vcs names; usedefaults=None, nothing added. -- Fix vcs.base.Base.vcs_check to search each directory for a vcs before moving up, to prevent nfs mount issues. -- Fix warnings where EGG-INFO files were added twice to the toadd list. PKcD%oo README.txt========= PyErector ========= An Erector set for building systems ----------------------------------- PyErector is more clear and Pythonic than using Distutils, but not as deep. The idea is to organize the work to be done in tasks (work units) and targets (organizational units). Targets can specify tasks to work on, or override the default operation (does nothing) and specify dependencies. Tasks can specify files or data to perform specialized operations on (Copy, Tar, etc.). Objects are classes and can be subclassed to create new work flows. Installation ============ Using file copy --------------- Copy the ``pyerector`` structure into the project directory, then create a ``pyerect`` script using the structure as a library. Using distutils --------------- Run the ``./setup.py install`` to install into Python's distribution. Starting out ============ PyErector is a library used by any program to create tasks and targets for performing build management steps. Traditionally, the main program is called ``pyerect``, but it can be any executable Python script. Start by importing from ``pyerector``:: #!/usr/bin/python from pyerector import * Next modify attributes of standard targets:: Compile.tasks = ( PyCompile(FileIterator('src', pattern='*.py')), ) Lastly, call the main routine:: PyErector() This small program would generate pyc files for each of the Python source files found under the ``src`` directory. Documentation ============= Please visit the `project site `_ for documentation. ---- Copyright (C) 2010-2013 Michael P. Reilly. All rights reserved. PKSCsetup.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. from setuptools import setup import os import release import sys program = sys.argv[0] progdir = os.path.dirname(program) # do not run if from the source directory if __name__ == '__main__' and os.path.exists(os.path.join(progdir, 'pyerect')): raise SystemExit('please run "pyerect" instead of %s.' % sys.argv[0]) setup( name=release.product, version=release.number, description="Self-contained Python build library", long_description="Python module to create simplified build scripts", author="Michael P. Reilly", author_email="arcege@gmail.com", url="http://code.google.com/p/pyerector", packages=[ "pyerector", "pyerector.py2", "pyerector.py3", "pyerector.vcs", ], download_url="http://code.google.com/p/pyerector/downloads/", license= "License :: OSI Approved :: GNU General Public License v3 (GPLv3)", classifiers=[ "Development Status :: 4 - Beta", "Environment :: Console", "Intended Audience :: Developers", "Operating System :: MacOS :: MacOS X", "Operating System :: Microsoft", "Operating System :: POSIX", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Topic :: Software Development :: Build Tools", ], ) PKSC|wfKK LICENSE.txt GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . PKD\'E"_ggEGG-INFO/PKG-INFOMetadata-Version: 1.1 Name: pyerector Version: 1.2.5 Summary: Self-contained Python build library Home-page: http://code.google.com/p/pyerector Author: Michael P. Reilly Author-email: arcege@gmail.com License: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Download-URL: http://code.google.com/p/pyerector/downloads/ Description: Python module to create simplified build scripts Platform: UNKNOWN Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console Classifier: Intended Audience :: Developers Classifier: Operating System :: MacOS :: MacOS X Classifier: Operating System :: Microsoft Classifier: Operating System :: POSIX Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Topic :: Software Development :: Build Tools PKD\'E2EGG-INFO/dependency_links.txt PKD\'E2EGG-INFO/zip-safe PKD\'Eu-v EGG-INFO/top_level.txtpyerector PKD\'EEGG-INFO/SOURCES.txtCHANGES.txt LICENSE.txt README.txt pyerector/__init__.py pyerector/__init__.pyc pyerector/api.py pyerector/base.py pyerector/base.pyc pyerector/config.py pyerector/config.pyc pyerector/exception.py pyerector/exception.pyc pyerector/execute.py pyerector/execute.pyc pyerector/helper.py pyerector/helper.pyc pyerector/iterators.py pyerector/iterators.pyc pyerector/main.py pyerector/main.pyc pyerector/metaclass.py pyerector/metaclass.pyc pyerector/py2/__init__.py pyerector/py2/__init__.pyc pyerector/py2/base.py pyerector/py2/base.pyc pyerector/py3/__init__.py pyerector/py3/base.py pyerector/py3/execfile.py pyerector/register.py pyerector/register.pyc pyerector/targets.py pyerector/targets.pyc pyerector/tasks.py pyerector/tasks.pyc pyerector/unit-test.py pyerector/variables.py pyerector/variables.pyc pyerector/vcs/__init__.py pyerector/vcs/__init__.pyc pyerector/vcs/base.py pyerector/vcs/base.pyc pyerector/vcs/git.py pyerector/vcs/git.pyc pyerector/vcs/mercurial.py pyerector/vcs/mercurial.pyc pyerector/vcs/none.py pyerector/vcs/none.pyc pyerector/vcs/subversion.py pyerector/vcs/subversion.pyc pyerector/version.py pyerector/version.py.orig pyerector/version.pyc release.py setup.py PKJ#E6V V pyerector/config.pyc ASc@swdZddlmZddlmZdddgZdefdYZd efd YZeZ eZ d S( suHandle configuration values. The Config class managed "variables" before global variables were introduced. This class has now been deprecated and exists only for backward compatibility. The State class is to give the same functionality as the now deprecated Verbose class. The "noop" and "noTimer" instances are initialized here and turned on in PyErector.arguments(). i(twarni(tVtConfigtnooptnoTimercBs eZdZdZdZRS(sCDeprecated, but still provide the same interface, but to variables.cCstdtt|S(NsConfig has been deprecated(RtDeprecationWarningR(tselftvariable((sB/home/arcege/src/pyerector/regression/parallel/pyerector/config.pyt __getattr__s cCstdt|t| s   PKJ#E S^|,|,pyerector/variables.pyc ASc@sdZddlZddlZddlmZddddgZd efd YZeZde fd YZ de fd YZ de fd YZ dS(s; Usage: Variable('name', 'value') Variable('name').value = 'value' print Variable('name') (Variable('name') == Variable('name')) (Variable('name1') < Variable('name2')) (hash(Variable('name')) == hash('name')) (Variable('name').name == 'name') (Variable('name').value == 'value') (Variable('name').toString() == Variable('name').name) (Variable('name').value is Variable('name').value) (Variable('name').value == str(Variable('name'))) (V['name'] == Variable('name').value) (V('name') == Variable('name')) # for backward compatibility fv = FileVariable('file', 'test.txt') (Variable('file').value == 'test.txt') fv.value == open('test.txt', 'rt').read().decode('UTF-8') fv.value = 'test2.txt' (Variable('file').value == 'test2.txt') fv.value != open('test.txt', 'rt').read().decode('UTF-8') # VariableSet is an augmented dictionary vs = VariableSet( Variable('name', 'michael'), Variable('number', '42'), Variable('address', '1313'), Variable('city', 'springfield') ) (vs['name'] == Variable('name')) (vs['name'] == vs[Variable('name')]) (vs == {Variable('name'): Variable('name'), Variable('number'): Variable('number'), Variable('address'): Variable('address'), Variable('city'): Variable('city')}) (len(vs) == 4) (sorted(vs.keys()) == ['address', 'city', 'name', 'number'] vs.add(Variable('zipcode', '00000')) vs.update(phone='888-555-1234') vs.update(('phone', '888-555-1234')) (vs['phone'] == 'phone') (vs['phone'].value == '888-555-1234') ('name' in vs == True) vs1 = VariableSet( Variable('email': 'michael@springfield.us') ) vs1.update(vs) (vs1['name'] == Variable('name')) ('email' not in vs and 'email' in vs1) iNi(tErrort FileVariabletVtVariablet VariableSett VariableCachecBsbeZdZdZdZdZdZdZdZdZ dZ d d Z RS( sMimic part of the functionality of a dictionary, but keep things minimalistic. And some functionality, like copy(), we don't want.cCsi|_tj|_dS(N(tcachet threadingtRLocktlock(tself((sD/home/arcege/src/pyerector/regression/minimal/pyerector/variables.pyt__init__Hs cCsd|jj|jfS(Ns%s(%s)(t __class__t__name__R(R ((sD/home/arcege/src/pyerector/regression/minimal/pyerector/variables.pyt__repr__LscCs t|jS(N(tlenR(R ((sD/home/arcege/src/pyerector/regression/minimal/pyerector/variables.pyt__len__OscCs6t|jdr|jjSt|jjSdS(Ntiterkeys(thasattrRRtitertkeys(R ((sD/home/arcege/src/pyerector/regression/minimal/pyerector/variables.pyt__iter__Rs cCs?t|tst|}n|j||jkSWdQXdS(N(t isinstanceRR R(R tname((sD/home/arcege/src/pyerector/regression/minimal/pyerector/variables.pyt __contains__Xs cCs_t|tst|}n|j2||jkrB|j|Std|jWdQXdS(Nsno such variable: %s(RRR RRR(R R((sD/home/arcege/src/pyerector/regression/minimal/pyerector/variables.pyt __getitem__^s   cCsNt|tst|}n|j!tjd}||j|R?R!((sD/home/arcege/src/pyerector/regression/minimal/pyerector/variables.pyR s cCsZt|tst|}ntjd|jjt|tt|j ||dS(sAdd a new variable to the set.s %s.add(%s)N( RRRtdebugR R treprR%RR(R R!((sD/home/arcege/src/pyerector/regression/minimal/pyerector/variables.pyR@scCs4t|tst|}ntt|j|S(N(RRR%RR(R titem((sD/home/arcege/src/pyerector/regression/minimal/pyerector/variables.pyRscCsyt|tst|}n||kr:|j|ntjdt|t|||k ru|||_ndS(Nssetitem(%s, %s)(RRR@RRARBR(R RCR((sD/home/arcege/src/pyerector/regression/minimal/pyerector/variables.pyRs  cCstt|j|dS(N(R%RR(R RC((sD/home/arcege/src/pyerector/regression/minimal/pyerector/variables.pyRscOsx`|D]X}t|dr>x@|D]}||||6s   ? > PKcDї5pyerector/api.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Package: pyerector.api While this module is mostly empty, it gives access to the routines that a PyErector developer would need most: Abort, Error, Iterator, Mapper, PyThread, Subcommand. Other routines are imported from the pyerector.__init__ module by default. XXX Future development may see more things moved here from pyerector. """ # This is almost no overhead, just the size of a module; everything would # have already been imported by 'pyerector'. from .exception import Abort, Error from .execute import get_current_stack, PyThread from .variables import V, Variable, VariableSet from .helper import Exclusions, normjoin, Subcommand, Timer from .base import Initer, Iterator, Mapper, Target, Task from .vcs.base import DVCS_Base, VCS_Base __all__ = [ 'Abort', 'Error', 'DVCS_Base', 'get_current_stack', 'Iterator', 'Mapper', 'PyThread', 'Subcommand', 'Target', 'Task', 'VCS_Base', ] PKJ#E;ׂpyerector/tasks.pyc ?Tc@sdZddlZddlZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z mZmZddlmZmZmZmZmZdd lmZmZyeWn!ek rdd lmZnXd d d ddddddddddddddddddd d!gZd e fd"YZd#e fd$YZd e fd%YZd e fd&YZ d'e fd(YZ!de fd)YZ"de fd*YZ#de fd+YZ$de fd,YZ%de fd-YZ&de fd.YZ'defd/YZ(de fd0YZ)d1e fd2YZ*de*fd3YZ+de*fd4YZ,de fd5YZ-de fd6YZ.defd7YZ/de fd8YZ0de fd9YZ1de fd:YZ2d;e fd<YZ3de3fd=YZ4d e3fd>YZ5d!efd?YZ6de6fd@YZ7dS(AsDefine the standard tasks.iNi(tError(t Exclusionst Subcommand(tnoTimer(tTasktMappertIterator(tBasenameMappertIdentityMappert FileMappert FileIteratortStaticIterator(tVt VariableSet(texecfiletChmodtCopytCopyTreetEchotEggtHashGentJavatMkdirt PyCompiletRemovetScptShebangtSpawntSsht SubPyErectortSymlinktTartTokenizetTouchtUnittesttUntartUnziptZipcBs,eZdZdZeddZdZRS(sHChange file permissions. constructor arguments: Chmod(*files, mode=0666)t666icCsddlm}|jdt}x_|j|jdD]E}|j|td|jj d||||j ||q;WdS(s$Change the permissions on the files.i(tchmodtmodetfiless chmod(%s, %o)N( tosR't get_kwargtintt get_filestget_argst asserttypetstrtloggertinfotjoin(tselfR'R(tfname((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pytrun's ((t__name__t __module__t__doc__R)R,R(R6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR st ContainercBsleZdZd ZddlmZd Zd Z e de Z dZ dZdZdZdZRS( s>An internal task for subclassing standard classes Tar and Zip.i(tcurdirt usedefaultsc Cs|jdtdt}|j|jdt}|jdtttf}t|tslt|}n|j|||g}ddl m }t|j d}x|r|d}|d=x||j||D]}|j |rqt j j|st j j|r)|j|qt j j|rgt j|D]} t j j|| ^qK} |j| qqWqW|j||||j||||j|||d S( s1Gather filenames and put them into the container.tnametnoNonetroottexcludei(tglobR)iN(R+R0tTrueR3Rttupletlistt isinstancetpreopRAR.tmatchR*tpathtislinktisfiletappendtisdirtlistdirtextendtmanifesttcontaintpostop( R4R=R?texcludesttoaddRAtqueuetentryR5tftfnames((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR69s0  $.cCsdS(sTo be overridden.N((R4R=R?RR((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRFUscCsdS(sTo be overridden.N((R4R=R?RR((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRQXscCsdS(sTo be overridden.N((R4R=R?RS((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRO[scCsdS(sTo be overridden.N((R4R=R?RS((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRP^sN((R7R8R9tNonetappnameR*R;R?R=R)RtFalseR@R6RFRQRORP(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR:1s    cBs)eZdZdZdZeZdZRS(sCopy files to a destination directory. Exclude standard hidden files. constructor arguments: Copy(*files, dest=, exclude=)c Cs|jdtdt}|jd}|jdtttf}t|ts`t|}nt|dkr|d krt|dt r|d}njt|dkr|d k rt j j | rt|j|d|}nt|j|d|}|jjdt|x|D]\}}|j|}|j|}|jt j j|s(t j j|r|j||r|jjd |q|jjd ||tj||q(q(Wd S( s&Copy files to a destination directory.tdestR>R)R@iitdestdirsCopy.fmap = %ss uptodate: %ss copy2(%s, %s)N(R+R0RZR.RRCRDREtlenRXRR*RHRLRR-R R1tdebugtvarsR3RGtbasenameRJt checkpairR2tshutiltcopy2( R4R[R)RRtfmaptsnametdnametsrcfiletdstfile((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6ks&1 1$(N( R7R8R9R)RXR[RZtnoglobR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRbs cBs2eZdZdZdZeZeZdZ RS(sCopy directory tree. Exclude standard hidden files. constructor arguments: CopyTree(srcdir=, dstdir=, exclude=)c Cs|jdtdt}|jdtdt}|jdtttf}t|tsit|}ntjj |j |st dd|n1tjj |j |st dd|nt d td|}t}tjg}x|r|d }|d =|j|s||j ||xtj|j ||D]}|j|sP|j |||} |j |||} tjj | r|jtjj ||q|| d | qPqPWqqWd S( sCopy a tree to a destination.tsrcdirR>tdstdirR@isNo such file or directory: isNot a directory: RiiR[N(R+R0RBRRCRDRER*RHtexistsR3tOSErrorRLRRR;RGRMRK( R4RjRkRRtcopy_ttmkdir_ttdirsttdirR5tspathtdpath((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6s0    "N( R7R8R9RXRjRkRR@RRR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRs  tDownloadcBs#eZdZdZdZdZRS(sNRetrieve contents of URLs. constructor arguments: Download(*urls, destdir=DIR)cCs tdS(N(tNotImplementedError(R4((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6sN((R7R8R9RXR[turlsR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRtscBseZdZdZdZRS(sDisplay a message, arguments are taken as with logger (msg, *args). This is displayed by the logging module, but at the internal 'DISPLAY' level created in pyerector.helper.cCs]|jd}|r7|d|d}}||}nd}|jjtjd|dS(sDisplay messages.tmsgsiittDISPLAYN(R.R1tlogtloggingt getLevelName(R4targstmsgtrestttext((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6s  ((R7R8R9RwR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRscBs#eZdZdZdZdZRS(sGenerate file(s) containing md5 or sha1 hash string. For example, generates foobar.txt.md5 and foobar.txt.sha1 for the contents of foobar.txt. By default, generates for both md5 and sha1. constructor arguments: HashGen(*files, hashs=('md5', 'sha1'))tmd5tsha1c Csddlm}m}|j|jd}|jdt}|jjd||g}d|kr|j |t |ddfnd |kr|j |t |dd fnx|D]\}}x|D]\}} |} t j j |r|j|j||j|  r| jt|j|d j|jjd | t|j| d j| jdqqWqWdS(s%Generate files with checksums inside.i(RRR)thashssfiles = %s; hashs = %sRtmappers %(name)s.md5Rs %(name)s.sha1trbs writing %stwts N(thashlibRRR-R.R+RCR1R^RKR R*RHRJRaR3tupdatetopentreadtwritet hexdigest( R4RRR)RtfmapsthashfuncRdReRfthashval((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6s*   %((smd5ssha1(R7R8R9R)RR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRscBsveZdZddlmZyedZWnek rCdZnXdZgZ [dZ dZ dZ dZ RS( spCall a Java routine. constructor arguments: Java(jar=, java_home=<$JAVA_HOME>, classpath=(), properties=[])i(tenviront JAVA_HOMEcCstj||jrLtjj|jrLtjj|jdd|_nftjjtjjtjjddrtjjtjjdddd|_n t dtj |jtj st dndS(Ntbintjavat~sno java program to execute( Rt__init__t java_homeR*RHRlR3t java_progt expanduserRtaccesstX_OK(R4((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRs !- $ cCs|jj||fdS(s'Add a Java system property to the list.N(t propertiesRK(R4tvartval((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pytaddprop#sc Cs"ddlm}ddlm}|jdtdt}|jrdg|jD]}d|^qK}nd }|jft |d|ft g|j D]}t|^q}|j }|j r|j |j |ds-D%s=%ss-jart CLASSPATHs%s failed with returncode %dN((R*Rtos.pathRR+R0RBRRRCR}tcopyt classpathR3Rt returncodeRt __class__R7tlower( R4RRRtxtsysproptstcmdtenvtproc((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6's #)     N((R7R8R9R*RRtKeyErrorRXRRRRRR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRs   cBs2eZdZdZeZdZedZRS(sDRecursively create directories. constructor arguments: Mkdir(*files)cCsV|j|jd}x7|D]/}|j|td|j|j|qWdS(sMake directories.R)N(R-R.R/R0tmkdirR3(R4R)targ((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6Cs cCsddlm}|d}tjj|s@tjj|rm|jd|tj||j|nT|svnKtjj |s|jtjj ||jd|tj|ndS(sRecursive mkdir.i(t getLoggerspyerector.executes remove(%s)s mkdir(%s)N( R{RR*RHRIRJR2tremoveRRLtdirname(tclsRHRR1((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRJs $ (( R7R8R9R)RBRiR6t classmethodR(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR<s  cBs)eZdZdZdZdZdZRS(s^Compile Python source files. constructor arguments: PyCompile(*files, dest=, version='2')t2c Csddl}|j|jd}|jd tjd kr{xI|D]/}|jjd||j|j|qEWn|jd dkrd}n"|jd dkrd }nd }|d d d ft g|D]}|j|^q}yt |}Wn`t k rdtj d}|j ddkr^|jjd|jj||j dqn&X|jdkr|jjd|ndS(sCompile Python source files.iNR)ispy_compile.compile(%s)Rtpython2t3tpython3tpythons-cs,import sys; from py_compile import compile; s"[compile(s) for s in sys.argv[1:]]itENOENTs%s: Error with %s: %sscould not compile files with %s(t py_compileR-R.tversiontsysR1R^tcompileR3RCRRtexc_infoR}terrorRR7RR2( R4RtfilesetR5RRtcmdpRtexc((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6cs2    )  (N(R7R8R9R)RXR[RR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR[s cBs)eZdZdZeZdZdZRS(sFRemove a file or directory tree. constructor arguments: Remove(*files)cCse|jd}|jdt}|jdtttf}t|trNndt|dkr|t|tr||d}n6t|ttfrt d|d|t|}nx|D]}|j |t d|j |}t jj|st jj|r(|jjd|t j|qt jj|r|jjd|tj|qqWdS( s Remove a file or directory tree.R)RiR@iis remove(%s)s rmtree(%s)N(R.R+tboolRRDRCRERR]R R/R0R3R*RHRJRIR1R2RRLRbtrmtree(R4R)RiRRR=R5((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6s$! ! $(N( R7R8R9R)RZRiRXR@R6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRs cBs/eZdZdZdZdZdZdZRS(sReplace the shebang string with a specific pathname. constructor arguments: Shebang(*files, dest=, token='#!', program=)s#!c Cs|jjd|jdtdt}|j|jd}|jdt}yddlm}Wn!t k rddl m }nXxV|D]N}|j |}|j |d }|dkr|}n|j ||j |d }t|j |d } |} | j} | j|jrd | krH| jd } n| jtj} | j | t|j| !|} | j| n | j| tj| | | j| jd t|d } tj| | qWdS(s4Replace the shebang string with a specific pathname.sstarting ShebangtprogramR>R)R[i(tBytesIO(tStringIORxtrt itwN(R1R2R+R0RBR-R.tioRt ImportErrorRR3treplaceRXRtreadlinet startswithttokentfindR*tlinesepR]RRbt copyfileobjtclosetseek( R4RtsrcsR[RR5tinfnametheadtoutfnametinftoutftfirsttwsp((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6s<       "   (N( R7R8R9R)RXR[RRR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRs cBs5eZdZdZdZdZdZiZdZ RS(sdSpawn a command. constructor arguments: Spawn(*cmd, infile=None, outfile=None, errfile=None, env={})cCs>|jdt}|jdt}|jdt}|rK|j|pNd }|rf|j|pid }|r|j|pd }|jdt}|jd}t|d|d|d|d|}|jd krtd d t|t |jfn4|jd kr:td d t||jfnd S(sSpawn a command.tinfiletoutfileterrfileRRtstdintstdouttstderriRs%s signal %d raiseds%s returned error = %dN( R+R0R3RXtdictR.RRRtabs(R4RRRRRR((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6s  " (N( R7R8R9RRXRRRRR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRst SshEnginecBsGeZdZdZdZdZdZedZ dZ dZ RS(s?Superclass for Scp and Ssh since there is the same basic logic.cCsd}|jdtdt}|jdt}d|krW|jdd\}}n|rkd||fS| r|rd||fS|SdS(NRxthostR>tusert@is%s@%s(R+R0RBtsplit(R4truserRR((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pytremhosts  cCsk|jdt}|r'd|f}nd }|jdddddddddddd dd dd f|S( Nt identfiles-is-os BatchMode=yessConnectTimeout=10sForwardAgent=nos ForwardX11=nosGSSAPIAuthentication=nosLogLevel=ERRORsPasswordAuthentication=nosStrictHostkeyChecking=no((R+R0t_cmdprog(R4tidfileR((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pytgencmds cCs~|jjd|t|dtjdtjdt}|jrgtt||jj j n|j j j SdS(Ns ssh.cmd = %sRRtwait( R1R^RtPIPERBRRR0RRtrstripR(R4RR((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyt_runs     'N( R7R8R9RXRRRRtpropertyRRR(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRs cBsGeZdZdZdZdZeZe Z dZ dZ dZ RS(sSpawn an scp command. constructor arguments: Scp(*files, dest=, host=, user=, identfile=, recurse=bool, down=bool)tscpcCsd|j|fS(Ns%s:%s(R(R4tfilename((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pytremfile+scCs |j|S(N(R3(R4R((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pytlclfile-sc Cs|jdt}|jdtdt}|jdt}|jd}|rg|j|j}}n|j|j}}|j}|r|d7}nd }x!|D]} ||| f7}qW|||f7}|jj d|||j ||dS( NtrecurseR[R>tdownR)s-rs%s%s(s-r(( R+RR0RBR.RRRR1R2R( R4RR[RR)tlefttrightRtfilelstR5((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6/s    (N(R7R8R9RR)RXR[RZRRBRRRR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR s  cBs#eZdZdZdZdZRS(sSpawn an ssh command and display the response in verbose mode. constructor arguments: Ssh(*cmd, host=, user=, identfile=) tsshcCs{|jd}|jd|jf|}|jjd|||j|}|jjd|jjdddS(NRs-Ts%s%ss s s ( R.RRR1R2RtwarningRR(R4tusercmdRtresponse((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6Is ((R7R8R9RRR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRBscBs/eZdZdZdZdZiZdZRS(sCall a PyErector program in a different directory. constructor arguments: SubPyErector(*targets, wdir=None, prog='pyerect', env={}) Adds PYERECTOR_PREFIX environment variable.tpyerectc Cs|jd}|jdt}g}tjd}|jtjrX|jdnD|jtjrz|jdn"|jtj r|jdnt r|jdnt j j d|ft|t|}|jd t}|jd t}d d l m}d } t j j|} | |kr[|| r[d|| | f|| , exclude=)c Cs|jdt}|jd}|jdtttf}t|tsZt|}nt|dkr|dkrt|dt r|d}nt|dkr|dk rt j j | rt |dd|d|}n@|dk rt |j|d|d|}ntd|x|D]\}}|jjd|||j|}|j|}|j|s/t j j|r|j||r|jjd |q|jjd ||t j||q/q/WdS( NR[R)R@iiR\smust supply dest to %ss"symlink.sname=%s; symlink.dname=%ss uptodate: %sssymlink(%s, %s)(R+R0R.RRCRDRER]RXRR*RHRLR R-RR1R^R3RGRIRaR2tsymlink( R4R[R)RRRdReRfRgRh((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6s,1 1 $(N(R7R8R9R)RXR[R@R6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR|s cBseZdZdZRS(sqGenerate a 'tar' archive file. Constructure arguments: Tar(*files, name=None, root=os.curdir, exclude=(defaults).cCsddl}y|j|j|d}Wn!tk rNtd|ndXxV|D]N}|j|tjd}|jj d|||j |j||qVW|j dS(s%Add a list of files to the container.iNsw:gzsno such file or directory: %sRxstar.add(%s, %s)( ttarfileRR3tIOErrort ValueErrorRR*tsepR1R^taddR(R4R=R?RSRttfileR5RH((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRPs   (R7R8R9RP(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRscBs5eZdZdZdZeZdZdZ RS(sReplace tokens found in tokenmap with their associated values in each file. constructor arguments: Tokenize(*files, dest=None, tokenmap=VariableSet())cCsdS(sTo be overridden.N((R4((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pytupdate_tokenmapscsjjdt}t|ts0tdnjddl}|fd}d}djg|D]}||^qn}|jd||j}j j d t |j j d }t|d jd t d t} xr| D]j\} } tj| dj} |j|| } | | krtj| dj| qqWdS(s>Replace tokens found in tokenmap with their associated values.ttokenmaps'tokenmap must be a VariableSet instanceiNcsPjjd|jd|j|jd}|dk rLt|pOdS(sReplace.sfound %siRxN(R1R^tgrouptgetRXR0(RGttmaptresult(R4(sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyt repltokenscSsL|jddjddjddjddjd d jd d S( sQuote special characters.s\s\\R s\.t$s\$t(s\(t)s\)t|s\|(R(tstring((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pytquotesR%s(%s)s patt = %sR)R\R[t iteratorclasstrtR(R+R REt TypeErrorRtreR3Rt MULTILINER1R^R0tpatternR.R R RRtsubR(R4RR+R!R'tktpattttokensR)RReRft realcontentstalteredcontents((R4sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6s&   "  (N( R7R8R9R)RXR[R RRR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR s   cBs#eZdZdZdZdZRS(sWCreate file if it didn't exist already. constructor arguments: Touch(*files, dest=None)cCsddlm}|jdt}xz|j|jdD]`}|j|td|dk rr|||}n|jj d|t |j |dq;WdS(Ni(tnormjoinR[R)s touch(%s)ta( thelperR4R+R0R-R.R/RXR1R2RR3(R4R4R[R5((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6s (N(R7R8R9R)RXR[R6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR!scBs#eZdZdZdZdZRS(sPCall Python unit tests found. constructor arguments: Unittest(*modules, path=())cCstjjt}tjj|d}tjj|sKt|dntit|j dd6|jd6t |j j t jd6t |j j t jd6}ttj||fdtdd id d 6d S( spCall the 'unit-test.py' script in the package directory with serialized parameters as the first argument string.s unit-test.pys&unable to find unittest helper programtmodulesRHtverbosetquietR tbasedirRs /dev/nulltCOVERAGE_PROCESS_STARTN(R*RHRt__file__R3RlRtreprRCR.RR1R R{RRRRt executableR (R4tbdirtsfiletparams((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6s " (((R7R8R9R7RHR6(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR"st UncontainercBsJeZdZdZdZdZdZdZdZ e dZ RS(s Super-class for Untar and Unzip.cCs|jdtdt}|jdt}|j|td|jd}y|j|}Wn!tk rtd|n0X|j||}|j ||||j dS(s#Extract members from the container.R=R>R?R)sno such file or directory: %sN( R+R0RBR/R.tget_fileRRtretrieve_memberstextract_membersR(R4R=R?R)tcontfileR((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR6s cCsdS(sTo be overridden.N(RX(R4R=((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRC#scCsdS(sTo be overridden.N((R4RFRR?((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRE'scCsdS(sTo be overridden.N(RX(RFR)((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRD*sN(( R7R8R9RXR=R?R)R6RCREt staticmethodRD(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRBs   cBs/eZdZdZedZdZRS(sJExtract a 'tar' archive file. Untar(*files, name=, root=None)cCs%ddl}|j|j|dS(sOpen the container.iNsr:gz(RRR3(R4R5R((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRC3s cCsg}t|}xg|jD]Y}|jjtjsx|jjtjrRq| sh|j|kr|j|qqW|S(s(Retrieve the members from the container.(RCt getmembersR=RR*RtpardirRK(RFR)Rtmember((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRD8s cCsDx=|D]5}|jjd|j|j|d|p8dqWdS(s#Extract members from the container.star.extract(%s)RHRxN(R1R^R=textract(R4RFRR?tfileinfo((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyREEs (R7R8R9RCRGRDRE(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR#0s  cBs/eZdZdZedZdZRS(sJExtract a 'zip' archive file. Unzip(*files, name=, root=None)cCs&ddlm}||j|dS(sOpen the container.i(tZipFileR(tzipfileRMR3(R4R5RM((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRCOscCswg}t|}x^|jD]P}|jtjso|jtjrLq| s_||kr|j|qqW|S(s(Retrieve the members from the container.(RCtnamelistRR*RRIRK(RFR)RRJ((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRDTs $cCs{xt|D]l}tjj||}tjtjj||jjd|t|d}|j |j |qWdS(s#Extract members from the container.szip.extract(%s)twbN( R*RHR3RRRR1R^RRR(R4RFRR?RJRftdfile((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRE`s  (R7R8R9RCRGRDRE(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR$Ls  cBseZdZdZRS(sdGenerate a 'zip' archive file. Zip(*files, name=(containername), root=os.curdir, exclude=(defaults).cCsddlm}y||j|d}Wn!tk rOtd|n[XxM|D]E}|j|tjd}|jj d|||j ||qWW|j dS(sAdd the files to the container.i(RMRsno such file or directory: %sRxszip.add(%s, %s)N( RNRMR3RRRR*RR1R^RR(R4R=R?RSRMtzfileR5RH((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRPms  (R7R8R9RP(((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyR%jscBsYeZdZdZedZdZdZdZdZ edZ RS(spGenerate an egg file for Python deployments. Egg(*files, name=, root=os.curdir, exclude=(defaults))cCstjjtjj|d}|jd}|dkrJ|| }ntjj|d}ytj|Wntk rnX|j||||j ||d|j ||d|j ||||j |||dS(sGenerate a manifest structure.it-isEGG-INFOsdependency_links.txtszip-safeN( R*RHtsplitextR`RR3RRmtdo_file_pkginfot do_file_dummytdo_file_top_leveltdo_file_sources(R4R=R?RSR5tpteggdir((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyROs"   cCs ||kr|j|ndS(N(RK(tseqRH((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pytadd_paths cCsBtjj||}t|djtj|j||dS(NR(R*RHR3RRRR\(R4trootdirRSR5tfn((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRVscCsFtjj|d}t|dj|tj|j||dS(Ns top_level.txtR(R*RHR3RRRR\(R4R]RSR=R^((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRWsc Cstjj|d}t|dj}x`tg|D]}|j|tjd^q4D],}|jdsZ|j|tj qZqZWWdQX|j ||dS(Ns SOURCES.txtRRxsEGG-INFO( R*RHR3RtsortedRRRRRR\(R4R]RSR?R^RVR((sA/home/arcege/src/pyerector/regression/parallel/pyerector/tasks.pyRXs 9!c Cs tjjtjj|drB|jtjj|d}ntddidd6}x_t|D]Q}|dkrdjg||D]}d|^q||sZ    ( 1#&'%4+//"+"."PKJ#EX{Q''pyerector/__init__.pyc \USc<@sdZddlmZmZddlmZmZmZmZddlm Z ddl m Z ddl m Z mZddlmZddlmZmZmZmZdd lmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-dd l.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;dd l<m=Z=m>Z>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFdd lGmHZHmIZImJZJmKZKd ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;d<d=d>d?d@dAdBdCdDdEdFdGdHg<ZLe jMdIS(Js Package: pyerector A framework to build products based on tasks and targets. The thought is, distutils/setuptools is to Maven as PyErector is to Ant. The former requires that you shoehorn project to fit into their model. PyErector is a toolkit/framework, like Make and Ant, giving you routines to set up what you want/need to do. It is geared more toward Python usage, but most tasks are generic enough for any system. The developer would create a driver file, commonly named 'pyerect', that would import the 'pyerector' module and call the PyErector routine. The smallest driver file would be: from pyerector import * PyErector() The PyErector main routine will scan the command-line arguments for target names, if none are found, then it will call the default target. Target references on the command-line are the same as the class, but with the first character lowercase. The command-line argument 'default' would refer to the 'Default' Target class. Command-line arguments must reference targets, not other objects, or variable assignments. Instead of referencing a Target, a command-line argument may also assign a string value to a variable, for example: "developer.name=Michael". After command-line arguments are processed, PyErector() will verify the classes are properly constructed and then call the target references in turn. The package is broken into three primary types of objects: standard targets, standard tasks and the API. All standard targets could be subclassed, but would not be, in general. Instead the class members would be modified in the driver file: InitDirs.files = ('build',) Compile.tasks = (Copy('foobar.txt', dest='build'),) As can be seen with the standard Copy task above, the most common usage is to create instances within a target's class members, like 'tasks'. The API allows the developer to extend or change what is being performed during the build. For example, in the product's "pyerect" program, there is a Registration task, which changes the SubPyErector task: class Regression(SubPyErector): wdir = 'regression' proddir = os.path.realpath(os.path.join('build', distfile)) if 'PYTHONPATH' in os.environ: paths = os.environ['PYTHONPATH'].split(os.pathsep) paths.append(proddir) env = {'PYTHONPATH': os.pathsep.join(paths)} del paths else: env = {'PYTHONPATH': proddir } This does not change the task's flow, but it does set up certain defaults which does change the behavior. i(tnormjoint Exclusions(tdisplaytwarntverbosetdebug(tInitialization(tVersion(t PyErectortpymain(tVCS(tTargettTaskt SequentialtParallel(tChmodtCopytCopyTreetEchotEggtHashGentJavatMkdirt PyCompiletRemovetScptShebangtSpawnt SubPyErectortSshtSymlinktTartTokenizetTouchtUnittesttUntartUnziptZip( tAlltBuildtCleantCompiletDefaulttDisttHelptInittInitDirstInitVCSt PackagingtTesttTestonly( tFileSettStaticIteratort FileIteratortFileListtDirListt FileMappertBasenameMappert MergeMappertIdentityMappertUptodate(t FileVariabletVtVariablet VariableSetRRRR RR R R RR>R=R?R@R RRRRRRRRRRRRRRRRRR R!R"R#R$R%R&R'R(R)R*R+R,R-R.R/R0R1R2R3R4R5R6R7R8R9R;R:R<N(Nt__doc__thelperRRRRRRtexecuteRtversionRtmainRR tvcsR tbaseR R R RttasksRRRRRRRRRRRRRRRRRR R!R"R#R$R%ttargetsR&R'R(R)R*R+R,R-R.R/R0R1R2t iteratorsR3R4R5R6R7R8R9R:R;R<t variablesR=R>R?R@t__all__tstart(((sC/home/arcege/src/pyerector/regression/minimal/pyerector/__init__.pyt:s""XF" PKcDS S pyerector/iterators.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Define various Iterator subclasses, for use in tasks to iterate over files and directories. """ import fnmatch import glob import os import re import sys from .helper import normjoin from .base import Iterator, Mapper from .variables import V try: reduce except NameError: try: from functools import reduce except NameError: # we shouldn't get here, but... def reduce(function, iterable, initializer=None): """Iterate over a sequence, calling function on each successive item, accumulating the result.""" accum = initializer for item in iterable: if accum is None: accum = item else: accum = function(accum, item) else: if accum is None and initializer is not None: return initializer return accum __all__ = [ 'FileSet', 'StaticIterator', 'FileIterator', 'FileList', 'DirList', 'FileMapper', 'BasenameMapper', 'MergeMapper', 'IdentityMapper', 'Uptodate', ] class FileIterator(Iterator): """File-based subclass of Iterator. Default parameters: pattern=None, noglob=False, recurse=False, fileonly=True, exclude=().""" def adjust(self, candidate): basedir = V['basedir'] noglob = self.get_kwarg('noglob', bool) if isinstance(candidate, Iterator): return list(candidate) elif not isinstance(candidate, str): raise TypeError('%s is not a string' % repr(candidate)) if noglob or not self.checkglobpatt(candidate): return super(FileIterator, self).adjust(candidate) else: items = glob.glob(os.path.join(basedir, candidate)) return [n.replace(os.path.join(basedir, ''), '') for n in items] def post_process_candidate(self, candidate): basedir = V['basedir'] recurse = self.get_kwarg('recurse', bool) fileonly = self.get_kwarg('fileonly', bool) if recurse and os.path.isdir(os.path.join(basedir, candidate)): fnames = [os.path.join(candidate, fn) for fn in os.listdir(os.path.join(basedir, candidate)) ] self._prepend(fnames) if fileonly: candidate = FileIterator.next(self) return candidate def check_candidate(self, candidate): basedir = V['basedir'] recurse = self.get_kwarg('recurse', bool) pattern = self.get_kwarg('pattern', str) if recurse and os.path.isdir(os.path.join(basedir, candidate)): return True elif not pattern or \ fnmatch.fnmatchcase(os.path.basename(candidate), pattern): return True else: return False @staticmethod def checkglobpatt(string): """Check if the string has any glob characters.""" try: from glob import match_check except ImportError: return re.search('[[*?]', string) is not None else: return match_check.search(string) is not None # a helper classes to handle file/directory lists better class StaticIterator(Iterator): """By default, noglob == True.""" noglob = True FileList = FileIterator FileSet = FileIterator class DirList(FileIterator): """By default, recurse and return both directory and file pathnames. Default params: recurse=True, fileonly=False.""" recurse = True fileonly = False class FileMapper(Mapper, FileIterator): """Maps source files to destination files, using a base path, destdir. The mapper member is either a string or callable that will adjust the basename; if None, then there is no adjustment. The map() method can also adjust the basename. An example of using a mapper would be: FileMapper( FileIterator('src', pattern='*.py'), destdir='build', mapper=lambda n: n+'c' ) This would map each py file in src to a pyc file in build: [('src/base.py', 'build/base.pyc'), ('src/main.py', 'build/main.pyc')] """ def checkpair(self, src, dst): sfile = self.join(src) dfile = self.join(dst) """Return True if destination is newer than source.""" if self.exclusion.match(os.path.basename(sfile)): return True try: srctime = round(os.path.getmtime(sfile), 4) except OSError: raise ValueError('no source:', src) try: dsttime = round(os.path.getmtime(dfile), 4) except OSError: self.logger.debug('%s not found', dst) return False self.logger.debug('%s(%0.4f) %s %s(%0.4f)', src, srctime, (srctime > dsttime and '>' or '<='), dst, dsttime) return srctime <= dsttime def checktree(self, src, dst): """Recursively check the files in both src and dst for their modification times, using checkpair above. """ dirs = [os.curdir] while dirs: cdir = dirs.pop(0) for fname in os.listdir(normjoin(src, cdir)): sname = normjoin(src, cdir, fname) dname = normjoin(dst, cdir, fname) if self.exclusion.match(fname): continue if os.path.isdir(sname): self.logger.debug('adding %s to fifo', normjoin(cdir, fname)) dirs.append(normjoin(cdir, fname)) else: self.logger.debug('checking %s with %s', sname, dname) result = self.checkpair(sname, dname) if not result: return result else: return True class BasenameMapper(FileMapper): """Remove the last file extension including the dot.""" def map(self, item): return os.path.splitext(item)[0] class MergeMapper(FileMapper): """Take only the base name, not subpaths.""" def map(self, item): return os.path.basename(item) class IdentityMapper(FileMapper): """Map to just the destdir value.""" def map(self, item): return '' # just use destdir sequencetypes = (list, tuple, Iterator, type(iter([])), type((None for i in ()))) class Uptodate(FileMapper): """For backward compatibility, should be using FileMapper or its other subclasses.""" # backward compatible interface sources = () destinations = () def __call__(self, *args): klsname = self.__class__.__name__ self.logger.debug('%s.__call__(*%s)', klsname, args) sources = self.get_kwarg('sources', sequencetypes) if isinstance(sources, Iterator): srcs = sources else: srcs = FileIterator(sources) destinations = self.get_kwarg('destinations', sequencetypes) if isinstance(destinations, Iterator): dsts = destinations else: dsts = FileIterator(destinations) files = self.get_args('files') if not files and (not srcs or not dsts): self.logger.debug('%s *> %s', klsname, False) return False elif srcs and dsts: def get_times(lst, slf=self): """Return the times as a list.""" times = [] for fname in lst: try: times.append(os.path.getmtime(slf.join(fname))) except OSError: pass return times maxval = float('inf') latest_src = reduce(max, get_times(srcs), 0) earliest_dst = reduce(min, get_times(dsts), maxval) if earliest_dst == maxval: # empty list case self.logger.debug('%s /> %s', klsname, False) return False result = round(earliest_dst, 4) >= round(latest_src, 4) self.logger.debug('%s => %s', klsname, result and 'False' or 'True') return result else: raise RuntimeError('call uptodate()') #return self.uptodate() PK!Ec pyerector/exception.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Exceptions and exception frame handling for displaying traceback objects "appropriately" (with the class name). """ import sys import linecache class Abort(Exception): """Roll back to the PyErector call.""" class Error(Exception): """Error within pyerector to raise.""" def __str__(self): return ': '.join([str(a) for a in self.args]) def __format__(self, format_spec): value = format(str(self), format_spec) if isinstance(format_spec, unicode): return unicode(value) else: return value # basically reproducing the extract_tb call but with conditionally # prepending the class name to the function/method name if the class # is a subclass of Target or Task (handled externally) def extract_tb(tb, limit=None, last_object=[None], valid_classes=()): """Return a list of the traceback objects, just as traceback.extract_tb(), but the function/method name would be augmented with the class name if a subclass of one of the valid_class entries.""" if limit is None: if hasattr(sys, 'tracebacklimit'): limit = getattr(sys, 'tracebacklimit') flist = [] n = 0 fself = None while tb is not None and (limit is None or n < limit): frame = tb.tb_frame lineno = frame.f_lineno code = frame.f_code filename = code.co_filename name = code.co_name linecache.checkcache(filename) line = linecache.getline(filename, lineno, frame.f_globals) if line: line = line.strip() else: line = None if 'self' in frame.f_locals: fself = frame.f_locals['self'] if not isinstance(fself, valid_classes): fself = None if fself is not None and last_object[0] != fself: last_object[0] = fself elif fself is None and last_object[0] is not None: fself = last_object[0] elif last_object[0] is not None: fself = last_object[0] del frame # delete the frame reference so there is circular reference if fself: name = '%s.%s' % (fself.__class__.__name__, name) flist.append((filename, lineno, name, line)) tb = tb.tb_next n += 1 del flist[:1] # first should be handle_error, unfortunately return flist PKJ#E B9pb&b&pyerector/iterators.pyc ASc @sdZddlZddlZddlZddlZddlZddlmZddlm Z m Z ddl m Z ye WnEek ryddlm Z Wqek rddZ qXnXd d d d d dddddg Zd e fdYZd e fdYZeZeZd efdYZde efdYZdefdYZdefdYZdefdYZeee eegeddDfZdefdYZ dS(s\Define various Iterator subclasses, for use in tasks to iterate over files and directories. iNi(tnormjoin(tIteratortMapper(tV(treducecCs[|}xN|D]*}|dkr(|}q |||}q W|dkrW|dk rW|S|S(s[Iterate over a sequence, calling function on each successive item, accumulating the result.N(tNone(tfunctiontiterablet initializertaccumtitem((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyRs   tFileSettStaticIteratort FileIteratortFileListtDirListt FileMappertBasenameMappert MergeMappertIdentityMappertUptodatecBs8eZdZdZdZdZedZRS(szFile-based subclass of Iterator. Default parameters: pattern=None, noglob=False, recurse=False, fileonly=True, exclude=().cCstd}|jdt}t|tr5t|St|ts]tdt|n|ss|j | rt t |j |St j tjj||}g|D]'}|jtjj|dd^qSdS(Ntbasedirtnoglobs%s is not a stringt(Rt get_kwargtboolt isinstanceRtlisttstrt TypeErrortreprt checkglobpatttsuperR tadjusttglobtostpathtjointreplace(tselft candidateRRtitemstn((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyR!1s  cCstd}|jdt}|jdt}|rtjjtjj||rgtjtjj||D]}tjj||^qt}|j||rt j |}qn|S(NRtrecursetfileonly( RRRR#R$tisdirR%tlistdirt_prependR tnext(R'R(RR+R,tfntfnames((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pytpost_process_candidate>s '= cCstd}|jdt}|jdt}|rYtjjtjj||rYtS| s~t j tjj ||rtSt SdS(NRR+tpattern( RRRRR#R$R-R%tTruetfnmatcht fnmatchcasetbasenametFalse(R'R(RR+R4((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pytcheck_candidateLs 'cCsRyddlm}Wn$tk r:tjd|dk SX|j|dk SdS(s,Check if the string has any glob characters.i(t match_checks[[*?]N(R"R;t ImportErrortretsearchR(tstringR;((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyRXs  (t__name__t __module__t__doc__R!R3R:t staticmethodR(((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyR -s   cBseZdZeZRS(sBy default, noglob == True.(R@RARBR5R(((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyR dscBseZdZeZeZRS(soBy default, recurse and return both directory and file pathnames. Default params: recurse=True, fileonly=False.(R@RARBR5R+R9R,(((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyRlscBs eZdZdZdZRS(s Maps source files to destination files, using a base path, destdir. The mapper member is either a string or callable that will adjust the basename; if None, then there is no adjustment. The map() method can also adjust the basename. An example of using a mapper would be: FileMapper( FileIterator('src', pattern='*.py'), destdir='build', mapper=lambda n: n+'c' ) This would map each py file in src to a pyc file in build: [('src/base.py', 'build/base.pyc'), ('src/main.py', 'build/main.pyc')] c Cs|j|}|j|}|jjtjj|r@tSyttjj|d}Wn t k rt d|nXyttjj|d}Wn%t k r|j j d|t SX|j j d||||krdpd||||kS(Nis no source:s %s not founds%s(%0.4f) %s %s(%0.4f)t>s<=(R%t exclusiontmatchR#R$R8R5troundtgetmtimetOSErrort ValueErrortloggertdebugR9(R'tsrctdsttsfiletdfiletsrctimetdsttime((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyt checkpairs$    c Cstjg}x|r|jd}xtjt||D]}t|||}t|||}|jj|rq=ntjj|r|j j dt|||j t||q=|j j d|||j ||}|s=|Sq=WqWt SdS(seRecursively check the files in both src and dst for their modification times, using checkpair above. isadding %s to fifoschecking %s with %sN(R#tcurdirtpopR.RRERFR$R-RKRLtappendRSR5( R'RMRNtdirstcdirtfnametsnametdnametresult((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyt checktrees"    (R@RARBRSR](((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyRss  cBseZdZdZRS(s1Remove the last file extension including the dot.cCstjj|dS(Ni(R#R$tsplitext(R'R ((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pytmaps(R@RARBR_(((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyRscBseZdZdZRS(s&Take only the base name, not subpaths.cCstjj|S(N(R#R$R8(R'R ((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyR_s(R@RARBR_(((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyRscBseZdZdZRS(sMap to just the destdir value.cCsdS(NR((R'R ((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyR_s(R@RARBR_(((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyRsccs|] }dVqdS(N(R(t.0ti((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pys scBs#eZdZdZdZdZRS(sOFor backward compatibility, should be using FileMapper or its other subclasses.c Gs|jj}|jjd|||jdt}t|trL|}n t|}|jdt}t|tr|}n t|}|j d}| r| s| r|jjd|t t S|r|r|d}t d} t t ||d} t t||| } | | krF|jjd |t t St| d t| d k} |jjd || rd pd | StddS(Ns%s.__call__(*%s)tsourcest destinationstfiless%s *> %scSsUg}xH|D]@}y&|jtjj|j|Wq tk rLq Xq W|S(sReturn the times as a list.(RVR#R$RHR%RI(tlsttslfttimesRY((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyt get_timess & tinfis%s /> %sis%s => %sR9R5scall uptodate()(t __class__R@RKRLRt sequencetypesRRR tget_argsR9tfloatRtmaxtminRGt RuntimeError( R'targstklsnameRbtsrcsRctdstsRdRhtmaxvalt latest_srct earliest_dstR\((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyt__call__s4        "(((R@RARBRbRcRx(((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyRs((!RBR6R"R#R=tsysthelperRtbaseRRt variablesRRt NameErrort functoolsRt__all__R R RR RRRRRRttuplettypetiterRkR(((sD/home/arcege/src/pyerector/regression/minimal/pyerector/iterators.pyts<         7= (PKcDppyerector/config.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Handle configuration values. The Config class managed "variables" before global variables were introduced. This class has now been deprecated and exists only for backward compatibility. The State class is to give the same functionality as the now deprecated Verbose class. The "noop" and "noTimer" instances are initialized here and turned on in PyErector.arguments(). """ from warnings import warn from .variables import V __all__ = [ 'Config', 'noop', 'noTimer', ] class Config(object): """Deprecated, but still provide the same interface, but to variables.""" def __getattr__(self, variable): warn('Config has been deprecated', DeprecationWarning) return V[variable] def __setattr__(self, variable, value): warn('Config has been deprecated', DeprecationWarning) V[variable] = value class State(object): """Create a mutable boolean object.""" def __init__(self, initial=False): self.state = initial def __bool__(self): return self.state __nonzero__ = __bool__ def on(self): """Change state to True.""" self.state = True def off(self): """Change state to False.""" self.state = False noop = State() # display timing information, changed in pyerector.main.PyErector noTimer = State() PKcDD%~MMpyerector/metaclass.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Need an easy way to map a class name (string) to the class. The metaclass here will use a global registry for the mapping. """ import logging from .register import registry # metaclass for instantiating Initer class IniterMetaClass(type): """Meta class to register the class for easy retrieval later.""" def __init__(cls, class_name, bases, namespace): type.__init__(cls, class_name, bases, namespace) registry[class_name] = cls logging.debug('registering %s', cls) PK E0N]t"8"8pyerector/helper.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Helper routines for the pyerector package. The Exclusions class can be used by Iterators to prevent matching against certain file (like .svn, .hg, .git) or patterns (*.pyc, *~). normjoin runs os.path.normpath(os.path.join(*args), as a convenience routine. The Subcommand class handles spawning processes through subprocess.Popen, but with a bit more backend control (like terminating the process when the object is deleted). Initialize the logging system, including setting up module specific formatters. It does not change the root ('') logger. """ import fnmatch import logging import os from sys import version, exc_info import traceback import warnings from .exception import Error from .execute import get_current_stack, Initialization __all__ = [ 'Exclusions', 'normjoin', 'Subcommand', ] # helper routines def normjoin(*args): """Join and normalize the arguments into a pathname.""" if not args: args = ('',) return os.path.normpath(os.path.join(*args)) class Exclusions(set): """A list of exclusion patterns. The usedefaults argument can take three values: True (default), False and None. A True will augment the set with the 'defaults' values. A False will augment with a set containing vcs_names. And None will not augment any additional values - this is dangerous when used with the Remove task.""" vcs_names = set(('.git', '.hg', '.svn', 'CVS')) cruft_patts = set(('*.pyc', '*~', '.*.swp', '__pycache__')) defaults = vcs_names | cruft_patts def __init__(self, items=(), usedefaults=True): if isinstance(items, Exclusions): if usedefaults != items.usedefaults: usedefaults = items.usedefaults items = set(items) elif not isinstance(items, (str, set, tuple, list, type(None))): raise TypeError('Exclusions: expecting str, set, tuple or list') if isinstance(items, str): # proper casting items = (str,) if items: initialset = set(items) else: initialset = set() self.usedefaults = usedefaults super(Exclusions, self).__init__(initialset) def match(self, string): """Return true if the given string matches one of the patterns in the set.""" # augment the possible matches with the defaults if self.usedefaults == True: matches = self | self.defaults elif self.usedefaults == False: matches = self | self.vcs_names else: matches = self values = [v for v in matches if fnmatch.fnmatchcase(os.path.basename(string), v)] return len(values) > 0 @classmethod def set_defaults(cls, items=(), reset=False): """Change or reset the defaults for all instances.""" if reset and hasattr(cls, 'real_defaults'): cls.defaults = cls.real_defaults del cls.real_defaults return elif reset: return if not hasattr(cls, 'real_defaults'): cls.real_defaults = cls.defaults if not isinstance(items, (set, tuple, list)): raise TypeError('Exclusions: expecting set, tuple or list') cls.defaults = set(items) class Subcommand(object): """Handles some of the subprocess details.""" try: import subprocess except ImportError: subprocess = None raise NotImplementedError("Earlier than Python 2.6 is unsupported") PIPE = subprocess.PIPE def __init__(self, cmd, wdir=os.curdir, env=None, wait=True, stdin=None, stdout=None, stderr=None): if env is None: env = {} assert isinstance(cmd, tuple), "must supply tuple as command" if (len(cmd) == 1 and (isinstance(cmd[0], tuple) or isinstance(cmd, list))): self.cmd = tuple(cmd[0]) else: self.cmd = cmd self.wdir = wdir self.env = env self.proc = None self.infile = stdin self.outfile = stdout self.errfile = stderr self.stdin = self.stdout = self.stderr = None self.returncode = None self.call_subprocess() if wait: self.wait() def __del__(self): import traceback logging.debug('starting %s.__del__()', self.__class__.__name__) self.close() if hasattr(self, 'returncode') and hasattr(self, 'proc') and \ self.returncode is None and self.proc is not None: self.proc.terminate() try: os.kill(self.proc.pid, 0) # check if there except OSError: pass else: self.proc.kill() self.proc.wait() self.proc = None def close(self): """Close the open files.""" if hasattr(self, 'infile') and \ hasattr(getattr(self, 'infile'), 'lower') and \ self.stdin is not None: self.stdin.close() self.stdin = None if hasattr(self, 'outfile') and \ hasattr(getattr(self, 'outfile'), 'lower') and \ self.stdout is not None: self.stdout.close() self.stdout = None if hasattr(self, 'errfile') and \ hasattr(getattr(self, 'errfile'), 'lower') and \ self.stderr is not None: self.stderr.close() self.stderr = None def terminate(self): """Send a SIGTERM signal to the process.""" assert self.proc is not None, 'subprocess not spawned' return self.proc.terminate() def kill(self): """Send a SIGKILL signal to the process.""" assert self.proc is not None, 'subprocess not spawned' return self.proc.kill() def poll(self): """Return True if the process has finished, setting returncode if applicable.""" assert self.proc is not None, 'subprocess not spawned' returncode = self.proc.poll() if returncode is not None: self.returncode = returncode return self.returncode is not None def wait(self): """Wait for the process to complete, and return the exit status.""" assert self.proc is not None, 'subprocess not spawned' if self.stdin: self.stdin.close() self.returncode = self.proc.wait() return self.returncode def call_subprocess(self): """Call subprocess.Popen and handle the I/O.""" from subprocess import Popen realenv = os.environ.copy() realenv.update(self.env) if hasattr(self.infile, 'write'): ifile = self.infile elif self.infile == self.PIPE: ifile = self.PIPE elif hasattr(self.infile, 'lower'): ifile = open(self.infile, 'r') else: ifile = None if hasattr(self.outfile, 'read'): ofile = self.outfile elif self.outfile == self.PIPE: ofile = self.PIPE elif hasattr(self.outfile, 'lower'): ofile = open(self.outfile, 'w') else: ofile = None if self.errfile == self.outfile: efile = ofile elif hasattr(self.errfile, 'read'): efile = self.errfile elif self.errfile == self.PIPE: efile = self.PIPE elif hasattr(self.errfile, 'lower'): efile = open(self.errfile, 'w') else: efile = None (self.stdin, self.stdout, self.stderr) = (ifile, ofile, efile) shellval = not isinstance(self.cmd, tuple) logging.debug('Popen(%s, shell=%s, cwd=%s, stdin=%s, stdout=%s,' 'stderr=%s, bufsize=0, env=%s)', self.cmd, shellval, self.wdir, ifile, ofile, efile, self.env) try: proc = Popen(self.cmd, shell=shellval, cwd=self.wdir, stdin=ifile, stdout=ofile, stderr=efile, bufsize=0, env=realenv) except (IOError, OSError): exc = exc_info()[1] if exc.args[0] == 2: # ENOENT raise Error('ENOENT', 'Program not found') else: raise self.proc = proc if ifile == self.PIPE: self.stdin = proc.stdin if ofile == self.PIPE: self.stdout = proc.stdout if efile == self.PIPE: self.stderr = proc.stderr class Timer(object): """Keep track of how long a section of code takes.""" def __init__(self): self.starttime = None self.duration = None def __repr__(self): name = self.__class__.__name__ if self.starttime is not None: return '<%s started>' % name elif self.duration is not None: return '<%s duration: %0.3f>' % (name, self.duration) else: return '<%s unstarted>' % name @staticmethod def now(): """Return the current time.""" from time import time return time() def start(self): """Start the timer.""" if self.starttime is not None: raise RuntimeError('cannot start more than once without stopping') self.starttime = self.now() def stop(self): """Stop the timer and set the duration.""" if self.starttime is None: raise RuntimeError('cannot stop without starting') self.duration = self.now() - self.starttime self.starttime = None def __enter__(self): self.start() def __exit__(self, etype, evalue, etb): self.stop() def __float__(self): if self.duration is not None: return float(self.duration) return 0.0 def __int__(self): return int(float(self)) class Verbose(object): # deprecated """Deprecated and should no longer be used.""" def __init__(self, state=False, level=logging.INFO): self.level = level @staticmethod def _getlogger(): """Return the default internal logger.""" return logging.getLogger('pyerector.execute') @staticmethod def _getlevelnum(level): """Return the logging level based on its name.""" return logging.getLevelName(level) def __bool__(self): return self._getlogger().isEnabledFor(self.level) def __call__(self, *args): warnings.warn("Use of Verbose is deprecated", DeprecationWarning) msg = ' '.join([str(s) for s in args]) self._getlogger().log(self._getlevelnum(self.level), msg) class LogFormatter(logging.Formatter, object): """Subclass to handle formatting messages from the library, including prepending text in the PYERECTOR_PREFIX envvar and showing thread if not the main threads (MainThread, PyErector).""" log_prefix = None def format(self, record): newrecord = super(LogFormatter, self).format(record) if self.log_prefix is None: try: prefix = os.environ['PYERECTOR_PREFIX'] except (AttributeError, KeyError): prefix = '' self.log_prefix = prefix else: prefix = self.log_prefix if prefix: if isinstance(newrecord, logging.LogRecord): message = '%s: %s' % (prefix, newrecord.message) else: message = '%s: %s' % (prefix, newrecord) elif isinstance(newrecord, logging.LogRecord): message = newrecord.message else: message = newrecord from threading import currentThread if currentThread().name not in ('MainThread', 'PyErector'): name = currentThread().name ident = currentThread().ident if ident is None: message = '(%s) %s' % (name, message) else: message = '(%s[%x]) %s' % (name, ident, message) return message def formatException(self, exc_info): """Return similar to the default, but using exception.extract_tb instead of traceback.extract_tb; the former adds the class being called. """ from .exception import extract_tb etype, exception, tb = exc_info exc = traceback.format_exception_only(etype, exception) stack = traceback.format_list(extract_tb(tb)) return ''.join(stack + exc) class LogExecFormatter(LogFormatter): """Subclass of LogFormatter that shows the pyerector execution stack instead of python's. """ def formatException(self, exc_info): """Extract the pyerector execution stack and return a string with that and the formatted exception. """ stack = get_current_stack() etype, exception = exc_info[:2] lines = stack.extract() + \ traceback.format_exception_only(etype, exception) return ''.join(lines).rstrip() class InitLogging(Initialization): """Initialize the logging, creating two loggers, pyerector and pyerector.execute.""" deflevel = logging.WARNING message = '%(message)s' @staticmethod def setup(name, handlerklass, formatterklass, message=message): """Set up a new logger with appropriate settings.""" fmtr = formatterklass(message) hndr = handlerklass() hndr.setFormatter(fmtr) lgr = logging.getLogger(name) lgr.addHandler(hndr) lgr.propagate = False return lgr def run(self): """Set the default logging level, create a 'DISPLAY' logging level, create two loggers: pyerector and pyerector.execute, and if available, set warnings to be captured by the logging module. """ logging.basicConfig(level=self.deflevel, message=self.message) logging.addLevelName(level=logging.ERROR + 5, levelName='DISPLAY') self.setup('pyerector', logging.StreamHandler, LogFormatter) self.setup('pyerector.execute', logging.StreamHandler, LogExecFormatter) warnings.simplefilter("default", DeprecationWarning) if hasattr(logging, 'captureWarnings'): logging.captureWarnings(True) InitLogging() display = Verbose(None, 'DISPLAY') warn = Verbose(None, 'ERROR') verbose = Verbose(None, 'INFO') debug = Verbose(None, 'DEBUG') PK˦!E* pyerector/tasks.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Define the standard tasks.""" import logging import os import shutil import sys from .exception import Error from .helper import Exclusions, Subcommand from .config import noTimer from .base import Task, Mapper, Iterator from .iterators import BasenameMapper, IdentityMapper, FileMapper, \ FileIterator, StaticIterator from .variables import V, VariableSet # Python 3.x removed the execfile function try: execfile except NameError: from .py3.execfile import execfile __all__ = [ 'Chmod', 'Copy', 'CopyTree', 'Echo', 'Egg', 'HashGen', 'Java', 'Mkdir', 'PyCompile', 'Remove', 'Scp', 'Shebang', 'Spawn', 'Ssh', 'SubPyErector', 'Symlink', 'Tar', 'Tokenize', 'Touch', 'Unittest', 'Untar', 'Unzip', 'Zip', ] class Chmod(Task): """Change file permissions. constructor arguments: Chmod(*files, mode=0666)""" files = () mode = int('666', 8) # gets around Python 2.x vs 3.x octal issue def run(self): """Change the permissions on the files.""" from os import chmod mode = self.get_kwarg('mode', int) for fname in self.get_files(self.get_args('files')): self.asserttype(fname, str, 'files') self.logger.info('chmod(%s, %o)', fname, mode) chmod(self.join(fname), mode) class Container(Task): """An internal task for subclassing standard classes Tar and Zip.""" appname = None from os import curdir as root name = None files = () exclude = Exclusions(usedefaults=False) def run(self): """Gather filenames and put them into the container.""" name = self.get_kwarg('name', str, noNone=True) root = self.join(self.get_kwarg('root', str)) excludes = self.get_kwarg('exclude', (Exclusions, tuple, list)) if not isinstance(excludes, Exclusions): excludes = Exclusions(excludes) self.preop(name, root, excludes) toadd = [] from glob import glob queue = list(self.get_args('files')) while queue: entry = queue[0] del queue[0] for fname in glob(self.join(root, entry)): if excludes.match(fname): # if true, then ignore pass elif os.path.islink(fname) or os.path.isfile(fname): toadd.append(fname) elif os.path.isdir(fname): fnames = [os.path.join(fname, f) for f in os.listdir(fname)] queue.extend(fnames) #verbose('toadd =', toadd) self.manifest(name, root, toadd) self.contain(name, root, toadd) self.postop(name, root, toadd) def preop(self, name, root, excludes): """To be overridden.""" def postop(self, name, root, excludes): """To be overridden.""" def manifest(self, name, root, toadd): """To be overridden.""" def contain(self, name, root, toadd): """To be overridden.""" class Copy(Task): """Copy files to a destination directory. Exclude standard hidden files. constructor arguments: Copy(*files, dest=, exclude=)""" files = () dest = None noglob = False def run(self): """Copy files to a destination directory.""" dest = self.get_kwarg('dest', str, noNone=False) files = self.get_args('files') excludes = self.get_kwarg('exclude', (Exclusions, tuple, list)) if not isinstance(excludes, Exclusions): excludes = Exclusions(excludes) if len(files) == 1 and dest is None and isinstance(files[0], Mapper): fmap = files[0] elif len(files) == 1 and dest is not None and not os.path.isdir(dest): fmap = IdentityMapper(self.get_files(files), destdir=dest) else: fmap = FileMapper(self.get_files(files), destdir=dest) self.logger.debug('Copy.fmap = %s', vars(fmap)) for (sname, dname) in fmap: #self.logger.error( repr( (sname, dname) ) ) srcfile = self.join(sname) dstfile = self.join(dname) if not excludes.match(os.path.basename(sname)): if os.path.isfile(dstfile) and fmap.checkpair(srcfile, dstfile): self.logger.debug('uptodate: %s', dstfile) else: self.logger.info('copy2(%s, %s)', sname, dname) shutil.copy2(srcfile, dstfile) class CopyTree(Task): """Copy directory tree. Exclude standard hidden files. constructor arguments: CopyTree(srcdir=, dstdir=, exclude=)""" srcdir = None dstdir = None exclude = Exclusions() excludes = exclude # deprecated def run(self): """Copy a tree to a destination.""" srcdir = self.get_kwarg('srcdir', str, noNone=True) dstdir = self.get_kwarg('dstdir', str, noNone=True) excludes = self.get_kwarg('exclude', (Exclusions, tuple, list)) if not isinstance(excludes, Exclusions): excludes = Exclusions(excludes) if not os.path.exists(self.join(srcdir)): raise OSError(2, "No such file or directory: " + srcdir) elif not os.path.isdir(self.join(srcdir)): raise OSError(20, "Not a directory: " + srcdir) copy_t = Copy(noglob=True, exclude=excludes) mkdir_t = Mkdir() dirs = [os.curdir] while dirs: tdir = dirs[0] del dirs[0] if not excludes.match(tdir): mkdir_t(self.join(dstdir, tdir)) for fname in os.listdir(self.join(srcdir, tdir)): if not excludes.match(fname): spath = self.join(srcdir, tdir, fname) dpath = self.join(dstdir, tdir, fname) if os.path.isdir(spath): dirs.append(os.path.join(tdir, fname)) else: copy_t(spath, dest=dpath) class Download(Task): """Retrieve contents of URLs. constructor arguments: Download(*urls, destdir=DIR)""" dest = None urls = () def run(self): # this is unfinished, it requires a more generic Mapper class # than one is available now raise NotImplementedError """ import urllib from posixpath import basename from .iterators import BaseIterator urls = self.get_files(self.get_args('urls')) urls = BaseIterators(self.get_args('urls'), noglob=True, fileonly=False) dest = self.get_kwarg('dest', str) if isinstance(urls, str): urls = (urls,) if isinstance(urls, FileMapper): urls = urls elif isinstance(dest, str) and os.path.isdir(dest): urls = BasenameMapper(urls, destdir=dest, mapper=lambda x: x or 'index.html') elif isinstance(dest, str): urls = IdentityMapper(urls, destdir=dest) else: urls = FileMapper(urls, destdir=dest) for url, fname in urls: path = urllib.splithost(urllib.splittype(url)[1])[1] self.logger.debug('Download.path=%s; Download.fname=%s', path, fname) try: urllib.urlretrieve(url, filename=fname) except Exception, e: raise Error(str(self), '%s: unable to retrieve %s' % (e, url)) """ class Echo(Task): """Display a message, arguments are taken as with logger (msg, *args). This is displayed by the logging module, but at the internal 'DISPLAY' level created in pyerector.helper.""" msgs = () def run(self): """Display messages.""" args = self.get_args('msgs') if args: msg, rest = args[0], args[1:] text = msg % rest else: text = '' self.logger.log(logging.getLevelName('DISPLAY'), text) class HashGen(Task): """Generate file(s) containing md5 or sha1 hash string. For example, generates foobar.txt.md5 and foobar.txt.sha1 for the contents of foobar.txt. By default, generates for both md5 and sha1. constructor arguments: HashGen(*files, hashs=('md5', 'sha1'))""" files = () hashs = ('md5', 'sha1') def run(self): """Generate files with checksums inside.""" from hashlib import md5, sha1 files = self.get_files(self.get_args('files')) hashs = self.get_kwarg('hashs', tuple) self.logger.debug('files = %s; hashs = %s', files, hashs) fmaps = [] if 'md5' in hashs: fmaps.append( (md5, FileMapper(files, mapper='%(name)s.md5')) ) if 'sha1' in hashs: fmaps.append( (sha1, FileMapper(files, mapper='%(name)s.sha1')) ) for hashfunc, fmap in fmaps: for sname, dname in fmap: hashval = hashfunc() if (os.path.isfile(sname) and not fmap.checkpair(self.join(sname), self.join(dname))): hashval.update(open(self.join(sname), 'rb').read()) self.logger.debug('writing %s', dname) open(self.join(dname), 'wt').write( hashval.hexdigest() + '\n' ) class Java(Task): """Call a Java routine. constructor arguments: Java(jar=, java_home=<$JAVA_HOME>, classpath=(), properties=[])""" from os import environ try: java_home = environ['JAVA_HOME'] except KeyError: java_home = None classpath = () properties = [] del environ jar = None def __init__(self): Task.__init__(self) if self.java_home and os.path.exists(self.java_home): self.java_prog = os.path.join(self.java_home, 'bin', 'java') elif os.path.exists(os.path.expanduser(os.path.join('~', 'java'))): self.java_prog = os.path.expanduser( os.path.join('~', 'java', 'bin', 'java') ) else: raise Error("no java program to execute") if not os.access(self.java_prog, os.X_OK): raise Error("no java program to execute") def addprop(self, var, val): """Add a Java system property to the list.""" self.properties.append((var, val)) def run(self): """Call java.""" from os import environ from os.path import pathsep jar = self.get_kwarg('jar', str, noNone=True) if self.properties: sysprop = ['-D%s=%s' % x for x in self.properties] else: sysprop = () cmd = (self.java_prog,) + tuple(sysprop) + ('-jar', jar,) + \ tuple([str(s) for s in self.args]) env = environ.copy() if self.classpath: env['CLASSPATH'] = pathsep.join(self.classpath) proc = Subcommand(cmd) if proc.returncode: raise Error(self, '%s failed with returncode %d' % (self.__class__.__name__.lower(), proc.returncode) ) class Mkdir(Task): """Recursively create directories. constructor arguments: Mkdir(*files)""" files = () noglob = True def run(self): """Make directories.""" files = self.get_files(self.get_args('files')) for arg in files: self.asserttype(arg, str, 'files') self.mkdir(self.join(arg)) @classmethod def mkdir(cls, path): """Recursive mkdir.""" from logging import getLogger logger = getLogger('pyerector.execute') if os.path.islink(path) or os.path.isfile(path): logger.info('remove(%s)', path) os.remove(path) cls.mkdir(path) elif not path: pass elif not os.path.isdir(path): cls.mkdir(os.path.dirname(path)) logger.info('mkdir(%s)', path) os.mkdir(path) class PyCompile(Task): """Compile Python source files. constructor arguments: PyCompile(*files, dest=, version='2')""" files = () dest = None version = '2' def run(self): """Compile Python source files.""" import py_compile fileset = self.get_files(self.get_args('files')) if self.version[:1] == sys.version[:1]: # compile inline for fname in fileset: self.logger.debug('py_compile.compile(%s)', fname) py_compile.compile(self.join(fname)) else: if self.version[:1] == '2': cmd = 'python2' elif self.version[:1] == '3': cmd = 'python3' else: cmd = 'python' cmdp = ( cmd, '-c', 'import sys; from py_compile import compile; ' + '[compile(s) for s in sys.argv[1:]]' ) + tuple([self.join(s) for s in fileset]) try: proc = Subcommand(cmdp) except Error: exc = sys.exc_info()[1] if exc.args[0] == 'ENOENT': self.logger.error('%s: Error with %s: %s', self.__class__.__name__, cmd, exc.args[1] ) else: raise else: if proc.returncode != 0: self.logger.info('could not compile files with %s', cmd) class Remove(Task): """Remove a file or directory tree. constructor arguments: Remove(*files)""" files = () noglob = False exclude = None def run(self): """Remove a file or directory tree.""" files = self.get_args('files') noglob = self.get_kwarg('noglob', bool) excludes = self.get_kwarg('exclude', (Exclusions, list, tuple)) if isinstance(files, Iterator): pass elif len(files) == 1 and isinstance(files, Iterator): files = files[0] elif isinstance(files, (tuple, list)): files = FileIterator(*tuple(files), noglob=noglob, exclude=excludes) for name in files: self.asserttype(name, str, 'files') fname = self.join(name) if os.path.isfile(fname) or os.path.islink(fname): self.logger.info('remove(%s)', fname) os.remove(fname) elif os.path.isdir(fname): self.logger.info('rmtree(%s)', fname) shutil.rmtree(fname) class Shebang(Copy): """Replace the shebang string with a specific pathname. constructor arguments: Shebang(*files, dest=, token='#!', program=)""" files = () dest = None token = '#!' program = None def run(self): """Replace the shebang string with a specific pathname.""" self.logger.info('starting Shebang') program = self.get_kwarg('program', str, noNone=True) srcs = self.get_files(self.get_args('files')) dest = self.get_kwarg('dest', str) try: from io import BytesIO as StringIO except ImportError: from StringIO import StringIO for fname in srcs: infname = self.join(fname) head = infname.replace(fname, '') if dest is None: outfname = infname else: outfname = self.join( dest, fname.replace(head, '') ) inf = open(self.join(fname), 'r') outf = StringIO() first = inf.readline() if first.startswith(self.token): if ' ' in first: wsp = first.find(' ') else: wsp = first.find(os.linesep) first = first.replace(first[len(self.token):wsp], program) outf.write(first) else: outf.write(first) shutil.copyfileobj(inf, outf) inf.close() outf.seek(0) inf = open(outfname, 'w') shutil.copyfileobj(outf, inf) class Spawn(Task): """Spawn a command. constructor arguments: Spawn(*cmd, infile=None, outfile=None, errfile=None, env={})""" cmd = () infile = None outfile = None errfile = None env = {} def run(self): """Spawn a command.""" infile = self.get_kwarg('infile', str) outfile = self.get_kwarg('outfile', str) errfile = self.get_kwarg('errfile', str) infile = infile and self.join(infile) or None outfile = outfile and self.join(outfile) or None errfile = errfile and self.join(errfile) or None env = self.get_kwarg('env', dict) cmd = self.get_args('cmd') proc = Subcommand(cmd, env=env, stdin=infile, stdout=outfile, stderr=errfile, ) if proc.returncode < 0: raise Error('Subcommand', '%s signal %d raised' % (str(self), abs(proc.returncode))) elif proc.returncode > 0: raise Error('Subcommand', '%s returned error = %d' % (str(self), proc.returncode)) class SshEngine(Task): """Superclass for Scp and Ssh since there is the same basic logic.""" _cmdprog = None host = None user = None identfile = None @property def remhost(self): ruser = '' host = self.get_kwarg('host', str, noNone=True) user = self.get_kwarg('user', str) if '@' in host: ruser, host = host.split('@', 1) if user: return '%s@%s' % (user, host) elif not user and ruser: return '%s@%s' % (ruser, host) else: return host def gencmd(self): idfile = self.get_kwarg('identfile', str) if idfile: identfile = ('-i', idfile) else: identfile = () return ( self._cmdprog, '-o', 'BatchMode=yes', '-o', 'ConnectTimeout=10', '-o', 'ForwardAgent=no', '-o', 'ForwardX11=no', '-o', 'GSSAPIAuthentication=no', '-o', 'LogLevel=ERROR', '-o', 'PasswordAuthentication=no', '-o', 'StrictHostkeyChecking=no', ) + identfile def _run(self, cmd): self.logger.debug('ssh.cmd = %s', cmd) proc = Subcommand(cmd, stdout=Subcommand.PIPE, stderr=Subcommand.PIPE, wait=True) if proc.returncode: raise Error(str(self), proc.stderr.read().rstrip()) else: return proc.stdout.read().rstrip() class Scp(SshEngine): """Spawn an scp command. constructor arguments: Scp(*files, dest=, host=, user=, identfile=, recurse=bool, down=bool)""" _cmdprog = 'scp' files = () dest = None recurse = False down=True def remfile(self, filename): return '%s:%s' % (self.remhost, filename) def lclfile(self, filename): return self.join(filename) def run(self): recurse = self.get_kwarg('recurse', bool) dest = self.get_kwarg('dest', str, noNone=True) down = self.get_kwarg('down', bool) files = self.get_args('files') if down: left, right = self.remfile, self.lclfile else: left, right = self.lclfile, self.remfile cmd = self.gencmd() if recurse: cmd += ('-r',) filelst = () for fname in files: filelst += (left(fname),) filelst += (right(dest),) self.logger.info('%s%s', self, filelst) self._run(cmd + filelst) class Ssh(SshEngine): """Spawn an ssh command and display the response in verbose mode. constructor arguments: Ssh(*cmd, host=, user=, identfile=) """ _cmdprog = 'ssh' cmd = () def run(self): usercmd = self.get_args('cmd') cmd = self.gencmd() + ('-T', self.remhost) + usercmd self.logger.info('%s%s', self, usercmd) response = self._run(cmd) #self.logger.info('Output from %s\n%s', usercmd, response) self.logger.warning('\t' + response.rstrip().replace('\n', '\n\t')) class SubPyErector(Task): """Call a PyErector program in a different directory. constructor arguments: SubPyErector(*targets, wdir=None, prog='pyerect', env={}) Adds PYERECTOR_PREFIX environment variable.""" targets = () prog = 'pyerect' wdir = None env = {} def run(self): """Call a PyErector program in a different directory.""" targets = self.get_args('targets') prog = self.get_kwarg('prog', str) # we explicitly add './' to prevent searching PATH options = [] logger = logging.getLogger('pyerector') if logger.isEnabledFor(logging.DEBUG): options.append('--DEBUG') elif logger.isEnabledFor(logging.INFO): options.append('--verbose') elif logger.isEnabledFor(logging.ERROR): options.append('--quiet') if noTimer: options.append('--timer') cmd = (os.path.join('.', prog),) + tuple(options) + tuple(targets) env = self.get_kwarg('env', dict) wdir = self.get_kwarg('wdir', str) from os import environ evname = 'PYERECTOR_PREFIX' nevname = os.path.basename(wdir) if evname in environ and environ[evname]: env[evname] = '%s: %s' % (environ[evname], nevname) else: env[evname] = nevname rc = Subcommand(cmd, wdir=wdir, env=env, wait=True) if rc.returncode < 0: raise Error('SubPyErector', '%s signal %d raised' % (str(self), abs(rc.returncode))) elif rc.returncode > 0: raise Error('SubPyErector', '%s returned error = %d' % (str(self), rc.returncode)) class Symlink(Task): """Generate a symbolic link. constructor arguments: Symlink(*files, dest=, exclude=)""" files = () dest = None exclude = None def run(self): dest = self.get_kwarg('dest', str) files = self.get_args('files') excludes = self.get_kwarg('exclude', (Exclusions, tuple, list)) if not isinstance(excludes, Exclusions): excludes = Exclusions(excludes) if len(files) == 1 and dest is None and isinstance(files[0], Mapper): fmap = files[0] elif len(files) == 1 and dest is not None and not os.path.isdir(dest): fmap = FileMapper(files[0], destdir=dest, exclude=excludes) elif dest is not None: fmap = FileMapper(self.get_files(files), destdir=dest, exclude=excludes) else: raise Error('must supply dest to %s' % self) for (sname, dname) in fmap: self.logger.debug('symlink.sname=%s; symlink.dname=%s', sname, dname) srcfile = self.join(sname) dstfile = self.join(dname) if not excludes.match(sname): if os.path.islink(srcfile) and fmap.checkpair(dstfile, srcfile): self.logger.debug('uptodate: %s', dstfile) else: self.logger.info('symlink(%s, %s)', dname, sname) os.symlink(dstfile, srcfile) class Tar(Container): """Generate a 'tar' archive file. Constructure arguments: Tar(*files, name=None, root=os.curdir, exclude=(defaults).""" def contain(self, name, root, toadd): """Add a list of files to the container.""" import tarfile try: tfile = tarfile.open(self.join(name), 'w:gz') except IOError: raise ValueError('no such file or directory: %s' % name) else: for fname in toadd: path = fname.replace( root + os.sep, '' ) self.logger.debug('tar.add(%s, %s)', fname, path) tfile.add(self.join(fname), path) tfile.close() class Tokenize(Task): """Replace tokens found in tokenmap with their associated values in each file. constructor arguments: Tokenize(*files, dest=None, tokenmap=VariableSet())""" files = () dest = None tokenmap = VariableSet() def update_tokenmap(self): """To be overridden.""" def run(self): """Replace tokens found in tokenmap with their associated values.""" tokenmap = self.get_kwarg('tokenmap', VariableSet) if not isinstance(tokenmap, VariableSet): raise TypeError('tokenmap must be a VariableSet instance') self.update_tokenmap() import re def repltoken(match, tmap=tokenmap): """Replace.""" self.logger.debug('found %s', match.group(0)) result = tmap.get(match.group(0)) return result is not None and str(result) or '' def quote(string): """Quote special characters.""" return string.replace('\\', r'\\').replace('.', r'\.')\ .replace('$', r'\$').replace('(', r'\(')\ .replace(')', r'\)').replace('|', r'\|') patt = '|'.join( [quote(k) for k in tokenmap] ) tokens = re.compile(r'(%s)' % patt, re.MULTILINE) self.logger.debug('patt = %s', str(tokens.pattern)) files = self.get_args('files') mapper = FileMapper(files, destdir=self.get_kwarg('dest', str), iteratorclass=StaticIterator) for (sname, dname) in mapper: realcontents = open(self.join(sname), 'rt').read() alteredcontents = tokens.sub(repltoken, realcontents) if alteredcontents != realcontents: open(self.join(dname), 'wt').write(alteredcontents) class Touch(Task): """Create file if it didn't exist already. constructor arguments: Touch(*files, dest=None)""" files = () dest = None def run(self): from .helper import normjoin """Create files, unless they already exist.""" dest = self.get_kwarg('dest', str) for fname in self.get_files(self.get_args('files')): self.asserttype(fname, str, 'files') if dest is not None: fname = normjoin(dest, fname) self.logger.info('touch(%s)', fname) open(self.join(fname), 'a') class Unittest(Task): """Call Python unit tests found. constructor arguments: Unittest(*modules, path=())""" modules = () path = () def run(self): """Call the 'unit-test.py' script in the package directory with serialized parameters as the first argument string.""" bdir = os.path.dirname(__file__) sfile = os.path.join(bdir, 'unit-test.py') if not os.path.exists(sfile): raise Error(self, 'unable to find unittest helper program') # create a parameter file with a serialized set of the arguments params = repr({ 'modules': tuple(self.get_args('modules')), 'path': self.path, 'verbose': bool(self.logger.isEnabledFor(logging.INFO)), 'quiet': bool(self.logger.isEnabledFor(logging.ERROR)), }) # call python Subcommand((sys.executable, sfile, params), wdir=V['basedir'], env={'COVERAGE_PROCESS_START': '/dev/null'}) class Uncontainer(Task): """Super-class for Untar and Unzip.""" name = None root = None files = () def run(self): """Extract members from the container.""" name = self.get_kwarg('name', str, noNone=True) root = self.get_kwarg('root', str) self.asserttype(root, str, 'root') files = self.get_args('files') try: contfile = self.get_file(name) except IOError: raise ValueError('no such file or directory: %s' % name) else: fileset = self.retrieve_members(contfile, files) self.extract_members(contfile, fileset, root) contfile.close() def get_file(self, name): """To be overridden.""" return None def extract_members(self, contfile, fileset, root): """To be overridden.""" @staticmethod def retrieve_members(contfile, files): """To be overridden.""" return None class Untar(Uncontainer): """Extract a 'tar' archive file. Untar(*files, name=, root=None)""" def get_file(self, fname): """Open the container.""" import tarfile return tarfile.open(self.join(fname), 'r:gz') @staticmethod def retrieve_members(contfile, files): """Retrieve the members from the container.""" fileset = [] files = tuple(files) # needed for contents test for member in contfile.getmembers(): if (member.name.startswith(os.sep) or member.name.startswith(os.pardir)): pass elif not files or member.name in files: fileset.append(member) return fileset def extract_members(self, contfile, fileset, root): """Extract members from the container.""" for fileinfo in fileset: self.logger.debug('tar.extract(%s)', fileinfo.name) contfile.extract(fileinfo, path=(root or "")) class Unzip(Uncontainer): """Extract a 'zip' archive file. Unzip(*files, name=, root=None)""" def get_file(self, fname): """Open the container.""" from zipfile import ZipFile return ZipFile(self.join(fname), 'r') @staticmethod def retrieve_members(contfile, files): """Retrieve the members from the container.""" fileset = [] files = tuple(files) # needed for contents test for member in contfile.namelist(): if member.startswith(os.sep) or member.startswith(os.pardir): pass elif not files or member in files: fileset.append(member) return fileset def extract_members(self, contfile, fileset, root): """Extract members from the container.""" for member in fileset: dname = os.path.join(root, member) Mkdir.mkdir(os.path.dirname(dname)) self.logger.debug('zip.extract(%s)', member) dfile = open(dname, 'wb') dfile.write(contfile.read(member)) class Zip(Container): """Generate a 'zip' archive file. Zip(*files, name=(containername), root=os.curdir, exclude=(defaults).""" def contain(self, name, root, toadd): """Add the files to the container.""" from zipfile import ZipFile try: zfile = ZipFile(self.join(name), 'w') except IOError: raise ValueError('no such file or directory: %s' % name) else: for fname in toadd: path = fname.replace( root + os.sep, '' ) self.logger.debug('zip.add(%s, %s)', fname, path) zfile.write(fname, path) zfile.close() class Egg(Zip): """Generate an egg file for Python deployments. Egg(*files, name=, root=os.curdir, exclude=(defaults))""" def manifest(self, name, root, toadd): """Generate a manifest structure.""" fname = os.path.splitext(os.path.basename(name))[0] p = fname.find('-') if p != -1: fname = fname[:p] eggdir = os.path.join(root, 'EGG-INFO') try: os.mkdir(eggdir) except OSError: pass self.do_file_pkginfo(eggdir, toadd, root) self.do_file_dummy(eggdir, toadd, 'dependency_links.txt') self.do_file_dummy(eggdir, toadd, 'zip-safe') self.do_file_top_level(eggdir, toadd, fname) self.do_file_sources(eggdir, toadd, root) @staticmethod def add_path(seq, path): if path not in seq: seq.append(path) def do_file_dummy(self, rootdir, toadd, fname): fn = os.path.join(rootdir, fname) open(fn, 'wt').write(os.linesep) self.add_path(toadd, fn) def do_file_top_level(self, rootdir, toadd, name): fn = os.path.join(rootdir, 'top_level.txt') open(fn, 'wt').write(name + os.linesep) self.add_path(toadd, fn) def do_file_sources(self, rootdir, toadd, root): fn = os.path.join(rootdir, 'SOURCES.txt') with open(fn, 'wt') as f: for s in sorted([s.replace(root+os.sep, '') for s in toadd]): if not s.startswith('EGG-INFO'): f.write(s + os.linesep) self.add_path(toadd, fn) def do_file_pkginfo(self, rootdir, toadd, root): if os.path.exists(os.path.join(root, 'setup.py')): setupvalue = self.get_setup_py(os.path.join(root, 'setup.py')) else: raise Error('Egg', 'unable to find a setup.py file') pkg_data = { 'classifiers': '', } for key in sorted(setupvalue): if key == 'classifiers': pkg_data[key] = '\n'.join( ['Classifier: %s' % c for c in setupvalue[key]] ) else: pkg_data[key] = setupvalue[key] pkg_info = '''\ Metadata-Version: 1.1 Name: %(name)s Version: %(version)s Summary: %(description)s Home-page: %(url)s Author: %(author)s Author-email: %(author_email)s License: %(license)s Download-URL: %(download_url)s Description: %(long_description)s Platform: UNKNOWN %(classifiers)s ''' % pkg_data fn = os.path.join(rootdir, 'PKG-INFO') open(fn, 'wt').write(pkg_info) self.add_path(toadd, fn) @staticmethod def get_setup_py(filename): """Simulate setup() in a fake distutils and setuptools.""" import imp backups = {} script = ''' def setup(**kwargs): global myvalue myvalue = dict(kwargs) ''' code = compile(script, 'setuptools.py', 'exec') try: for modname in ('setuptools', 'distutils'): if modname in sys.modules: backups[modname] = sys.modules[modname] else: backups[modname] = None mod = sys.modules[modname] = imp.new_module(modname) exec(code, mod.__dict__, mod.__dict__) mod = {'__builtins__': __builtins__, 'myvalue': None} execfile(filename, mod, mod) for modname in ('setuptools', 'distutils'): if sys.modules[modname].myvalue is not None: return sys.modules[modname].myvalue else: return None finally: for modname in backups: if backups[modname] is None: del sys.modules[modname] else: sys.modules[modname] = backups[modname] PKJ#E5? ? pyerector/register.pyc ASc@s>dZddlZdgZdefdYZeZdS(s>A thread-safe mapping object with a single instance, register.iNtregistrytRegistercBsqeZdZdZdZdZdZdZdZdZ dZ d Z d Zd Z RS( s*Emulate a dicutionary, but thread-safe(?).cCs%tj|_i|_i|_dS(N(t threadingtRLocktlocktmapt_cache(tself((sD/home/arcege/src/pyerector/regression/parallel/pyerector/register.pyt__init__s cCs!|jt|jSWdQXdS(N(RtreprR(R((sD/home/arcege/src/pyerector/regression/parallel/pyerector/register.pyt__repr__s cCs!|j||j|s  =PKcD5ypyerector/variables.py#!/usr/bin/python """ Usage: Variable('name', 'value') Variable('name').value = 'value' print Variable('name') (Variable('name') == Variable('name')) (Variable('name1') < Variable('name2')) (hash(Variable('name')) == hash('name')) (Variable('name').name == 'name') (Variable('name').value == 'value') (Variable('name').toString() == Variable('name').name) (Variable('name').value is Variable('name').value) (Variable('name').value == str(Variable('name'))) (V['name'] == Variable('name').value) (V('name') == Variable('name')) # for backward compatibility fv = FileVariable('file', 'test.txt') (Variable('file').value == 'test.txt') fv.value == open('test.txt', 'rt').read().decode('UTF-8') fv.value = 'test2.txt' (Variable('file').value == 'test2.txt') fv.value != open('test.txt', 'rt').read().decode('UTF-8') # VariableSet is an augmented dictionary vs = VariableSet( Variable('name', 'michael'), Variable('number', '42'), Variable('address', '1313'), Variable('city', 'springfield') ) (vs['name'] == Variable('name')) (vs['name'] == vs[Variable('name')]) (vs == {Variable('name'): Variable('name'), Variable('number'): Variable('number'), Variable('address'): Variable('address'), Variable('city'): Variable('city')}) (len(vs) == 4) (sorted(vs.keys()) == ['address', 'city', 'name', 'number'] vs.add(Variable('zipcode', '00000')) vs.update(phone='888-555-1234') vs.update(('phone', '888-555-1234')) (vs['phone'] == 'phone') (vs['phone'].value == '888-555-1234') ('name' in vs == True) vs1 = VariableSet( Variable('email': 'michael@springfield.us') ) vs1.update(vs) (vs1['name'] == Variable('name')) ('email' not in vs and 'email' in vs1) """ import logging import threading from .exception import Error __all__ = [ 'FileVariable', 'V', 'Variable', 'VariableSet', ] class VariableCache(object): """Mimic part of the functionality of a dictionary, but keep things minimalistic. And some functionality, like copy(), we don't want.""" def __init__(self): self.cache = {} self.lock = threading.RLock() def __repr__(self): return '%s(%s)' % (self.__class__.__name__, self.cache) def __len__(self): return len(self.cache) def __iter__(self): if hasattr(self.cache, 'iterkeys'): return self.cache.iterkeys() else: return iter(self.cache.keys()) def __contains__(self, name): if not isinstance(name, Variable): name = Variable(name) with self.lock: return name in self.cache def __getitem__(self, name): if not isinstance(name, Variable): name = Variable(name) with self.lock: if name in self.cache: return self.cache[name] else: raise Error('no such variable: %s' % name.name) def __setitem__(self, name, value): if not isinstance(name, Variable): name = Variable(name) with self.lock: logger = logging.getLogger('pyerector.execute') # to prevent a really long string, just the first 25 characters #s = repr(value) #if len(s) > 25: # logger.debug('name = %s; value = %s', repr(name), # s[:25] + '...' + s[-1:]) #else: # logger.debug('name = %s; value= %s', repr(name), s) self.cache[name] = value def __delitem__(self, name): if not isinstance(name, Variable): name = Variable(name) with self.lock: if name in self.cache: del self.cache[name] else: raise Error('no such variable: %s', name) def __call__(self, name, value=None): var = Variable(name) if value is not None: self[var] = value return var V = VariableCache() class Variable(str): """Create a persistent string with a mutable value.""" cache = V def __new__(cls, name, value=None): return super(Variable, cls).__new__(cls, name) def __init__(self, name, value=None): super(Variable, self).__init__() if value is not None: self.cache[self] = value # this gets around the concatenating the name instead of the value def __add__(self, other): return self.value + other def __radd__(self, other): return other + self.value def __str__(self): """Return the value instead of the name.""" return str(self.value) def __repr__(self): return 'Var(%s)' % super(Variable, self).__repr__() def toString(self): """Deprecated.""" return self.name # use the 'name' property @property def name(self): """Return the string (name).""" return super(Variable, self).__str__() def retrieve_value(self): try: return self.cache[self] except Error: return '' @property def value(self): """Retrieve the value from the cache.""" return self.retrieve_value() @value.setter def value(self, value): """Change the value in the cache.""" self.cache[self] = value @value.deleter def value(self): """Delete the variable from the cache.""" del self.cache[self] @classmethod def list(cls): """Return the cache as a tuple (not a list).""" return tuple(cls.cache) class FileVariable(Variable): """Load a variables value from a file, but only when we need it. Setting to the value changes the filename, getting the value reads the file. File contents are decoded by default as UTF-8.""" encoding = 'UTF-8' def retrieve_value(self): filename = self.cache[self] with open(str(filename), 'rt') as infile: contents = infile.read().decode(self.encoding) return contents class VariableSet(dict): """A dictionary where all keys are variables and all values are the same as the corresponding keys, i.e. {a: a, b: b, c: c, ...}""" def __new__(cls, *variables, **kwargs): # assign the variables in the initializer routine, not creater return super(VariableSet, cls).__new__(cls) def __init__(self, *variables, **kwargs): super(VariableSet, self).__init__() for var in variables: self.add(var) def add(self, var): """Add a new variable to the set.""" if not isinstance(var, Variable): var = Variable(var) logging.debug('%s.add(%s)', self.__class__.__name__, repr(var)) super(VariableSet, self).__setitem__(var, var) def __getitem__(self, item): if not isinstance(item, Variable): item = Variable(item) return super(VariableSet, self).__getitem__(item) def __setitem__(self, item, value): if not isinstance(item, Variable): item = Variable(item) if item not in self: self.add(item) logging.debug('setitem(%s, %s)', repr(item), repr(value)) if item is not value: # didn't pass in the same object self[item].value = value def __delitem__(self, item): #del self[item].value # also clear out the value? super(VariableSet, self).__delitem__(item) def update(self, *args, **kwargs): """Augment the set with additional values.""" for datum in args: if hasattr(datum, 'keys'): for name in datum: self[name] = datum[name] else: for (name, value) in datum: self[name] = value for name in kwargs: self[name] = kwargs[name] PKcDZPTTpyerector/register.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """A thread-safe mapping object with a single instance, register.""" import threading __all__ = [ 'registry', ] class Register(object): """Emulate a dicutionary, but thread-safe(?).""" def __init__(self): self.lock = threading.RLock() self.map = {} self._cache = {} def __repr__(self): with self.lock: return repr(self.map) def append(self, name, cls): """Add a new mapping.""" with self.lock: self.map[name] = cls def __iter__(self): with self.lock: return self.map.iterkeys() def __contains__(self, name): with self.lock: return name in self.map def __getitem__(self, name): with self.lock: return self.map[name] def __setitem__(self, name, value): with self.lock: self.map[name] = value def __delitem__(self, name): with self.lock: del self.map[name] def __len__(self): with self.lock: return len(self.map) def __iter__(self): with self.lock: return iter(self.map) def get(self, name): """Return a dict of all items of the same type as the one being given. For example, if "All" is given, then return a dict of all Target subclasses.""" with self.lock: cls = self[name] if cls in self._cache: return self._cache[cls] else: # build up the cache clscache = self._cache[cls] = {} for name in self: kls = self[name] if issubclass(kls, cls) and cls is not kls: clscache[name] = kls return clscache registry = Register() PKq"Eq]]pyerector/targets.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Define the standard targets.""" from .register import registry from .base import Target, Iterator from .tasks import Mkdir, Remove, Unittest from .iterators import StaticIterator from .variables import V from .vcs import VCS __all__ = [ 'All', 'Build', 'Clean', 'Compile', 'Default', 'Dist', 'Help', 'Init', 'InitDirs', 'InitVCS', 'Packaging', 'Test', ] # standard targets class Help(Target): """This information. Tasks: internal Dependencies: None Members: None Methods: None """ def run(self): """Display callable targets with the first line of their docstrings. If --verbose, then also show the contents of the VariableCache.""" def firstline(string): """Return only up to the first newline.""" try: pos = string.index('\n') except (AttributeError, ValueError, IndexError): return string or '' else: return string[:pos] for name, obj in sorted(registry.get('Target').items()): if name[1:].lower() != name[1:]: continue # ignore non-callable targets self.display( '%-20s %s' % (obj.__name__.lower(), firstline(obj.__doc__)) ) for var in sorted(V): self.logger.info('var %s = "%s"' % (var.name, var.value)) class Clean(Target): """Clean directories and files used by the build. Tasks: internal [Remove(files)] Dependencies: None Members: files = () Methods: None """ files = () def run(self): from .iterators import DirList files = self.get_args('files') if isinstance(files, Iterator): pass elif isinstance(files, (list, tuple)) and \ len(files) == 1 and isinstance(files[0], Iterator): files = files[0] elif isinstance(files, (list, tuple)): files = DirList(*tuple(files)) assert isinstance(files, Iterator) Remove()(files) class InitVCS(Target): """Initialize information about the version control system, VCS. The VCS instance is stored as a global Variable. Tasks: None Dependencies: None Members: None Methods: None This functionality should now be handled by pyerector.vcs.__init__.InitVCS before pyerector finishes being imported. """ def run(self): self.logger.warning('Target %s has been deprecated.', self) class InitDirs(Target): """Create initial directories. Tasks: internal [Mkdir(files)] Dependencies: None Members: files = () Methods: None """ files = () def run(self): Mkdir()(StaticIterator(self.files)) class Init(Target): """Initialize the build. Tasks: None Dependencies: InitDirs Members: None Methods: None """ dependencies = (InitDirs,) class Compile(Target): """Compile source files. Tasks: None Dependencies: None Members: None Methods: None """ # meant to be overriden class Build(Target): """The primary build. Tasks: None Dependencies: (Init, Compile) Members: None Methods: None """ dependencies = (Init, Compile) class Packaging(Target): """Package for distribution. Tasks: None Dependencies: None Members: None Methods: None """ # meant to be overriden class Dist(Target): """The primary packaging. Tasks: None Dependencies: (Build, Packaging) Members: None Methods: None """ dependencies = (Build, Packaging) # may be overriden class Testonly(Target): """Run unittest, without dependencies. Tasks: None Dependencies: None Members: None Methods: None """ class Test(Target): """Run (unit)tests. Tasks: None Dependencies: Build, Testonly Members: None Methods: None """ dependencies = (Build, Testonly) class All(Target): """Do it all. Tasks. None Dependencies: (Clean, Dist, Test) Members: None Methods: None """ dependencies = (Clean, Dist, Test) # default target class Default(Target): """When no target is specified. Tasks: None Dependencies: Dist Members: None Methods: None """ dependencies = (Dist,) PKJ#Eeepyerector/base.pyc Tc@sdZddlZddlZddlmZddlZyeWn!ek reddlmZnXeddkrddl m Z nddl m Z dd l m Z mZmZdd lmZmZdd lmZdd lmZmZdd lmZmZmZddlmZmZddddgZde fdYZ de fdYZ!de fdYZ"de fdYZ#de#fdYZ$de fdYZ%de%fdYZ&dS(sDefine the primary constructs for use within the package; namely Initer (base class), Target, Task, Iterator, Mapper, Sequential, Parallel. Iterator and Mapper have most of their logic defined in iterators. iN(tversion(treduceit2i(tBase(t ExclusionstnormjointTimer(tget_current_stacktPyThread(tregistry(tAborttError(tConfigtnooptnoTimer(tVtVariabletTargettTaskt SequentialtParalleltInitercBseZdZeZd ZeZeZ e Z d Z dZ d dZdZdZedZdZedZdZRS( sPrimary base class for everything. This is responsible for establishing the common framework for the API, including argument handling. cOstjd|_|jjd|jj||y|d}Wntk rXd}nX|d=y|d}Wntk rtj }nX|d=|r||_ n|rx%|D]}t ||||qWn|dk r|p|t dmssMust supply %s to '%s' in '%s'N( R4ttypeRR<RR1tlisttcallabletremovet TypeError(R#tvaluettypevaltvalnamettypenamettexttlst((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt asserttypehs !  #cCst|dr:|t|dkr:t|d|}nt||}|s[|dk rq|j|||n1|r|dkrtd||jjfn|S(swReturn a item in saved kwargs or an attribute of the name name. If noNone, then raise ValueError if the value is None. R$sno '%s' for '%s'N(thasattrR2RRJt ValueErrorRR(R#tnameREtnoNoneRD((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyR.|s$cCst|dr0t|dr0t|d}n4t||r`t||r`t||}ndS|j|tttf||S(s;Return the saved argument list or an attribute of the name.R!((RKR2RJR1R@R5(R#RMRD((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pytget_argsscCsdS(sTo be overridden in subclasses.N((tcls((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt validate_treescOs3ddlm}|jj|d|||dS(snDisplay a message at the DISPLAY log level, which should be above the level that the --quiet option would set.i(t getLevelNametDISPLAYN(RRRRtlog(R#tmsgR!R$RR((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pytdisplaysN((Rt __module__t__doc__R tconfigRR*tFalseR'R(tTrueR)R+R&R:R<RJR.ROt classmethodRQRV(((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyR#s       cBseZdZd Zd Zd ZeZeZe j Z e dZ e jdZ dZedZdZdZdZdZRS( sA representation of an element of "organization". There are three primary members and a method to be overridden: uptodates - a Mapper instance to check if the target should be started. dependencies - sequence of Targets (or Variable instances) to be called before tasks or the run() method. tasks - sequence of Tasks (or Variable instances) to be called before the run() method. run() - perform Python code. A timer tracks how long the tasks and run() method take. Exceptions raised: There are two primary exceptions, Abort and code-base (KeyError, etc.). The abort is captured by PyErector.run(), but other exceptions are handled as normal (using getLogger('pyerector').exception()). cCs(|j|j o|jjSWdQXdS(s-Return if the Target has been called already.N(t_been_called_lockt allow_reexecRt _been_called(R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt been_calleds cCs |j||j_WdQXdS(s"Set if the Target has been called.N(R]RR_(R#RD((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyR`s cCs |jjS(N(RR(R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt__str__scs^fd|j|jdd|j|jdd|j|jdddS( sRecursively validate the contents of 'dependencies' (Target), 'uptodates' (Mapper) and 'tasks' (Task). Also allowable are Variable instances.c s-t|}tj|}x |D]}t|tr;q n|dkr_t||r_|}nt|tr|j|||q nt|tr||kr||}nit|trt||r||j}n;t||r||j j}nt d|||f|j q WdS(s-Validate that the object is the correct type.tMappers%s: invalid %s: %sN( R tgetR4RRRR0R?t issubclassRRLRQ(tkobjtksettktypetktnametklassobjtklassesRMtobj(tvalidate_class(s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRls&    Rt dependencyRbtuptodateRttaskN(Rt dependenciest uptodatesttasks(RP((Rls@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRQscCst|jts't|j|_nt|jtsNt|j|_n!t|jtrotdnt|jtst|j|_nt|jtrt|jt stt|jtstt|jtstdS(sCast each member as Sequential.suptodates cannot be ParallelN(R4RpRRqRRLRrtAssertionError(R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt member_castsc Gs|jj}|jjd||t}|jr8dS|jt}|j|z~|j r|jjd||j r|j ddSn|j r|jjd||j n||j r|jjd||j ny!|jjd||j Wntttttfk r6nitk rInVtk rr|jjd|tn-tk rtjd jd tnXWdQXtr|j d n|j d |t|_Wd|jXdS( s6Call the chain: uptodates, dependencies, tasks, run().s%s.__call__(*%s)Nscalling %s.uptodates()s uptodate.scalling %s.dependencies()scalling %s.tasks()sstarting %s.runsException in %s.runt pyerectort Exceptionsdone.s done. (%0.3f)(RRRRRR`RtRtpushRqtverboseRpRrtrunRRLRCt RuntimeErrorR3R R t exceptionRvRRRR[tpop(R#R!tmynamettimertstack((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt__call__sP                   cCsdS(sTo be overridden.N((R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRy0scGs=dt|djd|Df}|jj|dS(s(Display the class name with the message.s%s: %st css|]}t|VqdS(N(R0(R=ts((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pys 5sN(R0R<Rtwarning(R#R!RU((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRx3s)((((RRWRXRpRqRrRZR^R_t threadingtRLockR]tpropertyR`tsetterRaR\RQRtRRyRx(((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRs  "  3 cBs8eZdZgZdZdZdZdZRS(sA representation of a unit of work. Generally performs Python code directly, either as one of the standard tasks or through the API. The run() method is meant to be overridden. cCs |jjS(N(RR(R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRaAsc OsW|jj}|jjd|||t}|j|z|j||trq|jjd|||dSy|j }Wnt t t t tfk rnitk rnVtk r|jjd|tn-tk rtjdjdtnXWd|jX|r@tt|d|n|jjd|dS(Ns%s.__call__(*%s, **%s)sCalling %s(*%s, **%s)sException in %s.runRuRvsreturn error = %ss %s: done.(RRRRRRwt handle_argsR RRyRRLRCRzR3R R R{RvRRR|R0tinfo(R#R!R$R}Rt returncode((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRDs6           cCsdS(sTo be overridden.N((R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRycscCst|dr|j s|rft|dkrTt|dtrT|d|_qft||_n|r~t||_ndS(s,"Put the arguments into their proper places.R!iiN(RKR!tlenR4R5R1tdictR$(R#R!R$((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRfs %(RRWRXR!RaRRyR(((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyR:s    R5cBszeZdZeZdZdZdZdZdZ dZ dZ dZ d Z d Zd ZRS( sThe base class for Iterators and Mappers. Processes arguments as sequences of files. Examples: i = Iterator('src', 'test', pattern='*.py') j = Iterator('build', pattern='*.py', recurse=True) k = Iterator('conf', ['etc/build.properties', 'tmp/dummy'], i, j) tuple(k) == ('conf/foo.cfg', 'etc/build.properties', 'tmp/dummy', 'src/foo.py', 'test/testfoo.py', 'build/foo.py', 'build/test/testfoo.py') i = Iterator('src', pattern='*.py', recurse=True) j = Iterator(i, pattern='test*') tuple(j) == ('src/test/testfoo.py',) c Ostt|j||d|_d|_|jdttt t t t df}t |trp||_nNt |tt t frt||_n$t |t rt|f|_ndS(NR+(tsuperR5R&RtpooltcursetR.RtsetR0R1R@R?R4t exclusion(R#R;R$R+((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyR&s   ! cCs tdS(sJIterators and Mappers do not get called as Targets, Tasks and Sequentials.N(tNotImplemented(R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRscCs+t|jd|_tg|_|S(NR;(R@RORtiterR(R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt__iter__scCs |jS(N(tnext(R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt__next__scCsxtryt|j}Wndtk r|jyt|j}Wqtk r~tjd}t|j|qXnXt|t rt |dkrPn|j j |rqn|j |rPqqW|j|S(sCycle through the curset, returning strings that would "match". Matching strings are not in the exclusions, if a pattern is set, would match the pattern, and if recursive and a directory. If it is a directory, then prepend the directory's contents to the pool (not the curset). ii(R[RRt StopIterationt getnextsetRCtsystexc_infoR4R1RRtmatchtcheck_candidatetpost_process_candidate(R#t candidatetexc((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRs"    !cCs|js"|jjdtn|jd}|jd=|jjdt|t|trst||_nt|t t frg|D]}|j |^q}|rtt d||_q td|_n*t|t r t|j ||_ndS(Ns nothing leftisnext item from pool is %scSs||S(N((tatb((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyts((RRRRtreprR4R5RRR1R@tadjustRR0(R#titemtititems((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRs    "cCsZt|jd}t|ttfr:|j|n |j|t||_dS(s#Add an item to the end of the pool.R;N(R@ROR4R1textendR7R;(R#RR;((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyR7s  cCsKt|tr|g}nt||jd*|jjdt|dS(s6Add a string or sequence to the beginning of the pool.isadding to pool: %sN(R4R0R@RRRR(R#R((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt_prepends cCs|S(N((R#R((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRscCs|gS(N((R#R((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRscCs:|jdt}|stStj||r2tStSdS(NR*(R.R0R[tfnmatcht fnmatchcaseRZ(R#RR*((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRs (RRWRXRRR&RRRRRR7RRRR(((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyR5qs           RbcBsYeZdZdZdZdZdZdZdZ dZ dZ dZ RS( s3Maps source files to destination files, using a base path, destdir. The mapper member is either a string or callable that will adjust the basename; if None, then there is no adjustment. The map() method can also adjust the basename. An example of using a mapper would be: FileMapper( FileIterator('src', pattern='*.py'), destdir='build', mapper=lambda n: n+'c' destdir='build', mapper='%(name)sc' ) This would map each py file in src to a pyc file in build: src/base.py -> build/base.pyc src/main.py -> build/main.pyc cOstt|j|||jdttf}|dkrLd|_nHt|rd||_n0t|tr|d|_nt d|dS(NtmappercSs|S(N((tx((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRscSs|i|d6S(NRM((RMtmapstr((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRssmap must be string or callable( RRbR&R.RAR0Rt mapper_funcR4RC(R#R,R$R((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyR&s   cCs |jdt}|dkr'd}ntt|j}t|tr~t|dkr~|\}}|j |}~n|j |}t|tst d|j j d||j ||j |}t|tst dt||}|j j d||||fS( s2Return the next item, with its mapped destination.tdestdirtismapper must return a strsself.map(%s) = %ssmap() must return a strsmapper yields (%s, %s)N(R.R0RRRbRR4R1RRRsRRtmapR(R#RRttemptmappedtresult((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRs   ! cCs|S(s%Identity routine, one-to-one mapping.((R#R((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRscGs[xA|D]"\}}|j||sPqqW|jjd|tS|jjd|tS(Ns %s() => Trues %s() => False(t checkpairRRR[RZ(R#R!tsrctdst((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyR!scCs|S(s]For each src,dst pair, check the modification times. If the dst is newer, then return True. ((R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRn+scCs|jjd|||tS(sTo be overridden.s%s.checkpair(%s, %s)(RRRZ(R#RR((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyR1scCstS(sTo be overridden.(RZ(R#RR((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt checktree6sN( RRWRXRRRR&RRRRnRR(((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRbs    cBseeZdZdZdZdZdZdZeZe dZ e dZ dZ RS( s4Class to sequentially call Target or Task instances.cCs'|jjd }d||jdfS(Nis%s%sR(RRRO(R#RM((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt__repr__?scCst|jdS(NR(RRO(R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRCscCst|jdS(NR(RRO(R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt__len__FscCst|dkS(Ni(R(R#((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyt__bool__IscCst|trdSt|tr=t|tr=|}nt|trU|}nt|dryt|}Wn6tt fk rt j dj d|t qX|}nt j dj d|t |S(s5Retrieve an instance. If an instance of a Variable, return None. If a subclass of Initer, then return an instance of the subclass. If an instance of Inter, return the object. If a string (has 'lower' attribute), then find in registry and return an instance. Otherwise log an exception and raise Abort error. tlowerRusCannot find %sspyerector.executescould not retrieve %sN(R4RRR?RdRRKR RRsRRR{R (RMRkRe((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pytretrieveMs$       cCs_t|trd||fSt|tr:d||fSt|trWd||fSdSdS(s+Generate the appropriate exception message.s Exception in %s.dependencies: %ssException in %s.tasks: %ssException in %s.uptodates: %sN(R4RRRbR(Rktparent((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pytget_exception_messagenscGstd}t}x|D]}|j|}|jjd||dkrUqnt|trmt}n|j ||}y||}Wn't k r|jj |t qX|r| rtSqW|rtSdSdS(sCall the items in the list.is Calling %sN( RRZRRRRR4RbR[RR R{R (R#R!RtabortiveRRktexcmsgR((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRzs(       (( RRWRXRRRRRt __nonzero__t staticmethodRRR(((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyR;s    ! cBseZdZdZRS(s4Class to concurrently call Target or Task instances.c Gstd}d|}g}xm|D]e}|j|}|dkrKq$n|j||}td|t|d|}|j|q$Wx|D]}|jqWx|D]}|jqWx |D]}|j rt qqW~dS(s0Call the items in the list, in separate threads.is%s.RMttargetN( RRRRRR0R7tstartR<R{R ( R#R!RtbnametthreadsRRktexgmsgtthread((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRs(          (RRWRXR(((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyRs('RXRR RRRRt NameErrort functoolstpy3.baseRtpy2.basethelperRRRtexecuteRRtregisterR R{R R RYR R Rt variablesRRt__all__RRRR5RbRR(((s@/home/arcege/src/pyerector/regression/parallel/pyerector/base.pyts2    7wSZPKM'EhQ  pyerector/version.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Hold the version control information about the product. The build replaces the %..% tokens with proper values. The constants (HG_* and RELEASE_*) at the end are for backward compatibility. """ from .variables import V class VersionClass(object): """Define variables with version/release information and retrieve using this interface.""" def __init__(self): V['pyerector.release.version'] = '03b41399d901' V['pyerector.release.branch'] = 'default' V['pyerector.release.tags'] = 'tip' V['pyerector.release.product'] = 'pyerector' V['pyerector.release.number'] = '1.2.5' @staticmethod def _validitem(item): """Ensure that the variable name is valid.""" return ( item.startswith('pyerector.vcs.') or item.startswith('pyerector.release.') ) def __len__(self): return 5 def __getitem__(self, itemname): if self._validitem(itemname): return V[itemname] else: return KeyError(itemname) def __setitem__(self, itemname, value): if self._validitem(itemname): V[itemname] = value else: return KeyError(itemname) def __delitem__(self, itemname): raise NotImplementedError @property def version(self): """Retrieve the version control information.""" vstr, bstr, tstr = ( V('pyerector.release.version'), V('pyerector.release.branch'), V('pyerector.release.tags'), ) version = vstr.value.replace('+', '') if bstr.value == 'default': branch = '' else: branch = ' (%s)' % bstr if tstr.value == 'tip': tags = '' else: tags = ' <%s>' % ','.join(tstr.value.split()) return 'r%s%s%s' % (version, branch, tags) @property def release(self): """Retrieve the release information.""" return '%s %s' % ( V('pyerector.release.product'), V('pyerector.release.number') ) def __call__(self, *args): from logging import getLogger logger = getLogger('pyerector') name = V['pyerector.release.product'] number = V['pyerector.release.number'] for desired in args: if desired and '%s %s' % (name, desired) > self.release: logger.error( 'Version: %s %s does not match desired %s, aborting', name, number, desired) raise SystemExit(1) HG_VERSION = '03b41399d901' HG_BRANCH = 'default' HG_TAGS = 'tip' RELEASE_PRODUCT = 'pyerector' RELEASE_NUMBER = '1.2.5' __all__ = [ 'Version', ] Version = VersionClass() PKJ#E@ pyerector/exception.pyc Tc@scdZddlZddlZdefdYZdefdYZddgddZdS( spExceptions and exception frame handling for displaying traceback objects "appropriately" (with the class name). iNtAbortcBseZdZRS(s Roll back to the PyErector call.(t__name__t __module__t__doc__(((sE/home/arcege/src/pyerector/regression/parallel/pyerector/exception.pyR stErrorcBs eZdZdZdZRS(s Error within pyerector to raise.cCs)djg|jD]}t|^qS(Ns: (tjointargststr(tselfta((sE/home/arcege/src/pyerector/regression/parallel/pyerector/exception.pyt__str__scCs6tt||}t|tr.t|S|SdS(N(tformatRt isinstancetunicode(Rt format_spectvalue((sE/home/arcege/src/pyerector/regression/parallel/pyerector/exception.pyt __format__s (RRRR R(((sE/home/arcege/src/pyerector/regression/parallel/pyerector/exception.pyRs c Cs|dkr0ttdr0ttd}q0ng}d}d}x|dk r|dksi||kr|j}|j}|j} | j} | j} t j | t j | ||j } | r| j } nd} d|jkr_|jd}t||s d}n|dk r3|d|kr3||ds   PKJ#E:g ! !pyerector/execute.pyc Tc@sdZddlZddlmZmZddlmZddgZdefd YZ d Z dej fd YZ d efd YZ de fdYZedS(s Implement an internal version of Python's frame stack, but for Initer instances. This allows for showing tracebacks of the targets and tasks, not the underlying Python calls (which are often recursive calls to the same Target.__call__ method). The execution stack, ExecStack, can take a parent, supposedly another thread which called this one. Operations are based on that stack along with the ancesters'. Also defines a specialized version of threading.Thread which creates a new execution stack based on the parent's. iNi(tAborttError(tVtget_current_stacktPyThreadt ExecStackcBskeZdZd dZdZdZdZdZdZ dZ dZ d Z d Z RS( sAn execution stack, with reference to caller's execution stack. Operations such as len(s) s[], iter() all access the ancester's stack(s) before performing the operations here. For example: s0 = ExecStack() s0.push('a') s0.push('b') s1 = ExecStack(s0) s1.push('c') s1.push('d') len(s0) == 2 len(s1) == 4 s0[0], s1[-1] == 'a', 'b' s1[0], s1[-1] == 'a', 'd' tuple(s1) == ('a', 'b', 'c', 'd') s1.pop() == 'd' cCsAtt|jg|_d|_||_tj|_ dS(N( tsuperRt__init__tstacktNonetpostparentt threadingtRLocktlock(tselfR ((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyR-s    cCsd|jj|jfS(Ns%s(%s)(t __class__t__name__R(R((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyt__repr__4scCs$|j|jj|WdQXdS(sAdd an item to the stack.N(RRtappend(Rtframe((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pytpush7s cCs!|j|jjSWdQXdS(s,Remove an item from the stack and return it.N(RRtpop(R((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyR<s cCsO|j@t|jtr.t|j}nd}|t|jSWdQXdS(Ni(Rt isinstanceR RtlenR(Rtparlen((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyt__len__As  cCs"|jt||SWdQXdS(N(Rttuple(Rtindex((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyt __getitem__Is cCsdS(N((RRtvalue((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyt __setitem__NscCsdS(N((RR((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyt __delitem__QscCsSddlm}|j4|jdk r<||j|jSt|jSWdQXdS(Ni(tchain(t itertoolsR!RR R Rtiter(RR!((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyt__iter__Ts  c Cs\g}d}|jAx9|D]1}|jdd||jjf|d7}qWWdQX|S(snReturn lines as traceback.extract_tb, but in relation to the execute stack, not Python's. Includes ancestors.is%s%s s iN(RRRR(Rtlinestindenttitem((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pytextract\s  N(Rt __module__t__doc__R RRRRRRRR R$R((((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyRs         cCs tjjS(s,Return the current thread's execution stack.(R t currentThreadR(((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyRjscBs&eZdZdZdZdZRS(s1Create a PyErector specific thread, with bounded semaphore and execution stack. Size of the bounded semaphore is from the global Variable "pyerector.pool.size". Calls to PyThread.run are wrapped so that exception.Error exceptions are caught and displayed using getLogger('pyerector.execute').exception. cOs|jjdkraddl}|jdjdtdtjt tdd|j_nt t |j ||t }t||_d|_dS(Nispyerector.executesBoundedSemaphore(%s)spyerector.pool.sizei(RtlimiterR tloggingt getLoggertdebugRR tBoundedSemaphoretintRRRRRRt exception(RtargstkwargsR-t parentstack((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyRys & c Csddl}ddl}|jd}z|j{|jdytt|jWnOtk r|j d}|j d|j ||_ dSt k rdSXWdQXWd|jdXdS(Nispyerector.executesPyThread.limiter.acquiredisException in %ssPyThread.limiter.released( R-tsysR.R,R/RRtrunRtexc_infoR2tnameR(RR-R6tloggertexc((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyR7s     N(RR)R*R R,RR7(((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyRos tInitializationcBs>eZdZgZdZedZdZdZRS(sRegister initialization routines and ensure that they are only called once. Otherwise, reimports of pyerector may result in overwriting configuration value or augmenting subsystems (like logging) in undesirable ways. cCst|_|jj|dS(N(tFalset been_calledtregistryR(R((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyRs cCsx|jD] }|q WdS(sStart calling the instances.N(R?(tclstinstance((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pytstartscCs,|jr dSz|jWdt|_XdS(N(R>R7tTrue(R((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyt__call__s  cCsdS(sTo be overriden.N((R((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyR7s( RR)R*R?Rt classmethodRBRDR7(((sC/home/arcege/src/pyerector/regression/parallel/pyerector/execute.pyR<s   t InitThreadingcBseZdZdZRS(sInitialize the module, including creating an initial execution stack on the MainThread (so get_current_stack will work in all instances). cCsMdtds  P + PKJ#E?)aapyerector/main.pyc Tc@sdZddlZddlZddlZddlmZmZddlmZddl m Z m Z ddl m Z mZddlmZdd lmZdd lmZdd lmZd d gZd efdYZeZde fdYZedS(s5The main driver of the library. The PyErector class acts like a void function, parsing the command-line arguments, setting up the environment, validating the 'updates', 'dependencies' and 'tasks' members of all registered targets and then processing the targets and variable assignments on the command-line. iNi(tAborttError(tTimer(tPyThreadtInitialization(tnooptnoTimer(tregistry(tTarget(tVersion(tVt PyErectortpymainc Bs*eZdZyddlZejddZ[ejddddd d d ejd d d dejddddddd dejddddd dejddddd dejddddd dejdddd d ejd!ddWnek rdZddl Z e j dd"Z[ ej d d d dej ddddddd dej ddddd dej ddddd dej ddddd dej dddd d ej d!ddnXd#Z d$Z d%Zd&Zd'ZRS((sThe main program of the library. Parses arguments, validates the calling tree, and starts the PyThread, which calls each target on the command-line. iNt descriptionsBPyErector build system; use the "help" target for more informationttargetstmetavartTARGETtnargst*thelpsJname of target to call or variable assignment, default target is "default"s --directorys-dsbase directorys --dry-runs-NtdestRtactiont store_truesdo not perform operationss--quiets-qs less outputs --verboses-vsmore verbose outputs --versions-Vsshow version informations --notimersdo not show timing informations--DEBUGsBPyerector build system; use the "help" target for more informationcGstjd}tjj|\|_|_|jdkrItj|_ntjj|j|_t j d|_ g|_ d|_ y|j|ptjdy ttddkstWn9tk rtdntk rtdnX|jtdd d |j}|j|jWn9tk rTtd n,tk rptdnXt|j dS( Nitt pyerectorispyerector.pool.sizes$pyerector.pool.size value is invalids,pyerector.pool.size must be positive integertnameR ttargetsCtrl-C(tsystargvtostpathtsplittprogdirtprognametcurdirtrealpathtloggingt getLoggertloggerRt returnstatust argumentstintR tAssertionErrort ValueErrort SystemExittvalidate_targetsRtruntstarttjointKeyboardInterruptR(tselftargstprogramt newthread((s@/home/arcege/src/pyerector/regression/parallel/pyerector/main.pyt__init__Ls2          cCsL|jj|}t|tr9|\}}||_n|j|tjd}x|jD]}d|kr|jdd\}}|j t |j s     PK3oDszpyerector/__init__.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Package: pyerector A framework to build products based on tasks and targets. The thought is, distutils/setuptools is to Maven as PyErector is to Ant. The former requires that you shoehorn project to fit into their model. PyErector is a toolkit/framework, like Make and Ant, giving you routines to set up what you want/need to do. It is geared more toward Python usage, but most tasks are generic enough for any system. The developer would create a driver file, commonly named 'pyerect', that would import the 'pyerector' module and call the PyErector routine. The smallest driver file would be: from pyerector import * PyErector() The PyErector main routine will scan the command-line arguments for target names, if none are found, then it will call the default target. Target references on the command-line are the same as the class, but with the first character lowercase. The command-line argument 'default' would refer to the 'Default' Target class. Command-line arguments must reference targets, not other objects, or variable assignments. Instead of referencing a Target, a command-line argument may also assign a string value to a variable, for example: "developer.name=Michael". After command-line arguments are processed, PyErector() will verify the classes are properly constructed and then call the target references in turn. The package is broken into three primary types of objects: standard targets, standard tasks and the API. All standard targets could be subclassed, but would not be, in general. Instead the class members would be modified in the driver file: InitDirs.files = ('build',) Compile.tasks = (Copy('foobar.txt', dest='build'),) As can be seen with the standard Copy task above, the most common usage is to create instances within a target's class members, like 'tasks'. The API allows the developer to extend or change what is being performed during the build. For example, in the product's "pyerect" program, there is a Registration task, which changes the SubPyErector task: class Regression(SubPyErector): wdir = 'regression' proddir = os.path.realpath(os.path.join('build', distfile)) if 'PYTHONPATH' in os.environ: paths = os.environ['PYTHONPATH'].split(os.pathsep) paths.append(proddir) env = {'PYTHONPATH': os.pathsep.join(paths)} del paths else: env = {'PYTHONPATH': proddir } This does not change the task's flow, but it does set up certain defaults which does change the behavior. """ from .helper import normjoin, Exclusions from .helper import display, warn, verbose, debug # being deprecated from .execute import Initialization from .version import Version from .main import PyErector, pymain from .vcs import VCS from .base import Target, Task, Sequential, Parallel from .tasks import Chmod, Copy, CopyTree, Echo, Egg, HashGen, \ Java, Mkdir, PyCompile, Remove, Scp, Shebang, Spawn, \ SubPyErector, Ssh, Symlink, Tar, Tokenize, Touch, \ Unittest, Untar, Unzip, Zip from .targets import All, Build, Clean, Compile, Default, Dist, Help, \ Init, InitDirs, InitVCS, Packaging, Test, Testonly from .iterators import FileSet, StaticIterator, FileIterator, FileList, \ DirList, FileMapper, BasenameMapper, MergeMapper, \ IdentityMapper, Uptodate from .variables import FileVariable, V, Variable, VariableSet # With the addition of pyerector.api, some of the lower-level objects, like # Target, Sequential, etc. should be removed in the future __all__ = [ # base routines 'Exclusions', 'normjoin', 'PyErector', 'pymain', 'Version', # remove 'Target' and 'Task' to move to pyerector.api 'Target', 'Task', 'Sequential', 'Parallel', 'V', # alias for VariableCache 'FileVariable', 'Variable', 'VariableSet', 'VCS', # tasks 'Chmod', 'Copy', 'CopyTree', 'Echo', 'Egg', 'HashGen', 'Java', 'Mkdir', 'PyCompile', 'Remove', 'Scp', 'Shebang', 'Spawn', 'Ssh', 'SubPyErector', 'Symlink', 'Tar', 'Tokenize', 'Touch', 'Unittest', 'Untar', 'Unzip', 'Zip', # targets 'All', 'Build', 'Clean', 'Compile', 'Default', 'Dist', 'Help', 'Init', 'InitDirs', 'InitVCS', 'Packaging', 'Test', 'Testonly', # iterators 'FileSet', 'StaticIterator', 'FileIterator', 'FileList', 'DirList', # mappers 'FileMapper', 'BasenameMapper', 'IdentityMapper', 'MergeMapper', 'Uptodate', ] Initialization.start() PKJ#E"pyerector/metaclass.pyc ASc@s<dZddlZddlmZdefdYZdS(s|Need an easy way to map a class name (string) to the class. The metaclass here will use a global registry for the mapping. iNi(tregistrytIniterMetaClasscBseZdZdZRS(s:Meta class to register the class for easy retrieval later.cCs4tj|||||t|s PKJ#EG@'(D(Dpyerector/helper.pyc Sc@s}dZddlZddlZddlZddlmZmZddlZddlZddl m Z ddl m Z m Z ddd gZd Zdefd YZd efd YZd efdYZdefdYZdejefdYZdefdYZde fdYZeeddZeddZeddZeddZdS(s'Helper routines for the pyerector package. The Exclusions class can be used by Iterators to prevent matching against certain file (like .svn, .hg, .git) or patterns (*.pyc, *~). normjoin runs os.path.normpath(os.path.join(*args), as a convenience routine. The Subcommand class handles spawning processes through subprocess.Popen, but with a bit more backend control (like terminating the process when the object is deleted). Initialize the logging system, including setting up module specific formatters. It does not change the root ('') logger. iN(tversiontexc_infoi(tError(tget_current_stacktInitializationt Exclusionstnormjoint SubcommandcGs+|sd}ntjjtjj|S(s1Join and normalize the arguments into a pathname.t(R(tostpathtnormpathtjoin(targs((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyR&s cBs]eZdZed Zed ZeeBZded Zd Z e de d Z RS(sBA list of exclusion patterns. The usedefaults argument can take three values: True (default), False and None. A True will augment the set with the 'defaults' values. A False will augment with a set containing vcs_names. And None will not augment any additional values - this is dangerous when used with the Remove task.s.gits.hgs.svntCVSs*.pycs*~s.*.swpt __pycache__cCst|tr9||jkr*|j}nt|}n3t|tttttdfslt dnt|trtf}n|rt|}n t}||_t t|j |dS(Ns-Exclusions: expecting str, set, tuple or list( t isinstanceRt usedefaultstsettstrttupletlistttypetNonet TypeErrortsupert__init__(tselftitemsRt initialset((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyR8s $   cCs|jtkr||jB}n%|jtkr>||jB}n|}g|D]*}tjtjj ||rK|^qK}t |dkS(sGReturn true if the given string matches one of the patterns in the set.i( RtTruetdefaultstFalset vcs_namestfnmatcht fnmatchcaseR R tbasenametlen(Rtstringtmatchestvtvalues((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pytmatchHs *cCs|r+t|dr+|j|_|`dS|r5dSt|dsS|j|_nt|tttfsztdnt||_dS(s/Change or reset the defaults for all instances.t real_defaultsNs(Exclusions: expecting set, tuple or list(thasattrR+RRRRRR(tclsRtreset((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyt set_defaultsVs (s.gits.hgs.svnR(s*.pycs*~s.*.swpR((( t__name__t __module__t__doc__RR!t cruft_pattsRRRR*t classmethodR R/(((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyR-s    cBseZdZyddlZWn#ek rAdZednXejZej de ddddZ dZ dZ dZdZd Zd Zd ZRS( s'Handles some of the subprocess details.iNs&Earlier than Python 2.6 is unsupportedcCs|dkri}nt|ts0tdt|dkrzt|dtsdt|trzt|d|_n ||_||_||_d|_ ||_ ||_ ||_ d|_ |_|_d|_|j|r|jndS(Nsmust supply tuple as commandii(RRRtAssertionErrorR%Rtcmdtwdirtenvtproctinfiletoutfileterrfiletstdintstdouttstderrt returncodetcall_subprocesstwait(RR6R7R8RBR=R>R?((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRos$  "         cCsddl}tjd|jj|jt|drt|dr|jdkr|j dk r|j j yt j |j j dWntk rnX|j j |j jd|_ ndS(Nisstarting %s.__del__()R@R9i(t tracebacktloggingtdebugt __class__R0tcloseR,R@RR9t terminateR tkilltpidtOSErrorRB(RRC((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyt__del__s      c Cst|drFtt|ddrF|jdk rF|jjnd|_t|drtt|ddr|jdk r|jjnd|_t|drtt|ddr|jdk r|jjnd|_dS(sClose the open files.R:tlowerR;R<N(R,tgetattrR=RRGR>R?(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRGs  cCs(|jdk std|jjS(s%Send a SIGTERM signal to the process.ssubprocess not spawnedN(R9RR5RH(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRHscCs(|jdk std|jjS(s%Send a SIGKILL signal to the process.ssubprocess not spawnedN(R9RR5RI(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRIscCsO|jdk std|jj}|dk rB||_n|jdk S(sJReturn True if the process has finished, setting returncode if applicable.ssubprocess not spawnedN(R9RR5tpollR@(RR@((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyROs   cCsM|jdk std|jr4|jjn|jj|_|jS(s=Wait for the process to complete, and return the exit status.ssubprocess not spawnedN(R9RR5R=RGRBR@(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRBs  c Csddlm}tjj}|j|jt|jdrM|j}nK|j|j krk|j }n-t|jdrt |jd}nd}t|j dr|j }nK|j |j kr|j }n-t|j drt |j d}nd}|j |j kr|}nit|j dr:|j }nK|j |j krX|j }n-t|j drt |j d}nd}||||_|_|_t|jt }tjd|j||j||||jy@||jd |d |jd |d |d |ddd|}WnLttfk rntd}|jddkrhtddqonX||_||j kr|j|_n||j kr|j|_n||j kr|j|_ndS(s)Call subprocess.Popen and handle the I/O.i(tPopentwriteRMtrtreadtwsMPopen(%s, shell=%s, cwd=%s, stdin=%s, stdout=%s,stderr=%s, bufsize=0, env=%s)tshelltcwdR=R>R?tbufsizeiR8iitENOENTsProgram not foundN(t subprocessRPR tenvirontcopytupdateR8R,R:tPIPEtopenRR;R<R=R>R?RR6RRDRER7tIOErrorRKRR RR9( RRPtrealenvtifiletofiletefiletshellvalR9texc((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRAs^          (R0R1R2RYt ImportErrorRtNotImplementedErrorR]R tcurdirRRRLRGRHRIRORBRA(((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRfs         tTimercBseeZdZdZdZedZdZdZdZ dZ dZ d Z RS( s/Keep track of how long a section of code takes.cCsd|_d|_dS(N(Rt starttimetduration(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRs cCsO|jj}|jdk r#d|S|jdk rCd||jfSd|SdS(Ns <%s started>s<%s duration: %0.3f>s<%s unstarted>(RFR0RjRRk(Rtname((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyt__repr__s  cCsddlm}|S(sReturn the current time.i(ttime(Rn(Rn((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pytnowscCs1|jdk rtdn|j|_dS(sStart the timer.s,cannot start more than once without stoppingN(RjRt RuntimeErrorRo(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pytstartscCsA|jdkrtdn|j|j|_d|_dS(s$Stop the timer and set the duration.scannot stop without startingN(RjRRpRoRk(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pytstopscCs|jdS(N(Rq(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyt __enter__scCs|jdS(N(Rr(Rtetypetevaluetetb((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyt__exit__scCs |jdk rt|jSdS(Ng(RkRtfloat(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyt __float__!s cCstt|S(N(tintRx(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyt__int__&s( R0R1R2RRmt staticmethodRoRqRrRsRwRyR{(((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRis       tVerbosecBsPeZdZeejdZedZedZ dZ dZ RS(s(Deprecated and should no longer be used.cCs ||_dS(N(tlevel(RtstateR~((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyR-scCs tjdS(s#Return the default internal logger.spyerector.execute(RDt getLogger(((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyt _getlogger0scCs tj|S(s+Return the logging level based on its name.(RDt getLevelName(R~((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyt _getlevelnum5scCs|jj|jS(N(Rt isEnabledForR~(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyt__bool__:scGs^tjdtdjg|D]}t|^q}|jj|j|j|dS(NsUse of Verbose is deprecatedt ( twarningstwarntDeprecationWarningR RRtlogRR~(RR tstmsg((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyt__call__=s(( R0R1R2R RDtINFORR|RRRR(((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyR}*s  t LogFormattercBs&eZdZdZdZdZRS(sSubclass to handle formatting messages from the library, including prepending text in the PYERECTOR_PREFIX envvar and showing thread if not the main threads (MainThread, PyErector).cCsEtt|j|}|jdkrdytjd}Wnttfk rWd}nX||_n |j}|rt |t j rd||j f}qd||f}n$t |t j r|j }n|}ddl m}|jd krA|j}|j}|dkr+d||f}qAd |||f}n|S( NtPYERECTOR_PREFIXRs%s: %si(t currentThreadt MainThreadt PyErectors(%s) %ss (%s[%x]) %s(s MainThreads PyErector(RRtformatt log_prefixRR RZtAttributeErrortKeyErrorRRDt LogRecordtmessaget threadingRRltident(Rtrecordt newrecordtprefixRRRlR((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRIs.       cCsWddlm}|\}}}tj||}tj||}dj||S(sReturn similar to the default, but using exception.extract_tb instead of traceback.extract_tb; the former adds the class being called. i(t extract_tbR(t exceptionRRCtformat_exception_onlyt format_listR (RRRRtRttbRetstack((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pytformatExceptionfs N(R0R1R2RRRR(((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRCs tLogExecFormattercBseZdZdZRS(sWSubclass of LogFormatter that shows the pyerector execution stack instead of python's. cCsHt}|d \}}|jtj||}dj|jS(saExtract the pyerector execution stack and return a string with that and the formatted exception. iR(RtextractRCRR trstrip(RRRRtRtlines((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRus   (R0R1R2R(((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRqst InitLoggingcBs8eZdZejZdZeedZdZ RS(sNInitialize the logging, creating two loggers, pyerector and pyerector.execute.s %(message)scCsK||}|}|j|tj|}|j|t|_|S(s.Set up a new logger with appropriate settings.(t setFormatterRDRt addHandlerR t propagate(Rlt handlerklasstformatterklassRtfmtrthndrtlgr((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pytsetups     cCstjd|jd|jtjdtjddd|jdtjt|jdtjt t j dt t td rtjtnd S( sSet the default logging level, create a 'DISPLAY' logging level, create two loggers: pyerector and pyerector.execute, and if available, set warnings to be captured by the logging module. R~Rit levelNametDISPLAYt pyerectorspyerector.executetdefaulttcaptureWarningsN(RDt basicConfigtdeflevelRt addLevelNametERRORRt StreamHandlerRRRt simplefilterRR,RR(R((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pytruns( R0R1R2RDtWARNINGRRR|RR(((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyRs   RRRtDEBUG(R2R"RDR tsysRRRCRRRtexecuteRRt__all__RRRtobjectRRiR}t FormatterRRRRtdisplayRtverboseRE(((sB/home/arcege/src/pyerector/regression/parallel/pyerector/helper.pyts0       91.PK#E0 pyerector/unit-test.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Driver script to call test cases from source files. First argument is a parameter file, containing a Python dictionary. Values in the parameters are: modules - tuple of possible modules path - string of path or None, verbose - boolean to display unittests """ import fnmatch import imp import logging import os import sys import traceback import unittest try: import coverage except ImportError: pass else: coverage.process_startup() try: loader = unittest.loader.TestLoader() except AttributeError: loader = unittest.TestLoader() if not hasattr(loader, 'discover'): # for before Py2.7 class TestLoaderNoDiscover(loader.__class__): # ported almost directly from Py2.7 unittest.loader module _top_level_dir = None def discover(self, start_dir, pattern='test*.py', top_level_dir=None): set_implicit_top = False if top_level_dir is None and self._top_level_dir is not None: top_level_dir = self._top_level_dir elif top_level_dir is None: set_implicit_top = True top_level_dir = start_dir self._top_level_dir = top_level_dir = os.path.abspath(top_level_dir) if top_level_dir not in sys.path: sys.path.insert(0, top_level_dir) is_not_importable = False if os.path.isdir(os.path.abspath(start_dir)): start_dir = os.path.abspath(start_dir) if start_dir != top_level_dir: is_not_importable = \ not os.path.isfile(os.path.join(start_dir, '__init__.py')) else: # a module/package name try: __import__(start_dir) except ImportError: is_not_importable = True else: the_module = sys.modules[start_dir] top_part = start_dir.split('.')[0] start_dir = os.path.abspath(os.path.dirname(the_module.__file__)) if set_implict_top: module = sys.modules[top_part] full_path = os.path.abspath(module.__file__) if os.path.basename(full_path).lower().__startswith('__init__.py'): self._top_level_dir = os.path.dirname(os.path.dirname(full_path)) else: self._top_level_dir = os.path.dirname(full_path) sys.path.remove(top_level_dir) if is_not_importable: raise ImportError('Start directory is not importable: %s' % start_dir) tests = list(self._find_tests(start_dir, pattern)) return self.suiteClass(tests) def _find_tests(self, start_dir, pattern): for path in os.listdir(start_dir): full_path = os.path.join(start_dir, path) if os.path.isfile(full_path): if not fnmatch.fnmatch(path, '[_a-z]*.py'): # value Python identifiers only continue if not fnmatch.fnmatch(path, pattern): continue try: name = self._get_name_from_path(full_path) module = self._get_module_from_name(name) except: yield self._make_failed_import_test(name, self.suiteClass) else: mod_file = os.path.abspath(getattr(module, '__file__', full_path)) realpath = os.path.splitext(mod_file)[0] fullpath_noext = os.path.splitext(full_path)[0] if realpath.lower() != fullpath_noext.lower(): module_dir = os.path.dirname(realpath) mod_name = os.path.splitext(os.path.basename(full_path))[0] expected_dir = os.path.dirname(full_path) msg = ('%r module incorrectly imported from %r. Expected %r. ' 'Is this moduled globally installed?') raise ImportError(msg % (mod_name, module_dir, expected_dir)) yield self.loadTestsFromModule(module) elif os.path.isdir(full_path): if not os.path.isfile(os.path.join(full_path, '__init__.py')): continue load_tests = None tests = None if fnmatch.fnmatch(path, pattern): name = self._get_name_from_path(full_path) package = self._get_module_from_name(name) if package is not None: load_tests = getattr(package, 'load_tests', None) tests = self.loadTestsFromModule(package, use_load_tests=False) if load_tests is None: if tests is not None: yield tests for test in self._find_tests(full_path, pattern): yield test else: try: yield load_tests(self, tests, pattern) except Exception: t, e, tb = sys.exc_info() yield self._make_failed_load_tests(package.__name__, e, self.suiteClass) def _get_name_from_path(self, path): path = os.path.splitext(os.path.normpath(path))[0] _relpath = os.path.relpath(path, self._top_level_dir) assert not os.path.isabs(_relpath), 'Path must be within the project' assert not _relpath.startswith('..'), 'Path must be within the project' name = _relpath.replace(os.path.sep, '.') return name def _get_module_from_name(self, name): try: __import__(name) except ImportError: return None else: return sys.modules[name] # return a TestClass subclass wrapped in a suite on error def _make_failed_import_test(self, name, suiteClass): message = r'Failed to import test module: %s\\n%s' % (name, traceback.format_exc()) return self._make_failed_test('ModuleImportFailure', name, ImportError(message), suiteClass) def _make_failed_load_tests(self, name, exception, suiteClass): return self._make_failed_test('LoadTestsFailure', name, exception, suiteClass) def _make_failed_test(self, classname, methodname, exception, suiteClass): def testFailure(self): raise exception attrs = {methodname: testFailure} TestClass = type(classname, (unittest.TestCase,), attrs) return suiteClass((TestClass(methodname),)) loader = TestLoaderNoDiscover() if __name__ == '__main__': params = eval(sys.argv[1]) verbose = ('verbose' in params and params['verbose']) and 1 or 0 try: runner = unittest.runner.TextTestRunner(verbosity=verbose) except AttributeError: runner = unittest.TextTestRunner(verbosity=verbose) try: suite = unittest.suite.TestSuite() except: suite = unittest.TestSuite() real_args = sys.argv[:] logging.getLogger('pyerector').setLevel(logging.ERROR) try: if params['modules']: path = [os.path.realpath(p) for p in params['path']] if not path: path = [os.curdir] for modname in params['modules']: sys.argv[:] = [modname] packet = imp.find_module(modname, path) mod = imp.load_module(modname, *packet) suite.addTests(loader.loadTestsFromModule(mod)) elif params['path']: for path in [os.path.realpath(p) for p in params['path']]: suite.addTests(loader.discover(path)) else: suite.addTests(loader.discover(os.curdir)) try: runner.run(suite) except KeyboardInterrupt: print raise SystemExit(5) finally: sys.argv[:] = real_args PKJ#Ebwpyerector/version.pyc ;Tc@s`dZddlmZdefdYZdZdZdZdZdZ d gZ eZ d S( sHold the version control information about the product. The build replaces the %..% tokens with proper values. The constants (HG_* and RELEASE_*) at the end are for backward compatibility. i(tVt VersionClasscBsqeZdZdZedZdZdZdZdZ e dZ e dZ d Z RS( sTDefine variables with version/release information and retrieve using this interface.cCs6dtdt,sr%s%s%s(RRtreplacetjointsplit(Rtvstrtbstrttstrtversiontbranchttags((sC/home/arcege/src/pyerector/regression/parallel/pyerector/version.pyR/s     cCsdtdtdfS(s!Retrieve the release information.s%s %sspyerector.release.productspyerector.release.number(R(R((sC/home/arcege/src/pyerector/regression/parallel/pyerector/version.pytreleaseBs cGsddlm}|d}td}td}xR|D]J}|r7d||f|jkr7|jd|||tdq7q7WdS( Ni(t getLoggerRspyerector.release.productspyerector.release.numbers%s %ss2Version: %s %s does not match desired %s, abortingi(tloggingR RRterrort SystemExit(RtargsR tloggertnametnumbertdesired((sC/home/arcege/src/pyerector/regression/parallel/pyerector/version.pyt__call__Js     (t__name__t __module__t__doc__Rt staticmethodR R R RRtpropertyRRR)(((sC/home/arcege/src/pyerector/regression/parallel/pyerector/version.pyR s     RRs1.2.5RtVersionN( R,t variablesRtobjectRt HG_VERSIONt HG_BRANCHtHG_TAGStRELEASE_PRODUCTtRELEASE_NUMBERt__all__R/(((sC/home/arcege/src/pyerector/regression/parallel/pyerector/version.pytsK PKL[&E) pyerector/main.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """The main driver of the library. The PyErector class acts like a void function, parsing the command-line arguments, setting up the environment, validating the 'updates', 'dependencies' and 'tasks' members of all registered targets and then processing the targets and variable assignments on the command-line. """ import logging import os import sys from .exception import Abort, Error from .helper import Timer from .execute import PyThread, Initialization from .config import noop, noTimer from .register import registry from .base import Target from .version import Version from .variables import V __all__ = [ 'PyErector', 'pymain', ] # the main program, an instance to be called by pyerect program class PyErector(object): """The main program of the library. Parses arguments, validates the calling tree, and starts the PyThread, which calls each target on the command-line. """ try: import argparse parser = argparse.ArgumentParser( description='PyErector build system; ' 'use the "help" target for more information') del argparse parser.add_argument('targets', metavar='TARGET', nargs='*', help='\ name of target to call or variable assignment, default target is "default"') parser.add_argument('--directory', '-d', help='base directory') parser.add_argument('--dry-run', '-N', dest='noop', action='store_true', help='do not perform operations') parser.add_argument('--quiet', '-q', action='store_true', help='less output') parser.add_argument('--verbose', '-v', action='store_true', help='more verbose output') parser.add_argument('--version', '-V', action='store_true', help='show version information') parser.add_argument('--notimer', action='store_true', help='do not show timing information') parser.add_argument('--DEBUG', action='store_true') except ImportError: argparse = None import optparse parser = optparse.OptionParser( description='Pyerector build system; ' 'use the "help" target for more information') del optparse parser.add_option('--directory', '-d', help='base directory') parser.add_option('--dry-run', '-N', dest='noop', action='store_true', help='do not perform operations') parser.add_option('--quiet', '-q', action='store_true', help='less output') parser.add_option('--verbose', '-v', action='store_true', help='more verbose output') parser.add_option('--version', '-V', action='store_true', help='show version information') parser.add_option('--notimer', action='store_true', help='do not show timing information') parser.add_option('--DEBUG', action='store_true') def __init__(self, *args): program = sys.argv[0] self.progdir, self.progname = os.path.split(program) if self.progdir == '': self.progdir = os.curdir else: self.progdir = os.path.realpath(self.progdir) self.logger = logging.getLogger('pyerector') self.targets = [] # returnstatus should not need mutex since it is only set by the # PyErector thread and only read after the thread completes, # adding a Condition object seems to cause issues with unittesting self.returnstatus = 0 # successfully completed try: self.arguments(args or sys.argv[1:]) try: assert int(V['pyerector.pool.size']) > 0 except ValueError: raise SystemExit('pyerector.pool.size value is invalid') except AssertionError: raise SystemExit('pyerector.pool.size must be positive integer') self.validate_targets() # run through a thread with an initial stack, wait for the thread # to finish newthread = PyThread(name='PyErector', target=self.run) newthread.start() newthread.join() except KeyboardInterrupt: raise SystemExit('Ctrl-C') except Abort: raise SystemExit(1) else: raise SystemExit(self.returnstatus) def arguments(self, args): """Process the command-line arguments. Not sure if using argparse or optparse, so handle both. """ args = self.parser.parse_args(args) if isinstance(args, tuple): args, arglist = args args.targets = arglist self.process_options(args) # process the arguments all_targets = registry.get('Target') for name in args.targets: if '=' in name: # variable assignment? var, val = name.split('=', 1) V[var.strip()] = val.strip() else: try: obj = all_targets[name.capitalize()] except KeyError: raise SystemExit('Error: unknown target: ' + str(name)) else: if not issubclass(obj, Target): raise SystemExit('Error: unknown target: ' + str(name)) self.targets.append(obj) if len(self.targets) == 0: self.targets.append(registry['Default']) def process_options(self, args): """Process the options.""" # check --verbose before --version if args.notimer: noTimer.on() if args.verbose: logging.getLogger().setLevel(logging.INFO) if args.DEBUG: logging.getLogger().setLevel(logging.DEBUG) # check --quiet after --verbose if args.quiet: logging.getLogger().setLevel(logging.ERROR) if args.noop: noop.on() if args.version: if logging.getLogger().isEnabledFor(logging.INFO): self.logger.log(logging.getLevelName('DISPLAY'), ('%s %s' % (Version.release, Version.version))) else: self.logger.log(logging.getLevelName('DISPLAY'), Version.release) raise SystemExit if args.directory: V['basedir'] = args.directory else: V['basedir'] = self.progdir def validate_targets(self): """Validate the dependency tree, make sure that all are subclasses of Target, validate all Uptodate values and all Task values. """ for target in self.targets: try: target.validate_tree() except ValueError: self.logger.exception('Validation') def run(self): """Call the targets in order.""" timer = Timer() # run all targets in the tree of each argument failed = True with timer: try: for target in self.targets: target()() except Abort: pass except ValueError: self.logger.exception(self.__class__.__name__) except KeyboardInterrupt: raise Abort except AssertionError: self.logger.exception('AssertionError') except Error: self.logger.exception(self.__class__.__name__) else: failed = False if noTimer: time = '' else: time = ' (%0.3f)' % timer if failed: msg = 'Failed.' else: msg = 'Done.' self.logger.warning('%s%s', msg, time) if failed: # passed to the root thread from (this) PyErector thread self.returnstatus = 1 pymain = PyErector class InitMain(Initialization): """Initialize the main module.""" basedir = os.curdir def run(self): V['basedir'] = os.path.realpath(self.basedir) V['homedir'] = os.path.normpath(os.path.expanduser('~')) InitMain() PKJ#E&0(==pyerector/targets.pyc 2Tc @sdZddlmZddlmZmZddlmZmZm Z ddl m Z ddl m Z ddlmZdd d d d d ddddddg ZdefdYZd efdYZdefdYZdefdYZdefdYZd efdYZd efdYZdefdYZd efdYZdefdYZdefdYZdefd YZd efd!YZd"S(#sDefine the standard targets.i(tregistry(tTargettIterator(tMkdirtRemovetUnittest(tStaticIterator(tV(tVCStAlltBuildtCleantCompiletDefaulttDisttHelptInittInitDirstInitVCSt PackagingtTestcBseZdZdZRS(sQThis information. Tasks: internal Dependencies: None Members: None Methods: None cCsd}xrttjdjD]U\}}|dj|dkrQq%n|jd|jj||jfq%Wx4ttD]&}|j j d|j |j fqWdS(sDisplay callable targets with the first line of their docstrings. If --verbose, then also show the contents of the VariableCache.cSsCy|jd}Wn!tttfk r6|p5dSX|| SdS(s$Return only up to the first newline.s tN(tindextAttributeErrort ValueErrort IndexError(tstringtpos((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyt firstline*s  Ris %-20s %ss var %s = "%s"N( tsortedRtgettitemstlowertdisplayt__name__t__doc__Rtloggertinfotnametvalue(tselfRR&tobjtvar((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pytrun&s ((R"t __module__R#R+(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyRscBseZdZdZdZRS(sClean directories and files used by the build. Tasks: internal [Remove(files)] Dependencies: None Members: files = () Methods: None cCsddlm}|jd}t|tr1nqt|ttfrxt|dkrxt|dtrx|d}n*t|ttfr|t|}nt|tstt |dS(Ni(tDirListtfilesi( t iteratorsR-tget_argst isinstanceRtlistttupletlentAssertionErrorR(R(R-R.((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyR+Gs% ((R"R,R#R.R+(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyR =scBseZdZdZRS(sInitialize information about the version control system, VCS. The VCS instance is stored as a global Variable. Tasks: None Dependencies: None Members: None Methods: None This functionality should now be handled by pyerector.vcs.__init__.InitVCS before pyerector finishes being imported. cCs|jjd|dS(NsTarget %s has been deprecated.(R$twarning(R(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyR+_s(R"R,R#R+(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyRUs cBseZdZdZdZRS(stCreate initial directories. Tasks: internal [Mkdir(files)] Dependencies: None Members: files = () Methods: None cCstt|jdS(N(RRR.(R(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyR+ms((R"R,R#R.R+(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyRcscBseZdZefZRS(sUInitialize the build. Tasks: None Dependencies: InitDirs Members: None Methods: None (R"R,R#Rt dependencies(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyRqscBseZdZRS(sQCompile source files. Tasks: None Dependencies: None Members: None Methods: None (R"R,R#(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyR {scBseZdZeefZRS(sYThe primary build. Tasks: None Dependencies: (Init, Compile) Members: None Methods: None (R"R,R#RR R7(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyR scBseZdZRS(sUPackage for distribution. Tasks: None Dependencies: None Members: None Methods: None (R"R,R#(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyRscBseZdZeefZRS(s`The primary packaging. Tasks: None Dependencies: (Build, Packaging) Members: None Methods: None (R"R,R#R RR7(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyRstTestonlycBseZdZRS(s_Run unittest, without dependencies. Tasks: None Dependencies: None Members: None Methods: None (R"R,R#(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyR8scBseZdZeefZRS(sWRun (unit)tests. Tasks: None Dependencies: Build, Testonly Members: None Methods: None (R"R,R#R R8R7(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyRscBseZdZeeefZRS(sUDo it all. Tasks. None Dependencies: (Clean, Dist, Test) Members: None Methods: None (R"R,R#R RRR7(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyR scBseZdZefZRS(sXWhen no target is specified. Tasks: None Dependencies: Dist Members: None Methods: None (R"R,R#RR7(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyR sN(R#tregisterRtbaseRRttasksRRRR/Rt variablesRtvcsRt__all__RR RRRR R RRR8RR R (((sB/home/arcege/src/pyerector/regression/minimal/pyerector/targets.pyts>        PK!EYpyerector/execute.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Implement an internal version of Python's frame stack, but for Initer instances. This allows for showing tracebacks of the targets and tasks, not the underlying Python calls (which are often recursive calls to the same Target.__call__ method). The execution stack, ExecStack, can take a parent, supposedly another thread which called this one. Operations are based on that stack along with the ancesters'. Also defines a specialized version of threading.Thread which creates a new execution stack based on the parent's. """ import threading from .exception import Abort, Error from .variables import V __all__ = [ 'get_current_stack', 'PyThread', ] class ExecStack(object): """An execution stack, with reference to caller's execution stack. Operations such as len(s) s[], iter() all access the ancester's stack(s) before performing the operations here. For example: s0 = ExecStack() s0.push('a') s0.push('b') s1 = ExecStack(s0) s1.push('c') s1.push('d') len(s0) == 2 len(s1) == 4 s0[0], s1[-1] == 'a', 'b' s1[0], s1[-1] == 'a', 'd' tuple(s1) == ('a', 'b', 'c', 'd') s1.pop() == 'd' """ def __init__(self, parent=None): super(ExecStack, self).__init__() self.stack = [] self.pos = None self.parent = parent self.lock = threading.RLock() def __repr__(self): return '%s(%s)' % (self.__class__.__name__, self.stack) def push(self, frame): """Add an item to the stack.""" with self.lock: self.stack.append(frame) def pop(self): """Remove an item from the stack and return it.""" with self.lock: return self.stack.pop() def __len__(self): with self.lock: if isinstance(self.parent, ExecStack): parlen = len(self.parent) else: parlen = 0 return parlen + len(self.stack) def __getitem__(self, index): with self.lock: # this includes the parent stack return tuple(self)[index] def __setitem__(self, index, value): pass def __delitem__(self, index): pass def __iter__(self): from itertools import chain with self.lock: if self.parent is not None: return chain(self.parent, self.stack) else: return iter(self.stack) def extract(self): """Return lines as traceback.extract_tb, but in relation to the execute stack, not Python's. Includes ancestors.""" lines = [] indent = 0 with self.lock: for item in self: lines.append( '%s%s\n' % (' ' * indent, item.__class__.__name__) ) indent += 1 return lines def get_current_stack(): """Return the current thread's execution stack.""" return threading.currentThread().stack class PyThread(threading.Thread): """Create a PyErector specific thread, with bounded semaphore and execution stack. Size of the bounded semaphore is from the global Variable "pyerector.pool.size". Calls to PyThread.run are wrapped so that exception.Error exceptions are caught and displayed using getLogger('pyerector.execute').exception. """ limiter = None def __init__(self, *args, **kwargs): if self.__class__.limiter is None: import logging logging.getLogger('pyerector.execute')\ .debug('BoundedSemaphore(%s)', V['pyerector.pool.size']) self.__class__.limiter = \ threading.BoundedSemaphore(int(V['pyerector.pool.size'])+1) super(PyThread, self).__init__(*args, **kwargs) # this works because at _this_ time, the new thread has not been # created, so currentThread will still have the parent stack parentstack = get_current_stack() self.stack = ExecStack(parentstack) self.exception = None def run(self): import logging, sys logger = logging.getLogger('pyerector.execute') try: with self.limiter: logger.debug('PyThread.limiter.acquired') try: super(PyThread, self).run() except Error: exc = sys.exc_info()[1] logger.exception('Exception in %s', self.name) self.exception = exc return except Abort: return finally: logger.debug('PyThread.limiter.released') class Initialization(object): """Register initialization routines and ensure that they are only called once. Otherwise, reimports of pyerector may result in overwriting configuration value or augmenting subsystems (like logging) in undesirable ways. """ registry = [] def __init__(self): self.been_called = False self.registry.append(self) @classmethod def start(cls): """Start calling the instances.""" for instance in cls.registry: instance() def __call__(self): if self.been_called: return try: self.run() finally: self.been_called = True def run(self): """To be overriden.""" pass class InitThreading(Initialization): """Initialize the module, including creating an initial execution stack on the MainThread (so get_current_stack will work in all instances). """ def run(self): V['pyerector.pool.size'] = 10 curthread = threading.currentThread() assert curthread.name == 'MainThread' if not hasattr(curthread, 'stack'): curthread.stack = ExecStack() InitThreading() PK!Evk_k_pyerector/base.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Define the primary constructs for use within the package; namely Initer (base class), Target, Task, Iterator, Mapper, Sequential, Parallel. Iterator and Mapper have most of their logic defined in iterators. """ import logging import os from sys import version import threading try: reduce except NameError: from functools import reduce if version[0] > '2': # python 3+ from .py3.base import Base else: from .py2.base import Base from .helper import Exclusions, normjoin, Timer from .execute import get_current_stack, PyThread from .register import registry from .exception import Abort, Error from .config import Config, noop, noTimer from .variables import V, Variable __all__ = [ 'Target', 'Task', 'Sequential', 'Parallel', ] # the base class to set up the others class Initer(Base): """Primary base class for everything. This is responsible for establishing the common framework for the API, including argument handling. """ config = Config() # for backward compatibility only # values to propagate to an iterator pattern = None noglob = False recurse = False fileonly = True exclude = () def __init__(self, *args, **kwargs): self.logger = logging.getLogger('pyerector.execute') self.logger.debug('%s.__init__(*%s, **%s)', self.__class__.__name__, args, kwargs) try: basedir = kwargs['basedir'] except KeyError: basedir = None else: del kwargs['basedir'] try: curdir = kwargs['curdir'] except KeyError: curdir = os.curdir else: del kwargs['curdir'] if args: self.args = args if kwargs: for key in kwargs: setattr(self, key, kwargs[key]) if basedir is not None: V['basedir'] = basedir or curdir def get_files(self, files=None): """Return an Iterator of either a given sequence or the "files" member. Iterator attributes define in the class will be propagated to the Iterator instance. """ # propagate 'noglob' keyword to the interator noglob = self.get_kwarg('noglob', bool) recurse = self.get_kwarg('recurse', bool) fileonly = self.get_kwarg('fileonly', bool) pattern = self.get_kwarg('pattern', str) exclude = self.get_kwarg('exclude', (Exclusions, tuple)) if files is None: try: files = getattr(self, 'files') except AttributeError: files = () if isinstance(files, Iterator): return files else: # import here to avoid recursive references from .iterators import FileIterator fset = FileIterator(noglob=noglob, recurse=recurse, fileonly=fileonly, pattern=pattern, exclude=exclude) for entry in files: fset.append(entry) return fset def join(self, *path): """Normjoin the basedir and the path.""" return normjoin(V['basedir'], *path) def asserttype(self, value, typeval, valname): """Assert that the value is a value type.""" if isinstance(typeval, type): typename = typeval.__name__ else: typename = ' or '.join(t.__name__ for t in typeval) text = "Must supply %s to '%s' in '%s'" % ( typename, valname, self.__class__.__name__ ) if isinstance(typeval, (tuple, list)) and callable in typeval: lst = list(typeval)[:] lst.remove(callable) #assert callable(value) or isinstance(value, tuple(lst)), text if not callable(value) and not isinstance(value, tuple(lst)): raise TypeError(value, text) else: #assert isinstance(value, typeval), text if not isinstance(value, typeval): raise TypeError(value, text) def get_kwarg(self, name, typeval, noNone=False): """Return a item in saved kwargs or an attribute of the name name. If noNone, then raise ValueError if the value is None. """ if hasattr(self, 'kwargs') and name in getattr(self, 'kwargs'): value = getattr(self, 'kwargs')[name] else: value = getattr(self, name) if noNone or value is not None: self.asserttype(value, typeval, name) elif noNone and value is None: raise ValueError("no '%s' for '%s'" % (name, self.__class__.__name__)) return value def get_args(self, name): """Return the saved argument list or an attribute of the name.""" if hasattr(self, 'args') and getattr(self, 'args'): value = getattr(self, 'args') elif hasattr(self, name) and getattr(self, name): value = getattr(self, name) else: return () self.asserttype(value, (tuple, list, Iterator), name) return value @classmethod def validate_tree(cls): """To be overridden in subclasses.""" pass # do nothing, Target will do something with this def display(self, msg, *args, **kwargs): """Display a message at the DISPLAY log level, which should be above the level that the --quiet option would set.""" from logging import getLevelName self.logger.log(getLevelName('DISPLAY'), msg, *args, **kwargs) class Target(Initer): """A representation of an element of "organization". There are three primary members and a method to be overridden: uptodates - a Mapper instance to check if the target should be started. dependencies - sequence of Targets (or Variable instances) to be called before tasks or the run() method. tasks - sequence of Tasks (or Variable instances) to be called before the run() method. run() - perform Python code. A timer tracks how long the tasks and run() method take. Exceptions raised: There are two primary exceptions, Abort and code-base (KeyError, etc.). The abort is captured by PyErector.run(), but other exceptions are handled as normal (using getLogger('pyerector').exception()). """ dependencies = () uptodates = () tasks = () # if True, then 'been_called' always returns False, allowing for # reexecution allow_reexec = False # if True, then 'been_called' returns True, preventing # reexecution _been_called = False _been_called_lock = threading.RLock() @property def been_called(self): """Return if the Target has been called already.""" with self._been_called_lock: # class member return not self.allow_reexec and self.__class__._been_called @been_called.setter def been_called(self, value): """Set if the Target has been called.""" with self._been_called_lock: # class member self.__class__._been_called = value def __str__(self): return self.__class__.__name__ @classmethod def validate_tree(cls): """Recursively validate the contents of 'dependencies' (Target), 'uptodates' (Mapper) and 'tasks' (Task). Also allowable are Variable instances.""" def validate_class(kobj, kset, ktype, ktname): """Validate that the object is the correct type.""" klassobj = registry[ktype] klasses = registry.get(ktype) for name in kset: if isinstance(name, Variable): # variables are valid, but we don't do anything with them continue if ktype == 'Mapper' and isinstance(name, klassobj): # special case, allow direct instance of Uptodate obj = name elif isinstance(name, Sequential): validate_class(klassobj.__name__, name, ktype, ktname) continue elif isinstance(name, str) and name in klasses: obj = klasses[name] elif isinstance(name, type) and issubclass(name, klassobj): obj = klasses[name.__name__] elif isinstance(name, klassobj): obj = klasses[name.__class__.__name__] else: raise ValueError( '%s: invalid %s: %s' % (kobj, ktname, name) ) obj.validate_tree() validate_class(cls.__name__, cls.dependencies, 'Target', 'dependency') validate_class(cls.__name__, cls.uptodates, 'Mapper', 'uptodate') validate_class(cls.__name__, cls.tasks, 'Task', 'task') def member_cast(self): """Cast each member as Sequential.""" if not isinstance(self.dependencies, Sequential): self.dependencies = Sequential(*self.dependencies) if not isinstance(self.uptodates, Sequential): self.uptodates = Sequential(*self.uptodates) elif isinstance(self.uptodates, Parallel): raise ValueError('uptodates cannot be Parallel') if not isinstance(self.tasks, Sequential): self.tasks = Sequential(*self.tasks) assert isinstance(self.uptodates, Sequential) and \ not isinstance(self.uptodates, Parallel) assert isinstance(self.dependencies, Sequential) assert isinstance(self.tasks, Sequential) def __call__(self, *args): """Call the chain: uptodates, dependencies, tasks, run().""" myname = self.__class__.__name__ self.logger.debug('%s.__call__(*%s)', myname, args) timer = Timer() if self.been_called: return self.member_cast() stack = get_current_stack() stack.push(self) # push me onto the execution stack try: # call uptodates if self.uptodates: self.logger.debug('calling %s.uptodates()', self) if self.uptodates(): self.verbose('uptodate.') return # call dependencies if self.dependencies: self.logger.debug('calling %s.dependencies()', self) self.dependencies() # call tasks, and run() with timer: if self.tasks: self.logger.debug('calling %s.tasks()', self) self.tasks() try: self.logger.debug('starting %s.run', myname) self.run() except (KeyError, ValueError, TypeError, RuntimeError, AttributeError): raise # reraise except Abort: raise # reraise except Error: self.logger.exception('Exception in %s.run', myname) raise Abort except Exception: logging.getLogger('pyerector').exception('Exception') raise Abort if noTimer: self.verbose('done.') else: self.verbose('done. (%0.3f)' % timer) self.been_called = True finally: stack.pop() def run(self): """To be overridden.""" def verbose(self, *args): """Display the class name with the message.""" msg = '%s: %s' % (str(self), ' '.join(str(s) for s in args)) self.logger.warning(msg) # Tasks class Task(Initer): """A representation of a unit of work. Generally performs Python code directly, either as one of the standard tasks or through the API. The run() method is meant to be overridden. """ args = [] def __str__(self): return self.__class__.__name__ def __call__(self, *args, **kwargs): myname = self.__class__.__name__ self.logger.debug('%s.__call__(*%s, **%s)', myname, args, kwargs) stack = get_current_stack() stack.push(self) # push me onto the execution stack try: self.handle_args(args, kwargs) if noop: self.logger.warning('Calling %s(*%s, **%s)', myname, args, kwargs) return try: returncode = self.run() except (KeyError, ValueError, TypeError, RuntimeError, AttributeError): raise except Abort: raise # reraise except Error: self.logger.exception('Exception in %s.run', myname) raise Abort except Exception: logging.getLogger('pyerector').exception('Exception') raise Abort finally: stack.pop() if returncode: raise Error(str(self), 'return error = %s' % returncode) else: self.logger.info('%s: done.', myname) def run(self): """To be overridden.""" def handle_args(self, args, kwargs): """"Put the arguments into their proper places.""" if (hasattr(self, 'args') and not self.args) or args: if len(args) == 1 and isinstance(args[0], Iterator): self.args = args[0] else: self.args = tuple(args) if kwargs: self.kwargs = dict(kwargs) class Iterator(Initer): """The base class for Iterators and Mappers. Processes arguments as sequences of files. Examples: i = Iterator('src', 'test', pattern='*.py') j = Iterator('build', pattern='*.py', recurse=True) k = Iterator('conf', ['etc/build.properties', 'tmp/dummy'], i, j) tuple(k) == ('conf/foo.cfg', 'etc/build.properties', 'tmp/dummy', 'src/foo.py', 'test/testfoo.py', 'build/foo.py', 'build/test/testfoo.py') i = Iterator('src', pattern='*.py', recurse=True) j = Iterator(i, pattern='test*') tuple(j) == ('src/test/testfoo.py',) """ exclusion = Exclusions() def __init__(self, *path, **kwargs): super(Iterator, self).__init__(*path, **kwargs) self.pool = None self.curset = None exclude = self.get_kwarg('exclude', (Exclusions, set, str, tuple, list, type(None)) ) if isinstance(exclude, Exclusions): self.exclusion = exclude elif isinstance(exclude, (set, tuple, list)): self.exclusion = Exclusions(exclude) elif isinstance(exclude, str): self.exclusion = Exclusions((exclude,)) def __call__(self): """Iterators and Mappers do not get called as Targets, Tasks and Sequentials.""" raise NotImplemented def __iter__(self): # this is a list so we can modify it later, if necessary self.pool = list(self.get_args('path')) self.curset = iter([]) return self def __next__(self): return self.next() def next(self): """Cycle through the curset, returning strings that would "match". Matching strings are not in the exclusions, if a pattern is set, would match the pattern, and if recursive and a directory. If it is a directory, then prepend the directory's contents to the pool (not the curset). """ while True: try: candidate = next(self.curset) except StopIteration: #self.logger.debug('caught StopIteration on next()') self.getnextset() # can raise StopIteration try: candidate = next(self.curset) # can raise StopIteration except TypeError: exc = sys.exc_info()[1] raise TypeError(self.curset, exc) if isinstance(candidate, tuple) and len(candidate) == 2: break if self.exclusion.match(candidate): continue if self.check_candidate(candidate): break return self.post_process_candidate(candidate) def getnextset(self): if not self.pool: self.logger.debug('nothing left') raise StopIteration item = self.pool[0] del self.pool[0] self.logger.debug('next item from pool is %s', repr(item)) if isinstance(item, Iterator): self.curset = iter(item) elif isinstance(item, (tuple, list)): items = [self.adjust(i) for i in item] if items: self.curset = iter(reduce(lambda a, b: a+b, items)) else: self.curset = iter(()) elif isinstance(item, str): self.curset = iter(self.adjust(item)) #self.logger.debug('curset = %s', repr(self.curset)) def append(self, item): """Add an item to the end of the pool.""" path = list(self.get_args('path')) if isinstance(item, (tuple, list)): path.extend(item) else: path.append(item) self.path = tuple(path) def _prepend(self, item): """Add a string or sequence to the beginning of the pool.""" if isinstance(item, str): item = [item] self.pool[:0] = list(item) self.logger.debug('adding to pool: %s', repr(item)) # text based def post_process_candidate(self, candidate): return candidate def adjust(self, candidate): return [candidate] def check_candidate(self, candidate): pattern = self.get_kwarg('pattern', str) if not pattern: return True elif fnmatch.fnmatchcase(candidate, pattern): return True else: return False class Mapper(Iterator): """Maps source files to destination files, using a base path, destdir. The mapper member is either a string or callable that will adjust the basename; if None, then there is no adjustment. The map() method can also adjust the basename. An example of using a mapper would be: FileMapper( FileIterator('src', pattern='*.py'), destdir='build', mapper=lambda n: n+'c' destdir='build', mapper='%(name)sc' ) This would map each py file in src to a pyc file in build: src/base.py -> build/base.pyc src/main.py -> build/main.pyc """ destdir = None mapper = None def __init__(self, *files, **kwargs): super(Mapper, self).__init__(*files, **kwargs) mapper = self.get_kwarg('mapper', (callable, str)) if mapper is None: # identity mapper self.mapper_func = lambda x: x elif callable(mapper): self.mapper_func = mapper elif isinstance(mapper, str): self.mapper_func = \ lambda name, mapstr = mapper: mapstr % {'name': name} else: raise TypeError('map must be string or callable', mapper) def next(self): """Return the next item, with its mapped destination.""" destdir = self.get_kwarg('destdir', str) if destdir is None: destdir = '' # do _not_ catch StopIteration item = super(Mapper, self).next() #self.logger.debug('super.next() = %s', item) if isinstance(item, tuple) and len(item) == 2: item, temp = item mapped = self.mapper_func(temp) del temp else: mapped = self.mapper_func(item) assert isinstance(mapped, str), "mapper must return a str" self.logger.debug('self.map(%s) = %s', mapped, self.map(mapped)) mapped = self.map(mapped) assert isinstance(mapped, str), 'map() must return a str' result = normjoin(destdir, mapped) self.logger.debug('mapper yields (%s, %s)', item, result) return item, result def map(self, item): """Identity routine, one-to-one mapping.""" return item def __call__(self, *args): for (src, dst) in self: if not self.checkpair(src, dst): break else: self.logger.debug('%s() => True', self) return True self.logger.debug('%s() => False', self) return False def uptodate(self): """For each src,dst pair, check the modification times. If the dst is newer, then return True. """ return self() # run __call__ def checkpair(self, src, dst): """To be overridden.""" self.logger.debug('%s.checkpair(%s, %s)', self, src, dst) return False def checktree(self, src, dst): """To be overridden.""" return False class Sequential(Initer): """Class to sequentially call Target or Task instances.""" items = () def __repr__(self): name = self.__class__.__name__[:1] return '%s%s' % (name, self.get_args('items')) def __iter__(self): return iter(self.get_args('items')) def __len__(self): return len(self.get_args('items')) def __bool__(self): return len(self) > 0 __nonzero__ = __bool__ @staticmethod def retrieve(name): """Retrieve an instance. If an instance of a Variable, return None. If a subclass of Initer, then return an instance of the subclass. If an instance of Inter, return the object. If a string (has 'lower' attribute), then find in registry and return an instance. Otherwise log an exception and raise Abort error. """ if isinstance(name, Variable): return None elif isinstance(name, type) and issubclass(name, Initer): obj = name() elif isinstance(name, Initer): obj = name elif hasattr(name, 'lower'): try: kobj = registry[name] except (KeyError, AssertionError): logging.getLogger('pyerector').exception( 'Cannot find %s', name ) raise Abort else: obj = kobj() else: logging.getLogger('pyerector.execute').exception( 'could not retrieve %s', name ) raise Abort return obj @staticmethod def get_exception_message(obj, parent): """Generate the appropriate exception message.""" if isinstance(obj, Target): return 'Exception in %s.dependencies: %s' % (parent, obj) elif isinstance(obj, Task): return 'Exception in %s.tasks: %s' % (parent, obj) elif isinstance(obj, Mapper): return 'Exception in %s.uptodates: %s' % (parent, obj) else: return None def __call__(self, *args): """Call the items in the list.""" parent = get_current_stack()[-1] abortive = False for item in self: obj = self.retrieve(item) self.logger.debug('Calling %s' % obj) if obj is None: # do not process Variable instances continue if isinstance(obj, Mapper): abortive = True excmsg = self.get_exception_message(obj, parent) try: result = obj(*args) except Error: self.logger.exception(excmsg) raise Abort else: if abortive and not result: return False else: if abortive: return True else: # this is just being explicit return class Parallel(Sequential): """Class to concurrently call Target or Task instances.""" def __call__(self, *args): """Call the items in the list, in separate threads.""" parent = get_current_stack()[-1] bname = '%s.' % parent threads = [] for item in self: obj = self.retrieve(item) if obj is None: # do not process Variable instances continue exgmsg = self.get_exception_message(obj, parent) thread = PyThread( name=bname + str(obj), target=obj, ) threads.append(thread) for thread in threads: thread.start() for thread in threads: thread.join() for thread in threads: if thread.exception: raise Abort del threads PK#'EpS S pyerector/version.py.orig#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Hold the version control information about the product. The build replaces the %..% tokens with proper values. The constants (HG_* and RELEASE_*) at the end are for backward compatibility. """ class VersionClass(object): """Define variables with version/release information and retrieve using this interface.""" def __init__(self): from .variables import V V['pyerector.release.version'] = '%hg.version%' V['pyerector.release.branch'] = '%hg.branch%' V['pyerector.release.tags'] = '%hg.tags%' V['pyerector.release.product'] = '%release.product%' V['pyerector.release.number'] = '%release.number%' @staticmethod def _validitem(item): """Ensure that the variable name is valid.""" return ( item.startswith('pyerector.vcs.') or item.startswith('pyerector.release.') ) def __len__(self): return 5 def __getitem__(self, itemname): if self._validitem(itemname): return V[itemname] else: return KeyError(itemname) def __setitem__(self, itemname, value): if self._validitem(itemname): V[itemname] = value else: return KeyError(itemname) def __delitem__(self, itemname): raise NotImplementedError @property def version(self): """Retrieve the version control information.""" vstr, bstr, tstr = ( V('pyerector.release.version'), V('pyerector.release.branch'), V('pyerector.release.tags'), ) version = vstr.value.replace('+', '') if bstr.value == 'default': branch = '' else: branch = ' (%s)' % bstr if tstr.value == 'tip': tags = '' else: tags = ' <%s>' % ','.join(tstr.value.split()) return 'r%s%s%s' % (version, branch, tags) @property def release(self): """Retrieve the release information.""" return '%s %s' % ( V('pyerector.release.product'), V('pyerector.release.number') ) def __call__(self, *args): from logging import getLogger logger = getLogger('pyerector') name = V['pyerector.release.product'] number = V['pyerector.release.number'] for desired in args: if desired and '%s %s' % (name, desired) > self.release: logger.error( 'Version: %s %s does not match desired %s, aborting', name, number, desired) raise SystemExit(1) HG_VERSION = '%hg.version%' HG_BRANCH = '%hg.branch%' HG_TAGS = '%hg.tags%' RELEASE_PRODUCT = '%release.product%' RELEASE_NUMBER = '%release.number%' __all__ = [ 'Version', ] Version = VersionClass() PKJ#EIVpyerector/py2/__init__.pyc cRc@sdS(N((((sG/home/arcege/src/pyerector/regression/minimal/pyerector/py2/__init__.pytsPKJ#E,(pyerector/py2/base.pyc ASc@s0dZddlmZdefdYZdS(sDefine the underlying base class for Initer as a metaclass instance for Python 2.x. The syntax for metaclass declarations is different between Python 2.x and Python 3.x.i(tIniterMetaClasstBasecBseZdZeZRS(s%Baseclass with metaclass declaration.(t__name__t __module__t__doc__Rt __metaclass__(((sC/home/arcege/src/pyerector/regression/minimal/pyerector/py2/base.pyR sN(Rt metaclassRtobjectR(((sC/home/arcege/src/pyerector/regression/minimal/pyerector/py2/base.pytsPKSCiW:RRpyerector/py2/__init__.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. PKcDfMgpyerector/py2/base.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Define the underlying base class for Initer as a metaclass instance for Python 2.x. The syntax for metaclass declarations is different between Python 2.x and Python 3.x.""" from ..metaclass import IniterMetaClass class Base(object): """Baseclass with metaclass declaration.""" __metaclass__ = IniterMetaClass PKJ#E5= = pyerector/vcs/__init__.pyc `Tc@s~dZddlZddlmZddlmZddlmZdgZd Z d Z d efd YZ e dS( s0Determine the version control system being used.iNi(tInitialization(tVi(tBasetVCScCsddlm}tjjt}gtj|D]7}|jdr2|d kr2tjj|d^q2}yIddl m }|t }x(|D] }dt |f}||qWWn!t k rt t d |nXd S( sMFind modules in pyerector.vcs (that are not __init__ and base) and load them.i(tmoduless.pys __init__.pysbase.pyi(t import_modules%s.%stfromlistN(s __init__.pysbase.py(tsysRtostpathtdirnamet__file__tlistdirtendswithtsplitextt importlibRt__name__t ImportErrort __import__(RtcurdirtftnamesRtwhereamitnametmodname((sH/home/arcege/src/pyerector/regression/parallel/pyerector/vcs/__init__.pyt load_pluginss(   cOsotytd}Wntk r1tj}nXtjd|}|jdkrbt dn|||S(sGDetermine the type of version control and return information about it. tbasedirtsrcdirsno version control foundN( RRtKeyErrorRRRt vcs_checkRtNonet RuntimeError(targstkwargsRtvcs((sH/home/arcege/src/pyerector/regression/parallel/pyerector/vcs/__init__.pyR"s  tInitVCScBseZdZdZRS(sPInitialize the version control system, assigning to the pyerector.vcs variable. cCsrddlm}|d}y t}Wntk rI|jdn%X|j|td<|jd|dS(Ni(t getLoggert pyerectors No VCS founds pyerector.vcssFound %s(tloggingR$RRtinfot current_infoR(tselfR$tloggerR"((sH/home/arcege/src/pyerector/regression/parallel/pyerector/vcs/__init__.pytrun4s     (Rt __module__t__doc__R+(((sH/home/arcege/src/pyerector/regression/parallel/pyerector/vcs/__init__.pyR#0s( R-RtexecuteRt variablesRtbaseRt__all__RRR#(((sH/home/arcege/src/pyerector/regression/parallel/pyerector/vcs/__init__.pyts    PKJ#Epn=pyerector/vcs/subversion.pyc ;Tc@sodZddlZddlmZddlmZddlmZdgZdefd YZ e j dS( s'Subversion based version control hooks.iNi(tVCS_Basei(t Subcommand(tVariablet SubversioncBs)eZdZdZdZdZdZRS(s%Version control class for Subversion.t subversiontsvns.svnc Cst|jd|jfdtdtjdtj}|jjj d}|j dkr{x|j tj j tj D]}|jdr|j d}td |d jqy|jd r|j d}|d jd }|d jd }|dkr~|d jd|d d }|d jd|d } | dkrctd|d |qtd|d || !qt|dkrt|d jd|d d } |d jd| d } | dkrtd|d | qtd|d | | !qtqy|jdr<|j d}td|d jqy|jdry|j d}td|d jqyqyWndS(s'Retrieve information from the workarea.tinfotwaittstdouttstderrsUTF-8is Revision: s: s svn.versionisURL: s /branches/s/tags/it/s svn.branchssvn.tagssLast Changed Author: ssvn.usersLast Changed Date: ssvn.dateN(RtprogtrootdirtTruetPIPEtostdevnullRtreadtdecodet returncodetrstriptlineseptsplitt startswithRtstriptfind( tselftproctsvnouttlinetpostpartstposbtposttposbetposbntpostetpostn((sI/home/arcege/src/pyerector/regression/minimal/pyerector/vcs/subversion.pyt current_infosB  %    (t__name__t __module__t__doc__tnameR t directoryR&(((sI/home/arcege/src/pyerector/regression/minimal/pyerector/vcs/subversion.pyRs ( R)RtbaseRthelperRt variablesRt__all__Rtregister(((sI/home/arcege/src/pyerector/regression/minimal/pyerector/vcs/subversion.pyts  -PKcDL[]pyerector/vcs/git.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Git based version control hooks.""" import os from ..helper import Subcommand from .base import DVCS_Base from ..variables import Variable __all__ = [ 'Git', ] class Git(DVCS_Base): """Version control class for Git.""" name = 'git' prog = 'git' directory = '.git' def current_info(self): """Retrieve information from the workarea.""" proc = Subcommand( (self.prog, 'log', '--max-count=1', '--format=%h%n%ce%n%cd'), wait=True, stdout=Subcommand.PIPE, stderr=os.devnull ) gitout = proc.stdout.read().decode('UTF-8') if proc.returncode == 0: assert gitout.count('\n') == 3 (vers, user, date) = gitout.rstrip().split('\n') Variable('git.version', vers) Variable('git.user', user) Variable('git.date', date) git_version = vers else: git_version = None proc = Subcommand( (self.prog, 'branch', '--no-color', '--list'), wait=True, stdout=Subcommand.PIPE, stderr=os.devnull ) gitout = proc.stdout.read().decode('UTF-8') if proc.returncode == 0: for line in gitout.rstrip(os.linesep).split(os.linesep): if line.startswith('*'): if line[2:].strip() != '(no branch)': Variable('git.branch', line[2:].strip()) if git_version: proc = Subcommand( (self.prog, 'tag', '--contains', git_version), wait=True, stdout=Subcommand.PIPE, stderr=os.devnull ) gitout = proc.stdout.read().decode('UTF-8') if proc.returncode == 0: if gitout.strip(): Variable('git.tag', gitout.strip()) # this is used by the package to get the primary class VCS_class = Git Git.register() PK/#EZ`pyerector/vcs/subversion.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Subversion based version control hooks.""" import os from .base import VCS_Base from ..helper import Subcommand from ..variables import Variable __all__ = [ 'Subversion', ] class Subversion(VCS_Base): """Version control class for Subversion.""" name = 'subversion' prog = 'svn' directory = '.svn' def current_info(self): """Retrieve information from the workarea.""" proc = Subcommand( (self.prog, 'info', self.rootdir), wait=True, stdout=Subcommand.PIPE, stderr=os.devnull ) svnout = proc.stdout.read().decode('UTF-8') if proc.returncode == 0: for line in svnout.rstrip(os.linesep).split(os.linesep): if line.startswith('Revision: '): pos = line.split(': ') Variable('svn.version', pos[1].strip()) elif line.startswith('URL: '): parts = line.split(': ') posb = parts[1].find('/branches/') post = parts[1].find('/tags/') if posb != -1: posbe = parts[1].find('/', posb+1) + 1 posbn = parts[1].find('/', posbe+1) if posbn == -1: Variable('svn.branch', parts[1][posbe:]) else: Variable('svn.branch', parts[1][posbe:posbn]) elif post != -1: poste = parts[1].find('/', post+1) + 1 postn = parts[1].find('/', poste+1) if postn == -1: Variable('svn.tags', parts[1][poste:]) else: Variable('svn.tags', parts[1][poste:postn]) elif line.startswith('Last Changed Author: '): parts = line.split(': ') Variable('svn.user', parts[1].strip()) elif line.startswith('Last Changed Date: '): parts = line.split(': ') Variable('svn.date', parts[1].strip()) Subversion.register() PKJ#EY  pyerector/vcs/mercurial.pyc USc@sodZddlZddlmZddlmZddlmZdgZdefd YZ e j dS( s*Mercurial(hg) based version control hooks.iNi(t Subcommandi(t DVCS_Base(tVariablet MercurialcBs)eZdZdZdZdZdZRS(s$Version control class for Mercurial.t mercurialthgs.hgc Cs ddlm}|d}t|jddfdtd|jdtjd tj}|j j j d j }|j d ||jd krtd n~t|jdd|jddddfdtd|jdtjd tj}|j j j d }|j dt||jd kr|jjd}|j d|y |dWntk r|jd nXy |dWntk r|jd nXtd|d td|dtd|dtd|dtd|dnd S(!s'Retrieve information from the workarea.i(t getLoggert pyerectortidentifys--idtwaittwdirtstdouttstderrsUTF-8s hgid = %sis!could not retrieve Mercurial nodetlogs-rt+ts --templates;{node|short} {author|email} {date|isodate} {branch} {tags} s hgout = %ss s parts = %siis hg.versionshg.userishg.dateis hg.branchshg.tagsN(tloggingRRtprogtTruetrootdirtPIPEtostdevnullR treadtdecodetstriptdebugt returncodet RuntimeErrortreplacetreprtrstriptsplitt IndexErrortappendtNoneR(tselfRtloggertprocthgidthgouttparts((sH/home/arcege/src/pyerector/regression/minimal/pyerector/vcs/mercurial.pyt current_infosL            (t__name__t __module__t__doc__tnameRt directoryR*(((sH/home/arcege/src/pyerector/regression/minimal/pyerector/vcs/mercurial.pyRs ( R-RthelperRtbaseRt variablesRt__all__Rtregister(((sH/home/arcege/src/pyerector/regression/minimal/pyerector/vcs/mercurial.pyts  2PKJ#E,{{pyerector/vcs/base.pyc ZTc@sdZddlZddlmZddlmZddlmZdefdYZ d e fd YZ d e fd YZ dS( svDefine the base classes for the various version control systems, including centralized (VCS) and decentralized (DVCS).iNi(t Subcommand(tError(tRegistertBasecBsweZdZdZdZeZedZ edZ e j dZ dZee j dZdZRS( s&Setup the structure of the subclasses.tcCs||j|js -PK\!E7pyerector/vcs/__init__.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Determine the version control system being used.""" import os from ..execute import Initialization from ..variables import V from .base import Base __all__ = [ 'VCS' ] def load_plugins(): """Find modules in pyerector.vcs (that are not __init__ and base) and load them.""" from sys import modules curdir = os.path.dirname(__file__) # only get the *.py files names = [os.path.splitext(f)[0] for f in os.listdir(curdir) if f.endswith('.py') and f not in ('__init__.py', 'base.py') ] try: from importlib import import_module whereami = modules[__name__] for name in names: modname = '%s.%s' % (__name__, name) import_module(modname) except ImportError: __import__(__name__, fromlist=names) def VCS(*args, **kwargs): """Determine the type of version control and return information about it. """ load_plugins() try: basedir = V['basedir'] except KeyError: basedir = os.curdir vcs = Base.vcs_check(srcdir=basedir) if vcs.name is None: raise RuntimeError('no version control found') return vcs(*args, **kwargs) class InitVCS(Initialization): """Initialize the version control system, assigning to the pyerector.vcs variable. """ def run(self): from logging import getLogger logger = getLogger('pyerector') try: vcs = VCS() except RuntimeError: logger.info('No VCS found') else: vcs.current_info() V['pyerector.vcs'] = vcs logger.info('Found %s', vcs) InitVCS() PKJ#E?ipyerector/vcs/none.pyc CSc@sSdZddlmZddlmZdgZdefdYZejdS(sNo version control hooks.i(tVCS_Basei(tVariabletNoVCScBseZdZdZdZRS(s-Version control class for no version control.tnoneN(t__name__t __module__t__doc__tnametNonet directory(((sC/home/arcege/src/pyerector/regression/minimal/pyerector/vcs/none.pyR sN(RtbaseRt variablesRt__all__Rtregister(((sC/home/arcege/src/pyerector/regression/minimal/pyerector/vcs/none.pyts  PK=DTxpyerector/vcs/mercurial.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Mercurial(hg) based version control hooks.""" import os from ..helper import Subcommand from .base import DVCS_Base from ..variables import Variable __all__ = [ 'Mercurial' ] class Mercurial(DVCS_Base): """Version control class for Mercurial.""" name = 'mercurial' prog = 'hg' directory = '.hg' def current_info(self): """Retrieve information from the workarea.""" from logging import getLogger logger = getLogger('pyerector') proc = Subcommand( (self.prog, 'identify', '--id'), wait=True, wdir=self.rootdir, stdout=Subcommand.PIPE, stderr=os.devnull, ) hgid = proc.stdout.read().decode('UTF-8').strip() logger.debug('hgid = %s', hgid) if proc.returncode != 0: raise RuntimeError('could not retrieve Mercurial node') del proc proc = Subcommand( (self.prog, 'log', '-r', hgid.replace('+', ''), '--template', '{node|short}\n{author|email}\n{date|isodate}\n{branch}\n{tags}\n' ), wait=True, wdir=self.rootdir, stdout=Subcommand.PIPE, stderr=os.devnull, ) hgout = proc.stdout.read().decode('UTF-8') logger.debug('hgout = %s', repr(hgout)) if proc.returncode == 0: parts = hgout.rstrip().split('\n') logger.debug('parts = %s', parts) try: parts[3] except IndexError: parts.append(None) try: parts[4] except IndexError: parts.append(None) Variable('hg.version', parts[0]) Variable('hg.user', parts[1]) Variable('hg.date', parts[2]) Variable('hg.branch', parts[3]) Variable('hg.tags', parts[4]) Mercurial.register() PKND{cTTpyerector/vcs/none.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """No version control hooks.""" from .base import VCS_Base from ..variables import Variable __all__ = [ 'NoVCS', ] class NoVCS(VCS_Base): """Version control class for no version control.""" name = 'none' directory = None NoVCS.register() PKk!E{pyerector/vcs/base.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Define the base classes for the various version control systems, including centralized (VCS) and decentralized (DVCS).""" import os from ..helper import Subcommand from ..exception import Error from ..register import Register class Base(object): """Setup the structure of the subclasses.""" name = None directory = '' _register = Register() @classmethod def register(cls): """Register this class for easier retrieval later.""" cls._register[cls.name] = cls @classmethod def registered(cls): """Retrieve the registered classes.""" return sorted([cls._register[name] for name in cls._register]) def __init__(self, rootdir=os.curdir): self.rootdir = rootdir self.current_info() def __str__(self): return self.name # used by the package to see which VCS system to use @classmethod def vcs_check(cls, srcdir=os.curdir): """Check directory tree in reverse for one of the registered Base subclasses. Return the subclass that 'fits'. If the directory attribute is None, use that subclass as default.""" srcdir = os.path.realpath(os.path.normpath(srcdir)) default = None klasses = cls.registered() while srcdir not in ('', os.sep): for c in klasses: if c.directory is None: default = c elif os.path.isdir(os.path.join(srcdir, c.directory)): return c srcdir = os.path.dirname(srcdir) return default def current_info(self): """To be overridden.""" pass class VCS_Base(Base): """Base class for VCS classes (e.g. Subversion).""" prog = None def checkout(self, url, destdir=os.curdir): """Perform a checkout.""" Subcommand( (self.prog, 'checkout', url, destdir), wait=True ) def update(self, destdir=None, rev=None): """Update the workarea from the repository.""" if rev is None: revopts = () else: revopts = ('-r', str(rev)) proc = Subcommand( (self.prog, 'update') + revopts + (destdir or self.rootdir,), wait=True, stderr=Subcommand.PIPE, ) proc.wait() if proc.returncode < 0: proc.stderr.read().rstrip() raise Error(self, 'signal received %d' % abs(proc.returncode)) elif proc.returncode > 0: errput = proc.stderr.read().rstrip() raise Error(self, 'error %d: %s' % (abs(proc.returncode), errput)) class DVCS_Base(Base): """Base class for DVCS classes (e.g. Mercurial).""" prog = None def checkout(self, url, destdir=None): """Perform a checkout (clone).""" Subcommand( (self.prog, 'clone', url, destdir or self.rootdir), wait=True ) def update(self, source=None, rev=None): """Update the workarea (pull).""" if rev is None: revopts = () else: revopts = ('-r', str(rev)) if source is None: sourcearg = () else: sourcearg = (source,) proc = Subcommand( (self.prog, 'pull', '-u') + revopts + sourcearg, wait=True, wdir=self.rootdir, stderr=Subcommand.PIPE, ) proc.wait() if proc.returncode < 0: raise Error(self, 'signal received %d' % abs(proc.returncode)) elif proc.returncode > 0: errput = proc.stderr.read().rstrip() raise Error(self, 'error %d: %s' % (abs(proc.returncode), errput)) PKJ#E#ddpyerector/vcs/git.pyc ASc@sudZddlZddlmZddlmZddlmZdgZdefd YZ e Z e j dS( s Git based version control hooks.iNi(t Subcommandi(t DVCS_Base(tVariabletGitcBs)eZdZdZdZdZdZRS(sVersion control class for Git.tgits.gitcCst|jdddfdtdtjdtj}|jjjd}|j dkr|j d d ksut |j j d \}}}td |td |td ||}nd}t|jdddfdtdtjdtj}|jjjd}|j dkrxk|j tjj tjD]H}|jdrB|djdkrtd|djqqBqBWn|rt|jdd|fdtdtjdtj}|jjjd}|j dkr|jrtd|jqqndS(s'Retrieve information from the workarea.tlogs --max-count=1s--format=%h%n%ce%n%cdtwaittstdouttstderrsUTF-8is is git.versionsgit.usersgit.datetbranchs --no-colors--listt*is (no branch)s git.branchttags --containssgit.tagN(RtprogtTruetPIPEtostdevnullRtreadtdecodet returncodetcounttAssertionErrortrstriptsplitRtNonetlinesept startswithtstrip(tselftproctgitouttverstusertdatet git_versiontline((sB/home/arcege/src/pyerector/regression/minimal/pyerector/vcs/git.pyt current_infosF        %$   (t__name__t __module__t__doc__tnameR t directoryR$(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/vcs/git.pyRs ( R'RthelperRtbaseRt variablesRt__all__Rt VCS_classtregister(((sB/home/arcege/src/pyerector/regression/minimal/pyerector/vcs/git.pyts  1PKcDӸpyerector/py3/execfile.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Redefine the execfile in Python 3.x, which was removed.""" import sys __all__ = [ 'execfile' ] def execfile(filename, globals=None, locals=None): """Reproduce the Python2 execfile() function.""" if globals is None: globals = sys._getframe(1).f_globals if locals is None: locals = sys._getframe(1).f_locals with open(filename, 'rt') as filehandle: exec(filehandle.read()+"\n", globals, locals) PKSCiW:RRpyerector/py3/__init__.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. PKcD{x͏pyerector/py3/base.py#!/usr/bin/python # Copyright @ 2012-2013 Michael P. Reilly. All rights reserved. """Define the underlying base class for Initer as a metaclass instance for Python 3.x. The syntax for metaclass declarations is different between Python 2.x and Python 3.x.""" from ..metaclass import IniterMetaClass class Base(metaclass = IniterMetaClass): """Baseclass with metaclass declaration.""" pass PKC"EdSyy release.pyPK:"E 6"6" CHANGES.txtPKcD%oo #README.txtPKSC큗)setup.pyPKSC|wfKK R/LICENSE.txtPKD\'E"_ggƸEGG-INFO/PKG-INFOPKD\'E2\EGG-INFO/dependency_links.txtPKD\'E2EGG-INFO/zip-safePKD\'Eu-v ȼEGG-INFO/top_level.txtPKD\'EEGG-INFO/SOURCES.txtPKJ#E6V V pyerector/config.pycPKJ#E S^|,|,ipyerector/variables.pycPKcDї5pyerector/api.pyPKJ#E;ׂ'pyerector/tasks.pycPKJ#EX{Q''8pyerector/__init__.pycPKcDS S pyerector/iterators.pyPK!Ec pyerector/exception.pyPKJ#E B9pb&b&pyerector/iterators.pycPKcDpvpyerector/config.pyPKcDD%~MM.pyerector/metaclass.pyPK E0N]t"8"8pyerector/helper.pyPK˦!E* ?pyerector/tasks.pyPKJ#E5? ? pyerector/register.pycPKcD5ybpyerector/variables.pyPKcDZPTTpyerector/register.pyPKq"Eq]]pyerector/targets.pyPKJ#Eee pyerector/base.pycPKM'EhQ  Yspyerector/version.pyPKJ#E@ ~pyerector/exception.pycPKJ#E:g ! !}pyerector/execute.pycPKJ#E?)aapyerector/main.pycPK3oDszMpyerector/__init__.pyPKJ#E"Qpyerector/metaclass.pycPKJ#EG@'(D(Dzpyerector/helper.pycPK#E0 "pyerector/unit-test.pyPKJ#EbwCpyerector/version.pycPKL[&E) FTpyerector/main.pyPKJ#E&0(==upyerector/targets.pycPK!EYppyerector/execute.pyPK!Evk_k_pyerector/base.pyPK#'EpS S ,pyerector/version.py.origPKJ#EIVpyerector/py2/__init__.pycPKJ#E,(pyerector/py2/base.pycPKSCiW:RRpyerector/py2/__init__.pyPKcDfMgpyerector/py2/base.pyPKJ#E5= = pyerector/vcs/__init__.pycPKJ#Epn=S#pyerector/vcs/subversion.pycPKcDL[]q,pyerector/vcs/git.pyPK/#EZ`4pyerector/vcs/subversion.pyPKJ#EY  =pyerector/vcs/mercurial.pycPKJ#E,{{Gpyerector/vcs/base.pycPK\!E7[pyerector/vcs/__init__.pyPKJ#E?ibpyerector/vcs/none.pycPK=DTxepyerector/vcs/mercurial.pyPKND{cTTmpyerector/vcs/none.pyPKk!E{opyerector/vcs/base.pyPKJ#E#dd~pyerector/vcs/git.pycPKcDӸpyerector/py3/execfile.pyPKSCiW:RR߈pyerector/py3/__init__.pyPKcD{x͏hpyerector/py3/base.pyPK<<*