{
    "info": {
        "author": "Markus Rother",
        "author_email": "python@markusrother.de",
        "bugtrack_url": null,
        "classifiers": [
            "Environment :: Console",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
            "Operating System :: Unix",
            "Programming Language :: Python :: 3.2",
            "Topic :: Software Development",
            "Topic :: Software Development :: Code Generators",
            "Topic :: Software Development :: Compilers",
            "Topic :: Software Development :: Debuggers",
            "Topic :: Software Development :: Testing",
            "Topic :: Utilities"
        ],
        "description": ".. _start:\n\n=====================\nREADME for ptTools.py\n=====================\n\n.. _intro:\n\nThe ptTools packages provides a domain specific language, for\npattern matching on trees - python parsetrees in particular.\n\nMatching parsetree nodes instead of regular expression based string\nsearch, allows fine-grained, semantic control over search results.\nThe second major benefit is that the matches are instantiated\nparsetree nodes, that can be manipulated and recompiled.  Parsetree\nbased search is of interest especially when manipulating or\nsubstituting selected nodes, e.g. for testing, or code analysis.\n\nThe domain specific language of ptTools patterns consists of a family\nof pattern objects, all of which can be expressed in valid python\ncode.  Binary and unary operators similar to those of regular\nexpressions are available. The resulting pattern language resembles\nthe grammar rules it matches.\n\nEmphasis was put on an intuitive interface, and a clean and easily\nextensible object-oriented design.  The components are loosely\ncoupled.  For example the pattern objects can be matched against\ntuples as well as against attributed parsetree nodes.  The parsetree\nnodes may or may not hold references to tokens, which again may or may\nnot be attributed.\n\n\n\n\n.. _contents:\n\n**Contents**\n\n    * Two simple examples_\n    * Description of the available operators_ (**TODO**)\n    * References_\n    * License_\n\n..  * A few implementation_ notes (**TODO**)\n..  * A todo_ list (**TODO**)\n\n\n.. _location:\n\nptTools was written by Markus Rother (python@markusrother.de) and is\nlocated at http://github.com/941design/pttools.  Feedback and\nsuggestions are welcome.  ptTools requires python3.2.\n\nThe object-oriented design of the RegularLanguageAcceptor classes in\n``ptTools.misc.acceptors`` was inspired by Lukas Renggli's `Petit\nParser <http://www.lukas-renggli.ch/blog/petitparser-1>`_ written in\nSmalltalk.\n\n.. _examples:\n.. _`previous example`:\n.. _`for_stmt`:\n\nA first example - Creating a simple pattern\n===========================================\n\n The following first example matches a for loop in a python source file.\n We will first `create the parsetree`_ as a nested tuple, then\n `create the pattern`_, and finally we will `match the pattern`_\n against the parsetree.\n\n Our starting point is the corresponding grammar rule, taken from\n http://docs.python.org/3.2/reference/grammar.html:\n ::\n\n    for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]\n\n Lowercase expressions are non-terminals that can be imported from the\n ``symbol`` module.  Strings represent terminals with a string value.\n Bracketed sub-rules are optional.  For the sake of simplicity we\n disregard the else option in our pattern.  Also, we will replace the\n rules ``exprlist``, ``testlist`` and ``suite`` by wildcards.  \n\n\n.. _`create the parsetree`:\n\nCreating the parse tree\n~~~~~~~~~~~~~~~~~~~~~~~\n\n Here, the parse tree is created from a short source segment.  The\n result is a nested tuple of the form:\n ::\n\n    node ::= (INT (, (node|STR))*)\n\n >>> import parser\n ...\n >>> source = '''\n ... for k in range(10):\n ...     print(k)\n ... '''\n >>> root = parser.suite(source).totuple()\n\n Note, that the ``for_stmt`` non-terminal seen in the rule above, is\n not the grammar's start symbol, but will be nested somewhere in the\n tuple.\n\n\n.. _`create the pattern`:\n\nCreating a pattern\n~~~~~~~~~~~~~~~~~~\n\n Next, we create a pattern that matches any for-statement without the\n else option.  As a universal wildcard pattern, we instantiate an\n ``AnyPattern()`` and assign it to the underscore.  The ``TuplePattern`` is\n - as the name suggests - a pattern that matches tuples.  It may hold\n other patterns or literal values.\n \n >>> from symbol import for_stmt\n >>> from ptTools.ptpatterns import TuplePattern as Tup\n >>> from ptTools.ptpatterns import AnyPattern as Any\n >>> _ = Any()\n ...\n >>> pattern = Tup(for_stmt, 'for', _, 'in', _, ':', _)\n >>> pattern\n (296, (1, 'for'), _, (1, 'in'), _, (11, ':'), _)\n\n where the integers originate from the ``symbol`` and ``token`` modules.\n\n >>> import symbol\n >>> import token\n >>> 296==symbol.for_stmt and 1==token.NAME and 11==token.COLON\n True\n\n Repeated for comparison:  The rule\n ::\n\n    for_stmt: 'for' exprlist 'in' testlist ':' suite\n\n is matched by:\n ::\n\n    ```TuplePattern(for_stmt, 'for', _, 'in', _, ':', _)```\n\n represented as:\n ::\n\n    ```(296, (1, 'for'), _, (1, 'in'), _, (11, ':'), _)```\n\n Note, that string literals in the pattern declaration are replaced by\n ``TuplePattern`` instances matching the appropriate parsetree nodes.\n If a token  exists, whose string value equals the given string, that\n token's identifier is inserted, otherwise the identifier defaults to\n ``token.NAME``.\n In the above example ``':'`` was replaced by\n ``TuplePattern(token.COLON, ':')``, because the colon exists as a token.\n The fact that ``in`` and ``for`` are keywords, does not mean they do\n exist as tokens, too.\n\n\n.. _`match the pattern`:\n\nMatching the pattern\n~~~~~~~~~~~~~~~~~~~~\n\n As mentioned earlier, the pattern does not match the entire parsetree.\n\n >>> pattern.matches(root)\n False\n\n It does however match somewhere in the parsetree.\n\n >>> pattern.matches_in(root)\n True\n >>> match = pattern.first_match_in(root)\n >>> isinstance(match, tuple)\n True\n\n We can retrieve the matched node, but formatting was lost during\n parsing:\n\n >>> from ptTools.parsetree import tupletraversal \n >>> ' '.join([s for s in tupletraversal.leaves(match)])\n 'for k in range ( 10 ) :   print ( k )  '\n\n\n.. _`another example`:\n.. _`classdef`:\n\nExample two - Processing matched nodes\n======================================\n\n This example demonstrates a slightly more complex pattern\n description, that matches any class definition in python source.  The\n example also shows how to capture matched regions, and apply\n functions to subpatterns.\n\n\nPattern definition\n~~~~~~~~~~~~~~~~~~\n\n >>> from symbol import classdef\n >>> from token import NAME\n >>> import ptTools\n >>> from ptTools.ptpatterns import TuplePattern as Tup\n >>> from ptTools.ptpatterns import _\n ...\n >>> pattern = Tup(classdef, 'class', (NAME, _), ['(', [ _ ], ')'], ':', _)\n\n Compare this pattern to the grammar rule definition of ``classdef`` taken from\n http://docs.python.org/3.2/reference/grammar.html:\n :: \n\n  classdef: 'class' NAME ['(' [arglist] ')'] ':' suite\n\n Unless we are interested in the class definition's body, we may\n replace the pattern's tail with another wildcard:\n\n >>> more = ~_ ## Creating a Kleene Closure over Any().\n ...\n >>> pattern = Tup(classdef, 'class', (NAME, _), more)\n\n..  (Under operators, see `Kleene closure`_).\n\n\nAdding callback functions\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n To send the matched regions to callback functions, simply append '``%\n callable``' to a pattern.  For instance, if we would like to print\n the class name from the above example, we would simply write:\n ::\n\n    ```pattern = Tup(classdef, 'class', (NAME, _%print), more)```\n\n Note, that the left hand side of the ``%`` operator must have been\n evaluated to a PatternObject prior to evaluating the entire\n expression.  If we would like to print the entire terminal node, we\n have to be explicit, and type:\n ::\n\n    ```pattern = Tup(classdef, 'class', Tup(NAME, _)%print, more)```\n\n Now, assume, one would like to count classes in a module.  For that\n purpose we define an ``inc`` function with the side effect of\n incrementing a counter, and which is to be called every time our\n classdef pattern matches.  All we have to do is to add a callback to\n ``inc`` to the pattern with the ``%`` operator.\n\n >>> count = 0\n >>> def inc(ignore):\n ...     globals()['count'] += 1\n ...\n >>> pattern = Tup(classdef, 'class', (NAME, _%print), more)%inc\n\n Note, that this does two things: It prints the classname *and then*\n increments the counter.  The order of the two is determined by the\n traversal order through our parsetree (which is preorder, and hence\n lexical order of the source code).  In other words, ``print`` will be\n called before ``inc``, because the class name is encountered before\n the entire class definition ends.\n\n Callback functions can be chained (e.g. '``%inc%input``'), and are\n executed in the order of appending.\n \n\nApplying patterns\n~~~~~~~~~~~~~~~~~\n\n As seen in the `previous example`_, PatternObjects have a\n ``.match(node)`` method, that tries to match a parsetree node, and if\n successful applies its callback functions to the matched regions.\n Its return value is ``None``.\n\n We now appply that method to every node in the parsetree, starting at\n its root.  If the pattern matches in a node, all queued callback\n functions will be called with the matched node as their only\n argument.\n \n >>> cls_lst = []\n ...\n >>> pattern = Tup(classdef, 'class', (NAME, _%cls_lst.append), more)%inc\n >>> pattern\n (329, (1, 'class'), (1, _), ~(_))\n \n >>> root = ptTools.parsetree.fromfile('ptpatterns/ptpatterns.py')\n >>> root.preorder_do(pattern.match)\n ...\n >>> 'TuplePattern' in cls_lst\n True\n\n Currently, five classes are defined in ``ptTools.ptpatterns.ptpatterns``.\n\n >>> count\n 5\n\n\n.. _`more examples`:\n\nMore examples\n=============\n\n Also, see ``ptTools.examples``:\n\n * ``ptTools.examples.pycat`` is a colorizer printing python source\n   code to the command line.  Performance is worse than that of e.g.\n   `pygmentize <http://pygments.org/>`_.  However, it may be a good starting point when\n   designing patterns and visualizing their matches in a source file.\n  \n * ``ptTools.examples.extract`` demonstrates recompilation of a\n   captured segment.\n\n Both examples are executable.\n\n\n.. _operators:\n\nAvailable operators\n===================\n\n **TODO**\n \n Please refer to the examples_.  (Also, see\n ``ptTools.misc.acceptors``).\n\n\n.. _references:\n\nReferences\n==========\n\n  * For the complete python grammar see:\n    http://docs.python.org/3.2/reference/grammar.html\n\n  * Nonterminals can be found in: ``/usr/include/python3.2/graminit.h`` as\n    well as in the module: ``symbol``.\n  \n  * Reference for terminals is found in the modules: ``token`` and\n    ``tokenize``.\n\n\n.. _license:\n\nLicense\n=======\n\n This program was published under the `GNU GENERAL PUBLIC LICENSE\n <http://www.gnu.org/licenses/gpl-3.0.txt>`_.",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://pypi.python.org/pypi/ptTools",
        "keywords": null,
        "license": "GNU-GPL",
        "maintainer": null,
        "maintainer_email": null,
        "name": "ptTools",
        "package_url": "https://pypi.org/project/ptTools/",
        "platform": "Linux",
        "project_url": "https://pypi.org/project/ptTools/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "http://pypi.python.org/pypi/ptTools"
        },
        "release_url": "https://pypi.org/project/ptTools/0.1.4/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "DSL for pattern matching on .py parse trees.",
        "version": "0.1.4"
    },
    "last_serial": 832108,
    "releases": {
        "0.1.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "31fa14f124263f602a923df6d2a31b17",
                    "sha256": "aefbd9b0272acbb2374fb3148160f1d5bd9953bd6e0d4ad6d11df388049a943e"
                },
                "downloads": -1,
                "filename": "ptTools-0.1.4.tar.gz",
                "has_sig": false,
                "md5_digest": "31fa14f124263f602a923df6d2a31b17",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 60475,
                "upload_time": "2013-07-31T19:22:53",
                "url": "https://files.pythonhosted.org/packages/4b/ca/93623a32dc0c2a75da2b10f3d162a7bb55a00b44981f99933f0c77f0dc38/ptTools-0.1.4.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "31fa14f124263f602a923df6d2a31b17",
                "sha256": "aefbd9b0272acbb2374fb3148160f1d5bd9953bd6e0d4ad6d11df388049a943e"
            },
            "downloads": -1,
            "filename": "ptTools-0.1.4.tar.gz",
            "has_sig": false,
            "md5_digest": "31fa14f124263f602a923df6d2a31b17",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 60475,
            "upload_time": "2013-07-31T19:22:53",
            "url": "https://files.pythonhosted.org/packages/4b/ca/93623a32dc0c2a75da2b10f3d162a7bb55a00b44981f99933f0c77f0dc38/ptTools-0.1.4.tar.gz"
        }
    ]
}