{
    "info": {
        "author": "asrp",
        "author_email": "asrp@email.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# pymetaterp\n\nThis is a python AST builder that uses no Python modules. A longer stackless version is available for easier porting. `single_file.py` is a stand-alone 502 lines script.\n\nIts (also) just another parsing expression grammar (PEG) based parser with one major difference. The parsed grammar is interpreted instead of compiled. This makes it easy to modify the language (by editing its grammar) *as well as* the language that grammar is written in (and the language of *that* grammar).\n\n[This is a **pre-release** of sorts. There are probably some errors and missing information.]\n\n## Download and run\n\n    git clone https://github.com/asrp/pymetaterp\n    cd pymetaterp\n    python single_file.py\n\nor\n\n    python single_file.py filename.py\n\nThis will print out the AST of the given file (or `single_file.py`'s own AST). Sample beginning of the output:\n\n    file_input\n      regular_assign\n        testlist\n          NAME\n            str 'NAME'\n          NAME\n            str 'FLAGS'\n\nTo run files from the library\n\n    python test/boot_test.py \n    python test/python_parse_test.py \n\n## Files\n\n`single_file.py` is mainly for demonstration. This module is otherwise separated into files. There are many files but they are mostly separate. The import dependencies is\n\n    util.py\n      boot.py\n      boot_stackless.py\n        python.py\n\nOther files have no imports. To get something useful, you'll have to import multiple files. See `test/python_parse_test.py` and `test/boot_test.py` for some examples.\n\n## Repl\n\nAn obvious thing *missing* is the grammar read-eval-print loop (repl) so the interpreter can be fed one rule at a time, parsing subsequence input using the rules seen so far.\n\n## Source reading order\n\nI'd suggest reading `boot.py` and `bootstrap` in `boot_grammar.py` first. The two form the core and together with `boot_tree.py`, they can regenerate `boot_tree`.\n\nThen `boot_stackless` is the same as `boot.py` but doesn't use the Python call stack/recursion for parsing.\n\n`python.py` adds functionality to the `boot.py` interpreter. `diff` in `boot_grammar.py` adds the syntax for those.\n\nFinally, `python_grammar.py` contains the python grammar to be finally parsed.\n\n## Python language parsed\n\nThe module builds the AST for Python 2.x programs. It is able to parse all of Python 2.x (in fact, it contains a slightly modified version of the Python 2.x grammar) but is less lenient with whitespaces. For example, parsing\n\n    from my_module import (var1, var2,\n                           var3, var4)\n\ngives an error.\n\n*Since this is a pre-release, there are likely bugs with parts of the language I don't use so often. It _can_ build the AST for all files included here.* \n\n## Gramamr language differences\n\nThe beginning of `boot_grammar.py` self-describes the grammar. Its a PEG so all \"or\" (`|`) returns the first match and \"and\" and \"quantified\" (`*, +, ?`) are greedy.\n\n    name = (letter | '_') (letter | digit | '_')*\n    expr = apply | exactly | token | parenthesis | output\n\n    exactly! = \"'\" {(escaped_char | ~'\\'' anything)*} \"'\"\n    token! = \"\\\"\" {(escaped_char | ~'\"' anything)*} \"\\\"\"\n    escaped_char! = '\\\\' {'n'|'r'|'t'|'b'|'f'|'\"'|'\\''|'\\\\'}\n    apply! = ('\\t'|' ')* {name}\n    parenthesis = \"(\" {or} \")\"\n    output! = \"{\" {or} \"}\"\n\n    not = \"~\" {expr=negation} | expr\n    quantified = not (('*' | '+' | '?')=quantifier)?\n    bound = quantified ('=' {name=inline})?\n    and = bound*\n    or = and (\"|\" {and})*\n\n    rule = spaces {name=rule_name '!'?=flags and=args (\"=\" {or})}\n    grammar = {rule*} spaces\n\nThe main difference from other PEG.\n\n- output rule: `a {b c} d` will match the concatenation of `a b c d` but only return what matched `b c`.\n- quantifier collapse: `letter letter*` returns a list rather than a pair with the second element being a list matching `letter*`.\n- nested and collapse: `a (b (c d)) e` has the same output as `a b c d e` (see inline below if some pairs need to be explicitly grouped).\n- node collapsing: nodes in the output with only one child are replaced by their parent, unless the `!` (\"don't collapse\") flag is set for that node.\n- inline: shamelessly taken [Ohm](https://github.com/ohm) but with a slightly different interpretation. `expression=name` creates a node named `name` wrapping the output of `expression`.\n- rule replacement: having a second `rule_name = expression` line replaces the first definition of `rule_name` (instead of appending into an or).\n- two basic tokens: there are two basic token types: `'a'` (single quote) and `\"a\"` (double quote). The double quoted token allows whitespace before matching.\n\n## Regenerating boot_tree.py\n\nCreate some tree `match_tree` using `Interpreter.match` and call `save` on the result.\n\n    match_tree.save(\"tree.py\")\n\n## Left recursion\n\nWhile \"[PEG/packrat parsers can support left-recursion]((http://www.vpri.org/pdf/tr2007002_packrat.pdf))\", the tree output isn't the one we want. The python functions `reformat_binary` and `reformat_atom` fixes a parsed tree's ouput.\n\n## Source oddities\n\n### Two hard-coded rules\n\n            if root[NAME] == \"anything\":\n                return pop(self.input)\n            elif root[NAME] == \"void\":\n                return\n\n### Hard-coded semantics for tokens\n\n            if name == \"token\":\n                while pop(self.input) in self.whitespace:\n                    if self.input[0][self.input[1]] == '\\\\':\n                        pop(self.input)\n                self.input[1] -= 1\n            if name == \"token\" and root[0].isalpha():\n                top = pop(self.input)\n                if top.isalnum() or top == '_':\n                    raise MatchError(\"Prefix matched but didn't end.\")\n                self.input[1] -= 1\n\n## Optimization\n\nSome effort were made to make these files short (especially `single_file.py`) but not too much. There are still some asserts around and commented print statements that can be useful for debugging. The final goal is, of course, to reduce the program's complexity and verbosity, not its line count.\n\n## Missing features\n\nFeatures/bloat from a longer version of this program not (yet?) moved over:\n\n- Debugging tree of nodes visited and their input and output\n- Function arguments (its in the grammar but not the interpreter)\n- Nested list inputs (its also in the grammar but not the interpreter)\n- name, args, flags, body as parameters instead of positional children\n- ~~Memoization~~\n- ~~Matched input start and end positions~~\n- Exact python expression matching for predicate, action and rule value. `balanced` is used as a simpler heuristic for now.\n\n## Removing features\n\nTo get a smaller file with just the basics.\n\n    patch -R pymetaterp/python.py < patches/python_pos.patch\n    patch -R pymetaterp/python.py < patches/python_memoizer.patch\n    patch -R pymetaterp/boot_stackless.py < patches/boot_pos.patch\n    patch -R pymetaterp/boot_stackless.py < patches/boot_memoizer.patch\n\n## Readings\n\n- [Ometa](http://www.tinlizzie.org/ometa/) - Warth's thesis reads very well.\n- [PEG and packrat parser](http://bford.info/packrat/)\n- [Packrat Parsers Can Support Left Recursion](http://www.vpri.org/pdf/tr2007002_packrat.pdf)\n\n## Other similar projects\n\n- [parsimonious](https://github.com/erikrose/parsimonious)\n- [Pymeta](https://pypi.python.org/pypi/PyMeta/)\n- [pyparsing](http://pyparsing.wikispaces.com/)\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/asrp/pymetaterp",
        "keywords": "parser peg python minimal",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "pymetaterp",
        "package_url": "https://pypi.org/project/pymetaterp/",
        "platform": "",
        "project_url": "https://pypi.org/project/pymetaterp/",
        "project_urls": {
            "Homepage": "https://github.com/asrp/pymetaterp"
        },
        "release_url": "https://pypi.org/project/pymetaterp/1.1/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "A python parser that builds python ASTs in 502 lines of python without using modules",
        "version": "1.1"
    },
    "last_serial": 4340234,
    "releases": {
        "1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2764d4cb9f91d68514a5326a3f7fba95",
                    "sha256": "f465b1007d925b847690f20b78ffe0a9981469cf85c3c724c80b10bdbcadfbbd"
                },
                "downloads": -1,
                "filename": "pymetaterp-1.0-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "2764d4cb9f91d68514a5326a3f7fba95",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 20071,
                "upload_time": "2018-10-04T13:05:08",
                "url": "https://files.pythonhosted.org/packages/d2/72/95ec8b52580d2eb305e951bfcf7cf715853763021a077f738aa59a44c87f/pymetaterp-1.0-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e6732454ccaf7d47cdd043a5cbef9449",
                    "sha256": "71d50e0a660802dea22b765cb0aa40fa19e0229d1545d8f851010f3a0daa142d"
                },
                "downloads": -1,
                "filename": "pymetaterp-1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "e6732454ccaf7d47cdd043a5cbef9449",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15371,
                "upload_time": "2018-10-04T12:49:45",
                "url": "https://files.pythonhosted.org/packages/53/5a/40c1431bd5813ebae4c7322d1aa248da992a59222196505749414d9e36b6/pymetaterp-1.0.tar.gz"
            }
        ],
        "1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "959edae9e9bf36216a550e0c2930de77",
                    "sha256": "3368cd8812adf7cfec0dcab54d334f2cd452ba090f39729e14a6016e800ef0d3"
                },
                "downloads": -1,
                "filename": "pymetaterp-1.1-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "959edae9e9bf36216a550e0c2930de77",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 20070,
                "upload_time": "2018-10-04T13:13:29",
                "url": "https://files.pythonhosted.org/packages/33/60/ef0cb3f6f9018fa566bde6812cf18790900a0a19078d60c2fe475cd2fdb1/pymetaterp-1.1-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "b2c2f8aeacca24b6f4d9c5055c152ad1",
                    "sha256": "3bf783acb81b735367e33b79c7fe94d6babb7325621cc2cf43d4429c0ce3673b"
                },
                "downloads": -1,
                "filename": "pymetaterp-1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "b2c2f8aeacca24b6f4d9c5055c152ad1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15224,
                "upload_time": "2018-10-04T13:13:43",
                "url": "https://files.pythonhosted.org/packages/19/77/6456a03deb466bd253db3fe3982fbb53e54cb7736457744fd80620169b93/pymetaterp-1.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "959edae9e9bf36216a550e0c2930de77",
                "sha256": "3368cd8812adf7cfec0dcab54d334f2cd452ba090f39729e14a6016e800ef0d3"
            },
            "downloads": -1,
            "filename": "pymetaterp-1.1-py2-none-any.whl",
            "has_sig": false,
            "md5_digest": "959edae9e9bf36216a550e0c2930de77",
            "packagetype": "bdist_wheel",
            "python_version": "py2",
            "requires_python": null,
            "size": 20070,
            "upload_time": "2018-10-04T13:13:29",
            "url": "https://files.pythonhosted.org/packages/33/60/ef0cb3f6f9018fa566bde6812cf18790900a0a19078d60c2fe475cd2fdb1/pymetaterp-1.1-py2-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "b2c2f8aeacca24b6f4d9c5055c152ad1",
                "sha256": "3bf783acb81b735367e33b79c7fe94d6babb7325621cc2cf43d4429c0ce3673b"
            },
            "downloads": -1,
            "filename": "pymetaterp-1.1.tar.gz",
            "has_sig": false,
            "md5_digest": "b2c2f8aeacca24b6f4d9c5055c152ad1",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 15224,
            "upload_time": "2018-10-04T13:13:43",
            "url": "https://files.pythonhosted.org/packages/19/77/6456a03deb466bd253db3fe3982fbb53e54cb7736457744fd80620169b93/pymetaterp-1.1.tar.gz"
        }
    ]
}