{
    "info": {
        "author": "David B. Curtis",
        "author_email": "davecurtis@sonic.net",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "==============\ntokenizertools\n==============\n\nText file tokenizers that support multiple start states\nand lexical tracking implemented as standard Python iterators.\n\nClasses provided:\n\n* Tokenizer -- Base class.\n* RegexTokenizer -- Python re-based tokenizer.\n* TokenizeAhead -- A look-ahead iterator that can wrap any tokenizer.\n\nOverview\n--------\n\nThe class ``RegexTokenizer`` implements a tokenizer using the ``re`` \nmodule to recognize tokens in the input stream.\nTokens and actions are defined by rules.\nThe tokenizer calls user action functions associated with each rule.\nIn most cases, the user action function can simply be a ``@classmethod``\nconstructor of a user-provided token class.\n\nEach rule is specified as a tuple.\nThe first element of the tuple is a regular expression that will\nbe compiled by ``re`` and used to match a token.\nThe second element of the tuple is a user-provided callable that \nwill be passed\nthe recognized text, along with the current lexical position.\n\nIn this example, the user class ``Token`` implements constructors\nas ``@classmethod`` functions, and these serve as the callables in\neach lexical rule.\n\nRules are specified in the class variable ``spec``, which is a\nlist of rules. ::\n\n  import tokenizertools as tt\n  class MyTokenizer(tt.RegexTokenizer):\n      spec = [\n          (r'[a-zA-Z][a-zA-Z0-9_]*',Token.type_ident), # idents and keywords\n          (r'[0-9]+\\.[0-9]+',Token.type_float), # floats\n          (r'[0-9]+', Token.type_int), # ints\n          (r'\\s*',None), # ignore white space\n      ]\n\nNothing else needs to be defined.  All methods are inherited.\nInstantiate a lexer and commence parsing.\nThe specification rules are compiled and cached on creation\nof the first instance.::\n\n  tokenizer = MyTokenizer()\n  with open('foo.bar') as f:\n      token_stream = Lookahead(tokenizer.lex(f, f.name))\n      compiled_stuff = my_parser.parse(token_stream)\n\n===================\ntokenizertools news\n===================\n\nRelease history.\n\n15-Sept-2016\n------------\n\n1.0 Time to put some mileage on it.\n\nClean ups, PEP8-ification. \nImproved exception handling for bad ``begin()`` state.\n\n16-June-2014\n------------\n\n0.1a Initial alpha.",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://github.com/dbc/tokenizertools",
        "keywords": null,
        "license": "BSD",
        "maintainer": null,
        "maintainer_email": null,
        "name": "tokenizertools",
        "package_url": "https://pypi.org/project/tokenizertools/",
        "platform": "any",
        "project_url": "https://pypi.org/project/tokenizertools/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "http://github.com/dbc/tokenizertools"
        },
        "release_url": "https://pypi.org/project/tokenizertools/1.0/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "Implements lexical analyzers as iterators yielding tokens.",
        "version": "1.0"
    },
    "last_serial": 2346676,
    "releases": {
        "0.1a": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "cdae97ab40f82c6c818e86a498413443",
                    "sha256": "bd0238545f7275b670d71ce3fd899c8b5a81febb0faefcf2d4052ad20d140301"
                },
                "downloads": -1,
                "filename": "tokenizertools-0.1a.tar.gz",
                "has_sig": false,
                "md5_digest": "cdae97ab40f82c6c818e86a498413443",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10139,
                "upload_time": "2014-06-17T04:44:16",
                "url": "https://files.pythonhosted.org/packages/41/b8/a06793517fc55f0865cb47f2e6d93c62bb0f980684a5f807eca59608a32a/tokenizertools-0.1a.tar.gz"
            }
        ],
        "1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "944878faa5c1859e3db1b1301592cfdb",
                    "sha256": "44074a442b456c48918ee99cb09ff9b7fccc3708f94e13d13d0ff637daf3f7f7"
                },
                "downloads": -1,
                "filename": "tokenizertools-1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "944878faa5c1859e3db1b1301592cfdb",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10858,
                "upload_time": "2016-09-16T17:13:16",
                "url": "https://files.pythonhosted.org/packages/42/81/3efe92537748299e3cd7cbfe519819855d17a6aecd92bb652d2310496bb7/tokenizertools-1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "944878faa5c1859e3db1b1301592cfdb",
                "sha256": "44074a442b456c48918ee99cb09ff9b7fccc3708f94e13d13d0ff637daf3f7f7"
            },
            "downloads": -1,
            "filename": "tokenizertools-1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "944878faa5c1859e3db1b1301592cfdb",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 10858,
            "upload_time": "2016-09-16T17:13:16",
            "url": "https://files.pythonhosted.org/packages/42/81/3efe92537748299e3cd7cbfe519819855d17a6aecd92bb652d2310496bb7/tokenizertools-1.0.tar.gz"
        }
    ]
}