{
    "info": {
        "author": "Anthony Sottile",
        "author_email": "asottile@umich.edu",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python :: 2",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7",
            "Programming Language :: Python :: Implementation :: CPython",
            "Programming Language :: Python :: Implementation :: PyPy"
        ],
        "description": "[![Build Status](https://dev.azure.com/asottile/asottile/_apis/build/status/asottile.tokenize-rt?branchName=master)](https://dev.azure.com/asottile/asottile/_build/latest?definitionId=25&branchName=master)\n[![Azure DevOps coverage](https://img.shields.io/azure-devops/coverage/asottile/asottile/25/master.svg)](https://dev.azure.com/asottile/asottile/_build/latest?definitionId=25&branchName=master)\n\ntokenize-rt\n===========\n\nThe stdlib `tokenize` module does not properly roundtrip.  This wrapper\naround the stdlib provides two additional tokens `ESCAPED_NL` and\n`UNIMPORTANT_WS`, and a `Token` data type.  Use `src_to_tokens` and\n`tokens_to_src` to roundtrip.\n\nThis library is useful if you're writing a refactoring tool based on the\npython tokenization.\n\n## Installation\n\n`pip install tokenize-rt`\n\n## Usage\n\n### datastructures\n\n#### `tokenize_rt.Offset(line=None, utf8_byte_offset=None)`\n\nA token offset, useful as a key when cross referencing the `ast` and the\ntokenized source.\n\n#### `tokenize_rt.Token(name, src, line=None, utf8_byte_offset=None)`\n\nConstruct a token\n\n- `name`: one of the token names listed in `token.tok_name` or\n  `ESCAPED_NL` or `UNIMPORTANT_WS`\n- `src`: token's source as text\n- `line`: the line number that this token appears on.  This will be `None` for\n   `ESCAPED_NL` and `UNIMPORTANT_WS` tokens.\n- `utf8_byte_offset`: the utf8 byte offset that this token appears on in the\n  line.  This will be `None` for `ESCAPED_NL` and `UNIMPORTANT_WS` tokens.\n\n#### `tokenize_rt.Token.offset`\n\nRetrieves an `Offset` for this token.\n\n### converting to and from `Token` representations\n\n#### `tokenize_rt.src_to_tokens(text: str) -> List[Token]`\n\n#### `tokenize_rt.tokens_to_src(Iterable[Token]) -> str`\n\n### additional tokens added by `tokenize-rt`\n\n#### `tokenize_rt.ESCAPED_NL`\n\n#### `tokenize_rt.UNIMPORTANT_WS`\n\n### helpers\n\n#### `tokenize_rt.NON_CODING_TOKENS`\n\nA `frozenset` containing tokens which may appear between others while not\naffecting control flow or code:\n- `COMMENT`\n- `ESCAPED_NL`\n- `NL`\n- `UNIMPORTANT_WS`\n\n#### `tokenize_rt.parse_string_literal(text: str) -> Tuple[str, str]`\n\nparse a string literal into its prefix and string content\n\n```pycon\n>>> parse_string_literal('f\"foo\"')\n('f', '\"foo\"')\n```\n\n#### `tokenize_rt.reversed_enumerate(Sequence[Token]) -> Iterator[Tuple[int, Token]]`\n\nyields `(index, token)` pairs.  Useful for rewriting source.\n\n#### `tokenize_rt.rfind_string_parts(Sequence[Token], i) -> Tuple[int, ...]`\n\nfind the indices of the string parts of a (joined) string literal\n\n- `i` should start at the end of the string literal\n- returns `()` (an empty tuple) for things which are not string literals\n\n```pycon\n>>> tokens = src_to_tokens('\"foo\" \"bar\".capitalize()')\n>>> rfind_string_parts(tokens, 2)\n(0, 2)\n>>> tokens = src_to_tokens('(\"foo\" \"bar\").capitalize()')\n>>> rfind_string_parts(tokens, 4)\n(1, 3)\n```\n\n## Differences from `tokenize`\n\n- `tokenize-rt` adds `ESCAPED_NL` for a backslash-escaped newline \"token\"\n- `tokenize-rt` adds `UNIMPORTANT_WS` for whitespace (discarded in `tokenize`)\n- `tokenize-rt` normalizes string prefixes, even if they are not parsed -- for\n  instance, this means you'll see `Token('STRING', \"f'foo'\", ...)` even in\n  python 2.\n- `tokenize-rt` normalizes python 2 long literals (`4l` / `4L`) and octal\n  literals (`0755`) in python 3 (for easier rewriting of python 2 code while\n  running python 3).\n\n## Sample usage\n\n- https://github.com/asottile/add-trailing-comma\n- https://github.com/asottile/future-fstrings\n- https://github.com/asottile/pyupgrade\n- https://github.com/asottile/yesqa\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/asottile/tokenize-rt",
        "keywords": "",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "tokenize-rt",
        "package_url": "https://pypi.org/project/tokenize-rt/",
        "platform": "",
        "project_url": "https://pypi.org/project/tokenize-rt/",
        "project_urls": {
            "Homepage": "https://github.com/asottile/tokenize-rt"
        },
        "release_url": "https://pypi.org/project/tokenize-rt/3.2.0/",
        "requires_dist": [
            "typing ; python_version == \"2.7\""
        ],
        "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
        "summary": "A wrapper around the stdlib `tokenize` which roundtrips.",
        "version": "3.2.0"
    },
    "last_serial": 5495978,
    "releases": {
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a54e5a3d7c19bfe21f2fa3a8b4953e70",
                    "sha256": "d4ee182772f863c2279ba335bd1e00b048ed2405a4c374ef62df036b9dfdb847"
                },
                "downloads": -1,
                "filename": "tokenize_rt-1.0.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a54e5a3d7c19bfe21f2fa3a8b4953e70",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 3241,
                "upload_time": "2017-06-02T18:11:30",
                "url": "https://files.pythonhosted.org/packages/87/11/419e1935bc0cd7bd4287d5c21269f8108d80435779eadaa5c15a1f1e3f75/tokenize_rt-1.0.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "80de52c8f2a9fa5c7eb5f98405675f55",
                    "sha256": "a62a0491533f00dd012f985014d249a46b470d87211655c9239abdecbc5ffe64"
                },
                "downloads": -1,
                "filename": "tokenize-rt-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "80de52c8f2a9fa5c7eb5f98405675f55",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1784,
                "upload_time": "2017-06-02T18:11:32",
                "url": "https://files.pythonhosted.org/packages/40/fd/971afe0c3e0bd8f292611cf8369a03374750988db99dacbfd6da4229ba6f/tokenize-rt-1.0.0.tar.gz"
            }
        ],
        "2.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2595e9ae81dba6f362d590583c46a581",
                    "sha256": "0d26a18e00fde31bb3c85ac3f3f3b5e9d3276853222bb8c8dcd210f7b3de5601"
                },
                "downloads": -1,
                "filename": "tokenize_rt-2.0.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "2595e9ae81dba6f362d590583c46a581",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 3358,
                "upload_time": "2017-07-14T15:25:14",
                "url": "https://files.pythonhosted.org/packages/30/d9/a02454b29e8cc7f39bff2620cbe8e1ecce377886e8956c7994a7a09e358e/tokenize_rt-2.0.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "db17488338f61e8a84e0ebf2ba2df93e",
                    "sha256": "282786bd226dc34114d5436e62ce8c6756d8051486086b81f85f3b25351b9289"
                },
                "downloads": -1,
                "filename": "tokenize-rt-2.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "db17488338f61e8a84e0ebf2ba2df93e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1899,
                "upload_time": "2017-07-14T15:25:15",
                "url": "https://files.pythonhosted.org/packages/eb/1a/718aff3dda46055977042137e7d22c817104c29e665ec8e259bd47b1b806/tokenize-rt-2.0.0.tar.gz"
            }
        ],
        "2.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9a196b374ee640d69223e0ce1a2cf277",
                    "sha256": "2310f462de4f57c0d4d2ee0b4e5c589f06a7ef25cdfc73e6531f815ba099b065"
                },
                "downloads": -1,
                "filename": "tokenize_rt-2.0.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "9a196b374ee640d69223e0ce1a2cf277",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 3467,
                "upload_time": "2017-07-26T11:35:22",
                "url": "https://files.pythonhosted.org/packages/04/59/092aa671d0e19a7b14419e12c518288c563d4532ce7d42da5b3a1dffc49e/tokenize_rt-2.0.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "b8953cfb17fc02dc219fe1119c56bb1c",
                    "sha256": "3cfa481f51ec1c6a1791964b14e4589c38d8877f2c74e5e50fe3e79dcf5f5bfa"
                },
                "downloads": -1,
                "filename": "tokenize-rt-2.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "b8953cfb17fc02dc219fe1119c56bb1c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1999,
                "upload_time": "2017-07-26T11:35:23",
                "url": "https://files.pythonhosted.org/packages/1a/89/49bd84fdc1c4af3cdfa18407a4c361005375d4368c2f38f1f6b34746c6f3/tokenize-rt-2.0.1.tar.gz"
            }
        ],
        "2.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f26c296234bf2126a69d1f22e9de2c2c",
                    "sha256": "608b26913b74e00d16f5eee36ea2975423be1684ac5bdf68480ba04d488513d4"
                },
                "downloads": -1,
                "filename": "tokenize_rt-2.1.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "f26c296234bf2126a69d1f22e9de2c2c",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 2707,
                "upload_time": "2018-10-07T23:16:01",
                "url": "https://files.pythonhosted.org/packages/76/82/0e6a9dda45dd76be22d74211443e199a330ac7e428b8dbbc5d116651be03/tokenize_rt-2.1.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "f7821302caafbbc464374b2c1cacde4f",
                    "sha256": "c218980dd483c407aa89683c9dfcee7935fc5efd05e8255a3eabfd8c81bcd437"
                },
                "downloads": -1,
                "filename": "tokenize-rt-2.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "f7821302caafbbc464374b2c1cacde4f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2789,
                "upload_time": "2018-10-07T23:16:03",
                "url": "https://files.pythonhosted.org/packages/e7/ad/a7e4eec5b55c878d0186b6ae29a06451c2b7a89332f1b17e2bf69340786c/tokenize-rt-2.1.0.tar.gz"
            }
        ],
        "2.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "73007d91a928060240fb486b71938a6b",
                    "sha256": "6b845036b52d430d395b02981fa4adaeb279c6914b57d8019be0d7d0a98b8a03"
                },
                "downloads": -1,
                "filename": "tokenize_rt-2.2.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "73007d91a928060240fb486b71938a6b",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
                "size": 4312,
                "upload_time": "2019-02-28T07:36:59",
                "url": "https://files.pythonhosted.org/packages/43/4b/c5df89ff5b38afffc04fb208c9b1fce30c1426788a368d7039b4cbcf524e/tokenize_rt-2.2.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "bc6c1fb937821f29c969ddcdba5cf383",
                    "sha256": "56a0a093db01e984297e4a15813b4196764dc09c5c352617f22ad6c26d4c8042"
                },
                "downloads": -1,
                "filename": "tokenize_rt-2.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "bc6c1fb937821f29c969ddcdba5cf383",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
                "size": 3832,
                "upload_time": "2019-02-28T07:37:00",
                "url": "https://files.pythonhosted.org/packages/d6/14/0168c11b729705fa64e531ae33c54c3b11a1e0f0b64fbe1d6c3d61898bb9/tokenize_rt-2.2.0.tar.gz"
            }
        ],
        "3.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e3fa038dcd76dacafe0da9331a76d095",
                    "sha256": "0ad773efe3325971ebb06231879166d63d0f64acbc224a78c91e3f5b5ec34c0a"
                },
                "downloads": -1,
                "filename": "tokenize_rt-3.0.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "e3fa038dcd76dacafe0da9331a76d095",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
                "size": 5147,
                "upload_time": "2019-06-16T02:20:20",
                "url": "https://files.pythonhosted.org/packages/95/e6/3d487dc522a442f88e5730538a3b9a380d04a331c305251e44f9e5ddabd5/tokenize_rt-3.0.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "5638ba5e02f7e34d5f85032322bbba37",
                    "sha256": "a599c322fd7096707e4419e280eccd80a878ee12a8365aa86cade7d153078982"
                },
                "downloads": -1,
                "filename": "tokenize_rt-3.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "5638ba5e02f7e34d5f85032322bbba37",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
                "size": 4737,
                "upload_time": "2019-06-16T02:20:21",
                "url": "https://files.pythonhosted.org/packages/55/66/e8b9dd283a2aaf6bd4e869675f88837b65381e76135e19f0dd9aa1b4e5dd/tokenize_rt-3.0.0.tar.gz"
            }
        ],
        "3.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "513fd53b382438b70ad39a385ab0367e",
                    "sha256": "c65559bd0574447e10348f0f18710e870f862e168d673157f9b5eb43fb1b7eff"
                },
                "downloads": -1,
                "filename": "tokenize_rt-3.0.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "513fd53b382438b70ad39a385ab0367e",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
                "size": 5002,
                "upload_time": "2019-06-16T02:49:06",
                "url": "https://files.pythonhosted.org/packages/ec/77/e37131d747e1b6be9c2279f0fa6312397cdfe45cf9741b25d1a5087fd4bb/tokenize_rt-3.0.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ff91b2d65bb1e56fb9c7115f1643529c",
                    "sha256": "ff2d4e39e924aa6998cf415efbfc4e8476b85215b4d094add79eacaeffd8bf12"
                },
                "downloads": -1,
                "filename": "tokenize_rt-3.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "ff91b2d65bb1e56fb9c7115f1643529c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
                "size": 4589,
                "upload_time": "2019-06-16T02:49:08",
                "url": "https://files.pythonhosted.org/packages/6b/28/ae055ca04483f2c484fa413cbe72e4ecb0d29040aab0911fb638c6cfce10/tokenize_rt-3.0.1.tar.gz"
            }
        ],
        "3.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4f553cf80fae388d93978bef1732f129",
                    "sha256": "862442a55cd21b24c62adbdf0ae7fa178f1fd289e532fe9f5ab902d227317b42"
                },
                "downloads": -1,
                "filename": "tokenize_rt-3.1.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "4f553cf80fae388d93978bef1732f129",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
                "size": 5169,
                "upload_time": "2019-07-05T21:01:05",
                "url": "https://files.pythonhosted.org/packages/87/88/edfba9ab2a34bc9ab557a7241e82aafec767d850852cce7a7f9425d8b47d/tokenize_rt-3.1.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "fdc1e0a371ca51c311a25648f40036eb",
                    "sha256": "0ac30f3386b212beb2a0e6dfaa6cb619f711587e9d05436907438ceb51319d58"
                },
                "downloads": -1,
                "filename": "tokenize_rt-3.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "fdc1e0a371ca51c311a25648f40036eb",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
                "size": 4668,
                "upload_time": "2019-07-05T21:01:06",
                "url": "https://files.pythonhosted.org/packages/da/15/689e5623915c3625d02e8ba3e763cb1928c96cf4d49c09e280cfc474331e/tokenize_rt-3.1.0.tar.gz"
            }
        ],
        "3.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fa4101d40efc5b06155c9ec9361286fb",
                    "sha256": "53f5c22d36e5c6f8e3fdbc6cb4dd151d1b3d38cea1b85b5fef6268f153733899"
                },
                "downloads": -1,
                "filename": "tokenize_rt-3.2.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "fa4101d40efc5b06155c9ec9361286fb",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
                "size": 6098,
                "upload_time": "2019-07-07T01:29:04",
                "url": "https://files.pythonhosted.org/packages/a9/50/1983329a9b82496326627bfb44d2f2604913311fe6d9aa426aa4261351ea/tokenize_rt-3.2.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "8dadb8261ebc148ffd346701dc3de8fe",
                    "sha256": "2f44eee8f620102f8a03c50142795121faf86e020d208896ea7a7047bbe933cf"
                },
                "downloads": -1,
                "filename": "tokenize_rt-3.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "8dadb8261ebc148ffd346701dc3de8fe",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
                "size": 5753,
                "upload_time": "2019-07-07T01:29:06",
                "url": "https://files.pythonhosted.org/packages/0c/ed/683a54fb3525c768331ec9c40e09e69a0d54e4ca55d1930e2df1d4b23afa/tokenize_rt-3.2.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "fa4101d40efc5b06155c9ec9361286fb",
                "sha256": "53f5c22d36e5c6f8e3fdbc6cb4dd151d1b3d38cea1b85b5fef6268f153733899"
            },
            "downloads": -1,
            "filename": "tokenize_rt-3.2.0-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "fa4101d40efc5b06155c9ec9361286fb",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
            "size": 6098,
            "upload_time": "2019-07-07T01:29:04",
            "url": "https://files.pythonhosted.org/packages/a9/50/1983329a9b82496326627bfb44d2f2604913311fe6d9aa426aa4261351ea/tokenize_rt-3.2.0-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "8dadb8261ebc148ffd346701dc3de8fe",
                "sha256": "2f44eee8f620102f8a03c50142795121faf86e020d208896ea7a7047bbe933cf"
            },
            "downloads": -1,
            "filename": "tokenize_rt-3.2.0.tar.gz",
            "has_sig": false,
            "md5_digest": "8dadb8261ebc148ffd346701dc3de8fe",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
            "size": 5753,
            "upload_time": "2019-07-07T01:29:06",
            "url": "https://files.pythonhosted.org/packages/0c/ed/683a54fb3525c768331ec9c40e09e69a0d54e4ca55d1930e2df1d4b23afa/tokenize_rt-3.2.0.tar.gz"
        }
    ]
}