{
    "info": {
        "author": "Mi\u00f0eind ehf.",
        "author_email": "mideind@mideind.is",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Natural Language :: Icelandic",
            "Operating System :: Microsoft :: Windows",
            "Operating System :: POSIX",
            "Operating System :: Unix",
            "Programming Language :: Python",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7",
            "Programming Language :: Python :: 3.8",
            "Programming Language :: Python :: 3.9",
            "Programming Language :: Python :: Implementation :: CPython",
            "Programming Language :: Python :: Implementation :: PyPy",
            "Topic :: Software Development :: Libraries :: Python Modules",
            "Topic :: Text Processing :: Linguistic",
            "Topic :: Utilities"
        ],
        "description": "-----------------------------------------\nTokenizer: A tokenizer for Icelandic text\n-----------------------------------------\n\n.. image:: https://github.com/mideind/Tokenizer/workflows/Python%20package/badge.svg\n   :target: https://github.com/mideind/Tokenizer\n\nOverview\n--------\n\nTokenization is a necessary first step in many natural language processing\ntasks, such as word counting, parsing, spell checking, corpus generation, and\nstatistical analysis of text.\n\n**Tokenizer** is a compact pure-Python (>= 3.6) executable\nprogram and module for tokenizing Icelandic text. It converts input text to\nstreams of *tokens*, where each token is a separate word, punctuation sign,\nnumber/amount, date, e-mail, URL/URI, etc. It also segments the token stream\ninto sentences, considering corner cases such as abbreviations and dates in\nthe middle of sentences.\n\nThe package contains a dictionary of common Icelandic abbreviations,\nin the file ``src/tokenizer/Abbrev.conf``.\n\nTokenizer is an independent spinoff from the `Greynir project <https://greynir.is>`_\n(GitHub repository `here <https://github.com/mideind/Greynir>`_), by the same authors.\nThe `Greynir natural language parser for Icelandic <https://github.com/mideind/GreynirPackage>`_\nuses Tokenizer on its input.\n\nNote that Tokenizer is licensed under the *MIT* license\nwhile Greynir is licensed under *GPLv3*.\n\n\nDeep vs. shallow tokenization\n-----------------------------\n\nTokenizer can do both *deep* and *shallow* tokenization.\n\n*Shallow* tokenization simply returns each sentence as a string (or as a line\nof text in an output file), where the individual tokens are separated\nby spaces.\n\n*Deep* tokenization returns token objects that have been annotated with\nthe token type and further information extracted from the token, for example\na *(year, month, day)* tuple in the case of date tokens.\n\nIn shallow tokenization, tokens are in most cases kept intact, although\nconsecutive white space is always coalesced. The input strings\n``\"800 MW\"``, ``\"21. jan\u00faar\"`` and ``\"800 7000\"`` thus become\ntwo tokens each, output with a single space between them.\n\nIn deep tokenization, the same strings are represented by single token objects,\nof type ``TOK.MEASUREMENT``, ``TOK.DATEREL`` and ``TOK.TELNO``, respectively.\nThe text associated with a single token object may contain spaces,\nalthough consecutive whitespace is always coalesced into a single space ``\" \"``.\n\nBy default, the command line tool performs shallow tokenization. If you\nwant deep tokenization with the command line tool, use the ``--json`` or\n``--csv`` switches.\n\nFrom Python code, call ``split_into_sentences()`` for shallow tokenization,\nor ``tokenize()`` for deep tokenization. These functions are documented with\nexamples below.\n\n\nInstallation\n------------\n\nTo install:\n\n.. code-block:: console\n\n    $ pip install tokenizer\n\n\nCommand line tool\n-----------------\n\nAfter installation, the tokenizer can be invoked directly from\nthe command line:\n\n.. code-block:: console\n\n    $ tokenize input.txt output.txt\n\nInput and output files are in UTF-8 encoding. If the files are not\ngiven explicitly, ``stdin`` and ``stdout`` are used for input and output,\nrespectively.\n\nEmpty lines in the input are treated as hard sentence boundaries.\n\nBy default, the output consists of one sentence per line, where each\nline ends with a single newline character (ASCII LF, ``chr(10)``, ``\"\\n\"``).\nWithin each line, tokens are separated by spaces.\n\nThe following (mutually exclusive) options can be specified\non the command line:\n\n+-------------------+---------------------------------------------------+\n| | ``--csv``       | Deep tokenization. Output token objects in CSV    |\n|                   | format, one per line. Sentences are separated by  |\n|                   | lines containing ``0,\"\",\"\"``                      |\n+-------------------+---------------------------------------------------+\n| | ``--json``      | Deep tokenization. Output token objects in JSON   |\n|                   | format, one per line.                             |\n+-------------------+---------------------------------------------------+\n\nOther options can be specified on the command line:\n\n+-----------------------------------+---------------------------------------------------+\n| | ``-n``                          | Normalize punctuation, causing e.g. quotes to be  |\n| |                                 | output in Icelandic form and hyphens to be        |\n| | ``--normalize``                 | regularized. This option is only applicable to    |\n|                                   | shallow tokenization.                             |\n+-----------------------------------+---------------------------------------------------+\n| | ``-s``                          | Input contains strictly one sentence per line,    |\n| |                                 | i.e. every newline is a sentence boundary.        |\n| | ``--one_sent_per_line``         |                                                   |\n+-----------------------------------+---------------------------------------------------+\n| | ``-o``                          | Output original token text, i.e. bypass shallow   |\n| |                                 | tokenization. This effectively runs the tokenizer |\n| | ``--original``                  | as a sentence splitter only.                      |\n+-----------------------------------+---------------------------------------------------+\n| | ``-m``                          | Degree signal in tokens denoting temperature      |\n| | ``--convert_measurements``      | normalized (200\u00b0 C -> 200 \u00b0C)                     |\n+-----------------------------------+---------------------------------------------------+\n| | ``-p``                          | Numbers combined into one token with the          |\n| | ``--coalesce_percent``          | following token denoting percentage word forms    |\n|                                   | (*pr\u00f3sent*, *pr\u00f3sentustig*, *hundra\u00f0shlutar*)     |\n+-----------------------------------+---------------------------------------------------+\n| | ``-g``                          | Do not replace composite glyphs using Unicode     |\n| | ``--keep_composite_glyphs``     | COMBINING codes with their accented/umlaut        |\n|                                   | counterparts                                      |\n+-----------------------------------+---------------------------------------------------+\n| | ``-e``                          | HTML escape codes replaced by their meaning,      |\n| | ``--replace_html_escapes``      | such as ``&aacute;`` -> ``\u00e1``                     |\n+-----------------------------------+---------------------------------------------------+\n| | ``-c``                          | English-style decimal points and thousands        |\n| | ``--convert_numbers``           | separators in numbers changed to Icelandic style  |\n+-----------------------------------+---------------------------------------------------+\n| | ``-k N``                        | Kludgy ordinal handling defined.                  |\n| | ``--handle_kludgy_ordinals N``  | 0: Returns the original mixed word form           |\n|                                   | 1. Kludgy ordinal returned as pure word forms     |\n|                                   | 2: Kludgy ordinals returned as pure numbers       |\n+-----------------------------------+---------------------------------------------------+\n\n\nType ``tokenize -h`` or ``tokenize --help`` to get a short help message.\n\nExample\n=======\n\n.. code-block:: console\n\n    $ echo \"3.jan\u00faar sl. keypti   \u00e9g 64kWst rafb\u00edl. Hann kosta\u00f0i \u20ac 30.000.\" | tokenize\n    3. jan\u00faar sl. keypti \u00e9g 64kWst rafb\u00edl .\n    Hann kosta\u00f0i \u20ac30.000 .\n\n    $ echo \"3.jan\u00faar sl. keypti   \u00e9g 64kWst rafb\u00edl. Hann kosta\u00f0i \u20ac 30.000.\" | tokenize --csv\n    19,\"3. jan\u00faar\",\"0|1|3\"\n    6,\"sl.\",\"s\u00ed\u00f0astli\u00f0inn\"\n    6,\"keypti\",\"\"\n    6,\"\u00e9g\",\"\"\n    22,\"64kWst\",\"J|230400000.0\"\n    6,\"rafb\u00edl\",\"\"\n    1,\".\",\".\"\n    0,\"\",\"\"\n    6,\"Hann\",\"\"\n    6,\"kosta\u00f0i\",\"\"\n    13,\"\u20ac30.000\",\"30000|EUR\"\n    1,\".\",\".\"\n    0,\"\",\"\"\n\n    $ echo \"3.jan\u00faar sl. keypti   \u00e9g 64kWst rafb\u00edl. Hann kosta\u00f0i \u20ac 30.000.\" | tokenize --json\n    {\"k\":\"BEGIN SENT\"}\n    {\"k\":\"DATEREL\",\"t\":\"3. jan\u00faar\",\"v\":[0,1,3]}\n    {\"k\":\"WORD\",\"t\":\"sl.\",\"v\":[\"s\u00ed\u00f0astli\u00f0inn\"]}\n    {\"k\":\"WORD\",\"t\":\"keypti\"}\n    {\"k\":\"WORD\",\"t\":\"\u00e9g\"}\n    {\"k\":\"MEASUREMENT\",\"t\":\"64kWst\",\"v\":[\"J\",230400000.0]}\n    {\"k\":\"WORD\",\"t\":\"rafb\u00edl\"}\n    {\"k\":\"PUNCTUATION\",\"t\":\".\",\"v\":\".\"}\n    {\"k\":\"END SENT\"}\n    {\"k\":\"BEGIN SENT\"}\n    {\"k\":\"WORD\",\"t\":\"Hann\"}\n    {\"k\":\"WORD\",\"t\":\"kosta\u00f0i\"}\n    {\"k\":\"AMOUNT\",\"t\":\"\u20ac30.000\",\"v\":[30000,\"EUR\"]}\n    {\"k\":\"PUNCTUATION\",\"t\":\".\",\"v\":\".\"}\n    {\"k\":\"END SENT\"}\n\nPython module\n-------------\n\nShallow tokenization example\n============================\n\nAn example of shallow tokenization from Python code goes something like this:\n\n.. code-block:: python\n\n    from tokenizer import split_into_sentences\n\n    # A string to be tokenized, containing two sentences\n    s = \"3.jan\u00faar sl. keypti   \u00e9g 64kWst rafb\u00edl. Hann kosta\u00f0i \u20ac 30.000.\"\n\n    # Obtain a generator of sentence strings\n    g = split_into_sentences(s)\n\n    # Loop through the sentences\n    for sentence in g:\n\n        # Obtain the individual token strings\n        tokens = sentence.split()\n\n        # Print the tokens, comma-separated\n        print(\"|\".join(tokens))\n\nThe program outputs::\n\n    3.|jan\u00faar|sl.|keypti|\u00e9g|64kWst|rafb\u00edl|.\n    Hann|kosta\u00f0i|\u20ac30.000|.\n\nDeep tokenization example\n=========================\n\nTo do deep tokenization from within Python code:\n\n.. code-block:: python\n\n    from tokenizer import tokenize, TOK\n\n    text = (\"M\u00e1linu var v\u00edsa\u00f0 til stj\u00f3rnskipunar- og eftirlitsnefndar \"\n        \"skv. 3. gr. XVII. kafla laga nr. 10/2007 \u00feann 3. jan\u00faar 2010.\")\n\n    for token in tokenize(text):\n\n        print(\"{0}: '{1}' {2}\".format(\n            TOK.descr[token.kind],\n            token.txt or \"-\",\n            token.val or \"\"))\n\nOutput::\n\n    BEGIN SENT: '-' (0, None)\n    WORD: 'M\u00e1linu'\n    WORD: 'var'\n    WORD: 'v\u00edsa\u00f0'\n    WORD: 'til'\n    WORD: 'stj\u00f3rnskipunar- og eftirlitsnefndar'\n    WORD: 'skv.' [('samkv\u00e6mt', 0, 'fs', 'skst', 'skv.', '-')]\n    ORDINAL: '3.' 3\n    WORD: 'gr.' [('grein', 0, 'kvk', 'skst', 'gr.', '-')]\n    ORDINAL: 'XVII.' 17\n    WORD: 'kafla'\n    WORD: 'laga'\n    WORD: 'nr.' [('n\u00famer', 0, 'hk', 'skst', 'nr.', '-')]\n    NUMBER: '10' (10, None, None)\n    PUNCTUATION: '/' (4, '/')\n    YEAR: '2007' 2007\n    WORD: '\u00feann'\n    DATEABS: '3. jan\u00faar 2010' (2010, 1, 3)\n    PUNCTUATION: '.' (3, '.')\n    END SENT: '-'\n\nNote the following:\n\n- Sentences are delimited by ``TOK.S_BEGIN`` and ``TOK.S_END`` tokens.\n- Composite words, such as *stj\u00f3rnskipunar- og eftirlitsnefndar*,\n  are coalesced into one token.\n- Well-known abbreviations are recognized and their full expansion\n  is available in the ``token.val`` field.\n- Ordinal numbers (*3., XVII.*) are recognized and their value (*3, 17*)\n  is available in the ``token.val``  field.\n- Dates, years and times, both absolute and relative, are recognized and\n  the respective year, month, day, hour, minute and second\n  values are included as a tuple in ``token.val``.\n- Numbers, both integer and real, are recognized and their value\n  is available in the ``token.val`` field.\n- Further details of how Tokenizer processes text can be inferred from the\n  `test module <https://github.com/mideind/Tokenizer/blob/master/test/test_tokenizer.py>`_\n  in the project's `GitHub repository <https://github.com/mideind/Tokenizer>`_.\n\n\nThe ``tokenize()`` function\n---------------------------\n\nTo deep-tokenize a text string, call ``tokenizer.tokenize(text, **options)``.\nThe ``text`` parameter can be a string, or an iterable that yields strings\n(such as a text file object).\n\nThe function returns a Python *generator* of token objects.\nEach token object is a simple ``namedtuple`` with three\nfields: ``(kind, txt, val)`` (further documented below).\n\nThe ``tokenizer.tokenize()`` function is typically called in a ``for`` loop:\n\n.. code-block:: python\n\n    import tokenizer\n    for token in tokenizer.tokenize(mystring):\n        kind, txt, val = token\n        if kind == tokenizer.TOK.WORD:\n            # Do something with word tokens\n            pass\n        else:\n            # Do something else\n            pass\n\nAlternatively, create a token list from the returned generator::\n\n    token_list = list(tokenizer.tokenize(mystring))\n\nThe ``split_into_sentences()`` function\n---------------------------------------\n\nTo shallow-tokenize a text string, call\n``tokenizer.split_into_sentences(text_or_gen, **options)``.\nThe ``text_or_gen`` parameter can be a string, or an iterable that yields\nstrings (such as a text file object).\n\nThis function returns a Python *generator* of strings, yielding a string\nfor each sentence in the input. Within a sentence, the tokens are\nseparated by spaces.\n\nYou can pass the option ``normalize=True`` to the function if you want\nthe normalized form of punctuation tokens. Normalization outputs\nIcelandic single and double quotes (\u201ethese\u201c) instead of English-style\nones (\"these\"), converts three-dot ellipsis ... to single character\nellipsis \u2026, and casts en-dashes \u2013 and em-dashes \u2014 to regular hyphens.\n\nThe ``tokenizer.split_into_sentences()`` function is typically called\nin a ``for`` loop:\n\n.. code-block:: python\n\n    import tokenizer\n    with open(\"example.txt\", \"r\", encoding=\"utf-8\") as f:\n        # You can pass a file object directly to split_into_sentences()\n        for sentence in tokenizer.split_into_sentences(f):\n            # sentence is a string of space-separated tokens\n            tokens = sentence.split()\n            # Now, tokens is a list of strings, one for each token\n            for t in tokens:\n                # Do something with the token t\n                pass\n\n\nThe ``correct_spaces()`` function\n---------------------------------\n\nThe ``tokenizer.correct_spaces(text)`` function returns a string after\nsplitting it up and re-joining it with correct whitespace around\npunctuation tokens. Example::\n\n    >>> import tokenizer\n    >>> tokenizer.correct_spaces(\n    ... \"Fr\u00e9tt \\n  dagsins:J\u00f3n\\t ,Fri\u00f0geir og P\u00e1ll ! 100  /  2  =   50\"\n    ... )\n    'Fr\u00e9tt dagsins: J\u00f3n, Fri\u00f0geir og P\u00e1ll! 100/2 = 50'\n\n\nThe ``detokenize()`` function\n---------------------------------\n\nThe ``tokenizer.detokenize(tokens, normalize=False)`` function\ntakes an iterable of token objects and returns a corresponding, correctly\nspaced text string, composed from the tokens' text. If the\n``normalize`` parameter is set to ``True``,\nthe function uses the normalized form of any punctuation tokens, such\nas proper Icelandic single and double quotes instead of English-type\nquotes. Example::\n\n    >>> import tokenizer\n    >>> toklist = list(tokenizer.tokenize(\"Hann sag\u00f0i: \u201e\u00de\u00fa ert \u00e1g\u00e6t!\u201c.\"))\n    >>> tokenizer.detokenize(toklist, normalize=True)\n    'Hann sag\u00f0i: \u201e\u00de\u00fa ert \u00e1g\u00e6t!\u201c.'\n\n\nThe ``normalized_text()`` function\n----------------------------------\n\nThe ``tokenizer.normalized_text(token)`` function\nreturns the normalized text for a token. This means that the original\ntoken text is returned except for certain punctuation tokens, where a\nnormalized form is returned instead. Specifically, English-type quotes\nare converted to Icelandic ones, and en- and em-dashes are converted\nto regular hyphens.\n\n\nThe ``text_from_tokens()`` function\n-----------------------------------\n\nThe ``tokenizer.text_from_tokens(tokens)`` function\nreturns a concatenation of the text contents of the given token list,\nwith spaces between tokens. Example::\n\n    >>> import tokenizer\n    >>> toklist = list(tokenizer.tokenize(\"Hann sag\u00f0i: \\\"\u00de\u00fa ert \u00e1g\u00e6t!\\\".\"))\n    >>> tokenizer.text_from_tokens(toklist)\n    'Hann sag\u00f0i : \" \u00de\u00fa ert \u00e1g\u00e6t ! \" .'\n\n\nThe ``normalized_text_from_tokens()`` function\n----------------------------------------------\n\nThe ``tokenizer.normalized_text_from_tokens(tokens)`` function\nreturns a concatenation of the normalized text contents of the given\ntoken list, with spaces between tokens. Example (note the double quotes)::\n\n    >>> import tokenizer\n    >>> toklist = list(tokenizer.tokenize(\"Hann sag\u00f0i: \\\"\u00de\u00fa ert \u00e1g\u00e6t!\\\".\"))\n    >>> tokenizer.normalized_text_from_tokens(toklist)\n    'Hann sag\u00f0i : \u201e \u00de\u00fa ert \u00e1g\u00e6t ! \u201c .'\n\n\nTokenization options\n--------------------\n\nYou can optionally pass one or more of the following options as\nkeyword parameters to the ``tokenize()`` and ``split_into_sentences()``\nfunctions:\n\n\n* ``convert_numbers=[bool]``\n\n  Setting this option to ``True`` causes the tokenizer to convert numbers\n  and amounts with\n  English-style decimal points (``.``) and thousands separators (``,``)\n  to Icelandic format, where the decimal separator is a comma (``,``)\n  and the thousands separator is a period (``.``). ``$1,234.56`` is thus\n  converted to a token whose text is ``$1.234,56``.\n\n  The default value for the ``convert_numbers`` option is ``False``.\n\n  Note that in versions of Tokenizer prior to 1.4, ``convert_numbers``\n  was ``True``.\n\n\n* ``convert_measurements=[bool]``\n\n  Setting this option to ``True`` causes the tokenizer to convert\n  degrees Kelvin, Celsius and Fahrenheit to a regularized form, i.e.\n  ``200\u00b0 C`` becomes ``200 \u00b0C``.\n\n  The default value for the ``convert_measurements`` option is ``False``.\n\n\n* ``replace_composite_glyphs=[bool]``\n\n  Setting this option to ``False`` disables the automatic replacement\n  of composite Unicode glyphs with their corresponding Icelandic characters.\n  By default, the tokenizer combines vowels with the Unicode\n  COMBINING ACUTE ACCENT and COMBINING DIAERESIS glyphs to form single\n  character code points, such as '\u00e1' and '\u00f6'.\n\n  The default value for the ``replace_composite_glyphs`` option is ``True``.\n\n\n* ``replace_html_escapes=[bool]``\n\n  Setting this option to ``True`` causes the tokenizer to replace common\n  HTML escaped character codes, such as ``&aacute;`` with the character being\n  escaped, such as ``\u00e1``. Note that ``&shy;`` (soft hyphen) is replaced by\n  an empty string, and ``&nbsp;`` is replaced by a normal space.\n  The ligatures ``&filig;`` and ``&fllig;`` are replaced by ``fi`` and ``fl``,\n  respectively.\n\n  The default value for the ``replace_html_escapes`` option is ``False``.\n\n\n* ``handle_kludgy_ordinals=[value]``\n\n  This options controls the way Tokenizer handles 'kludgy' ordinals, such as\n  *1sti*, *4\u00f0u*, or *2ja*. By default, such ordinals are returned unmodified\n  ('passed through') as word tokens (``TOK.WORD``).\n  However, this can be modified as follows:\n\n  * ``tokenizer.KLUDGY_ORDINALS_MODIFY``: Kludgy ordinals are corrected\n    to become 'proper' word tokens, i.e. *1sti* becomes *fyrsti* and\n    *2ja* becomes *tveggja*.\n\n  * ``tokenizer.KLUDGY_ORDINALS_TRANSLATE``: Kludgy ordinals that represent\n    proper ordinal numbers are translated to ordinal tokens (``TOK.ORDINAL``),\n    with their original text and their ordinal value. *1sti* thus\n    becomes a ``TOK.ORDINAL`` token with a value of 1, and *3ja* becomes\n    a ``TOK.ORDINAL`` with a value of 3.\n\n  * ``tokenizer.KLUDGY_ORDINALS_PASS_THROUGH`` is the default value of\n    the option. It causes kludgy ordinals to be returned unmodified as\n    word tokens.\n\n  Note that versions of Tokenizer prior to 1.4 behaved as if\n  ``handle_kludgy_ordinals`` were set to\n  ``tokenizer.KLUDGY_ORDINALS_TRANSLATE``.\n\n\nThe token object\n----------------\n\nEach token is an instance of the class ``Tok`` that has three main properties:\n``kind``, ``txt`` and ``val``.\n\n\nThe ``kind`` property\n=====================\n\nThe ``kind`` property contains one of the following integer constants,\ndefined within the ``TOK`` class:\n\n+---------------+---------+---------------------+---------------------------+\n| Constant      |  Value  | Explanation         | Examples                  |\n+===============+=========+=====================+===========================+\n| PUNCTUATION   |    1    | Punctuation         | . ! ; % &                 |\n+---------------+---------+---------------------+---------------------------+\n| TIME          |    2    | Time (h, m, s)      | | 11:35:40                |\n|               |         |                     | | kl. 7:05                |\n|               |         |                     | | klukkan 23:35           |\n+---------------+---------+---------------------+---------------------------+\n| DATE *        |    3    | Date (y, m, d)      | [Unused, see DATEABS and  |\n|               |         |                     | DATEREL]                  |\n+---------------+---------+---------------------+---------------------------+\n| YEAR          |    4    | Year                | | \u00e1ri\u00f0 874 e.Kr.          |\n|               |         |                     | | 1965                    |\n|               |         |                     | | 44 f.Kr.                |\n+---------------+---------+---------------------+---------------------------+\n| NUMBER        |    5    | Number              | | 100                     |\n|               |         |                     | | 1.965                   |\n|               |         |                     | | 1.965,34                |\n|               |         |                     | | 1,965.34                |\n|               |         |                     | | 2\u215e                      |\n+---------------+---------+---------------------+---------------------------+\n| WORD          |    6    | Word                | | kattaeftirlit           |\n|               |         |                     | | hunda- og kattaeftirlit |\n+---------------+---------+---------------------+---------------------------+\n| TELNO         |    7    | Telephone number    | | 5254764                 |\n|               |         |                     | | 699-4244                |\n|               |         |                     | | 410 4000                |\n+---------------+---------+---------------------+---------------------------+\n| PERCENT       |    8    | Percentage          | 78%                       |\n+---------------+---------+---------------------+---------------------------+\n| URL           |    9    | URL                 | | https://greynir.is      |\n|               |         |                     | | http://tiny.cc/28695y   |\n+---------------+---------+---------------------+---------------------------+\n| ORDINAL       |    10   | Ordinal number      | | 30.                     |\n|               |         |                     | | XVIII.                  |\n+---------------+---------+---------------------+---------------------------+\n| TIMESTAMP *   |    11   | Timestamp           | [Unused, see              |\n|               |         |                     | TIMESTAMPABS and          |\n|               |         |                     | TIMESTAMPREL]             |\n+---------------+---------+---------------------+---------------------------+\n| CURRENCY *    |    12   | Currency name       | [Unused]                  |\n+---------------+---------+---------------------+---------------------------+\n| AMOUNT        |    13   | Amount              | | \u20ac2.345,67               |\n|               |         |                     | | 750 \u00fe\u00fas.kr.             |\n|               |         |                     | | 2,7 mr\u00f0. USD            |\n|               |         |                     | | kr. 9.900               |\n|               |         |                     | | EUR 200                 |\n+---------------+---------+---------------------+---------------------------+\n| PERSON *      |    14   | Person name         | [Unused]                  |\n+---------------+---------+---------------------+---------------------------+\n| EMAIL         |    15   | E-mail              | ``fake@news.is``          |\n+---------------+---------+---------------------+---------------------------+\n| ENTITY *      |    16   | Named entity        | [Unused]                  |\n+---------------+---------+---------------------+---------------------------+\n| UNKNOWN       |    17   | Unknown token       |                           |\n+---------------+---------+---------------------+---------------------------+\n| DATEABS       |    18   | Absolute date       | | 30. desember 1965       |\n|               |         |                     | | 30/12/1965              |\n|               |         |                     | | 1965-12-30              |\n|               |         |                     | | 1965/12/30              |\n+---------------+---------+---------------------+---------------------------+\n| DATEREL       |    19   | Relative date       | | 15. mars                |\n|               |         |                     | | 15/3                    |\n|               |         |                     | | 15.3.                   |\n|               |         |                     | | mars 1911               |\n+---------------+---------+---------------------+---------------------------+\n| TIMESTAMPABS  |    20   | Absolute timestamp  | | 30. desember 1965 11:34 |\n|               |         |                     | | 1965-12-30 kl. 13:00    |\n+---------------+---------+---------------------+---------------------------+\n| TIMESTAMPREL  |    21   | Relative timestamp  | | 30. desember kl. 13:00  |\n+---------------+---------+---------------------+---------------------------+\n| MEASUREMENT   |    22   | Value with a        | | 690 MW                  |\n|               |         | measurement unit    | | 1.010 hPa               |\n|               |         |                     | | 220 m\u00b2                  |\n|               |         |                     | | 80\u00b0 C                   |\n+---------------+---------+---------------------+---------------------------+\n| NUMWLETTER    |    23   | Number followed by  | | 14a                     |\n|               |         | a single letter     | | 7B                      |\n+---------------+---------+---------------------+---------------------------+\n| DOMAIN        |    24   | Domain name         | | greynir.is              |\n|               |         |                     | | Reddit.com              |\n|               |         |                     | | www.wikipedia.org       |\n+---------------+---------+---------------------+---------------------------+\n| HASHTAG       |    25   | Hashtag             | | #MeToo                  |\n|               |         |                     | | #12stig                 |\n+---------------+---------+---------------------+---------------------------+\n| MOLECULE      |    26   | Molecular formula   | | H2SO4                   |\n|               |         |                     | | CO2                     |\n+---------------+---------+---------------------+---------------------------+\n| SSN           |    27   | Social security     | | 591213-1480             |\n|               |         | number (*kennitala*)|                           |\n+---------------+---------+---------------------+---------------------------+\n| USERNAME      |    28   | Twitter user handle | | @username_123           |\n|               |         |                     |                           |\n+---------------+---------+---------------------+---------------------------+\n| SERIALNUMBER  |    29   | Serial number       | | 394-5388                |\n|               |         |                     | | 12-345-6789             |\n+---------------+---------+---------------------+---------------------------+\n| COMPANY *     |    30   | Company name        | [Unused]                  |\n+---------------+---------+---------------------+---------------------------+\n| S_BEGIN       |  11001  | Start of sentence   |                           |\n+---------------+---------+---------------------+---------------------------+\n| S_END         |  11002  | End of sentence     |                           |\n+---------------+---------+---------------------+---------------------------+\n\n(*) The token types marked with an asterisk are reserved for the Greynir package\nand not currently returned by the tokenizer.\n\nTo obtain a descriptive text for a token kind, use\n``TOK.descr[token.kind]`` (see example above).\n\n\nThe ``txt`` property\n====================\n\nThe ``txt`` property contains the original source text for the token,\nwith the following exceptions:\n\n* All contiguous whitespace (spaces, tabs, newlines) is coalesced\n  into single spaces (``\" \"``) within the ``txt`` string. A date\n  token that is parsed from a source text of ``\"29.  \\n   jan\u00faar\"``\n  thus has a ``txt`` of ``\"29. jan\u00faar\"``.\n\n* Tokenizer automatically merges Unicode ``COMBINING ACUTE ACCENT``\n  (code point 769) and ``COMBINING DIAERESIS`` (code point 776)\n  with vowels to form single code points for the Icelandic letters\n  \u00e1, \u00e9, \u00ed, \u00f3, \u00fa, \u00fd and \u00f6, in both lower and upper case. (This behavior\n  can be disabled; see the ``replace_composite_glyphs`` option described\n  above.)\n\n* If the appropriate options are specified (see above), it converts\n  kludgy ordinals (*3ja*) to proper ones (*\u00feri\u00f0ja*), and English-style\n  thousand and decimal separators to Icelandic ones\n  (*10,345.67* becomes *10.345,67*).\n\n* If the ``replace_html_escapes`` option is set, Tokenizer replaces\n  HTML-style escapes (``&aacute;``) with the characters\n  being escaped (``\u00e1``).\n\n\nThe ``val`` property\n====================\n\nThe ``val`` property contains auxiliary information, corresponding to\nthe token kind, as follows:\n\n- For ``TOK.PUNCTUATION``, the ``val`` field contains a tuple with\n  two items: ``(whitespace, normalform)``. The first item (``token.val[0]``)\n  specifies the whitespace normally found around the symbol in question,\n  as an integer::\n\n    TP_LEFT = 1   # Whitespace to the left\n    TP_CENTER = 2 # Whitespace to the left and right\n    TP_RIGHT = 3  # Whitespace to the right\n    TP_NONE = 4   # No whitespace\n\n  The second item (``token.val[1]``) contains a normalized representation of the\n  punctuation. For instance, various forms of single and double\n  quotes are represented as Icelandic ones (i.e. \u201ethese\u201c or \u201athese\u2018) in\n  normalized form, and ellipsis (\"...\") are represented as the single\n  character \"\u2026\".\n\n- For ``TOK.TIME``, the ``val`` field contains an\n  ``(hour, minute, second)`` tuple.\n\n- For ``TOK.DATEABS``, the ``val`` field contains a\n  ``(year, month, day)`` tuple (all 1-based).\n\n- For ``TOK.DATEREL``, the ``val`` field contains a\n  ``(year, month, day)`` tuple (all 1-based),\n  except that a least one of the tuple fields is missing and set to 0.\n  Example: *3. j\u00fan\u00ed* becomes ``TOK.DATEREL`` with the fields ``(0, 6, 3)``\n  as the year is missing.\n\n- For ``TOK.YEAR``, the ``val`` field contains the year as an integer.\n  A negative number indicates that the year is BCE (*fyrir Krist*),\n  specified with the suffix *f.Kr.* (e.g. *\u00e1ri\u00f0 33 f.Kr.*).\n\n- For ``TOK.NUMBER``, the ``val`` field contains a tuple\n  ``(number, None, None)``.\n  (The two empty fields are included for compatibility with Greynir.)\n\n- For ``TOK.WORD``, the ``val`` field contains the full expansion\n  of an abbreviation, as a list containing a single tuple, or ``None``\n  if the word is not abbreviated.\n\n- For ``TOK.PERCENT``, the ``val`` field contains a tuple\n  of ``(percentage, None, None)``.\n\n- For ``TOK.ORDINAL``, the ``val`` field contains the ordinal value\n  as an integer. The original ordinal may be a decimal number\n  or a Roman numeral.\n\n- For ``TOK.TIMESTAMP``, the ``val`` field contains\n  a ``(year, month, day, hour, minute, second)`` tuple.\n\n- For ``TOK.AMOUNT``, the ``val`` field contains\n  an ``(amount, currency, None, None)`` tuple. The amount is a float, and\n  the currency is an ISO currency code, e.g. *USD* for dollars ($ sign),\n  *EUR* for euros (\u20ac sign) or *ISK* for Icelandic kr\u00f3na\n  (*kr.* abbreviation). (The two empty fields are included for\n  compatibility with Greynir.)\n\n- For ``TOK.MEASUREMENT``, the ``val`` field contains a ``(unit, value)``\n  tuple, where ``unit`` is a base SI unit (such as ``g``, ``m``,\n  ``m\u00b2``, ``s``, ``W``, ``Hz``, ``K`` for temperature in Kelvin).\n\n- For ``TOK.TELNO``, the ``val`` field contains a tuple: ``(number, cc)``\n  where the first item is the phone number\n  in a normalized ``NNN-NNNN`` format, i.e. always including a hyphen,\n  and the second item is the country code, eventually prefixed by ``+``.\n  The country code defaults to ``354`` (Iceland).\n\n\nAbbreviations\n-------------\n\nAbbreviations recognized by Tokenizer are defined in the ``Abbrev.conf``\nfile, found in the ``src/tokenizer/`` directory. This is a text file with\nabbreviations, their definitions and explanatory comments.\n\nWhen an abbreviation is encountered, it is recognized as a word token\n(i.e. having its ``kind`` field equal to ``TOK.WORD``).\nIts expansion(s) are included in the token's\n``val`` field as a list containing tuples of the format\n``(ordmynd, utg, ordfl, fl, stofn, beyging)``.\nAn example is *o.s.frv.*, which results in a ``val`` field equal to\n``[('og svo framvegis', 0, 'ao', 'frasi', 'o.s.frv.', '-')]``.\n\nThe tuple format is designed to be compatible with the\n*Database of Icelandic Morphology* (*DIM*),\n*Beygingarl\u00fdsing \u00edslensks n\u00fat\u00edmam\u00e1ls*, i.e. the so-called *Sigr\u00fanarsni\u00f0*.\n\n\nDevelopment installation\n------------------------\n\nTo install Tokenizer in development mode, where you can easily\nmodify the source files (assuming you have ``git`` available):\n\n.. code-block:: console\n\n    $ git clone https://github.com/mideind/Tokenizer\n    $ cd Tokenizer\n    $ # [ Activate your virtualenv here, if you have one ]\n    $ pip install -e .\n\n\nTest suite\n----------\n\nTokenizer comes with a large test suite.\nThe file ``test/test_tokenizer.py`` contains built-in tests that\nrun under ``pytest``.\n\nTo run the built-in tests, install `pytest <https://docs.pytest.org/en/latest/>`_,\n``cd`` to your ``Tokenizer`` subdirectory (and optionally\nactivate your virtualenv), then run:\n\n.. code-block:: console\n\n    $ python -m pytest\n\nThe file ``test/toktest_large.txt`` contains a test set of 13,075 lines.\nThe lines test sentence detection, token detection and token classification.\nFor analysis, ``test/toktest_large_gold_perfect.txt`` contains\nthe expected output of a perfect shallow tokenization, and\n``test/toktest_large_gold_acceptable.txt`` contains the current output of the\nshallow tokenization.\n\nThe file ``test/Overview.txt`` (only in Icelandic) contains a description\nof the test set, including line numbers for each part in both\n``test/toktest_large.txt`` and ``test/toktest_large_gold_acceptable.txt``,\nand a tag describing what is being tested in each part.\n\nIt also contains a description of a perfect shallow tokenization for each part,\nacceptable tokenization and the current behaviour.\nAs such, the description is an analysis of which edge cases the tokenizer\ncan handle and which it can not.\n\nTo test the tokenizer on the large test set the following needs to be typed\nin the command line:\n\n.. code-block:: console\n\n    $ tokenize test/toktest_large.txt test/toktest_large_out.txt\n\nTo compare it to the acceptable behaviour:\n\n.. code-block:: console\n\n    $ diff test/toktest_large_out.txt test/toktest_large_gold_acceptable.txt > diff.txt\n\nThe file ``test/toktest_normal.txt`` contains a running text from recent\nnews articles, containing no edge cases. The gold standard for that file\ncan be found in the file ``test/toktest_normal_gold_expected.txt``.\n\n\nChangelog\n---------\n\n* Version 3.4.1: Improved performance on long input chunks.\n* Version 3.4.0: Improved handling and normalization of punctuation.\n* Version 3.3.2: Internal refactoring; bug fixes in paragraph handling.\n* Version 3.3.1: Fixed bug where opening quotes at the start of paragraphs\n  were sometimes incorrectly recognized and normalized.\n* Version 3.2.0: Numbers and amounts that consist of word tokens only ('sex hundru\u00f0')\n  are now returned as the original ``TOK.WORD`` s ('sex' and 'hundru\u00f0'), not as single\n  coalesced ``TOK.NUMBER`` / ``TOK.AMOUNT`` /etc. tokens.\n* Version 3.1.2: Changed paragraph markers to ``[[`` and ``]]`` (removing spaces).\n* Version 3.1.1: Minor fixes; added Tok.from_token().\n* Version 3.1.0: Added ``-o`` switch to the ``tokenize`` command to return original\n  token text, enabling the tokenizer to run as a sentence splitter only.\n* Version 3.0.0: Added tracking of character offsets for tokens within the\n  original source text. Added full type annotations. Dropped Python 2.7 support.\n* Version 2.5.0: Added arguments for all tokenizer options to the\n  command-line tool. Type annotations enhanced.\n* Version 2.4.0: Fixed bug where certain well-known word forms (*f\u00e1*, *f\u00e6r*, *m\u00edn*, *s\u00e1*...)\n  were being interpreted as (wrong) abbreviations. Also fixed bug where certain\n  abbreviations were being recognized even in uppercase and at the end\n  of a sentence, for instance *\u00d6rn.*\n* Version 2.3.1: Various bug fixes; fixed type annotations for Python 2.7;\n  the token kind ``NUMBER WITH LETTER`` is now ``NUMWLETTER``.\n* Version 2.3.0: Added the ``replace_html_escapes`` option to\n  the ``tokenize()`` function.\n* Version 2.2.0: Fixed ``correct_spaces()`` to handle compounds such as\n  *Atvinnu-, n\u00fdsk\u00f6punar- og fer\u00f0am\u00e1lar\u00e1\u00f0uneyti\u00f0* and\n  *bens\u00ednst\u00f6\u00f0var, -d\u00e6lur og -tankar*.\n* Version 2.1.0: Changed handling of periods at end of sentences if they are\n  a part of an abbreviation. Now, the period is kept attached to the abbreviation,\n  not split off into a separate period token, as before.\n* Version 2.0.7: Added ``TOK.COMPANY`` token type; fixed a few abbreviations;\n  renamed parameter ``text`` to ``text_or_gen`` in functions that accept a string\n  or a string iterator.\n* Version 2.0.6: Fixed handling of abbreviations such as *m.v.* (*mi\u00f0a\u00f0 vi\u00f0*)\n  that should not start a new sentence even if the following word is capitalized.\n* Version 2.0.5: Fixed bug where single uppercase letters were erroneously\n  being recognized as abbreviations, causing prepositions such as '\u00cd' and '\u00c1'\n  at the beginning of sentences to be misunderstood in GreynirPackage.\n* Version 2.0.4: Added imperfect abbreviations (*amk.*, *osfrv.*); recognized\n  *klukkan h\u00e1lf tv\u00f6* as a ``TOK.TIME``.\n* Version 2.0.3: Fixed bug in ``detokenize()`` where abbreviations, domains\n  and e-mails containing periods were wrongly split.\n* Version 2.0.2: Spelled-out day ordinals are no longer included as a part of\n  ``TOK.DATEREL`` tokens. Thus, *\u00feri\u00f0ji j\u00fan\u00ed* is now a ``TOK.WORD``\n  followed by a ``TOK.DATEREL``. *3. j\u00fan\u00ed* continues to be parsed as\n  a single ``TOK.DATEREL``.\n* Version 2.0.1: Order of abbreviation meanings within the ``token.val`` field\n  made deterministic; fixed bug in measurement unit handling.\n* Version 2.0.0: Added command line tool; added ``split_into_sentences()``\n  and ``detokenize()`` functions; removed ``convert_telno`` option;\n  splitting of coalesced tokens made more robust;\n  added ``TOK.SSN``, ``TOK.MOLECULE``, ``TOK.USERNAME`` and\n  ``TOK.SERIALNUMBER`` token kinds; abbreviations can now have multiple\n  meanings.\n* Version 1.4.0: Added the ``**options`` parameter to the\n  ``tokenize()`` function, giving control over the handling of numbers,\n  telephone numbers, and 'kludgy' ordinals.\n* Version 1.3.0: Added ``TOK.DOMAIN`` and ``TOK.HASHTAG`` token types;\n  improved handling of capitalized month name *\u00c1g\u00fast*, which is\n  now recognized when following an ordinal number; improved recognition\n  of telephone numbers; added abbreviations.\n* Version 1.2.3: Added abbreviations; updated GitHub URLs.\n* Version 1.2.2: Added support for composites with more than two parts, i.e.\n  *\u201ed\u00f3msm\u00e1la-, fer\u00f0am\u00e1la-, i\u00f0na\u00f0ar- og n\u00fdsk\u00f6punarr\u00e1\u00f0herra\u201c*; added support for\n  ``\u00b1`` sign; added several abbreviations.\n* Version 1.2.1: Fixed bug where the name *\u00c1g\u00fast* was recognized\n  as a month name; Unicode nonbreaking and invisible space characters\n  are now removed before tokenization.\n* Version 1.2.0: Added support for Unicode fraction characters;\n  enhanced handing of degrees (\u00b0, \u00b0C, \u00b0F); fixed bug in cubic meter\n  measurement unit; more abbreviations.\n* Version 1.1.2: Fixed bug in liter (``l`` and ``ltr``) measurement units.\n* Version 1.1.1: Added ``mark_paragraphs()`` function.\n* Version 1.1.0: All abbreviations in ``Abbrev.conf`` are now\n  returned with their meaning in a tuple in ``token.val``;\n  handling of 'mbl.is' fixed.\n* Version 1.0.9: Added abbreviation 'MAST'; harmonized copyright headers.\n* Version 1.0.8: Bug fixes in ``DATEREL``, ``MEASUREMENT`` and ``NUMWLETTER``\n  token handling; added 'kWst' and 'MWst' measurement units; blackened.\n* Version 1.0.7: Added ``TOK.NUMWLETTER`` token type.\n* Version 1.0.6: Automatic merging of Unicode ``COMBINING ACUTE ACCENT`` and\n  ``COMBINING DIAERESIS`` code points with vowels.\n* Version 1.0.5: Date/time and amount tokens coalesced to a further extent.\n* Version 1.0.4: Added ``TOK.DATEABS``, ``TOK.TIMESTAMPABS``,\n  ``TOK.MEASUREMENT``.\n\n\n\n\n",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/mideind/Tokenizer",
        "keywords": "nlp,tokenizer,icelandic",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "tokenizer",
        "package_url": "https://pypi.org/project/tokenizer/",
        "platform": null,
        "project_url": "https://pypi.org/project/tokenizer/",
        "project_urls": {
            "Homepage": "https://github.com/mideind/Tokenizer"
        },
        "release_url": "https://pypi.org/project/tokenizer/3.4.1/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "A tokenizer for Icelandic text",
        "version": "3.4.1",
        "yanked": false,
        "yanked_reason": null
    },
    "last_serial": 13699142,
    "releases": {
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "51f8450a721544630aef1e6fb6a25b5c",
                    "sha256": "e24eed67a141272d0cb88dfd3fa730082ca1aa119a38c9df57b87fda4fdc30f8"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "51f8450a721544630aef1e6fb6a25b5c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 49706,
                "upload_time": "2017-10-01T16:08:19",
                "upload_time_iso_8601": "2017-10-01T16:08:19.234402Z",
                "url": "https://files.pythonhosted.org/packages/61/50/b35a23d91503fa7ba757129c1bc04d37cfb7e0ead2b19729307f2a6dafb2/tokenizer-0.1.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.1.10": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a2f38d4f1eb5117315ce3af84cf0f14f",
                    "sha256": "79bc0eb2771bc7883333affc4275ae4bb818d97056bb8d1188b29cffdfa07a7d"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.10-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a2f38d4f1eb5117315ce3af84cf0f14f",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 57425,
                "upload_time": "2018-03-13T11:36:36",
                "upload_time_iso_8601": "2018-03-13T11:36:36.586163Z",
                "url": "https://files.pythonhosted.org/packages/6f/f7/eaafb5a04f3f116d7b9ffc6a14a8ac86d8a835ae7c1f416562e15e9cd35a/tokenizer-0.1.10-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c690540e9f85ce46709157156cd48d6a",
                    "sha256": "2f3db1d5889d6cf1e49112e664043285efec2f33fd3f4cb1dfe68ff90b739024"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.10.tar.gz",
                "has_sig": false,
                "md5_digest": "c690540e9f85ce46709157156cd48d6a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 37558,
                "upload_time": "2018-03-13T11:36:37",
                "upload_time_iso_8601": "2018-03-13T11:36:37.835101Z",
                "url": "https://files.pythonhosted.org/packages/36/e2/cb7682af6888cef2853024b20a937cf7c26174a98140f2bbac836603d361/tokenizer-0.1.10.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.1.11": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "75e9601df29e6bb4a14bad0a17fab077",
                    "sha256": "fae881c189cf22d141fca48f0820b16a75fa004e953ccc96ce7ecbb8ef861def"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.11-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "75e9601df29e6bb4a14bad0a17fab077",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 44084,
                "upload_time": "2018-03-27T16:32:27",
                "upload_time_iso_8601": "2018-03-27T16:32:27.304299Z",
                "url": "https://files.pythonhosted.org/packages/3f/0e/54313c2acb3fe97935a34620840d876886bf2a672199b74a71015c0caa8d/tokenizer-0.1.11-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "def6926838a7c7fbdc92598ec8f9883c",
                    "sha256": "c18b968218f68aeac2c1c042fbba05d535e9de3958f7a1f224f4a172469bf504"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.11.tar.gz",
                "has_sig": false,
                "md5_digest": "def6926838a7c7fbdc92598ec8f9883c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 39475,
                "upload_time": "2018-03-27T16:32:28",
                "upload_time_iso_8601": "2018-03-27T16:32:28.616837Z",
                "url": "https://files.pythonhosted.org/packages/98/73/162789b761ad9ccd53717d092d1272a10cbbfcae2f48efd3d45fd3d423ae/tokenizer-0.1.11.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.1.12": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "790e599aed2b5826f421b84ebc2c8a40",
                    "sha256": "8f66449e46c6d9f5e609179eb6fcafca13ec115a934945e8412f30b19da529a1"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.12-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "790e599aed2b5826f421b84ebc2c8a40",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 44084,
                "upload_time": "2018-04-05T16:30:27",
                "upload_time_iso_8601": "2018-04-05T16:30:27.437479Z",
                "url": "https://files.pythonhosted.org/packages/82/9a/a353c42e59c38402f55fe052510bc06d7427dba51b6a7503387bd734227c/tokenizer-0.1.12-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "467a869d471614155c0c8a3039bbbac6",
                    "sha256": "fc2fdf6eea920ca78647f207af036c2ea8dc1c651603ccba3ad6fd4bcb88fa81"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.12.tar.gz",
                "has_sig": false,
                "md5_digest": "467a869d471614155c0c8a3039bbbac6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 39484,
                "upload_time": "2018-04-05T16:30:28",
                "upload_time_iso_8601": "2018-04-05T16:30:28.709703Z",
                "url": "https://files.pythonhosted.org/packages/c3/a7/051d352e0534a87681b72e9af96ef38e27f0f553b0470c36416bd3ff5088/tokenizer-0.1.12.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.1.14": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "746ab3b7eb77d126b37e96368f887d0b",
                    "sha256": "9dc8f8ba760261fdc1b19ccca400ac7eb2353228a35258c2a27b07b8ee2c37ee"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.14-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "746ab3b7eb77d126b37e96368f887d0b",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 44324,
                "upload_time": "2018-04-11T22:17:49",
                "upload_time_iso_8601": "2018-04-11T22:17:49.982268Z",
                "url": "https://files.pythonhosted.org/packages/88/4b/99f6342b53797effd801560dd46b6294b77c26b91d57124a935fbf1faecb/tokenizer-0.1.14-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "021f88b2912e71dd18d4d3b5112b772d",
                    "sha256": "4ded9ba1c0bbd4da325028a675ef960e0b1a4017f857e6c6a004ca084edc56d5"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.14.tar.gz",
                "has_sig": false,
                "md5_digest": "021f88b2912e71dd18d4d3b5112b772d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 39726,
                "upload_time": "2018-04-11T22:17:51",
                "upload_time_iso_8601": "2018-04-11T22:17:51.224002Z",
                "url": "https://files.pythonhosted.org/packages/b7/58/47e66e5fd526955f0f81a08a60b6712a41fada6b7075c6a512c621e144cb/tokenizer-0.1.14.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bc8ad18886c03b320edff4280e1144f8",
                    "sha256": "427b2aa6acedb3c8e89d78503ef4f22a9545cb455e1b48700084121d625d86e4"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "bc8ad18886c03b320edff4280e1144f8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 49676,
                "upload_time": "2017-10-01T16:30:42",
                "upload_time_iso_8601": "2017-10-01T16:30:42.490305Z",
                "url": "https://files.pythonhosted.org/packages/43/92/bf087046d1ee49258c35448cc34608660a705ce5e01c0924b02518da3ed6/tokenizer-0.1.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fb16da94ce5a387c80d2cb00a2cd53eb",
                    "sha256": "9c4da27fcb68f7c5959923f78765f4b775fe337d5d955dfeb74a58214c11d86c"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.3-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "fb16da94ce5a387c80d2cb00a2cd53eb",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 58150,
                "upload_time": "2017-10-01T18:41:08",
                "upload_time_iso_8601": "2017-10-01T18:41:08.312004Z",
                "url": "https://files.pythonhosted.org/packages/37/5e/35ddaa5e3fac34d0a440efd09cd7237fa94ef48047e7b4eca067e8ad51e2/tokenizer-0.1.3-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "0f2d98d55110d35ca63b45ba4b853878",
                    "sha256": "eb1cf95c6f4fcfa87bcb8addc2acf5955b284647398b5fd216e3bcfdf94be4d8"
                },
                "downloads": -1,
                "filename": "tokenizer-0.1.3.tar.gz",
                "has_sig": false,
                "md5_digest": "0f2d98d55110d35ca63b45ba4b853878",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 50373,
                "upload_time": "2017-10-01T18:27:19",
                "upload_time_iso_8601": "2017-10-01T18:27:19.538457Z",
                "url": "https://files.pythonhosted.org/packages/2d/8e/38a33f2e6ed3f39eab6d6d0074ac292386afb431ffc09fd9327069451425/tokenizer-0.1.3.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "453703199bea7e5cf41e5c545b6a9d0a",
                    "sha256": "4669489d162d809e33d4edacbeaa2eacb437cad5e3e9392e0724c1a816cc3183"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "453703199bea7e5cf41e5c545b6a9d0a",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 44402,
                "upload_time": "2018-04-20T12:09:39",
                "upload_time_iso_8601": "2018-04-20T12:09:39.753523Z",
                "url": "https://files.pythonhosted.org/packages/46/88/02b2c7b9e2fb680e8cea4203a2ef3ed2aaa73330e44d3448804aa8ae2ead/tokenizer-1.0.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "20c3d88f6c36fcf67002ec39751e8ff3",
                    "sha256": "f80f57dfb1c8977411069963c8fc0113534a37d08a30745a83770119f71028ce"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "20c3d88f6c36fcf67002ec39751e8ff3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 39800,
                "upload_time": "2018-04-20T12:09:41",
                "upload_time_iso_8601": "2018-04-20T12:09:41.408446Z",
                "url": "https://files.pythonhosted.org/packages/1e/f2/32830931d4fe86996f3a4c895288063997941100fdf0a0bbbc596c024df5/tokenizer-1.0.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "923b667f809b9e0fec4f3a56a0a2a77d",
                    "sha256": "fb0a5c329ddb400919e8409bea724357b27b965a8b020e248bff425453d3170f"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "923b667f809b9e0fec4f3a56a0a2a77d",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 44898,
                "upload_time": "2018-04-24T15:03:01",
                "upload_time_iso_8601": "2018-04-24T15:03:01.539689Z",
                "url": "https://files.pythonhosted.org/packages/c4/a7/ef324aedcaf4b4f7f5fe3244e70e6df569ab12b742b030fbdc6b80d7f108/tokenizer-1.0.1-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "16c52b54690264bc2d4844c2cdf23ca4",
                    "sha256": "1cc9a9d238ba9c5612a7a769cbb2b8501f83faf3c263adbaedb922a953db7d9c"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "16c52b54690264bc2d4844c2cdf23ca4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 40136,
                "upload_time": "2018-04-24T15:03:02",
                "upload_time_iso_8601": "2018-04-24T15:03:02.999513Z",
                "url": "https://files.pythonhosted.org/packages/9a/ec/5d409842b01318dbdf6912a4c64962092b7e977bce2cf7ea9d50bcd93c93/tokenizer-1.0.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8445af548b6ca05b4953a2f562067bcf",
                    "sha256": "5cbab5a776babf070b90a444044180cfc405567d416d0b9506f85213870d9549"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "8445af548b6ca05b4953a2f562067bcf",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 44885,
                "upload_time": "2018-04-25T14:02:25",
                "upload_time_iso_8601": "2018-04-25T14:02:25.029319Z",
                "url": "https://files.pythonhosted.org/packages/98/6f/f2518190080a83ce452f17d78c347f89c29b0efe522521db6302cb27e515/tokenizer-1.0.2-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "200011e7e8c52bbeb5fde02617703dfd",
                    "sha256": "dfc9431549c0bfab0f55e2dd52d76580d5304e85df028925839ed4b7b362de93"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "200011e7e8c52bbeb5fde02617703dfd",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 40193,
                "upload_time": "2018-04-25T14:02:26",
                "upload_time_iso_8601": "2018-04-25T14:02:26.450333Z",
                "url": "https://files.pythonhosted.org/packages/da/58/1ed6a7a021d11ca1675513b56f07ebede69be69600e3329fd4832b5fa2f8/tokenizer-1.0.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7da12128e7925b68519609ab88ded473",
                    "sha256": "e6a5a9812c0efa7a8de676d06a0f5c26b4feaea53ca536b1aef13a20072dee71"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.3-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "7da12128e7925b68519609ab88ded473",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 45309,
                "upload_time": "2018-04-30T13:51:40",
                "upload_time_iso_8601": "2018-04-30T13:51:40.056805Z",
                "url": "https://files.pythonhosted.org/packages/01/f4/c3d461945ef3dce6f725a01639176122b25d72bd33a8746c61438c7f8f33/tokenizer-1.0.3-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "9fa4da39315201d69959ae3564a29da3",
                    "sha256": "d2c4f4d356d1a82b8755d7b7e730b1060e9f82e1f6bcf4752781f7133bf409e8"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "9fa4da39315201d69959ae3564a29da3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 40493,
                "upload_time": "2018-04-30T13:51:41",
                "upload_time_iso_8601": "2018-04-30T13:51:41.388895Z",
                "url": "https://files.pythonhosted.org/packages/5a/83/e2584026ae01b2eade0be1779d75c5d2ac7f1c9f2557387473c22b1e0b2d/tokenizer-1.0.3.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ecc37be1f81e799c68c51fb19a14f0f7",
                    "sha256": "573d4f754949c96661e7ba250e637cda3106064580cea6404d1e4d3b4aff3034"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.4-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ecc37be1f81e799c68c51fb19a14f0f7",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 55258,
                "upload_time": "2018-06-08T17:48:32",
                "upload_time_iso_8601": "2018-06-08T17:48:32.123278Z",
                "url": "https://files.pythonhosted.org/packages/45/88/b623f12bb31f4cb54c406596128060458cfe9b8a16832b69a20468cfe0f4/tokenizer-1.0.4-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "9f2ffd544902591bb5fdaae91f9b7c0c",
                    "sha256": "dc2e64a3c9d8068a77b737a94bd48b2c7053dd39bce426d70e68ef95be027163"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "9f2ffd544902591bb5fdaae91f9b7c0c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 54462,
                "upload_time": "2018-06-08T17:48:33",
                "upload_time_iso_8601": "2018-06-08T17:48:33.599661Z",
                "url": "https://files.pythonhosted.org/packages/0c/93/0b982ee194128528fa1a3aebb371fbd461ce1479323a0c4b3e55d2a2c692/tokenizer-1.0.4.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0c85816a67fb172fae9af62fa24d25d6",
                    "sha256": "386a8ccab98cff94752ac92736f35fead6d122ba63ad9d770ca73672e22b0b15"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.5-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "0c85816a67fb172fae9af62fa24d25d6",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 60633,
                "upload_time": "2018-07-23T15:50:40",
                "upload_time_iso_8601": "2018-07-23T15:50:40.306607Z",
                "url": "https://files.pythonhosted.org/packages/58/21/4b8618eecc77b2aa60505dfcd660e1148682c2a8c101e37dd0667550c0b1/tokenizer-1.0.5-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "6f71ba3996e65e33852ae776ecfb9eaf",
                    "sha256": "aebcc8b4bd3da2323ef4965558e27d5cb11682376a7897f096feb0a8a4fa5fe8"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "6f71ba3996e65e33852ae776ecfb9eaf",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 45680,
                "upload_time": "2018-07-23T15:50:41",
                "upload_time_iso_8601": "2018-07-23T15:50:41.494234Z",
                "url": "https://files.pythonhosted.org/packages/7d/50/d57447db0eea1ab161f53b7b23119426ed6d26399d9f70e33b2520b7a8b2/tokenizer-1.0.5.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fde46bf8210ac46ff033aaa85760ad65",
                    "sha256": "77ae1fd23e3823509dd089eef361932c588aaf4b7842dbc1d84eb15b66564466"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.6-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "fde46bf8210ac46ff033aaa85760ad65",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 68110,
                "upload_time": "2018-08-23T17:09:54",
                "upload_time_iso_8601": "2018-08-23T17:09:54.167380Z",
                "url": "https://files.pythonhosted.org/packages/96/a3/7ed7c578ed2ce3ce466863c73acb8c6d580edf3985d81e79401e01538ee5/tokenizer-1.0.6-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "aa47d826904a99ba59ca6cd4eaee7957",
                    "sha256": "50cc5858da1832aefbd31b7ee8cc94866eee7d9af792bdea5971ae3114a757a0"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.6.tar.gz",
                "has_sig": false,
                "md5_digest": "aa47d826904a99ba59ca6cd4eaee7957",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 54501,
                "upload_time": "2018-08-23T17:09:55",
                "upload_time_iso_8601": "2018-08-23T17:09:55.983566Z",
                "url": "https://files.pythonhosted.org/packages/16/a9/a9b0aa81bdacb9486305c96d6ef2e9f94819cc9892304ec425986a73b30b/tokenizer-1.0.6.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "452ba09e534b5ae07666ee276e109626",
                    "sha256": "c651c3343be5d33f5a4c33a6064b2c8abbf28695459717a00be35fac2491230e"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.7-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "452ba09e534b5ae07666ee276e109626",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 68324,
                "upload_time": "2018-09-25T12:13:01",
                "upload_time_iso_8601": "2018-09-25T12:13:01.215018Z",
                "url": "https://files.pythonhosted.org/packages/b4/3c/6f444db3966704a76403feffd393803c3234295a213e6d5b680cfc6b9785/tokenizer-1.0.7-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "aa21d580f468cbabd38ba937bafba06f",
                    "sha256": "0d597a5c13be013b83d3f7291d3e454402252df9e92068940f7b0a0419bb8476"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.7.tar.gz",
                "has_sig": false,
                "md5_digest": "aa21d580f468cbabd38ba937bafba06f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 54965,
                "upload_time": "2018-09-25T12:13:03",
                "upload_time_iso_8601": "2018-09-25T12:13:03.297483Z",
                "url": "https://files.pythonhosted.org/packages/bb/11/7130c7f478207147514ff3e9456c5f64b7999125e3731ede797c1b363303/tokenizer-1.0.7.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.0.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c8601eecb9d2dc008a12647921b8edee",
                    "sha256": "daddad477333c46c55f5f6e9c9e0f9a8cf3cf10d1624a1e1b1073812778c5d97"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.8-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "c8601eecb9d2dc008a12647921b8edee",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 60704,
                "upload_time": "2018-10-30T18:33:10",
                "upload_time_iso_8601": "2018-10-30T18:33:10.670692Z",
                "url": "https://files.pythonhosted.org/packages/af/8f/fc44a4ce5268382b87df43638a83d41a729cf3b0821a9c1b2494f7113e35/tokenizer-1.0.8-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "555fe52af91f4edc0071631f4841635b",
                    "sha256": "282ef136ab122c58a1c16b4f2926928f23c6f9e106f8b0c92a932c4c001eafaa"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.8.tar.gz",
                "has_sig": false,
                "md5_digest": "555fe52af91f4edc0071631f4841635b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 55783,
                "upload_time": "2018-10-30T18:33:12",
                "upload_time_iso_8601": "2018-10-30T18:33:12.559897Z",
                "url": "https://files.pythonhosted.org/packages/fe/49/da364494aee3526490bfea378f6c14c6a05bdbca030f4966addc4a9e6504/tokenizer-1.0.8.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.0.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2ade3b4246258baa37a743db78cb38a0",
                    "sha256": "a31a7e3983a0ecdc362ec50c494649698bc2602c5e43ba3d81612d2212688168"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.9-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "2ade3b4246258baa37a743db78cb38a0",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 61221,
                "upload_time": "2018-12-29T13:04:15",
                "upload_time_iso_8601": "2018-12-29T13:04:15.862248Z",
                "url": "https://files.pythonhosted.org/packages/d8/98/989d2fd86f8e6e92ffb19ff990ec2f0a3a908b731a540d1794b549c65cad/tokenizer-1.0.9-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c5b13e81b84aecc46a7904eed108bcf0",
                    "sha256": "244c8c7e7606e9410f2dedf1e359fb73ab30f131eb3c8350a16ed8248313ded1"
                },
                "downloads": -1,
                "filename": "tokenizer-1.0.9.tar.gz",
                "has_sig": false,
                "md5_digest": "c5b13e81b84aecc46a7904eed108bcf0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 59879,
                "upload_time": "2018-12-29T13:04:17",
                "upload_time_iso_8601": "2018-12-29T13:04:17.658148Z",
                "url": "https://files.pythonhosted.org/packages/b1/cf/a542f3972f28ca5017ac77cf6a8168288bbfc93075c94b93c04fb0c8b0eb/tokenizer-1.0.9.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9ea98409a3285355b1c04d37196e861c",
                    "sha256": "538d28ab767eb417a8b2318d4e1b4b437f7bf2699381042ec4a96bb3af00d13f"
                },
                "downloads": -1,
                "filename": "tokenizer-1.1.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "9ea98409a3285355b1c04d37196e861c",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 61732,
                "upload_time": "2019-01-02T14:37:20",
                "upload_time_iso_8601": "2019-01-02T14:37:20.070458Z",
                "url": "https://files.pythonhosted.org/packages/1f/50/d27c3fea1f3e367d1e4eab7600b1ee3797eba230f06baaf2d77f13bedb95/tokenizer-1.1.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "1f3921126b2b2c8627f8c3c8dd796d58",
                    "sha256": "4dcbecc913eac115d7c72c9a58bd8a33978a55559492eb1b8af8a4acd539abbd"
                },
                "downloads": -1,
                "filename": "tokenizer-1.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "1f3921126b2b2c8627f8c3c8dd796d58",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 60674,
                "upload_time": "2019-01-02T14:37:22",
                "upload_time_iso_8601": "2019-01-02T14:37:22.125265Z",
                "url": "https://files.pythonhosted.org/packages/dc/7e/8322e7b1c8ba315813eb81fbcc91e8e26b576ba53433af955c4859ab772d/tokenizer-1.1.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f098d23c3f8e0ccc1a86ab89e97804b8",
                    "sha256": "e382dbb0f0f3d936817aa100ca02e90e976ce11ee2635df9208d1dc0d6cd8a05"
                },
                "downloads": -1,
                "filename": "tokenizer-1.1.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "f098d23c3f8e0ccc1a86ab89e97804b8",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 61993,
                "upload_time": "2019-01-04T18:24:20",
                "upload_time_iso_8601": "2019-01-04T18:24:20.016852Z",
                "url": "https://files.pythonhosted.org/packages/ad/e1/543445781df1b7b6a40332f522d19b857bd004a51717a4495062eb5bf7fe/tokenizer-1.1.1-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c97d9dbbbfd8c942229267ce7c98e768",
                    "sha256": "845a47213fe4accdcb7b8bd4ff7652fb53b9475b7d9cfc5f07d54143ff512dfe"
                },
                "downloads": -1,
                "filename": "tokenizer-1.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "c97d9dbbbfd8c942229267ce7c98e768",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 60925,
                "upload_time": "2019-01-04T18:24:22",
                "upload_time_iso_8601": "2019-01-04T18:24:22.044407Z",
                "url": "https://files.pythonhosted.org/packages/8e/3d/971d40b67ceeb8ae48b54e95dfa7251c021d5b39645aa393bcf28fd1b50f/tokenizer-1.1.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "64bf5770c6aaa9156fb017c904e84b64",
                    "sha256": "d7889a728852f95dfe93ecd1c6e07f4dec99d0767a4b062ef01b5ddc89e972e9"
                },
                "downloads": -1,
                "filename": "tokenizer-1.1.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "64bf5770c6aaa9156fb017c904e84b64",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 62053,
                "upload_time": "2019-01-10T11:36:20",
                "upload_time_iso_8601": "2019-01-10T11:36:20.219682Z",
                "url": "https://files.pythonhosted.org/packages/e1/ad/b927b6d4a3af94416a5e5598ddbc73daa3e2ab7dc8570643254e619ad18d/tokenizer-1.1.2-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "231e865e2eb4d899f283e916a5e69de4",
                    "sha256": "16014d50c6e80489bc36de7a76de10b5ce7eca6a286979cdb9bd015699ece319"
                },
                "downloads": -1,
                "filename": "tokenizer-1.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "231e865e2eb4d899f283e916a5e69de4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 61049,
                "upload_time": "2019-01-10T11:36:22",
                "upload_time_iso_8601": "2019-01-10T11:36:22.330985Z",
                "url": "https://files.pythonhosted.org/packages/44/3e/0da2f03137478903dd588f20167d7ae56d45a2ab195ab3a67c61436a7b88/tokenizer-1.1.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "54298593197adaf685d8c50b83063747",
                    "sha256": "6c794eb5e288d5b976772625a5d5b8b2aca83b3fea4b3a3e10c08c1304338cbe"
                },
                "downloads": -1,
                "filename": "tokenizer-1.2.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "54298593197adaf685d8c50b83063747",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 63043,
                "upload_time": "2019-02-07T16:35:11",
                "upload_time_iso_8601": "2019-02-07T16:35:11.381587Z",
                "url": "https://files.pythonhosted.org/packages/11/76/50c693df95bcec3f28203d7837d5ce0e5a7e28c3f31fb379e743843fb292/tokenizer-1.2.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c2d3013d3ba32a204c49a2e92768b9ce",
                    "sha256": "464d6a0a6c0adb3ccb2c616c7fe18bbe817fed25534f34a6b6895464c7d59afa"
                },
                "downloads": -1,
                "filename": "tokenizer-1.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "c2d3013d3ba32a204c49a2e92768b9ce",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 62460,
                "upload_time": "2019-02-07T16:35:13",
                "upload_time_iso_8601": "2019-02-07T16:35:13.562615Z",
                "url": "https://files.pythonhosted.org/packages/39/29/ae65b522395357a91d92d6d857fd899eb5e4171dd2f2f0fe1d32bef979b2/tokenizer-1.2.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "aacb37c62ea6b13051c5f44edee14c0c",
                    "sha256": "cdc859ba3c732dd9f076fc0ad3ea970d4092fe362b802586e4dd47d04f79ac48"
                },
                "downloads": -1,
                "filename": "tokenizer-1.2.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "aacb37c62ea6b13051c5f44edee14c0c",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 63783,
                "upload_time": "2019-02-18T19:17:49",
                "upload_time_iso_8601": "2019-02-18T19:17:49.756457Z",
                "url": "https://files.pythonhosted.org/packages/1b/46/dee3e5628effb7efc13fcbad082097fd29c6c6c1960b3d757d529d90720b/tokenizer-1.2.1-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "98c01956cdf35315835b59d07cbdc186",
                    "sha256": "e361c713641c32f91659e8e4771dfa974c0b5f170f08145074142bfc1fbdab63"
                },
                "downloads": -1,
                "filename": "tokenizer-1.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "98c01956cdf35315835b59d07cbdc186",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 63605,
                "upload_time": "2019-02-18T19:17:51",
                "upload_time_iso_8601": "2019-02-18T19:17:51.567305Z",
                "url": "https://files.pythonhosted.org/packages/7a/18/36770f34671bfc8ba4c3c3829f51485eea502b1c4b3c0cb13972f780f7ea/tokenizer-1.2.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.2.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d8b1e652565dd3ca5fa5e8e3bbe8353f",
                    "sha256": "e6f0deb2f5322b71b82326eaf9d52249b8acca6d0fd7d8087197f3e2d32622b0"
                },
                "downloads": -1,
                "filename": "tokenizer-1.2.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d8b1e652565dd3ca5fa5e8e3bbe8353f",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 64478,
                "upload_time": "2019-04-26T13:15:51",
                "upload_time_iso_8601": "2019-04-26T13:15:51.798885Z",
                "url": "https://files.pythonhosted.org/packages/1c/aa/be419418313c47c4b6eaa6666ebbb164c896e33a70e74faa45f0e473a619/tokenizer-1.2.2-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ce22c0d23ffc9ce14505e4e49a4b2637",
                    "sha256": "d0f4550ddcabf617e904f3d429195df4a623912dfbf97e7780e0d0088703f5a8"
                },
                "downloads": -1,
                "filename": "tokenizer-1.2.2.tar.gz",
                "has_sig": false,
                "md5_digest": "ce22c0d23ffc9ce14505e4e49a4b2637",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 59840,
                "upload_time": "2019-04-26T13:15:55",
                "upload_time_iso_8601": "2019-04-26T13:15:55.374881Z",
                "url": "https://files.pythonhosted.org/packages/bd/7e/ecb493d085189baa8ab0987d74ed882fb72ff8e8351cf36eaca9540c751d/tokenizer-1.2.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.2.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c03c445508dddcb97470028dff8b27ec",
                    "sha256": "5132f0ccee0a613e9a48106a816f1abdafd90e86656bac3f2e4277ab3eeb301a"
                },
                "downloads": -1,
                "filename": "tokenizer-1.2.3-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "c03c445508dddcb97470028dff8b27ec",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 72466,
                "upload_time": "2019-05-03T11:42:01",
                "upload_time_iso_8601": "2019-05-03T11:42:01.007615Z",
                "url": "https://files.pythonhosted.org/packages/f3/b1/eb1715a312b949ac1001e23d387f39833e4683f136239e193ae4fa2f73ec/tokenizer-1.2.3-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "845f4770a1a7e5b3985918d6cdec5c67",
                    "sha256": "44195dbc1c312773bcc2476cfdb32ff5ea2ea57244edd58d2857034cd3cc37e1"
                },
                "downloads": -1,
                "filename": "tokenizer-1.2.3.tar.gz",
                "has_sig": false,
                "md5_digest": "845f4770a1a7e5b3985918d6cdec5c67",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 59921,
                "upload_time": "2019-05-03T11:42:02",
                "upload_time_iso_8601": "2019-05-03T11:42:02.898999Z",
                "url": "https://files.pythonhosted.org/packages/f5/7e/a157345a652781fcfdb4dfc6e06eb9b8514499968c6fef8779f28c35cee2/tokenizer-1.2.3.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c7d85265e66e07b940621b083fb5392f",
                    "sha256": "65e20f53974b936018a4a9f55af8b5bf45927f84e22f723d6cb8b95c8120d961"
                },
                "downloads": -1,
                "filename": "tokenizer-1.3.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "c7d85265e66e07b940621b083fb5392f",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 75955,
                "upload_time": "2019-05-21T11:19:42",
                "upload_time_iso_8601": "2019-05-21T11:19:42.335699Z",
                "url": "https://files.pythonhosted.org/packages/08/ad/bdf7cac9415855ce61249be3549c8d32f781c3d3b54ad754d42078f152ce/tokenizer-1.3.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ce22cb92db7eb619ac7c3a16978fc3ee",
                    "sha256": "b6b68b3873dbb946d97331b2d43c0198530a4b4ca4d42e18e2bb9122ffefed02"
                },
                "downloads": -1,
                "filename": "tokenizer-1.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "ce22cb92db7eb619ac7c3a16978fc3ee",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 63563,
                "upload_time": "2019-05-21T11:19:44",
                "upload_time_iso_8601": "2019-05-21T11:19:44.682776Z",
                "url": "https://files.pythonhosted.org/packages/e1/8c/039292bb0855fb7543e20ac028dca0cc593a819a26128d8cb4c8106af22e/tokenizer-1.3.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7a0b61472bb1f11913eafca4eedbf9b1",
                    "sha256": "53545f701e257e409014f995976d06f2183432fceb1c9a80bc18b708510ef8a0"
                },
                "downloads": -1,
                "filename": "tokenizer-1.4.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "7a0b61472bb1f11913eafca4eedbf9b1",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 78231,
                "upload_time": "2019-07-16T17:05:41",
                "upload_time_iso_8601": "2019-07-16T17:05:41.673451Z",
                "url": "https://files.pythonhosted.org/packages/4d/af/46342e59df9dbb195f24554b26e082e40a8bc2689a33d72a011c1a3b78df/tokenizer-1.4.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "09145bed4265e30b02340ea97acc8772",
                    "sha256": "9da08dce06565a5fca3f3ab472cd6608a98830c1d5165dd53ec13a340f555b12"
                },
                "downloads": -1,
                "filename": "tokenizer-1.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "09145bed4265e30b02340ea97acc8772",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 67144,
                "upload_time": "2019-07-16T17:05:43",
                "upload_time_iso_8601": "2019-07-16T17:05:43.549996Z",
                "url": "https://files.pythonhosted.org/packages/cf/00/20a5bb856e781ba5305e46ba134b1597b52caea4ca5dc5fe62210a8ad987/tokenizer-1.4.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "1.4.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0571cb711d00f3c575f4c9b3923bc6c3",
                    "sha256": "1ec6325327d45a82841356e343b434c89df96484abe9c6ba2109f85ec65cba7f"
                },
                "downloads": -1,
                "filename": "tokenizer-1.4.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "0571cb711d00f3c575f4c9b3923bc6c3",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 80019,
                "upload_time": "2019-10-22T17:18:46",
                "upload_time_iso_8601": "2019-10-22T17:18:46.923408Z",
                "url": "https://files.pythonhosted.org/packages/1a/ef/2e9a3f764907a46e965d0c311eab8312290dfe723ae10ed2d776583174e0/tokenizer-1.4.1-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "49a70198793bc9c1d75bf58c6f2e7d0b",
                    "sha256": "75e9ebc0f6dadad3cb82ba6076a418fcea0b8370d879b5f6b9e3317c8484ebe8"
                },
                "downloads": -1,
                "filename": "tokenizer-1.4.1.tar.gz",
                "has_sig": false,
                "md5_digest": "49a70198793bc9c1d75bf58c6f2e7d0b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 68905,
                "upload_time": "2019-10-22T17:18:48",
                "upload_time_iso_8601": "2019-10-22T17:18:48.903658Z",
                "url": "https://files.pythonhosted.org/packages/4a/3f/ac6ae2595e2ddba612d5815bc8c86527cd8d865719f7b9ff3a2db8e43f7c/tokenizer-1.4.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "af3f7c81bffa6621dffa63b7b2a7f9a1",
                    "sha256": "6728d0eccc1f5386817bf28859d70e16512f2b9fb8053b2159e5f54c5b071214"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "af3f7c81bffa6621dffa63b7b2a7f9a1",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 106484,
                "upload_time": "2019-12-04T16:05:24",
                "upload_time_iso_8601": "2019-12-04T16:05:24.068148Z",
                "url": "https://files.pythonhosted.org/packages/5e/0e/1e253c56542934b053c6021d7e5464206c8c2445fbcd72e76b4168b9499d/tokenizer-2.0.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "7af130f4f4bc4b107c7f4d4f6bb99ae0",
                    "sha256": "ef4005840a573ebb4787dbaedc0bb6b4aecf6b10fa359bb2d1d12d41b34b5c15"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "7af130f4f4bc4b107c7f4d4f6bb99ae0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 112974,
                "upload_time": "2019-12-04T16:05:26",
                "upload_time_iso_8601": "2019-12-04T16:05:26.390468Z",
                "url": "https://files.pythonhosted.org/packages/7d/48/597b086edab63dfc38a34c0853244f740587c7cee8edadf3d160691aa2e1/tokenizer-2.0.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "41e297b3304195a76b6c29e89def559b",
                    "sha256": "790b938d2b509e832ae4d89a0bfb9f45548d05c9e0de6e48b60a581d0294ff7e"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "41e297b3304195a76b6c29e89def559b",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 107344,
                "upload_time": "2019-12-09T15:33:20",
                "upload_time_iso_8601": "2019-12-09T15:33:20.518929Z",
                "url": "https://files.pythonhosted.org/packages/6f/4e/613393a8ba3154f8b58cef5af1585dab33cc08430fdd0cc72896c131b79a/tokenizer-2.0.1-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "da5e6685af94114cdfc5d45a60d272d8",
                    "sha256": "e0773953ca9a47bdedb10863539e41ba623519e0341a9a89541fedc91ce85963"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "da5e6685af94114cdfc5d45a60d272d8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 114828,
                "upload_time": "2019-12-09T15:33:23",
                "upload_time_iso_8601": "2019-12-09T15:33:23.180765Z",
                "url": "https://files.pythonhosted.org/packages/08/36/3e75dd86ec5beb11c8f9e07b4e446f7c4372ba9edb123d7c73ec8d8bfc46/tokenizer-2.0.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "03da0d25ed265200c8133ed4f3c91fa9",
                    "sha256": "1a169066ca9f34e89fa646089f34656e69d983186b0af2fafd9a421f68174808"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "03da0d25ed265200c8133ed4f3c91fa9",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 107466,
                "upload_time": "2019-12-11T11:34:19",
                "upload_time_iso_8601": "2019-12-11T11:34:19.938985Z",
                "url": "https://files.pythonhosted.org/packages/97/bd/83b997aa8f360b813ddab004adff36f711812f0a498f957871b0c281bb84/tokenizer-2.0.2-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "963ee4dfa402cece0c595fb1ee9467de",
                    "sha256": "0d53b9035af9b217f4cc9c31beb52c15e75fe6bfbef250125e588c05645d7731"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "963ee4dfa402cece0c595fb1ee9467de",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 115211,
                "upload_time": "2019-12-11T11:34:22",
                "upload_time_iso_8601": "2019-12-11T11:34:22.689694Z",
                "url": "https://files.pythonhosted.org/packages/c0/0c/0733cd853170fb3fd6cccd180ef54cde6c4c2e9667e8e733605bc2e7d386/tokenizer-2.0.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "eaffcb4f0d1449e2e56bccbd3ab2938c",
                    "sha256": "49693ea39fd4bcd74bff4087d62c7020e75b6e30950b6c02a597438e6c3ce908"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.3-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "eaffcb4f0d1449e2e56bccbd3ab2938c",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 107557,
                "upload_time": "2019-12-17T15:54:23",
                "upload_time_iso_8601": "2019-12-17T15:54:23.714034Z",
                "url": "https://files.pythonhosted.org/packages/65/48/49fd393348a4dcb796e901732ebab32d7a37e826fa35b72702eeea2e2f77/tokenizer-2.0.3-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e7ee17809fda7563a56ad89ed707a926",
                    "sha256": "ff9ff525c32cbec46e254bf343ed1f5ba0e39436bbca33a5912277b0c463e0b1"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "e7ee17809fda7563a56ad89ed707a926",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 115677,
                "upload_time": "2019-12-17T15:54:26",
                "upload_time_iso_8601": "2019-12-17T15:54:26.383162Z",
                "url": "https://files.pythonhosted.org/packages/f0/3a/8f391d16cf47bc3847d5212c88dfa7ad3e27d071d6ae8e088a307c1c62db/tokenizer-2.0.3.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9a5a2d7b06b46a41b100922abbf296b0",
                    "sha256": "3085d93464967b6d79b4eaa5e06707cdbcef0543951cc0f2e444eab9d6d539f8"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.4-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "9a5a2d7b06b46a41b100922abbf296b0",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 104908,
                "upload_time": "2020-02-17T16:17:02",
                "upload_time_iso_8601": "2020-02-17T16:17:02.906031Z",
                "url": "https://files.pythonhosted.org/packages/b2/ec/d049556036ba76c3002d08410c3864b1b6daa175562156a6d0431dc7e0cd/tokenizer-2.0.4-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "8c7ee8954a2d87cc42d6e17161aa094e",
                    "sha256": "5a99d8ca4673786b0ca4e4802e7ee2937da5a091932eb6c45894fa893da7779e"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "8c7ee8954a2d87cc42d6e17161aa094e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 116228,
                "upload_time": "2020-02-17T16:17:05",
                "upload_time_iso_8601": "2020-02-17T16:17:05.382783Z",
                "url": "https://files.pythonhosted.org/packages/36/a0/0efac5c15129829815016056a21db306b04fd74ec44a22f653d619ff663f/tokenizer-2.0.4.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "aaa66bf866477127fe0705bbe11e5f73",
                    "sha256": "7c1ac32ef74e1da0be0a1b864b78cf5e13b002f494139faec21562b49a9997a4"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.5-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "aaa66bf866477127fe0705bbe11e5f73",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 105378,
                "upload_time": "2020-03-26T23:28:27",
                "upload_time_iso_8601": "2020-03-26T23:28:27.872018Z",
                "url": "https://files.pythonhosted.org/packages/a5/22/440325f97a442a2d5e2d01703dbedc80deb168f8f8f5a5480fc2e7a8979e/tokenizer-2.0.5-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "0dc2f1137db2c347dbcf1135498a21c7",
                    "sha256": "75961c560e6f8baed2128598220cc0177da5d0733d6a33c26922e5e4e59582e4"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "0dc2f1137db2c347dbcf1135498a21c7",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 116863,
                "upload_time": "2020-03-26T23:28:30",
                "upload_time_iso_8601": "2020-03-26T23:28:30.374174Z",
                "url": "https://files.pythonhosted.org/packages/4a/bb/82268b7bf0dee4a27a0db53b62d8c60af295ebb37396b2f4910079a2df7c/tokenizer-2.0.5.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9740cf95644c76c2f81f99edd4ea0388",
                    "sha256": "deb66812d4ac460f2fbf96fe7170e510973138810fe23abe9283b81562675484"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.6-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "9740cf95644c76c2f81f99edd4ea0388",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 473873,
                "upload_time": "2020-05-29T12:58:18",
                "upload_time_iso_8601": "2020-05-29T12:58:18.536733Z",
                "url": "https://files.pythonhosted.org/packages/66/c5/f3c2f6163f48eaac476e03ec99ade1aa642071431c95fd895a0ce966b80e/tokenizer-2.0.6-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "5c31ee14345f056c646b2a2b66ade6c5",
                    "sha256": "142664e6dc0d257fee208fa52a7e1b9c63436479a683b4c2c57987570d07e646"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.6.tar.gz",
                "has_sig": false,
                "md5_digest": "5c31ee14345f056c646b2a2b66ade6c5",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 405769,
                "upload_time": "2020-05-29T12:58:20",
                "upload_time_iso_8601": "2020-05-29T12:58:20.991813Z",
                "url": "https://files.pythonhosted.org/packages/18/d3/a9eb7eccf072129e9b586946850468a8282d394afeacc25f1481a5a1c07d/tokenizer-2.0.6.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f4e8d8a5d76f1f18d50dac04f20798b3",
                    "sha256": "64c47f24d0abc156b934e0fc854edd2bbf789f7201d66d6e016e80b4c990db92"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.7-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "f4e8d8a5d76f1f18d50dac04f20798b3",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 471369,
                "upload_time": "2020-06-24T15:50:44",
                "upload_time_iso_8601": "2020-06-24T15:50:44.058662Z",
                "url": "https://files.pythonhosted.org/packages/a5/2c/8457eda3a168ff197e9c4d5492e4d198dac759ba8d7b164f4d5420355a87/tokenizer-2.0.7-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "552a0060e88118f72783000e6eab55aa",
                    "sha256": "ae0d8931a7391510d5c75089f6c2dca7667f0749f62a5672299a3ec85bb1ef58"
                },
                "downloads": -1,
                "filename": "tokenizer-2.0.7.tar.gz",
                "has_sig": false,
                "md5_digest": "552a0060e88118f72783000e6eab55aa",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 406102,
                "upload_time": "2020-06-24T15:50:46",
                "upload_time_iso_8601": "2020-06-24T15:50:46.015787Z",
                "url": "https://files.pythonhosted.org/packages/34/bc/21a31b7193593966bd39f177f41fc6c479a78f44278d545e14ad392fa3a2/tokenizer-2.0.7.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "13326a7e4b689eaf44e2f90972e5b0f2",
                    "sha256": "5c248c2abd11def4598872abef6b1f432c5ebe8ec23724f1656a8fed35b65f6b"
                },
                "downloads": -1,
                "filename": "tokenizer-2.1.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "13326a7e4b689eaf44e2f90972e5b0f2",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 471580,
                "upload_time": "2020-07-02T16:07:03",
                "upload_time_iso_8601": "2020-07-02T16:07:03.424902Z",
                "url": "https://files.pythonhosted.org/packages/a8/7e/e68e6b91c13028b12aa0e7079f962888a453cf9240d168ae25ded612a3e1/tokenizer-2.1.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "cc40d0ac369f77e97637597696176026",
                    "sha256": "4f1723f024b03ee649af8688b08e001011c6bc6c0580c50e3a2452f0aa6e5bc9"
                },
                "downloads": -1,
                "filename": "tokenizer-2.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "cc40d0ac369f77e97637597696176026",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 120778,
                "upload_time": "2020-07-02T16:07:05",
                "upload_time_iso_8601": "2020-07-02T16:07:05.640954Z",
                "url": "https://files.pythonhosted.org/packages/4e/b9/f46b45b0a56667aa80eb40d569384141b634dab551ca067e87a6b6724e65/tokenizer-2.1.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "320210edb6b35ded06a5878b7ee61365",
                    "sha256": "64e3592e3dbbd430e08c4d8428db39e9d42f014e9310ce29b6be05aaf093491a"
                },
                "downloads": -1,
                "filename": "tokenizer-2.2.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "320210edb6b35ded06a5878b7ee61365",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 477560,
                "upload_time": "2020-08-20T22:18:25",
                "upload_time_iso_8601": "2020-08-20T22:18:25.206194Z",
                "url": "https://files.pythonhosted.org/packages/08/8b/95e2e9947e5523bf6d839d0bfa2844e24b8970106e2d69dbc9c3bba651b3/tokenizer-2.2.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "73c82c45afc5f39592460db5feb3134c",
                    "sha256": "f6694c5afc05516e5f10e233bca56dfc03b178a414fcdc173099f4635ef1526c"
                },
                "downloads": -1,
                "filename": "tokenizer-2.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "73c82c45afc5f39592460db5feb3134c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 127211,
                "upload_time": "2020-08-20T22:18:27",
                "upload_time_iso_8601": "2020-08-20T22:18:27.458454Z",
                "url": "https://files.pythonhosted.org/packages/3f/e1/5a39729853dec0104ecf91cebbc80232ffcc1d73b2b6f7296d5d70055f6a/tokenizer-2.2.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ccc9c28a0bb514606cac1a1e7f4a1ae9",
                    "sha256": "52bfcab46e6c9ac5dbbaaa679485c743243e0db40c3e0c78aeb8512fea3ca078"
                },
                "downloads": -1,
                "filename": "tokenizer-2.3.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ccc9c28a0bb514606cac1a1e7f4a1ae9",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 116861,
                "upload_time": "2020-09-03T18:05:33",
                "upload_time_iso_8601": "2020-09-03T18:05:33.511639Z",
                "url": "https://files.pythonhosted.org/packages/ba/09/117167b7535d6b965e98c55b8c18d6d4c2f98fb06f6989de31f1baf6e746/tokenizer-2.3.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e25ef921195a66257bc35025e20944ba",
                    "sha256": "4466a27af0549a7e2066b06725b1b176aa896cfa52a79deb4e4ae78bf542dd7e"
                },
                "downloads": -1,
                "filename": "tokenizer-2.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "e25ef921195a66257bc35025e20944ba",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 129513,
                "upload_time": "2020-09-03T18:05:35",
                "upload_time_iso_8601": "2020-09-03T18:05:35.978782Z",
                "url": "https://files.pythonhosted.org/packages/85/90/dab7976dd0ff4eeeb764e86bcd65ed6138960e7d3743ef6080969369c13b/tokenizer-2.3.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.3.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "480972574cece4d8ce4ef88089196d85",
                    "sha256": "290a39d2cef1b1bd436079c0da975309d8555bc41af6afaf944496ffe2f419ee"
                },
                "downloads": -1,
                "filename": "tokenizer-2.3.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "480972574cece4d8ce4ef88089196d85",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 117187,
                "upload_time": "2020-09-21T12:05:27",
                "upload_time_iso_8601": "2020-09-21T12:05:27.925444Z",
                "url": "https://files.pythonhosted.org/packages/da/2d/760135ffeaba5d250d23c39fd6785f965c396768f342c688d68c8dab2f9d/tokenizer-2.3.1-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "a3a470c37b1608d872b6ead75b687e1a",
                    "sha256": "ccad126f982411e3e1fe41461c033c009cf12338e6e40df767f070f39fdaeab9"
                },
                "downloads": -1,
                "filename": "tokenizer-2.3.1.tar.gz",
                "has_sig": false,
                "md5_digest": "a3a470c37b1608d872b6ead75b687e1a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 129925,
                "upload_time": "2020-09-21T12:05:31",
                "upload_time_iso_8601": "2020-09-21T12:05:31.075875Z",
                "url": "https://files.pythonhosted.org/packages/ac/44/959c10a5b780b8ebd22da066010d4f33bee9b95d3a0428c4ee30fb7b04b9/tokenizer-2.3.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3bed062483380e6067f1977a350c40fd",
                    "sha256": "45656060f31467f9bbf21e394118b09b6a06f6847ac956992d325271dc1b7da2"
                },
                "downloads": -1,
                "filename": "tokenizer-2.4.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "3bed062483380e6067f1977a350c40fd",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 105584,
                "upload_time": "2020-10-08T12:03:52",
                "upload_time_iso_8601": "2020-10-08T12:03:52.611387Z",
                "url": "https://files.pythonhosted.org/packages/a9/88/76356c78aef6b527db91362ed92a0da2d18a0271921d79559fc0bc8c49c3/tokenizer-2.4.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "4b2f29340709ede23c74511bd0c8f9b3",
                    "sha256": "b189f12b565766213d2350c91cb0702be82753fe452c82c9178489712740e36f"
                },
                "downloads": -1,
                "filename": "tokenizer-2.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "4b2f29340709ede23c74511bd0c8f9b3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 129678,
                "upload_time": "2020-10-08T12:03:54",
                "upload_time_iso_8601": "2020-10-08T12:03:54.886120Z",
                "url": "https://files.pythonhosted.org/packages/9c/a3/657a7e92c7e45c9e0123cb801817910a36538a581873e2239bfe04ee43cb/tokenizer-2.4.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "2.5.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f77b1d958bfd1fd9ab290c35e7e45d47",
                    "sha256": "56c3b90a5633242e7b7bddb251c7f3493cb3dc374e699d95e799b5667f9694c5"
                },
                "downloads": -1,
                "filename": "tokenizer-2.5.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "f77b1d958bfd1fd9ab290c35e7e45d47",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 107983,
                "upload_time": "2021-03-08T11:49:44",
                "upload_time_iso_8601": "2021-03-08T11:49:44.382622Z",
                "url": "https://files.pythonhosted.org/packages/fc/e9/3c617c16df4a60a52e53e77e66c58819a5023de29387a8f8f018d2c105c7/tokenizer-2.5.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e131ed908dd518986f9a9a8b745b6250",
                    "sha256": "eaf27ae80c81b4c202ba7c4deda1c49f19acd5d42d0ffaa7d33049f39dc86a0e"
                },
                "downloads": -1,
                "filename": "tokenizer-2.5.0.tar.gz",
                "has_sig": false,
                "md5_digest": "e131ed908dd518986f9a9a8b745b6250",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 132425,
                "upload_time": "2021-03-08T11:49:46",
                "upload_time_iso_8601": "2021-03-08T11:49:46.576706Z",
                "url": "https://files.pythonhosted.org/packages/c8/e1/d1c8299704a89cd002d28de60a01d60c1dd17f2080d6605e3b7e9c2488de/tokenizer-2.5.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "3.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "321c836e9ee0707fa15d64a313688856",
                    "sha256": "338c10710151e4f4ed2d0d801af8bf3e4e618c25b977134b2bee709a3f7d5aa7"
                },
                "downloads": -1,
                "filename": "tokenizer-3.0.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "321c836e9ee0707fa15d64a313688856",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 76571,
                "upload_time": "2021-04-09T16:08:52",
                "upload_time_iso_8601": "2021-04-09T16:08:52.615472Z",
                "url": "https://files.pythonhosted.org/packages/7b/d7/b7367d1cef4d6e10fc17e0b89dcfbf3ee10c1c6f159bf202d3395429d139/tokenizer-3.0.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "11f80d21fc86b0b4419253b8e000dc43",
                    "sha256": "e3073cd1ae6582f8ea193570e217213288e6d04a51871a5d0a7a8b58f9321161"
                },
                "downloads": -1,
                "filename": "tokenizer-3.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "11f80d21fc86b0b4419253b8e000dc43",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 112568,
                "upload_time": "2021-04-09T16:08:54",
                "upload_time_iso_8601": "2021-04-09T16:08:54.595450Z",
                "url": "https://files.pythonhosted.org/packages/a0/06/2633c2d2f9d2fe76a1254488a5ab3b0dcb90fa7fb393f6cd00a31cf4243c/tokenizer-3.0.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "3.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c37177028dac39f5a5eac3207f5d2d12",
                    "sha256": "b3eb2af5626e82e1b18f1042f2a94fa1ebf9c9f6f38ea2265a436528c0647080"
                },
                "downloads": -1,
                "filename": "tokenizer-3.1.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "c37177028dac39f5a5eac3207f5d2d12",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 77180,
                "upload_time": "2021-04-14T15:20:01",
                "upload_time_iso_8601": "2021-04-14T15:20:01.682471Z",
                "url": "https://files.pythonhosted.org/packages/42/69/547bc7518c90259f7f4127eee15a18c16c11ccc4fb05ce8b3ca2e47b24d0/tokenizer-3.1.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "db8481309b4934fd9fd330fec7f3e29e",
                    "sha256": "3833c4681d6715fbb909c5c2777fd84e3c71e799260477c6f8946f9093a6b72e"
                },
                "downloads": -1,
                "filename": "tokenizer-3.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "db8481309b4934fd9fd330fec7f3e29e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 113420,
                "upload_time": "2021-04-14T15:20:03",
                "upload_time_iso_8601": "2021-04-14T15:20:03.818791Z",
                "url": "https://files.pythonhosted.org/packages/29/a2/ebe95c5f67c4dabd74d16b3fbea625d32819147b7307e91c31c0054bace0/tokenizer-3.1.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "3.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ccf93bcb8a6a969db480b2b7179592c2",
                    "sha256": "aa8e560037f4da1f343ef2c28ed95b64d7f2422becf303ad3ed5c9ab89bdcd03"
                },
                "downloads": -1,
                "filename": "tokenizer-3.1.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ccf93bcb8a6a969db480b2b7179592c2",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 77342,
                "upload_time": "2021-05-10T14:35:10",
                "upload_time_iso_8601": "2021-05-10T14:35:10.924850Z",
                "url": "https://files.pythonhosted.org/packages/f8/da/2434ed47ebf679f2c8fd796e94b4cb6f3824fc86a0b4516b744d252f3198/tokenizer-3.1.1-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "947cb24ef93c316fb2097d78c7ab1263",
                    "sha256": "5fecc53c76cd9fce1210844a0e36b841c23fec79653a8d0f9f7e4b7c4f1b94e7"
                },
                "downloads": -1,
                "filename": "tokenizer-3.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "947cb24ef93c316fb2097d78c7ab1263",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 113608,
                "upload_time": "2021-05-10T14:35:13",
                "upload_time_iso_8601": "2021-05-10T14:35:13.094784Z",
                "url": "https://files.pythonhosted.org/packages/f1/63/e9d349cee089828afc44e4c8ee634b8e19fdf4f1fb96c1715c42d6cae1fe/tokenizer-3.1.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "3.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e7193ae8b1e6eccb84fb3d208a691155",
                    "sha256": "d9cab93353a6606e191fd3a418d1b51e7484b6451a86988fdc8115786118af17"
                },
                "downloads": -1,
                "filename": "tokenizer-3.1.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "e7193ae8b1e6eccb84fb3d208a691155",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 77386,
                "upload_time": "2021-06-02T17:02:01",
                "upload_time_iso_8601": "2021-06-02T17:02:01.989653Z",
                "url": "https://files.pythonhosted.org/packages/eb/55/077d1d428bd6fd6e3d306ff375c06320ccd20d36b29542cff8ab700d45d0/tokenizer-3.1.2-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "b0fc3318036ba419bef8d518c88dca1e",
                    "sha256": "be5f59d22c1ed88d6e777a876155f73a1657ab3ccfeafba2b8099cdb0cb7b81b"
                },
                "downloads": -1,
                "filename": "tokenizer-3.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "b0fc3318036ba419bef8d518c88dca1e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 113722,
                "upload_time": "2021-06-02T17:02:04",
                "upload_time_iso_8601": "2021-06-02T17:02:04.116298Z",
                "url": "https://files.pythonhosted.org/packages/70/b0/e6b428751ac5f2ded9956fb38c907a48bda9ea520882220c57d13803866c/tokenizer-3.1.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "3.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "43b41e00941da83f3de37e19952f3a94",
                    "sha256": "99134ed31b62043bf66b6ad493b67f586a84bfe49abcbf0bb1d48c6f34a45866"
                },
                "downloads": -1,
                "filename": "tokenizer-3.2.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "43b41e00941da83f3de37e19952f3a94",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 77354,
                "upload_time": "2021-08-16T16:40:59",
                "upload_time_iso_8601": "2021-08-16T16:40:59.382879Z",
                "url": "https://files.pythonhosted.org/packages/6e/f3/04bb14cdd528be563975bea75bb8b04707273ba5a975de57dc8058d612a4/tokenizer-3.2.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "72894059b8ebb19f71abb0d001fb8832",
                    "sha256": "69798dc44dac021784be99af139acab2bac6c71f9503bedb7091a742dae7f24c"
                },
                "downloads": -1,
                "filename": "tokenizer-3.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "72894059b8ebb19f71abb0d001fb8832",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 115949,
                "upload_time": "2021-08-16T16:41:01",
                "upload_time_iso_8601": "2021-08-16T16:41:01.291228Z",
                "url": "https://files.pythonhosted.org/packages/62/e2/403632f5acd87f51d2d53c60676524a03866d8c063c1169ca49bf87116a3/tokenizer-3.2.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "3.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ba5bbbec4567f7fa37092a301bb6c31e",
                    "sha256": "38ca62d612bc9c06f57bef779127bed9e4334a02ecf629c5395402037c2d32ad"
                },
                "downloads": -1,
                "filename": "tokenizer-3.3.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ba5bbbec4567f7fa37092a301bb6c31e",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 77410,
                "upload_time": "2021-09-08T16:02:22",
                "upload_time_iso_8601": "2021-09-08T16:02:22.771367Z",
                "url": "https://files.pythonhosted.org/packages/05/d7/af8d4b3bd9173045effdb392dd95de02b8c68717210ee39076c439f6f618/tokenizer-3.3.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "2bbee5d7f53c2d5bd52595bea083b7c6",
                    "sha256": "975778f62085bd30b999021af64a3e6abd24f4ebc1270abc12c485808bbabc46"
                },
                "downloads": -1,
                "filename": "tokenizer-3.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "2bbee5d7f53c2d5bd52595bea083b7c6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 116170,
                "upload_time": "2021-09-08T16:02:24",
                "upload_time_iso_8601": "2021-09-08T16:02:24.599223Z",
                "url": "https://files.pythonhosted.org/packages/77/cd/501b8eb468b717e5855bd199f0961ab01fd5b455cd4d3c3863ba861a3f3a/tokenizer-3.3.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "3.3.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7e48fd5a3d868b31ec8771e14f14e0bd",
                    "sha256": "cf71f7c742aaed56e81a013ba31f1215e701e3edf67e844099847c7bc5fd948a"
                },
                "downloads": -1,
                "filename": "tokenizer-3.3.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "7e48fd5a3d868b31ec8771e14f14e0bd",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 77489,
                "upload_time": "2021-09-08T21:17:30",
                "upload_time_iso_8601": "2021-09-08T21:17:30.708405Z",
                "url": "https://files.pythonhosted.org/packages/fb/89/dd0af0f8c759606f69934d0694a78a52bf185d8ea9463a923611863ead54/tokenizer-3.3.1-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "08cf7c03dd26544d7078dddad188477d",
                    "sha256": "50235de86f0f10e4d16ee8c6193ba307ec68c24ceecd38e2ddb9817752c18563"
                },
                "downloads": -1,
                "filename": "tokenizer-3.3.1.tar.gz",
                "has_sig": false,
                "md5_digest": "08cf7c03dd26544d7078dddad188477d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 116161,
                "upload_time": "2021-09-08T21:17:32",
                "upload_time_iso_8601": "2021-09-08T21:17:32.915480Z",
                "url": "https://files.pythonhosted.org/packages/66/60/e70ef3ffe36db1af0f5779f5dd86cfc3a1c863e7d7d0950fc01733754b16/tokenizer-3.3.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "3.3.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5e8c41b2ba6574418cdf4baa063cf61a",
                    "sha256": "c18082e836b96f143ac331b83994ecb8ec60c30c75c685dd12361d02838136b7"
                },
                "downloads": -1,
                "filename": "tokenizer-3.3.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "5e8c41b2ba6574418cdf4baa063cf61a",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 77638,
                "upload_time": "2021-09-27T15:00:03",
                "upload_time_iso_8601": "2021-09-27T15:00:03.897769Z",
                "url": "https://files.pythonhosted.org/packages/52/9f/91df1023113d5b9f8f3db2dc3f6059f4998b68cd6b0465b6f6a298c2385d/tokenizer-3.3.2-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "0e90567fd3b348cf74872eb5bceb44ed",
                    "sha256": "4dcaf8952d30f27a6280e9ad947e5f7625cf5e57e35d045342d45416805f097a"
                },
                "downloads": -1,
                "filename": "tokenizer-3.3.2.tar.gz",
                "has_sig": false,
                "md5_digest": "0e90567fd3b348cf74872eb5bceb44ed",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 117051,
                "upload_time": "2021-09-27T15:00:06",
                "upload_time_iso_8601": "2021-09-27T15:00:06.287968Z",
                "url": "https://files.pythonhosted.org/packages/8d/3d/09ada0b9d1b5e131675db8b6f8b0f932a62d4796d2ab8c481112063b107e/tokenizer-3.3.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "3.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "345a094e1c614c4114d2622174e533ee",
                    "sha256": "ec9e113e43661e4d614b74bd5572e34a3f2fa6ac83c4a074a313aca8110f7406"
                },
                "downloads": -1,
                "filename": "tokenizer-3.4.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "345a094e1c614c4114d2622174e533ee",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 77978,
                "upload_time": "2022-03-10T14:45:51",
                "upload_time_iso_8601": "2022-03-10T14:45:51.542003Z",
                "url": "https://files.pythonhosted.org/packages/59/33/2255f9fadccf19342afb8c2979af190c48f399a39b964ae70ca212602e4b/tokenizer-3.4.0-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "420217a37a95f8b98ee55a3bac6d92f2",
                    "sha256": "332552b60377baf60ab29bbe3cd3e751815fe67bd5cdb36bd8a37f53eae6bf4c"
                },
                "downloads": -1,
                "filename": "tokenizer-3.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "420217a37a95f8b98ee55a3bac6d92f2",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 118137,
                "upload_time": "2022-03-10T14:45:53",
                "upload_time_iso_8601": "2022-03-10T14:45:53.863345Z",
                "url": "https://files.pythonhosted.org/packages/6d/04/68e5f45ad6ea0a00d93d982071c6f175971c7389304fa4afd1f645bc75e8/tokenizer-3.4.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "3.4.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "582c759bc6ee12ac8a29c4301bc59b67",
                    "sha256": "69325cf36c64d058109dd1d1a6cf665ff67204f1e8d67933ceea7983735a555e"
                },
                "downloads": -1,
                "filename": "tokenizer-3.4.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "582c759bc6ee12ac8a29c4301bc59b67",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 78856,
                "upload_time": "2022-05-03T13:43:04",
                "upload_time_iso_8601": "2022-05-03T13:43:04.461548Z",
                "url": "https://files.pythonhosted.org/packages/16/0c/14728e0fcf97c8b7ee32197dc035bede5e2de414d66468f8c95b58422f62/tokenizer-3.4.1-py2.py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "86d90b624672c6729173f5c0f0279a97",
                    "sha256": "dc0343ed8b87a7d04b9eedb29f25bbbe3c41906fa2231862077b780aa8537f3e"
                },
                "downloads": -1,
                "filename": "tokenizer-3.4.1.tar.gz",
                "has_sig": false,
                "md5_digest": "86d90b624672c6729173f5c0f0279a97",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 119826,
                "upload_time": "2022-05-03T13:43:07",
                "upload_time_iso_8601": "2022-05-03T13:43:07.016213Z",
                "url": "https://files.pythonhosted.org/packages/92/e2/595f6f69256beb325c409cc01d4361f7bef4f6c62b72985ab8d8e236fed3/tokenizer-3.4.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "582c759bc6ee12ac8a29c4301bc59b67",
                "sha256": "69325cf36c64d058109dd1d1a6cf665ff67204f1e8d67933ceea7983735a555e"
            },
            "downloads": -1,
            "filename": "tokenizer-3.4.1-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "582c759bc6ee12ac8a29c4301bc59b67",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 78856,
            "upload_time": "2022-05-03T13:43:04",
            "upload_time_iso_8601": "2022-05-03T13:43:04.461548Z",
            "url": "https://files.pythonhosted.org/packages/16/0c/14728e0fcf97c8b7ee32197dc035bede5e2de414d66468f8c95b58422f62/tokenizer-3.4.1-py2.py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "86d90b624672c6729173f5c0f0279a97",
                "sha256": "dc0343ed8b87a7d04b9eedb29f25bbbe3c41906fa2231862077b780aa8537f3e"
            },
            "downloads": -1,
            "filename": "tokenizer-3.4.1.tar.gz",
            "has_sig": false,
            "md5_digest": "86d90b624672c6729173f5c0f0279a97",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 119826,
            "upload_time": "2022-05-03T13:43:07",
            "upload_time_iso_8601": "2022-05-03T13:43:07.016213Z",
            "url": "https://files.pythonhosted.org/packages/92/e2/595f6f69256beb325c409cc01d4361f7bef4f6c62b72985ab8d8e236fed3/tokenizer-3.4.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "vulnerabilities": []
}