{
    "info": {
        "author": "Konstantin Lopukhin",
        "author_email": "kostia.lopuhin@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Natural Language :: English",
            "Programming Language :: Python :: 2",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7"
        ],
        "description": "============\nHTML to Text\n============\n\n\n.. image:: https://img.shields.io/pypi/v/html-text.svg\n   :target: https://pypi.python.org/pypi/html-text\n   :alt: PyPI Version\n\n.. image:: https://img.shields.io/travis/TeamHG-Memex/html-text.svg\n   :target: https://travis-ci.org/TeamHG-Memex/html-text\n   :alt: Build Status\n\n.. image:: http://codecov.io/github/TeamHG-Memex/soft404/coverage.svg?branch=master\n   :target: http://codecov.io/github/TeamHG-Memex/html-text?branch=master\n   :alt: Code Coverage\n\nExtract text from HTML\n\n* Free software: MIT license\n\nHow is html_text different from ``.xpath('//text()')`` from LXML\nor ``.get_text()`` from Beautiful Soup?\n\n* Text extracted with ``html_text`` does not contain inline styles,\n  javascript, comments and other text that is not normally visible to users;\n* ``html_text`` normalizes whitespace, but in a way smarter than\n  ``.xpath('normalize-space())``, adding spaces around inline elements\n  (which are often used as block elements in html markup), and trying to\n  avoid adding extra spaces for punctuation;\n* ``html-text`` can add newlines (e.g. after headers or paragraphs), so\n  that the output text looks more like how it is rendered in browsers.\n\nInstall\n-------\n\nInstall with pip::\n\n    pip install html-text\n\nThe package depends on lxml, so you might need to install additional\npackages: http://lxml.de/installation.html\n\n\nUsage\n-----\n\nExtract text from HTML::\n\n    >>> import html_text\n    >>> html_text.extract_text('<h1>Hello</h1> world!')\n    'Hello\\n\\nworld!'\n\n    >>> html_text.extract_text('<h1>Hello</h1> world!', guess_layout=False)\n    'Hello world!'\n\nPassed html is first cleaned from invisible non-text content such\nas styles, and then text is extracted.\n\nYou can also pass an already parsed ``lxml.html.HtmlElement``:\n\n    >>> import html_text\n    >>> tree = html_text.parse_html('<h1>Hello</h1> world!')\n    >>> html_text.extract_text(tree)\n    'Hello\\n\\nworld!'\n\nIf you want, you can handle cleaning manually; use lower-level\n``html_text.etree_to_text`` in this case:\n\n    >>> import html_text\n    >>> tree = html_text.parse_html('<h1>Hello<style>.foo{}</style>!</h1>')\n    >>> cleaned_tree = html_text.cleaner.clean_html(tree)\n    >>> html_text.etree_to_text(cleaned_tree)\n    'Hello!'\n\nparsel.Selector objects are also supported; you can define\na parsel.Selector to extract text only from specific elements:\n\n    >>> import html_text\n    >>> sel = html_text.cleaned_selector('<h1>Hello</h1> world!')\n    >>> subsel = sel.xpath('//h1')\n    >>> html_text.selector_to_text(subsel)\n    'Hello'\n\nNB parsel.Selector objects are not cleaned automatically, you need to call\n``html_text.cleaned_selector`` first.\n\nMain functions and objects:\n\n* ``html_text.extract_text`` accepts html and returns extracted text.\n* ``html_text.etree_to_text`` accepts parsed lxml Element and returns\n  extracted text; it is a lower-level function, cleaning is not handled\n  here.\n* ``html_text.cleaner`` is an ``lxml.html.clean.Cleaner`` instance which\n  can be used with ``html_text.etree_to_text``; its options are tuned for\n  speed and text extraction quality.\n* ``html_text.cleaned_selector`` accepts html as text or as\n  ``lxml.html.HtmlElement``, and returns cleaned ``parsel.Selector``.\n* ``html_text.selector_to_text`` accepts ``parsel.Selector`` and returns\n  extracted text.\n\nIf ``guess_layout`` is True (default), a newline is added before and after\n``newline_tags``, and two newlines are added before and after\n``double_newline_tags``. This heuristic makes the extracted text\nmore similar to how it is rendered in the browser. Default newline and double\nnewline tags can be found in `html_text.NEWLINE_TAGS`\nand `html_text.DOUBLE_NEWLINE_TAGS`.\n\nIt is possible to customize how newlines are added, using ``newline_tags`` and\n``double_newline_tags`` arguments (which are `html_text.NEWLINE_TAGS` and\n`html_text.DOUBLE_NEWLINE_TAGS` by default). For example, don't add a newline\nafter ``<div>`` tags:\n\n    >>> newline_tags = html_text.NEWLINE_TAGS - {'div'}\n    >>> html_text.extract_text('<div>Hello</div> world!',\n    ...                        newline_tags=newline_tags)\n    'Hello world!'\n\nApart from just getting text from the page (e.g. for display or search),\none intended usage of this library is for machine learning (feature extraction).\nIf you want to use the text of the html page as a feature (e.g. for classification),\nthis library gives you plain text that you can later feed into a standard text\nclassification pipeline.\nIf you feel that you need html structure as well, check out\n`webstruct <http://webstruct.readthedocs.io/en/latest/>`_ library.\n\n----\n\n.. image:: https://hyperiongray.s3.amazonaws.com/define-hg.svg\n\t:target: https://www.hyperiongray.com/?pk_campaign=github&pk_kwd=html-text\n\t:alt: define hyperiongray\n\n\n=======\nHistory\n=======\n\n0.5.1 (2019-05-27)\n------------------\n\nFixed whitespace handling when ``guess_punct_space`` is False: html-text was\nproducing unnecessary spaces after newlines.\n\n0.5.0 (2018-11-19)\n------------------\n\nParsel dependency is removed in this release,\nthough parsel is still supported.\n\n* ``parsel`` package is no longer required to install and use html-text;\n* ``html_text.etree_to_text`` function allows to extract text from\n  lxml Elements;\n* ``html_text.cleaner`` is an ``lxml.html.clean.Cleaner`` instance with\n  options tuned for text extraction speed and quality;\n* test and documentation improvements;\n* Python 3.7 support.\n\n0.4.1 (2018-09-25)\n------------------\n\nFixed a regression in 0.4.0 release: text was empty when\n``html_text.extract_text`` is called with a node with text, but\nwithout children.\n\n0.4.0 (2018-09-25)\n------------------\n\nThis is a backwards-incompatible release: by default html_text functions\nnow add newlines after elements, if appropriate, to make the extracted text\nto look more like how it is rendered in a browser.\n\nTo turn it off, pass ``guess_layout=False`` option to html_text functions.\n\n* ``guess_layout`` option to to make extracted text look more like how\n  it is rendered in browser.\n* Add tests of layout extraction for real webpages.\n\n\n0.3.0 (2017-10-12)\n------------------\n\n* Expose functions that operate on selectors,\n  use ``.//text()`` to extract text from selector.\n\n\n0.2.1 (2017-05-29)\n------------------\n\n* Packaging fix (include CHANGES.rst)\n\n\n0.2.0 (2017-05-29)\n------------------\n\n* Fix unwanted joins of words with inline tags: spaces are added for inline\n  tags too, but a heuristic is used to preserve punctuation without extra spaces.\n* Accept parsed html trees.\n\n\n0.1.1 (2017-01-16)\n------------------\n\n* Travis-CI and codecov.io integrations added\n\n\n0.1.0 (2016-09-27)\n------------------\n\n* First release on PyPI.",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/TeamHG-Memex/html-text",
        "keywords": "",
        "license": "MIT license",
        "maintainer": "",
        "maintainer_email": "",
        "name": "html-text",
        "package_url": "https://pypi.org/project/html-text/",
        "platform": "",
        "project_url": "https://pypi.org/project/html-text/",
        "project_urls": {
            "Homepage": "https://github.com/TeamHG-Memex/html-text"
        },
        "release_url": "https://pypi.org/project/html-text/0.5.1/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Extract text from HTML",
        "version": "0.5.1"
    },
    "last_serial": 5328374,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a65d4c965a41a7dfffdb33bc52985ea4",
                    "sha256": "a7c11d1f1ab1aa2f69c9b0a76b1bdd1ae988c39838b1b6d9f7164a2277b58f8d"
                },
                "downloads": -1,
                "filename": "html_text-0.1.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a65d4c965a41a7dfffdb33bc52985ea4",
                "packagetype": "bdist_wheel",
                "python_version": "3.5",
                "requires_python": null,
                "size": 3846,
                "upload_time": "2016-09-27T12:22:41",
                "url": "https://files.pythonhosted.org/packages/bc/e6/edc21667da6c0dc408c6b25f05e8170a8f42d26aeda8f4f1842ba01e1766/html_text-0.1.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "69eccc6f87ba90c7b1b40100883dd7b5",
                    "sha256": "e4d36d6b0359af0241740972632388c5e3aa00f3507021100d25f3dd9cd2e986"
                },
                "downloads": -1,
                "filename": "html_text-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "69eccc6f87ba90c7b1b40100883dd7b5",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10230,
                "upload_time": "2016-09-27T12:22:38",
                "url": "https://files.pythonhosted.org/packages/8c/f5/5a579ed618d5d8a3267ae54c1238b49770cff326b104a7faff5007c4d990/html_text-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "1e60cd771c4ce1b71985239f83b7e82a",
                    "sha256": "c2ab75db89a3198fa54f4d833d9fbe24c8c5d92324656ca10d5a0d76a5136724"
                },
                "downloads": -1,
                "filename": "html_text-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "1e60cd771c4ce1b71985239f83b7e82a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10384,
                "upload_time": "2017-01-16T15:56:18",
                "url": "https://files.pythonhosted.org/packages/dd/9c/2c189c3f54ca8bf83dfb3e1c0f1c44799784ba817eb2cdbd590859b8e2a8/html_text-0.1.1.tar.gz"
            }
        ],
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "76ff6d933adedff054fbc5813dbe6501",
                    "sha256": "3d1cc8a58055750365e9b7699709a680674eda71e0f4f7535b5d64cfc5fe9d4b"
                },
                "downloads": -1,
                "filename": "html_text-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "76ff6d933adedff054fbc5813dbe6501",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5230,
                "upload_time": "2017-05-29T12:42:02",
                "url": "https://files.pythonhosted.org/packages/fa/f7/ac66d7cff63f82c0950ed16b67954f0bf54720239ee647e331b89ffd5027/html_text-0.2.0.tar.gz"
            }
        ],
        "0.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d4418b823d016bb6a95bb5f2ff1f602e",
                    "sha256": "5bc38ea393cccf5dcbcb38441021c3c7f957e94f7a736ddcf20af1dec25c102b"
                },
                "downloads": -1,
                "filename": "html_text-0.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "d4418b823d016bb6a95bb5f2ff1f602e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5341,
                "upload_time": "2017-05-29T13:05:55",
                "url": "https://files.pythonhosted.org/packages/64/5a/26172b094a0292b7e7886ebff5b367516bb74283d08452b945256fdfee98/html_text-0.2.1.tar.gz"
            }
        ],
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "31e91ee64c5a948e76942d32d3419594",
                    "sha256": "039a305fae8c0367738861e7ce7d5775be9039b163a74d01fc9387c8f92155e4"
                },
                "downloads": -1,
                "filename": "html_text-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "31e91ee64c5a948e76942d32d3419594",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5679,
                "upload_time": "2017-10-12T08:09:06",
                "url": "https://files.pythonhosted.org/packages/c5/83/ec044994d674b922746964fa7ce20690f505fc72af84257f0d7ee27413fc/html_text-0.3.0.tar.gz"
            }
        ],
        "0.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3c4911bbdaf039ca920cac933df40d95",
                    "sha256": "fe17427767d9c1c7eb2df9de77b9833debbf00737624291e471d9955eb86c8e7"
                },
                "downloads": -1,
                "filename": "html_text-0.4.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "3c4911bbdaf039ca920cac933df40d95",
                "packagetype": "bdist_wheel",
                "python_version": "3.6",
                "requires_python": null,
                "size": 8860,
                "upload_time": "2018-09-25T15:33:28",
                "url": "https://files.pythonhosted.org/packages/01/cd/56bbe57b945b4fc6e77de3cf59de1e5d3612dfe0b1f4a14b05ce2640c866/html_text-0.4.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "327ef92f2b71a780d8dc9d0d4f48be32",
                    "sha256": "93e329054bcbbe390601ddb77444b5b00a9f7121d12eef4864e8a6e2e24bb7d7"
                },
                "downloads": -1,
                "filename": "html_text-0.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "327ef92f2b71a780d8dc9d0d4f48be32",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 55701,
                "upload_time": "2018-09-25T15:33:18",
                "url": "https://files.pythonhosted.org/packages/d8/c0/2fe04e706e3205734aa5b76a3693104ab5c930ffcb776a07a9676d03a56a/html_text-0.4.0.tar.gz"
            }
        ],
        "0.4.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d11f6285407d16405bccc53c2a612adc",
                    "sha256": "4549c68e15396d17a3eb68fdbdebcebb6101bca9fb748bdf4c7036436f052b63"
                },
                "downloads": -1,
                "filename": "html_text-0.4.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d11f6285407d16405bccc53c2a612adc",
                "packagetype": "bdist_wheel",
                "python_version": "3.6",
                "requires_python": null,
                "size": 8962,
                "upload_time": "2018-09-25T18:49:57",
                "url": "https://files.pythonhosted.org/packages/b8/82/55b11f734d5e7982fe4f84e27499c7860eb3acb6c3ed284605de4677c691/html_text-0.4.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "fb9a8f8e16a34f3bdfe7cfea24aeeab0",
                    "sha256": "0876a2ca198ddfe3cc13e5002f4de7462c3669a55707ae3af14c8e7bfd9dfc89"
                },
                "downloads": -1,
                "filename": "html_text-0.4.1.tar.gz",
                "has_sig": false,
                "md5_digest": "fb9a8f8e16a34f3bdfe7cfea24aeeab0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 55877,
                "upload_time": "2018-09-25T18:49:49",
                "url": "https://files.pythonhosted.org/packages/76/45/30cea0ef7060f2e7e6286c325673ffc34cd9458d46e3c76376a8374f9f44/html_text-0.4.1.tar.gz"
            }
        ],
        "0.5.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8f1548bc06a57e8188bf6ee6af044078",
                    "sha256": "d18ae371f721651b445efc0cfe67396421e110158f866c11524f41e78fb7c0fe"
                },
                "downloads": -1,
                "filename": "html_text-0.5.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "8f1548bc06a57e8188bf6ee6af044078",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 9849,
                "upload_time": "2018-11-19T18:03:43",
                "url": "https://files.pythonhosted.org/packages/38/6f/9826e54bf0c1d3ff398ecfc6150a9dc43dda9ac09c3041dd53b6c5ab02ef/html_text-0.5.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "f39b515cff90dff996e802c551615ee3",
                    "sha256": "104660335e5a992e5d412f5840d310a74edc19d8d8a23e2d4c0c6d3675f54b36"
                },
                "downloads": -1,
                "filename": "html_text-0.5.0.tar.gz",
                "has_sig": false,
                "md5_digest": "f39b515cff90dff996e802c551615ee3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 57061,
                "upload_time": "2018-11-19T18:03:44",
                "url": "https://files.pythonhosted.org/packages/9b/7f/160b96e79896aa72da1e0c8879294c900644990affb8877a0bcaec50a2ab/html_text-0.5.0.tar.gz"
            }
        ],
        "0.5.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "82642d05ff12b2d29ec9c13bf0cff44c",
                    "sha256": "7cef12a85f6ea63b3bf5fd4e0be1c1e061dda7aa48e5c343e0a4b3670ea34d86"
                },
                "downloads": -1,
                "filename": "html_text-0.5.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "82642d05ff12b2d29ec9c13bf0cff44c",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 9978,
                "upload_time": "2019-05-28T18:48:19",
                "url": "https://files.pythonhosted.org/packages/e1/8d/e43d6dcec7432732ac1955b73dcccb536d09211af343de0f1c8eb892df45/html_text-0.5.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e4149571cfe7cc464886438b676509ae",
                    "sha256": "58ab7cfd99d50f3220ae3bb8b6e507f5d5d0dfaf2adfd9848689a3cc5c01ffeb"
                },
                "downloads": -1,
                "filename": "html_text-0.5.1.tar.gz",
                "has_sig": false,
                "md5_digest": "e4149571cfe7cc464886438b676509ae",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 57230,
                "upload_time": "2019-05-28T18:47:59",
                "url": "https://files.pythonhosted.org/packages/4f/e6/f056f7269d36088a5e9674c50b6db3c7dccef5b9c7481eec6f40f06420ce/html_text-0.5.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "82642d05ff12b2d29ec9c13bf0cff44c",
                "sha256": "7cef12a85f6ea63b3bf5fd4e0be1c1e061dda7aa48e5c343e0a4b3670ea34d86"
            },
            "downloads": -1,
            "filename": "html_text-0.5.1-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "82642d05ff12b2d29ec9c13bf0cff44c",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 9978,
            "upload_time": "2019-05-28T18:48:19",
            "url": "https://files.pythonhosted.org/packages/e1/8d/e43d6dcec7432732ac1955b73dcccb536d09211af343de0f1c8eb892df45/html_text-0.5.1-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "e4149571cfe7cc464886438b676509ae",
                "sha256": "58ab7cfd99d50f3220ae3bb8b6e507f5d5d0dfaf2adfd9848689a3cc5c01ffeb"
            },
            "downloads": -1,
            "filename": "html_text-0.5.1.tar.gz",
            "has_sig": false,
            "md5_digest": "e4149571cfe7cc464886438b676509ae",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 57230,
            "upload_time": "2019-05-28T18:47:59",
            "url": "https://files.pythonhosted.org/packages/4f/e6/f056f7269d36088a5e9674c50b6db3c7dccef5b9c7481eec6f40f06420ce/html_text-0.5.1.tar.gz"
        }
    ]
}