{
    "info": {
        "author": "Dylan Jay",
        "author_email": "software@pretaweb.com",
        "bugtrack_url": null,
        "classifiers": [
            "Programming Language :: Python",
            "Topic :: Software Development :: Libraries :: Python Modules"
        ],
        "description": "Introduction\n============\n\nHelpful transmogrifier blueprints to extract text or html out of html content.\n\n\ntransmogrify.htmlcontentextractor.auto\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis blueprint has a clustering algorithm that tries to automatically extract the content from the HTML template.\nThis is slow and not always effective. Often you will need to input your own template extraction rules.\nIn addition to extracting Title, Description and Text of items the blueprint will output\nthe rules it generates to a logger with the same name as the blueprint.\n\nSetting debug mode on templateauto will give you details about the rules it uses. ::\n\n  ...\n  DEBUG:templateauto:'icft.html' discovered rules by clustering on 'http://...'\n  Rules:\n\ttext= html //div[@id = \"dal_content\"]//div[@class = \"content\"]//p\n\ttitle= text //div[@id = \"dal_content\"]//div[@class = \"content\"]//h3\n  Text:\n\tTITLE: ...\n\tMAIN-10: ...\n\tMAIN-10: ...\n\tMAIN-10: ...\n\nOptions\n-------\n\ncondition\n  TAL Expression to control use of this blueprint\n\ndebug\n  default is ''\n\ntransmogrify.htmlcontentextractor\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis blueprint extracts out title, description and body from html\neither via xpath, TAL or by automatic cluster analysis\n\n\nRules are in the form of ::\n\n  (title|description|text|anything) = (text|html|optional|tal) Expression\n\nWhere expression is either TAL or XPath\n\nFor example ::\n\n  [template1]\n  blueprint = transmogrify.htmlcontentextractor\n  title       = text //div[@class='body']//h1[1]\n  _delete1    = optional //div[@class='body']//a[@class='headerlink']\n  _delete2    = optional //div[contains(@class,'admonition-description')]\n  description = text //div[contains(@class,'admonition-description')]//p[@class='last']\n  text        = html //div[@class='body']\n\nNote that for a single template e.g. template1, ALL of the XPaths need to match otherwise\nthat template will be skipped and the next template tried. If you'd like to make it\nso that a single XPath isn't nessary for the template to match then use the keyword `optional` or `optionaltext`\ninstead of `text` or `html` before the XPath.\n\n\nWhen an XPath is applied within a single template, the HTML it matches will be removed from the page.\nAnother rule in that same template can't match the same HTML fragment.\n\nIf a content part is not useful (e.g. redundant text, title or description) it is a way to effectively remove that HTML\nfrom the content.\n\nTo help debug your template rules you can set debug mode.\n\n\nFor more information about XPath see\n\n- http://www.w3schools.com/xpath/default.asp\n- http://blog.browsermob.com/2009/04/test-your-selenium-xpath-easily-with-firebug/\n\nHTMLContentExtractor\n====================\n\nThis blueprint extracts out fields from html either via xpath rules or by automatic cluster\nanalysis\n\ntransmogrify.htmlcontentextractor\n---------------------------------\n\nYou can define a series of rules which will get applied to the to the '_text'\nof the input item. The rules use a XPATH expression or a TAL expression to\nextract html or text out of the html and adds it as key to the outputted item.\n\nEach option of the blueprint is a rule of the following form ::\n\n    (N-)field = (optional)(text|html|delete|optional) xpath\n\n    OR\n\n    (N-)field = (optional)tal tal-expression\n\n\n\"field\" is the attribute that will be set with the results of the xpath\n\n\"format\" is what to do with the results of the xpath. \"optional\" means the same\nas \"delete\" but won't cause the group to not match. if the format is delete or optional\nthen the field name doesn't matter but will still need to be unique\n\n\"xpath' is an xpath expression\n\nIf the format is 'tal' then instead of an XPath use can use a TAL expression. TAL expression\nis evaluated on the item object AFTER the XPath expressions have been applied.\n\nFor example ::\n\n    [template]\n    blueprint = transmogrify.htmlcontentextractor\n    title = text //div[@class='body']//h1[1]\n    _permalink = text //div[@class='body']//a[@class='headerlink']\n    _text = html //div[@class='body']\n    _label = optional //p[contains(@class,'admonition-title')]\n    description = optional //div[contains(@class,'admonition-description')]/p[@class='last']/text()\n    _remove_useless_links = optional //div[@id = 'indices-and-tables']\n    mimetype = tal string:text/html\n    text = tal python:item['_text'].replace('id=\"blah\"','')\n\nYou can delete a number of parts of the html by extracting content to fields such as _permalink and _label.\nThese items won't get used be set used to set any properties on the final content so are effective as a means\nof deleting parts of the html.\nTAL expressions are evaluated after XPath expressions so we can post process the _text XPath to produce a text\nstripped of a certain id.\n\n\nN is the group number. Groups are run in order of group number. If\nany rule doesn't match (unless its marked optional) then the next group\nwill be tried instead. Group numbers are optional.\n\nInstead of groups you can also chain several blueprints togeather. The blueprint\nwill set '_template' on the item. If another blueprint finds the '_template' key in an item\nit will ignore that item.\n\nThe '_template' field is the remainder of the html once all the content selected by the\nXPATH expressions have been applied.\n\n\n\ntransmogrify.htmlcontentextractor.auto\n--------------------------------------\nThis blueprint will analyse the html and attempt to discover the rules to extract out the\ntitle, description and body of the html.\n\nIf the logger output is in DEBUG mode then the XPaths used by the auto extrator will be output\nto the logger.\n\n\nChangelog\n=========\n\n1.0 (2012-04-18)\n----------------\n\n- include datetime in tal expression. [djay]\n- fix bug in drop_tree when removing html [djay]\n- better logging [djay]\n- better handling of text nodes [djay]\n- added iterating accention of a text node in order uniquifiy [aterry]\n\n\n1.0b5 (2011-06-29)\n------------------\n\n- include docs\n- now can use TAL expressions\n\n1.0b4 (2011-02-06)\n------------------\n\n- handle '/text()' in  xpaths\n- new 'optionaltext' rule format \n\n1.0b3 (2010-12-13)\n------------------\n\n- simpler autogenerated xpath\n- better logging\n\n1.0b2 (2010-11-09)\n------------------\n\n- Put condition on autofinder so can be turned off\n\n1.0b1 (2010-11-03)\n------------------\n\n- ignore already found items. better debug\n  [\"Dylan Jay\"]\n\n- skip templates if item already parsed\n  [\"Dylan Jay\"]\n  \n- print automaticly found XPaths\n  [\"Dylan Jay\"]\n  \n- make text fields strip tail text\n  [\"Vitaliy Podoba\"]\n  \n1.0dev (2010-03-22)\n-------------------\n\n- split the auto templatefinder out to it's own blueprint\n  [\"Dylan Jay\"]",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://github.com/djay/transmogrify.htmlcontentextractor",
        "keywords": "transmogrifier blueprint funnelweb source plone import conversion microsoft office",
        "license": "GPL",
        "maintainer": null,
        "maintainer_email": null,
        "name": "transmogrify.htmlcontentextractor",
        "package_url": "https://pypi.org/project/transmogrify.htmlcontentextractor/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/transmogrify.htmlcontentextractor/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "http://github.com/djay/transmogrify.htmlcontentextractor"
        },
        "release_url": "https://pypi.org/project/transmogrify.htmlcontentextractor/1.0/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "This blueprint extracts out title, description and body from html either via xpath or by automatic cluster analysis",
        "version": "1.0"
    },
    "last_serial": 800890,
    "releases": {
        "1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "79ba21c3d976f1b022cc4c3f6b0fc4c1",
                    "sha256": "fabeb367cab348dafed25c4a200c10306bfd6bbd6e358648b9ce830ba37c8cbf"
                },
                "downloads": -1,
                "filename": "transmogrify.htmlcontentextractor-1.0.zip",
                "has_sig": false,
                "md5_digest": "79ba21c3d976f1b022cc4c3f6b0fc4c1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 388534,
                "upload_time": "2012-04-18T14:56:51",
                "url": "https://files.pythonhosted.org/packages/18/1c/83d6797f1ce58a5ee1db7fdbbbb42f630d4465105287f1e0763c48483796/transmogrify.htmlcontentextractor-1.0.zip"
            }
        ],
        "1.0b1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "74cf35ddd26825c6acc3a2c92f2da7ba",
                    "sha256": "1698e6ef619b670ba16ed72492aa05e649bbc7b78dd951b07d79f11ec161f04f"
                },
                "downloads": -1,
                "filename": "transmogrify.htmlcontentextractor-1.0b1.zip",
                "has_sig": false,
                "md5_digest": "74cf35ddd26825c6acc3a2c92f2da7ba",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 381179,
                "upload_time": "2010-11-07T16:31:10",
                "url": "https://files.pythonhosted.org/packages/be/87/2d68d34c6d889d0a230ba259b57ce4cc1898992acd021d4e5f295636be0d/transmogrify.htmlcontentextractor-1.0b1.zip"
            }
        ],
        "1.0b2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e25106a641142bd4fd819e08ee7ea063",
                    "sha256": "86d54156d3288be7f4d3ce89afc23268303c1413c839bfc10c09bc18dbe389c0"
                },
                "downloads": -1,
                "filename": "transmogrify.htmlcontentextractor-1.0b2.zip",
                "has_sig": false,
                "md5_digest": "e25106a641142bd4fd819e08ee7ea063",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 381338,
                "upload_time": "2010-11-08T15:52:33",
                "url": "https://files.pythonhosted.org/packages/0c/8f/9ee2aa426bb904ca3d41771e6be823a9d17b429c840548aba9b44994c008/transmogrify.htmlcontentextractor-1.0b2.zip"
            }
        ],
        "1.0b3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7a656cc5a6c36a6938c36089b49a1af1",
                    "sha256": "39c7a17c99481b7e376d8b63b38a9c7a18692676289519dc48b2f0506d2696a6"
                },
                "downloads": -1,
                "filename": "transmogrify.htmlcontentextractor-1.0b3.zip",
                "has_sig": false,
                "md5_digest": "7a656cc5a6c36a6938c36089b49a1af1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 381842,
                "upload_time": "2010-12-13T16:42:15",
                "url": "https://files.pythonhosted.org/packages/ae/b1/2ef7cbb6524cdc1d7e4ba97843b8c2848aa4f1d9294bf159c71d29c38105/transmogrify.htmlcontentextractor-1.0b3.zip"
            }
        ],
        "1.0b4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ebc7034460f778191f04138b475a7c33",
                    "sha256": "39c6e1d90f10d5cdeee56ca026a6eb9693f1ad612d3f649aab4a88147945a6ae"
                },
                "downloads": -1,
                "filename": "transmogrify.htmlcontentextractor-1.0b4.zip",
                "has_sig": false,
                "md5_digest": "ebc7034460f778191f04138b475a7c33",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 382198,
                "upload_time": "2011-02-06T17:19:16",
                "url": "https://files.pythonhosted.org/packages/c3/ee/d0c1e7bd62bc84b3b86d54ea2e4743094e87d40201cd1a6c9f2c12c2c1a1/transmogrify.htmlcontentextractor-1.0b4.zip"
            }
        ],
        "1.0b5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f49b4c013e76c514f21e392f90a663af",
                    "sha256": "2da78e8a477820a0e2df69b4eb022fea1a5e2563bfe5fa0367f7c73fb8318520"
                },
                "downloads": -1,
                "filename": "transmogrify.htmlcontentextractor-1.0b5.zip",
                "has_sig": false,
                "md5_digest": "f49b4c013e76c514f21e392f90a663af",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 385404,
                "upload_time": "2011-06-29T16:09:38",
                "url": "https://files.pythonhosted.org/packages/c3/c2/a1ea2ca3fe052aeeb51b2f82735255b3a145045cf3b8392a28ba4ad06ad6/transmogrify.htmlcontentextractor-1.0b5.zip"
            }
        ],
        "1.0b5dev": [],
        "1.0dev": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "59129d802c7a4278670fe56b41c9c0b1",
                    "sha256": "ecbc866748a159984b6238b9d09d1b9bca937a8c050048cc0a860e6a9f20f109"
                },
                "downloads": -1,
                "filename": "transmogrify.htmlcontentextractor-1.0dev.tar.gz",
                "has_sig": false,
                "md5_digest": "59129d802c7a4278670fe56b41c9c0b1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 355094,
                "upload_time": "2010-03-22T17:07:41",
                "url": "https://files.pythonhosted.org/packages/c7/85/f3ce5979114592ad8df0a3df40e2efa4660fedbf681228e6d4424ce17ae7/transmogrify.htmlcontentextractor-1.0dev.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "79ba21c3d976f1b022cc4c3f6b0fc4c1",
                "sha256": "fabeb367cab348dafed25c4a200c10306bfd6bbd6e358648b9ce830ba37c8cbf"
            },
            "downloads": -1,
            "filename": "transmogrify.htmlcontentextractor-1.0.zip",
            "has_sig": false,
            "md5_digest": "79ba21c3d976f1b022cc4c3f6b0fc4c1",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 388534,
            "upload_time": "2012-04-18T14:56:51",
            "url": "https://files.pythonhosted.org/packages/18/1c/83d6797f1ce58a5ee1db7fdbbbb42f630d4465105287f1e0763c48483796/transmogrify.htmlcontentextractor-1.0.zip"
        }
    ]
}