{
    "info": {
        "author": "cr0hn[-at-]cr0hn.com",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [
            "Environment :: Console",
            "Intended Audience :: Other Audience",
            "Intended Audience :: System Administrators",
            "License :: OSI Approved :: BSD License",
            "Operating System :: MacOS",
            "Operating System :: Microsoft :: Windows",
            "Operating System :: POSIX",
            "Programming Language :: Python :: 3"
        ],
        "description": "Feed to Wordpress Post\n======================\n\nThis project allow to get RSS Feed and create new Wordpress post from them.\n\nPre-requisites\n--------------\n\nWordpress Mode\n++++++++++++++\n\n- You must install the Wordpress plugin: `Application Passwords <https://es.wordpress.org/plugins/application-passwords/>`_\n- You must create a new Application Password\n- **Python 3.6 or above**\n\n\nmapping.json format\n-------------------\n\nWordpress Mode\n++++++++++++++\n\nThe mapping file has this format:\n\n.. code-block:: json\n\n    {\n      \"feed\": \"http://www.xxxx.es/blogs/xxxx/feed/\",\n      \"exportMethod\": \"wordpress\",\n      \"mapping\": {\n        \"body\": \"summary\"\n      },\n      \"fixed\": {\n        \"tags\": [\"one\", \"two\", \"general\"],\n        \"categories\": [\n          {\n            \"category\": \"subcategory\",\n            \"parent\": \"top-category\"\n          }\n        ],\n        \"post_status\": \"publish\"\n      }\n    }\n\nMongo Mode\n++++++++++\n\n\n.. code-block:: json\n\n    {\n      \"feed\": \"http://www.xxxx.es/blogs/xxxx/feed/\",\n      \"exportMethod\": \"mongo\",\n      \"fixed\": {\n        \"categories\": [\n          {\n            \"category\": \"subcategory\",\n            \"parent\": \"top-category\"\n          }\n        ]\n      }\n    }\n\nMapping key\n+++++++++++\n\n``mapping`` indicates how ``f2e`` must match the input feed values to the wordpress result.\n\n- Left values of mapping will be the variables names of exported objects.\n- Right values are the key names in feed where ``f2e`` must map to the output.\n\nFor example:\n\nSuppose this RSS:\n\n.. code-block:: xml\n\n    <?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>\n    <rss version=\"2.0\">\n        <channel>\n            <title>Trabajo y empleo - BOE - Bolet\u00edn Oficial del Estado</title>\n            <link>http://www.boe.es/diario_boe/</link>\n            <description>Legislaci\u00f3n relativa a Trabajo y empleo ingresada en los \u00faltimos dos meses en la base de datos del Bolet\u00edn Oficial del Estado</description>\n            <language>es-es</language>\n            <pubDate>Mon, 06 Aug 2018 00:00:00 +0200</pubDate>\n            <lastBuildDate>Mon, 06 Aug 2018 14:55:03 +0200</lastBuildDate>\n            <webMaster>webmaster@boe.es</webMaster>\n            <item>\n              <title>Pleno. Sentencia 78/2018, de 5 de julio de 2018. Recurso de inconstitucionalidad 3720-2017. Interpuesto por el Presidente del Gobierno en relaci\u00f3n con los art\u00edculos 13 y 36 de la Ley 10/2016, de 27 de diciembre, del presupuesto de la Comunidad Aut\u00f3noma de Andaluc\u00eda para el a\u00f1o 2017. Competencias sobre ordenaci\u00f3n general de la econom\u00eda, hacienda general y funci\u00f3n p\u00fablica: nulidad parcial del precepto legal auton\u00f3mico relativo a la oferta de empleo p\u00fablico de 2017 u otro instrumento similar de gesti\u00f3n de la provisi\u00f3n de necesidades de personal (STC 142/2017). Voto particular.</title>\n              <link>http://www.boe.es/diario_boe/txt.php?id=BOE-A-2018-11276</link>\n              <description>Tribunal Constitucional - Publicado el 06/08/2018 - Referencia: BOE-A-2018-11276</description>\n              <guid isPermaLink=\"true\">http://www.boe.es/boe/dias/2018/08/06/pdfs/BOE-A-2018-11276.pdf</guid>\n              <pubDate>Mon, 06 Aug 2018 00:00:00 +0200</pubDate>\n            </item>\n        </channel>\n    </rss>\n\nWhere each item has this format:\n\n.. code-block:: xml\n\n    <item>\n      <title>Pleno. Sentencia 78/2018, de 5 de julio de 2018. Recurso de inconstitucionalidad 3720-2017. Interpuesto por el Presidente del Gobierno en relaci\u00f3n con los art\u00edculos 13 y 36 de la Ley 10/2016, de 27 de diciembre, del presupuesto de la Comunidad Aut\u00f3noma de Andaluc\u00eda para el a\u00f1o 2017. Competencias sobre ordenaci\u00f3n general de la econom\u00eda, hacienda general y funci\u00f3n p\u00fablica: nulidad parcial del precepto legal auton\u00f3mico relativo a la oferta de empleo p\u00fablico de 2017 u otro instrumento similar de gesti\u00f3n de la provisi\u00f3n de necesidades de personal (STC 142/2017). Voto particular.</title>\n      <link>http://www.boe.es/diario_boe/txt.php?id=BOE-A-2018-11276</link>\n      <description>Tribunal Constitucional - Publicado el 06/08/2018 - Referencia: BOE-A-2018-11276</description>\n      <guid isPermaLink=\"true\">http://www.boe.es/boe/dias/2018/08/06/pdfs/BOE-A-2018-11276.pdf</guid>\n      <pubDate>Mon, 06 Aug 2018 00:00:00 +0200</pubDate>\n    </item>\n\nThis implies that we'll have these keys:\n\n- title\n- link\n- description\n- pubDate\n- guid\n\nThen, if we want to recover the title, description and published date, we must write this ``mapping.json``:\n\n.. code-block:: json\n\n    {\n      \"mapping\": {\n        \"body\": \"description\"\n      }\n    }\n\nFixed keys\n++++++++++\n\nSome times it should be interesting to add some static values to the result dict object. You can use fixed keys for this purpose.\n\n**Wordpress mode**\n\nIn Wordpress there're two special keys: ``tags`` and ``categories``.\n\nYou can specify *tags* and *categories*. ``f2e`` will try to resolve the tag/category or create if it doesn't exits in the Wordpress site.\n\nFilters\n-------\n\nThere's situations where you may want to apply some advanced filters. To do that we must add some Python code.\n\nYou can use any name for the filter file, but for convention we'll use ``filters.py``. There a basic example:\n\nBasics\n++++++\n\n.. code-block:: python\n\n    from feed_to_exporter.filters import FeedInfo\n    from feed_to_exporter.exceptions import FeedToWordpressNotValidInfoFound\n\n\n\n    def link_filter(field_value: str) -> dict:\n        \"\"\"\n        this filter will download the link pointing by the field and replace\n        the content of the web page.\n\n        Also try to check if some keywords are available and generate some tags\n\n        Content filter must return a dictionary type, otherwise, engine will\n        release an exception\n        \"\"\"\n        response = request.get(field_value)\n\n        results = {\n            'body': response.content\n        }\n\n        # Try to find tags\n        if any(x in response.content for x in ('hacking', 'security',\n            'pentesting')):\n            results['tags'] = ['security']\n\n        return results\n\n    def body_filter(field_value: str) -> dict:\n        \"\"\"\n        This filter remove the words 'SEO' from the body field and return\n        the new 'body' value for the field. The engine will update that\n        with this information.\n\n        Content filter must return a dictionary type, otherwise, engine will\n        release an exception\n        \"\"\"\n        return {'body': field_value.replace('SEO', '')}\n\n\n    def global_filter(feed_info: FeedInfo) -> \\\n            dict or FeedToWordpressNotValidInfoFound:\n        \"\"\"\n        Global filter enables a validation with the context of all of fields\n        values. This filter must return a dictionary or an exception.\n\n        If one exception is returned, engine will interpret that the current\n        feed must not be processed and continue to the next feed.\n\n        Global filter will executed after the individual filters.\n        \"\"\"\n\n        if not feed_info.title or not feed_info.body:\n            raise FeedToWordpressNotValidInfoFound()\n\n        if \"security\" in feed_info.title and \"hacking\" in feed_info.body:\n            return {\"category\": [\"hard-security\"]}\n        elif \"ciso\" in feed_info.body.lower():\n            return {\"category\": [\"ciso-news\"]}\n        else:\n            return {}\n\n    #\n    # Order of filters are following the definition in the bellow dictionary\n    #\n    # The name of the variable must be the following for the individual filters\n    INDIVIDUAL_VALIDATORS = {\n        'link': link_filter,\n        'body': body_filter\n    }\n\n    # The name of the variable must be the following for global validator\n    GLOBAL_VALIDATOR = global_filter\n\nAs you can see you must define the var name ``INDIVIDUAL_VALIDATORS`` indicates the field where it will apply the filter.\n\nFilters **always** must return a dictionary and it can overwrite the original content of a field.\n\nFilters execution order are defined by the order indicated in the ``INDIVIDUAL_VALIDATORS`` var.\n\nThe parameters passed in each individual filter function is the value of the field.\n\nResult data structure\n+++++++++++++++++++++\n\n``f2e`` export collected data from feed to a dictionary. Depending of the export method you want, it need some different keys:\n\n**Wordpress**\n\nFeedInfoWordpress has these properties:\n\n- title: str\n- slug: str\n- link: str\n- feed_source: str\n- body: str -> raw information from Feed mapping\n- content: str -> content that will send to the Wordpress Post. By default is a composition of: body + html link + feed_source. You can see at internal filters (``feed_to_exporter.filters.py``)\n- raw_feed_info: dict -> raw content of feed\n- ping_status: str (default: closed)\n- feed_source: str (default: closed)\n- post_status: str (default: draft)\n- comment_status: str (default: closed)\n- date: str (default: now time, with format: %Y-%m-%dT%H:%M:%S)\n\nFor fields ``ping_status``, ``feed_source``, ``post_status`` and ``comment_status`` you can check valid values at Wordpress REST API doc: https://developer.wordpress.org/rest-api/\n\n**Mongo**\n\nMongo doesn't need any special value for the result dictionary. The whole dict will be stored into Mongo \"as is\".\n\n\nValidation rule\n+++++++++++++++\n\nSome times you could want to use a global validation rule. This validation could implies more than one field. If this is the case then you must use the a new function and map to ``GLOBAL_VALIDATOR`` variable.\n\nThis function must returns a **dict** value or a Exception.\n\nWorking modes\n-------------\n\nSimple\n++++++\n\nSimple mode is the usual mode. Explained above.\n\nDiscovery mode\n++++++++++++++\n\nDiscover mode discover recursively the directories, form a base dir given. The engine will get each directory and manage it as and independent running.\n\nFor this mode works well each crawler must in an independent directory and have only 2 files: ``filters.py`` and ``mapping.json``.\n\nTo enable this mode you must use the ``-D`` option and each m\u00a1``mapping.json`` must have an additional entry: ``feed``:\n\n.. code-block:: json\n\n    {\n      \"feed\": \"http://www.mysite.com/feed/\",\n      \"mapping\": {\n        \"body\": \"summary\"\n      },\n      \"fixed\": {\n        \"categories\": [\"myCategory\"]\n      }\n    }\n\n**Example of directory structure**\n\n.. code-block:: bash\n\n    > tree examples/\n    examples\n    \u251c\u2500\u2500 __init__.py\n    \u251c\u2500\u2500 site1.com\n    \u2502\u00a0\u00a0 \u251c\u2500\u2500 __ini__.py\n    \u2502\u00a0\u00a0 \u251c\u2500\u2500 filters.py\n    \u2502\u00a0\u00a0 \u2514\u2500\u2500 mapping.json\n    \u2514\u2500\u2500 other-site.com\n    \u2502\u00a0\u00a0 \u251c\u2500\u2500 __ini__.py\n        \u251c\u2500\u2500 f2eSkip\n        \u251c\u2500\u2500 filters.py\n        \u2514\u2500\u2500 mapping.json\n\n**IMPORTANT**: all of folders need the file ``__init__.py`` con convert it into a Python package.\n\n**Ignoring directory**\n\nIf you want that a directory will be ignored, only create a file called ``f2eSkip`` into the directory and the engine will ignore it.\n\n\nRunning Examples\n----------------\n\nWithout Docker\n++++++++++++++\n\nInstall:\n\n.. code-block:: bash\n\n    > pip install -U feed-to-exporter\n\n**Wordpress mode**\n\n\nBasic Usage:\n\n\n.. code-block:: bash\n\n    > f2e wordpress -W https://mywordpress.com -U user -A \"XXXX XXXX XXXX XXXX XXXX XXXX\" examples/\n\nWhere ``-A`` indicates the Application Password\n\nFor more help type ``-h``:\n\n.. code-block:: bash\n\n    > f2e wordpress -h\n\n    usage: f2e wordpress [-h] --wordpress-url WORDPRESS_URL --user USER --app-auth\n                     APP_AUTH [--devel]\n                     [feed_source [feed_source ...]]\n\n    positional arguments:\n      feed_source           target url or path\n\n    optional arguments:\n      -h, --help            show this help message and exit\n      --wordpress-url WORDPRESS_URL, -W WORDPRESS_URL\n                            wordpress url\n      --user USER, -U USER  user to access to Wordpress\n      --app-auth APP_AUTH, -A APP_AUTH\n                            app auth code (from \"Application Passwords\" plugin)\n      --devel               running in develop mode doesn't publish Wordpress Post\n\n\n**MongoDB mode**\n\nWith default parameters (mongo in localhost without authentication, database=f2e, collection=f2e)\n\n.. code-block:: bash\n\n    > f2e mongo examples/\n\nSetting some parameters:\n\n\n.. code-block:: bash\n\n    > f2e mongo -U mongoAdmin -M mongodb://10.0.0.1:27017 examples/\n\nFor more help type ``-h``:\n\n.. code-block:: bash\n\n    > f2e mongo -h\n\n    usage: f2e mongo [-h] [--user USER] [--password PASSWORD]\n                 [--collection COLLECTION] [--database DATABASE]\n                 [--mongo-url MONGO_URN]\n                 [feed_source [feed_source ...]]\n\n    positional arguments:\n      feed_source           target url or path\n\n    optional arguments:\n      -h, --help            show this help message and exit\n      --user USER, -U USER  mongodb user\n      --password PASSWORD, -P PASSWORD\n                            mongodb password\n      --collection COLLECTION, -C COLLECTION\n                            mongo collection\n      --database DATABASE, -D DATABASE\n                            mongo database\n      --mongo-url MONGO_URN, -M MONGO_URN\n                            mongo URL. (Default: mongodb://127.0.0.1:27017/f2e)\n\n\nUsing Docker\n++++++++++++\n\nDocker only run in discovery mode and can schedule a new run each some seconds.\n\nYou can mount a dir with the filters/mapping, but it's highly recommended to put it into a git repository.\n\n**Environment vars**\n\n- f2e_CMD_PARAMETERS: f2e running options\n- f2e_CHECK_TIME: time to launch in seconds\n- f2e_FILTERS_GIT: git where to download filters and mapping\n\n**Running examples**\n\nRun feed each 3600 seconds:\n\n.. code-block:: bash\n\n    > docker run --rm \\\n        -e f2e_FILTERS_GIT=https://XXXXXXXXXXXXXX@github.com/cr0hn/myfeeds-repo.git \\\n        -e f2e_CMD_PARAMETERS='wordpress -W https://mywordpress.com -U admin -A \"XXXX XXXX XXXX XXXX XXXX XXXX\"' \\\n        -e f2e_CHECK_TIME=3600 f2e\n\n\nContributing\n============\n\nAny collaboration is welcome!\n\nThere're many tasks to do.You can check the `Issues <https://github.com/cr0hn/feed-to-exporter/issues/>`_ and send us a Pull Request.\n\nLicense\n=======\n\nThis project is distributed under `BSD 3 license <https://github.com/cr0hn/feed-to-exporter/blob/master/LICENSE>`_",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/cr0hn/feed-to-exporter",
        "keywords": "",
        "license": "BSD 3 License",
        "maintainer": "",
        "maintainer_email": "",
        "name": "feed-to-exporter",
        "package_url": "https://pypi.org/project/feed-to-exporter/",
        "platform": "",
        "project_url": "https://pypi.org/project/feed-to-exporter/",
        "project_urls": {
            "Homepage": "https://github.com/cr0hn/feed-to-exporter"
        },
        "release_url": "https://pypi.org/project/feed-to-exporter/1.0.3/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Export RSS Feed as different outputs: Wordpress, MongoDB",
        "version": "1.0.3"
    },
    "last_serial": 4223045,
    "releases": {
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2a07bcbbdfcd09133df310653cec8931",
                    "sha256": "025c421a0103be015f2778061732929ebb275d8469ace07d75384c6184f95305"
                },
                "downloads": -1,
                "filename": "feed-to-exporter-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "2a07bcbbdfcd09133df310653cec8931",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 23321,
                "upload_time": "2018-08-29T11:06:33",
                "url": "https://files.pythonhosted.org/packages/b1/27/f5b28ea9207aad386a0d9c2cd5c4162d4c801eb7ad715446598e8c42db2b/feed-to-exporter-1.0.0.tar.gz"
            }
        ],
        "1.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c6be5c70640cc37dadb189fd662703b8",
                    "sha256": "7c950d96dd2a4c1710412e48618b3d2a3339e4a04b2beb0d0bb7fd4a95686f14"
                },
                "downloads": -1,
                "filename": "feed-to-exporter-1.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "c6be5c70640cc37dadb189fd662703b8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 20219,
                "upload_time": "2018-08-29T13:09:56",
                "url": "https://files.pythonhosted.org/packages/71/8a/bae893914525ba78c1b8624d469e1652765595f68f97f8769f0cd6d8f2ae/feed-to-exporter-1.0.1.tar.gz"
            }
        ],
        "1.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "175c07fb36aba7a7b2dc664856b76c62",
                    "sha256": "59bf0528fb5ed5ee64764d36dff38c5cdd8e1a84acad50506d689fb31dfa9ddd"
                },
                "downloads": -1,
                "filename": "feed-to-exporter-1.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "175c07fb36aba7a7b2dc664856b76c62",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 20410,
                "upload_time": "2018-08-30T15:06:39",
                "url": "https://files.pythonhosted.org/packages/56/58/cb133500b06546d9e40cd2eb18a00f9f0fc5acc572cddb569fae59d9c136/feed-to-exporter-1.0.2.tar.gz"
            }
        ],
        "1.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "1dafbd07adb8ec408d35b336605d0ee9",
                    "sha256": "358d2b9f76634080eac24641dea8ebdae74518bd7757af61bc29edc603e2d3fd"
                },
                "downloads": -1,
                "filename": "feed-to-exporter-1.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "1dafbd07adb8ec408d35b336605d0ee9",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15834,
                "upload_time": "2018-08-30T15:18:43",
                "url": "https://files.pythonhosted.org/packages/e7/9a/6ff4e32c4d350837266fc5d560c582bae5b8a1d3e6d890889ae4628c1a88/feed-to-exporter-1.0.3.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "1dafbd07adb8ec408d35b336605d0ee9",
                "sha256": "358d2b9f76634080eac24641dea8ebdae74518bd7757af61bc29edc603e2d3fd"
            },
            "downloads": -1,
            "filename": "feed-to-exporter-1.0.3.tar.gz",
            "has_sig": false,
            "md5_digest": "1dafbd07adb8ec408d35b336605d0ee9",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 15834,
            "upload_time": "2018-08-30T15:18:43",
            "url": "https://files.pythonhosted.org/packages/e7/9a/6ff4e32c4d350837266fc5d560c582bae5b8a1d3e6d890889ae4628c1a88/feed-to-exporter-1.0.3.tar.gz"
        }
    ]
}