{
    "info": {
        "author": "Anne L'H\u00f4te, Bruno Thomas",
        "author_email": "alhote@icij.org, bthomas@icij.org",
        "bugtrack_url": null,
        "classifiers": [
            "Programming Language :: Python",
            "Programming Language :: Python :: 3.6"
        ],
        "description": "solr2es\n=======\n\n.. image:: https://circleci.com/gh/ICIJ/solr2es.png?style=shield&circle-token=846c844f540fb3746b80b8f12656ddde665b5037\n   :alt: Circle CI\n   :target: https://circleci.com/gh/ICIJ/solr2es\n\nMigration script from solr to elasticsearch via a queue, that could be either redis or postgresql.\n\n\nCLI\n---\n\nHere are the option to use as a command line :\n\n* -m | --migrate : to migrate from a solr index to an elasticsearch index\n* -r | --resume : to resume from a given queue to an elasticsearch index. By default, the queue will be redis. If the parameter \"postgresqldsn\" is set, the queue will be postgresql.\n* -d | --dump : to dump from a solr index into a queue. By default, the queue will be redis.  If the parameter \"postgresqldsn\" is set, the queue will be postgresql.\n* -t | --test : to test the solr and elasticsearch connections\n* -a | --async : to use python 3 asyncio\n* --solrhost : to set solr host (by default: 'solr')\n* --solrfq: to set solr filter query (by default: '*')\n* --solrid: to set solr id field name (by default: 'id')\n* --core: to set solr core name (by default: 'solr2es')\n* --index: to set index name for solr and elasticsearch (by default: solr core name, see --core parameter)\n* --redishost: to set redis host (by default: 'redis')\n* --postgresqldsn: to set postgresql Data Source Name (by default: None, by example: 'dbname=solr2es user=test password=test host=postgresql')\n* --eshost: to set elasticsearch host (by default: 'elasticsearch')\n* --translationmap: dict string or file path (starting with @) to translate fields from queue into elasticsearch (by default: None, by example: '{\"postgresql_field\": {\"name\": \"es_field\"}}')\n* --esmapping: dict string or file path (starting with @) to set elasticsearch mapping (by default: None)\n* --essetting: dict string or file path (starting with @) to set elasticsearch setting (by default: None)\n\n\n.. image:: examples/solr2es_process.png\n    :alt: solr2es process\n    :align: center\n\n\nUse\n---\n\n**translation_map**\n\n.. image:: examples/migration.jpg\n    :alt: migration map\n    :align: center\n\nThe purpose of a translation_map is to create a mapping between the fields coming from the queue (either Redis or Postgresql) to the ones inserted to Elasticsearch.\n\n1. If a field from the queue doesn't exist in the translation_map, it will be inserted as it is into Elasticsearch.\n\n2. Use the property *name* to rename a field in Elasticsearch :\n\n::\n\n    {\"queue_name\": {\"name\": \"elasticsearch_name\"}}\n\n\n3. Use the property *default* if you want to set a default value into a field in Elasticsearch.\n\nIf the field exists into the queue and has a value, it won't be changed by the translation_map.\nOtherwise a field *queue_name* willl be added to Elasticsearch with value *john doe*.\n\n::\n\n    {\"queue_name\": {\"default\": \"john doe\"}}\n\n4. Use the property *name* with some *.* in it, to create a nested field in Elasticsearch.\n\nIf the queue record has a field *nested_a_b*, the Elasticsearch record will get a field *nested*, that will have a nested field *a*, that will have a nested field *b* that will get the content of *nested_a_b*.\n\n::\n\n    {\"nested_a_b\": {\"name\": \"nested.a.b\"}}\n\n5. Use the property *name* with some regex groups capture to rename a bulk of queue fields in Elasticsearch by adding `[regexp]` at the beginning of the field.\nThis will rename all the fields prefixed by *queue_* into *elasticsearch_*.\n\n::\n\n    {\"[regexp]queue_(.*)\": {\"name\": \"elasticsearch_\\\\1\"}}\n\n6. Use the property *ignore* at *true* to ignore some fields from the queue to Elasticsearch.\n\n::\n\n    {\"ignored_field\": {\"ignore\": true}}\n\n7. Use the property *routing_field* set to *true* to use one field for routing in elasticsearch. An exception will be raised if several fields are set to true.\n\n::\n\n    {\"my_root_doc\": {\"routing_field\": true}}\n\n8. Use the property *multivalued* set to *false* to ignore multi valued array field. Get the first value instead. By default the array is copied.\n\n::\n\n    {\"my_array\": {\"multivalued\": false}}\n\n\n\n**execution**\n\n1. Execute a dump from Solr into Postgresql specifying the Solr host, the Solr core, the Solr id and the Postgresql DSN\n\n::\n\n    solr2es --postgresqldsn 'dbname=solr2es user=test password=test host=localhost' --solrhost 127.0.0.1 --core test_core --solrid solr_id -d -a\n\n2. Execute a resume from Postgresql into Elasticsearch specifying the Postgresql DSN, the Elasticsearch index, the Elasticsearch mapping, the Elasticsearch settings and the translation map\n\n::\n\n    solr2es --postgresqldsn 'dbname=solr2es user=test password=test host=localhost' --index es-index --translationmap @examples/translation-map.json --esmapping @examples/datashare_index_mappings.json --essetting @examples/datashare_index_settings.json -r -a\n\n\nDevelop\n-------\n\nTo build and run tests you can make :\n\n::\n\n    virtualenv --python=python3.6 venv\n    source venv/bin/activate\n    python setup.py develop\n    python setup.py test\n\nTo release :\n\n::\n\n    python setup.py  sdist bdist_egg upload\n\n\nMisc\n----\n\nSome features are not implemented yet :\n\n- Resume from the redis queue to elasticsearch in asynchronous mode (function aioresume_from_redis)\n- Resume from the redis queue to elasticsearch in synchronous mode (function resume_from_redis)\n- Resume from the postgresql queue to elasticsearch in synchronous mode (function resume_from_postgresql)\n\nChanges\n=======\n\nv. 0.7\n------\n\n* multivalued field : flatten the array if it has one value\n* multivalued field : ignore multi valuated field in translation map\n* multivalued field : copy the array into elasticsearch\n\nv. 0.6\n------\n\n* error handling : logs ids that have failed when resuming from postgresql\n* adds a the possibility to specify a routing field in the translation map\n\nv. 0.5\n------\n\n* adds postgresql resume\n* elasticsearch : adds mappings and settings support\n* better logs and progress marks\n* doc : README\n* translation map : support for empty default list\n* adds postgresql blocking queue\n* translation map : ignore field\n* translation map : default value\n\nv. 0.4\n------\n\n* [solr2es] wildcard support in translation_map\n* [solr2es] nested fields support in translation_map\n* [solr2es] adds solrid parameter to change sort field\n* [solr2es] adds solrfq parameter to parallelize solr reading\n\nv. 0.3\n------\n\n* [solr2es] adds translation map for fields\n* [solr2es] adds elasticsearch mapping for index creation\n* [test] compatible with 6.6.0\n\nv. 0.2\n------\n\n* [log] adds logger and progression feedbacks\n* [cli] exit if no args\n\nv. 0.1\n------\n\n* [solr2es] initial version",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/ICIJ/solr2es",
        "keywords": "migration search engine solr elasticsearch",
        "license": "GPL-3.0",
        "maintainer": "",
        "maintainer_email": "",
        "name": "solr2es",
        "package_url": "https://pypi.org/project/solr2es/",
        "platform": "",
        "project_url": "https://pypi.org/project/solr2es/",
        "project_urls": {
            "Homepage": "https://github.com/ICIJ/solr2es"
        },
        "release_url": "https://pypi.org/project/solr2es/0.7/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Python solr/elasticsearch migration script",
        "version": "0.7"
    },
    "last_serial": 4656619,
    "releases": {
        "0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0836200a97a01a73d2dbb550fc8a03a2",
                    "sha256": "2c3d13f5040d8a7e222805a441e8e3e00911ed85240938cc313eb3491d2a45ae"
                },
                "downloads": -1,
                "filename": "solr2es-0.1-py3.6.egg",
                "has_sig": false,
                "md5_digest": "0836200a97a01a73d2dbb550fc8a03a2",
                "packagetype": "bdist_egg",
                "python_version": "3.6",
                "requires_python": null,
                "size": 16700,
                "upload_time": "2018-11-14T11:56:38",
                "url": "https://files.pythonhosted.org/packages/17/cd/d021bd64c4fa4d6152e2e4e7cf5011bf2b25c1f08ca7b346b6e756d74487/solr2es-0.1-py3.6.egg"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d1c01dce9ed5b557f039e3131fa2f552",
                    "sha256": "3f746acf399ea18c912a0afb88a6584dc2c386ea615f945679b96a095da29371"
                },
                "downloads": -1,
                "filename": "solr2es-0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "d1c01dce9ed5b557f039e3131fa2f552",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 4453,
                "upload_time": "2018-11-14T11:56:35",
                "url": "https://files.pythonhosted.org/packages/43/ca/ea0778e1c67244454f0526a68f716abf9f36600c8b205c0f7e8dd510d805/solr2es-0.1.tar.gz"
            }
        ],
        "0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ad2b3381be4a9b2bf0349988adb2f80f",
                    "sha256": "7716e273f52cc75478ad1fe314004a633de86aefb1ea9e047b6d5c0c12c2830a"
                },
                "downloads": -1,
                "filename": "solr2es-0.2-py3.6.egg",
                "has_sig": false,
                "md5_digest": "ad2b3381be4a9b2bf0349988adb2f80f",
                "packagetype": "bdist_egg",
                "python_version": "3.6",
                "requires_python": null,
                "size": 18000,
                "upload_time": "2018-11-14T15:18:13",
                "url": "https://files.pythonhosted.org/packages/6d/22/2d89f232ecbbbdadb7e9ba424fe208529d7d4a368fef1be9b293b4d8a2c8/solr2es-0.2-py3.6.egg"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "963fbabe7d8156a380e1b3e5382e245d",
                    "sha256": "205b38905a8d5175ea7fe3096bd669ce9a01ab31fbd4835dea0df6d2a144a66c"
                },
                "downloads": -1,
                "filename": "solr2es-0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "963fbabe7d8156a380e1b3e5382e245d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 4900,
                "upload_time": "2018-11-14T15:18:11",
                "url": "https://files.pythonhosted.org/packages/33/7d/adb839d046f5259a3585ec5507ec1f00a23bc9fd0b285f57a798b75cb607/solr2es-0.2.tar.gz"
            }
        ],
        "0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8cc99e69675b78b4d8fc3a8f8151a646",
                    "sha256": "94ad5891169b2ee09535f273a6a29dfdf29e2015099416aa77465b3891e1a401"
                },
                "downloads": -1,
                "filename": "solr2es-0.3-py3.6.egg",
                "has_sig": false,
                "md5_digest": "8cc99e69675b78b4d8fc3a8f8151a646",
                "packagetype": "bdist_egg",
                "python_version": "3.6",
                "requires_python": null,
                "size": 23160,
                "upload_time": "2018-11-19T11:01:50",
                "url": "https://files.pythonhosted.org/packages/5e/91/4f1d1d0ca72b5716caa85d604ea23c9829b64b968a134370642208cc6a14/solr2es-0.3-py3.6.egg"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "432df4bc5d2ba1bb20dcff83acd91da8",
                    "sha256": "04ec335733ae0f54c36bd28175e289f53de4441b68e5c148f4d724a7c8656a78"
                },
                "downloads": -1,
                "filename": "solr2es-0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "432df4bc5d2ba1bb20dcff83acd91da8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6938,
                "upload_time": "2018-11-19T11:01:48",
                "url": "https://files.pythonhosted.org/packages/05/1f/b7121791ac776efc18a7ce4fb016a0b8c6ad42c1ead3f0326f0eb7114d75/solr2es-0.3.tar.gz"
            }
        ],
        "0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3197cb48f3b32821d7a1802335d9e1c5",
                    "sha256": "41d1ebd6be465dd856c53e20a9ef8062946309918f27834ad8b13ffd81421d9d"
                },
                "downloads": -1,
                "filename": "solr2es-0.4-py3.6.egg",
                "has_sig": false,
                "md5_digest": "3197cb48f3b32821d7a1802335d9e1c5",
                "packagetype": "bdist_egg",
                "python_version": "3.6",
                "requires_python": null,
                "size": 24999,
                "upload_time": "2018-11-19T17:50:14",
                "url": "https://files.pythonhosted.org/packages/df/27/a42269f08ef3acbf2d4ff9dca0f7f76095e25940757e6fcf07498f112825/solr2es-0.4-py3.6.egg"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "6c9e954ff263e7a27d10c13ec47ec241",
                    "sha256": "6f17eadc3f788e61f2be478880dcf1842c335cf3e7256d0456954fb5d51cdb93"
                },
                "downloads": -1,
                "filename": "solr2es-0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "6c9e954ff263e7a27d10c13ec47ec241",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7486,
                "upload_time": "2018-11-19T17:50:12",
                "url": "https://files.pythonhosted.org/packages/dc/07/0ff078da06b6c02c8722800d1cd99b8fcd944a08a2152e66250b063ef388/solr2es-0.4.tar.gz"
            }
        ],
        "0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "86e6e5d04d2c1c536de4283eee135779",
                    "sha256": "0d2fadc0a3200f28327915fe2a14abdb66cc5292bffa66bd095dc205c8722757"
                },
                "downloads": -1,
                "filename": "solr2es-0.5-py3.6.egg",
                "has_sig": false,
                "md5_digest": "86e6e5d04d2c1c536de4283eee135779",
                "packagetype": "bdist_egg",
                "python_version": "3.6",
                "requires_python": null,
                "size": 21056,
                "upload_time": "2018-11-29T16:38:20",
                "url": "https://files.pythonhosted.org/packages/45/ab/afe651fcd8bf94f36e7c07b331d41333b39e870e9e085f6b421fddfe03d6/solr2es-0.5-py3.6.egg"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "2cf25cbb928790562f80e4365f6f6084",
                    "sha256": "0a8e0d335806a8d2e2a2f81e5c888f4829ed793e2ffe14b1172a3723e3adfc8c"
                },
                "downloads": -1,
                "filename": "solr2es-0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "2cf25cbb928790562f80e4365f6f6084",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9127,
                "upload_time": "2018-11-29T16:38:17",
                "url": "https://files.pythonhosted.org/packages/40/5b/6489133ecba4eb20f7a9f54724f5d9e98a925adbab03045e5280b0718f1c/solr2es-0.5.tar.gz"
            }
        ],
        "0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9e8f9532458bd986b11f59948a18a47f",
                    "sha256": "3855469053cc1eec45dfed421a8e716a7d323c5f96afe987b87b9a1b02709333"
                },
                "downloads": -1,
                "filename": "solr2es-0.6-py3.6.egg",
                "has_sig": false,
                "md5_digest": "9e8f9532458bd986b11f59948a18a47f",
                "packagetype": "bdist_egg",
                "python_version": "3.6",
                "requires_python": null,
                "size": 22168,
                "upload_time": "2018-12-12T14:49:51",
                "url": "https://files.pythonhosted.org/packages/d3/78/99e7959fbd686d1daf027db1f43c2d2fcab492510550e7321012b37d27d7/solr2es-0.6-py3.6.egg"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "df6b33fb6e210310e9f75d1a7a8819d8",
                    "sha256": "369eb3ace2e8d78946872f94a5587d48940a5b6109e82fa1b4467df6929a2540"
                },
                "downloads": -1,
                "filename": "solr2es-0.6.tar.gz",
                "has_sig": false,
                "md5_digest": "df6b33fb6e210310e9f75d1a7a8819d8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9941,
                "upload_time": "2018-12-12T14:49:49",
                "url": "https://files.pythonhosted.org/packages/79/e4/b2dfba51dcfe7ef8b76e969e77d0d18a0ca8caf5604cabbe3970ffcf1ee0/solr2es-0.6.tar.gz"
            }
        ],
        "0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "401ef426d9863e74bf104e7548bbc0ff",
                    "sha256": "f72b63833f4cf1ced4bcc01d1991a064d8a74239fa485c57ab2b755cda4b41b9"
                },
                "downloads": -1,
                "filename": "solr2es-0.7-py3.6.egg",
                "has_sig": false,
                "md5_digest": "401ef426d9863e74bf104e7548bbc0ff",
                "packagetype": "bdist_egg",
                "python_version": "3.6",
                "requires_python": null,
                "size": 22500,
                "upload_time": "2019-01-03T14:37:58",
                "url": "https://files.pythonhosted.org/packages/fb/1f/81f9fa24429d3d07c65b340b42766cd8ee05d684eeacb5ce1f9a0c9c89a4/solr2es-0.7-py3.6.egg"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "24cc51cf9dcddd50f79365da5f01f4a8",
                    "sha256": "e3be67054c72e1423f80ae2aa3a9da9d9e82b02cd15c5efe29001b7cc29dbb59"
                },
                "downloads": -1,
                "filename": "solr2es-0.7.tar.gz",
                "has_sig": false,
                "md5_digest": "24cc51cf9dcddd50f79365da5f01f4a8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10208,
                "upload_time": "2019-01-03T14:37:56",
                "url": "https://files.pythonhosted.org/packages/6c/ac/645baf4f5e04d6fc73ac8cc1e2ce50515a98b445009d45188839be62a0ef/solr2es-0.7.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "401ef426d9863e74bf104e7548bbc0ff",
                "sha256": "f72b63833f4cf1ced4bcc01d1991a064d8a74239fa485c57ab2b755cda4b41b9"
            },
            "downloads": -1,
            "filename": "solr2es-0.7-py3.6.egg",
            "has_sig": false,
            "md5_digest": "401ef426d9863e74bf104e7548bbc0ff",
            "packagetype": "bdist_egg",
            "python_version": "3.6",
            "requires_python": null,
            "size": 22500,
            "upload_time": "2019-01-03T14:37:58",
            "url": "https://files.pythonhosted.org/packages/fb/1f/81f9fa24429d3d07c65b340b42766cd8ee05d684eeacb5ce1f9a0c9c89a4/solr2es-0.7-py3.6.egg"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "24cc51cf9dcddd50f79365da5f01f4a8",
                "sha256": "e3be67054c72e1423f80ae2aa3a9da9d9e82b02cd15c5efe29001b7cc29dbb59"
            },
            "downloads": -1,
            "filename": "solr2es-0.7.tar.gz",
            "has_sig": false,
            "md5_digest": "24cc51cf9dcddd50f79365da5f01f4a8",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 10208,
            "upload_time": "2019-01-03T14:37:56",
            "url": "https://files.pythonhosted.org/packages/6c/ac/645baf4f5e04d6fc73ac8cc1e2ce50515a98b445009d45188839be62a0ef/solr2es-0.7.tar.gz"
        }
    ]
}