{ "info": { "author": "Lev Veshnyakov", "author_email": "lev@lev.website", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Topic :: Software Development :: Build Tools" ], "description": "Django import data\n==================\n\n**Django import data** is a command-line tool for importing XML, HTML or\nJSON data to django models via XSLT mapping.\n\nSource code is located here -\nhttps://github.com/lev-veshnyakov/django-import-data.\n\nBasic features\n--------------\n\nDjango import data can take any XML, HTML or JSON source file or URL as\nan input and save entities from it to the django models without need to\nmodify an existing code.\n\nIt also supports saving of a related data in form one-to-many and\nmany-to-many.\n\nDependencies\n------------\n\nIt uses `lxml `__ library for all XML manipulations and\n`dicttoxml `__ library for\nthe transformation from JSON to XML.\n\nInstallation\n------------\n\nFirst you need to install dependencies for lxml library:\n\n.. code:: bash\n\n sudo apt-get install libxml2-dev libxslt-dev python-dev\n\nThen install django-import-data using pip:\n\n.. code:: bash\n\n pip install django-import-data\n\nIf you want the latest version you can install it from Github:\n\n.. code:: bash\n\n pip install git+https://github.com/lev-veshnyakov/django-import-data\n\nAdd import\\_data to INSTALLED\\_APPS:\n\n.. code:: python\n\n INSTALLED_APPS = [\n ...\n 'import_data',\n ]\n\nUsage\n-----\n\n**Django import data** is a management command-line tool, that can be\nused from the code as well.\n\nToo see the list of console commands type:\n\n.. code:: bash\n\n python manage.py help\n\nIn the output you will find import\\_data section like below:\n\n.. code:: bash\n\n [import_data]\n process_xslt\n validate_xml\n\nTo get help for the particular command type:\n\n.. code:: bash\n\n python manage.py help process_xslt\n\n.. code:: bash\n\n python manage.py help validate_xml\n\n.. code:: bash\n\n python manage.py help json_to_xml\n\nTo call console commands from your code use\n`django.core.management.call\\_command `__:\n\n.. code:: python\n\n from django.core.management import call_command\n\n call_command('process_xslt', 'http://stackoverflow.com/', 'transform.xslt', '--save')\n\nHow it works\n------------\n\nIn a few words it takes a source in either XML or HTML, then takes\nprovided by you XSLT file, transforms the source into the specific XML\nrepresentation, and then saves the data from this XML to the database\nusing models.\n\nThe point is, that you don't need to write procedural code for saving\ndata. You only need to write XSLT files, which is actually XML. One file\nfor one source. By the source I mean a range of XML or HTML files in the\nsame format. For example all google search result pages have one schema.\nThat means that you can write only one XSLT transformation file to\nimport all search pages data.\n\nThe difficult moment is that you have to be familiar with XSLT and\nXpath.\n\nXSLT and XPath\n~~~~~~~~~~~~~~\n\nXSLT is a language for transforming XML documents into XHTML documents\nor to other XML documents.\n\nXSLT uses XPath to find information in an XML document. XPath is used to\nnavigate through elements and attributes in XML documents.\n\nIf you are not familiar with that I reccomend you to read a `short\ntutorial on\nwww.w3school.com `__.\n\nMoreover, you have to know what an XML Schema is and a particular schema\nlanguage RELAX NG.\n\nXML Schema and RELAX NG\n~~~~~~~~~~~~~~~~~~~~~~~\n\n**Django import data** uses RELAX NG to validate resuls of\ntransformations. That means if you write XSLT file wrong, it wouldn't be\naccepted.\n\nBut you dont have to write RELAX NG schema yoursef, it's already\n`included in the\nmodule `__.\n\nResulting XML\n~~~~~~~~~~~~~\n\nAfter XSLT transformation and schema validation the resulting XML file\nshould be like following:\n\n.. code:: xml\n\n \n \n \n \n Andrew Tanenbaum\n \n \n Donald Knuth\n \n \n \n \n Computer Networks\n 0130661023\n \n \n \n The Art of Computer Programming\n 0321751043\n \n \n \n \n\nThis XML can be automatically saved to the models.\n\nIt contains the root element ````. Into it are nested\n```` elements. Each model element represents a particular django\nmodel. You must provide ``model=\"\"`` attributes, in which specify a\nrelated model. Path to the model is in following format:\napplication\\_name.ModelName, the same format like ``manage.py dumpdata``\nuses.\n\nModel elements don't have to be unique. If you specify several model\nelements with the same model attribute, they will be merged together.\nThis concerns to item elements as well.\n\nModel elements contain ```` elements, representing particular\nrecords in the database. They have only one required attribute\n``name=\"\"``, which sets the name of a related model field.\n\nForeign keys\n~~~~~~~~~~~~\n\n**Django import data** supports import of related entities in the form\none-to-many and many-to-many. To save such entities your models should\nhave appropriate foreign keys.\n\nIn a resulting XML you can use ```` and ```` elements (see\nabove). They have ``model=\"\"`` and ``key=\"\"`` attributes, pointing to\nthe related ```` elements.\n\nSetting key attribute\n~~~~~~~~~~~~~~~~~~~~~\n\nThe ``key=\"\"`` attribute of ```` elements must be unique by each\nunique record. It has not to be the same as a primary key value in the\ndatabase. It even will not be stored (if you want to store a primary key\nvalue, use ```` element).\n\nTherefore, the value of the ``key=\"\"`` attribute not obliged to be\ninteger. You can use any sring. Often it's convenient to use an URL as\nthe key.\n\nYou can even omit filling that attribute if you don't have related\nitems.\n\n**But one case is special**. That's when you don't have any unique\nattributes in the source. In that case you can use ``generate-id(..)``\nXPath function. It will generate unique IDs for every separate XML node\nin the source.\n\nUsing JSON sources\n~~~~~~~~~~~~~~~~~~\n\nIt's possible to use JSON sources. Because the transformation is\nXSLT-based, JSON is converted to the appropriate XML.\n\nFor example the following JSON code:\n\n.. code:: javascript\n\n {\n \"firstName\": \"John\",\n \"lastName\": \"Smith\",\n \"age\": 25,\n \"address\": {\n \"streetAddress\": \"21 2nd Street\",\n \"city\": \"New York\",\n \"state\": \"NY\",\n \"postalCode\": \"10021\"\n },\n \"phoneNumber\": [\n {\n \"type\": \"home\",\n \"number\": \"212 555-1234\"\n },\n {\n \"type\": \"fax\",\n \"number\": \"646 555-4567\"\n }\n ],\n \"gender\": {\n \"type\": \"male\"\n }\n }\n\nwill be converted to this XML:\n\n.. code:: xml\n\n \n \n John\n Smith\n 25\n
\n 10021\n New York\n 21 2nd Street\n NY\n
\n \n \n home\n 212 555-1234\n \n \n fax\n 646 555-4567\n \n \n \n male\n \n
\n\nThat XML is supposed to be used for writing an XSLT transformation.\n\nIf you use some JSON source and want to find out which XML is related\nfor it, then use the command:\n\n.. code:: bash\n\n python manage.py json_to_xml \n\nAfter writing an XSLT transformation file you can use ``process_xslt``\nspecifying the URL of the JSON source.\n\nJSON to XML transformations is performed by dicttoxml library written by\nRyan McGreal https://github.com/quandyfactory/dicttoxml.\n\nExamples\n--------\n\nSave data to one model\n~~~~~~~~~~~~~~~~~~~~~~\n\nIn this simple example we will parse the main page of\n`stackoverflow.com `__ and save titles of\nrecent questions to this model:\n\n.. code:: python\n\n from django.db import models\n\n class Question(models.Model):\n title = models.CharField(max_length=255)\n\nFirst we need to write an XSLT file:\n\n.. code:: xml\n\n \n \n \n \n \n \n \n \n \n \n \n \n\nName it transform.xslt and perform the following command:\n\n.. code:: bash\n\n python manage.py process_xslt http://stackoverflow.com/questions transform.xslt --validate\n\nThe output will be like this (but longer):\n\n.. code:: xml\n\n \n \n \n \n customizing soap response attribute format\n \n \n Second fragment loaded but not visible on screen\n \n \n django-oscar :first time use \"python manage.py migrate\" gets error\n \n \n JTable fireTableDataChanged() method doesn't refresh table\n \n \n why the dynamic nodes dont respond to click in jstree?\n \n \n Connecting kdb+ to R\n \n \n \n\nParameter ``--validate`` adds to output ``Document is valid``.\n\nTo save the result add the parameter ``--save`` to the command above.\n\nSave data to related models\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nUse the same source and add two other models with foreign keys:\n\n.. code:: python\n\n class Question(models.Model):\n title = models.CharField(max_length=255)\n user = models.ForeignKey('User', null=True)\n tags = models.ManyToManyField('Tag')\n\n class Tag(models.Model):\n title = models.CharField(max_length=255)\n\n class User(models.Model):\n title = models.CharField(max_length=255)\n\nThe XSLT file will be like following:\n\n.. code:: xml\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\nTake notice how calculated attributes are set and how to use generate-id\nfunction. In tis example we use both types of relationship: one-to-many\nand many-to-many. This means that one question can have several tags,\nbut only one related user (which has changed the question last).\n\nThe output will be like following (but essential longer):\n\n.. code:: xml\n\n \n \n \n \n C TCP Server doesn't send data before closing\n \n \n \n \n \n \n \n Ninject Factory Extension\n \n \n \n \n \n \n \n \n \n c\n \n \n linux\n \n \n sockets\n \n \n tcp\n \n \n c#\n \n \n dependency-injection\n \n \n ninject\n \n \n ninject.web.mvc\n \n \n ninject-extensions\n \n \n \n \n user3809727\n \n \n user2119597\n \n \n ", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://django-import-data.readthedocs.org", "keywords": "django import mapping parser xml html json xslt", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "django-import-data", "package_url": "https://pypi.org/project/django-import-data/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/django-import-data/", "project_urls": { "Homepage": "http://django-import-data.readthedocs.org" }, "release_url": "https://pypi.org/project/django-import-data/0.4.2/", "requires_dist": [ "lxml", "django", "dicttoxml", "pypandoc; extra == 'dev'" ], "requires_python": "", "summary": "A Django command line tool for importing HTML, XML and JSON data to models via XSLT mapping", "version": "0.4.2" }, "last_serial": 1945016, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "78f794494eaa47e7dde81aa72e9f10af", "sha256": "835630ac1520db4e94c4c1c9121e9028b4fdcd89afb275305be793e2228587af" }, "downloads": -1, "filename": "django-import-data-0.1.0.tar.gz", "has_sig": false, "md5_digest": "78f794494eaa47e7dde81aa72e9f10af", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12759, "upload_time": "2016-02-04T16:20:15", "url": "https://files.pythonhosted.org/packages/7c/f3/d582095f5f382d3b8b8fdb917146004b440671b7bc589bf06603f104e1dc/django-import-data-0.1.0.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "ae800a337b446943601e0204ce8bc2c2", "sha256": "8b79442f036ff1a4cf690e917ca6b0990f04c0d13ecff6baee3638dfe0930b27" }, "downloads": -1, "filename": "django-import-data-0.1.1.tar.gz", "has_sig": false, "md5_digest": "ae800a337b446943601e0204ce8bc2c2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12765, "upload_time": "2016-02-05T13:21:31", "url": "https://files.pythonhosted.org/packages/a2/26/e9a3d6ab540ebc226de1d3fa0f77779210d24e61486ae1e5d2f04b892897/django-import-data-0.1.1.tar.gz" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "6d5eaeadff1dd38aa4ab0f084171f2eb", "sha256": "ded83e922351fbc552d3d6cb5bd6fc1505220d11dd50eca9ac727ebf4900ec15" }, "downloads": -1, "filename": "django-import-data-0.1.2.tar.gz", "has_sig": false, "md5_digest": "6d5eaeadff1dd38aa4ab0f084171f2eb", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13076, "upload_time": "2016-02-05T13:41:06", "url": "https://files.pythonhosted.org/packages/b4/c7/785869aea7ee414321ddff741a6d1d272a4c64e52b784872e8a8f1661204/django-import-data-0.1.2.tar.gz" } ], "0.1.4": [ { "comment_text": "", "digests": { "md5": "9a7b9f4667316c055cc7f04f7f59e15e", "sha256": "e84a5b8581bad767bac6b81c45fb2e77a8b6b1bab22a7ec40d437182d57fd72e" }, "downloads": -1, "filename": "django-import-data-0.1.4.tar.gz", "has_sig": false, "md5_digest": "9a7b9f4667316c055cc7f04f7f59e15e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13175, "upload_time": "2016-02-06T12:24:45", "url": "https://files.pythonhosted.org/packages/ee/bd/84dd42c7988640a54e3b418320a2e49e7dff065ab1011fff450ff032c580/django-import-data-0.1.4.tar.gz" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "df006eca45f3311620509e8181a44e39", "sha256": "a519fb22269a94e98d7111df614569dbd92130811426c9a94f035f55dd465666" }, "downloads": -1, "filename": "django-import-data-0.2.0.tar.gz", "has_sig": false, "md5_digest": "df006eca45f3311620509e8181a44e39", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14483, "upload_time": "2016-02-07T13:34:55", "url": "https://files.pythonhosted.org/packages/43/2f/73f234a9c34f070682ccb8ce900ddb77bca07b575267c74ddf86e34d810f/django-import-data-0.2.0.tar.gz" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "b53d391c4d7898dc3d117fa05fc38cc9", "sha256": "04f7ea3827de88542bb0ed6409cf691866f143e499e45be0ce85b340c7785685" }, "downloads": -1, "filename": "django-import-data-0.2.1.tar.gz", "has_sig": false, "md5_digest": "b53d391c4d7898dc3d117fa05fc38cc9", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14482, "upload_time": "2016-02-07T13:43:24", "url": "https://files.pythonhosted.org/packages/ed/b4/eaacac4c960064dbc54d1440821f19c1ef0d24c2ed2bb365ae2ba373f235/django-import-data-0.2.1.tar.gz" } ], "0.2.2": [ { "comment_text": "", "digests": { "md5": "3b26c3caa0589c46606253bb4a711b77", "sha256": "51122ace0be37b975bf37adebaf75c538e98f6601277c709ab4e517070f04758" }, "downloads": -1, "filename": "django_import_data-0.2.2-py2-none-any.whl", "has_sig": false, "md5_digest": "3b26c3caa0589c46606253bb4a711b77", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 16600, "upload_time": "2016-02-07T15:02:59", "url": "https://files.pythonhosted.org/packages/be/a4/413ba68477dd8d97bf6056fff5227df87297b3aea8d032d3f89096c49a7f/django_import_data-0.2.2-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "f694584f060f0d8bcb3622e5caaa6613", "sha256": "cc8c408498515c8560c16585119e000967363bc88a133b705b3cbb42370cf015" }, "downloads": -1, "filename": "django-import-data-0.2.2.tar.gz", "has_sig": false, "md5_digest": "f694584f060f0d8bcb3622e5caaa6613", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14714, "upload_time": "2016-02-07T13:47:16", "url": "https://files.pythonhosted.org/packages/d5/c1/875fca0938953a82bab2a78862794bd3d174ea998f712e77edd26cfc6bdc/django-import-data-0.2.2.tar.gz" } ], "0.3.0": [ { "comment_text": "", "digests": { "md5": "6a9373ec2391813d1230e580505f79e8", "sha256": "5e55321bd5e431569d4da53ff96cf7c735da00d67818bc41190136101ced472e" }, "downloads": -1, "filename": "django_import_data-0.3.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "6a9373ec2391813d1230e580505f79e8", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 16608, "upload_time": "2016-02-07T16:26:22", "url": "https://files.pythonhosted.org/packages/e6/fc/1469e74ba5090f607f70995e316ca514a23a395d33c459308cec41370320/django_import_data-0.3.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "d310f2b197a2174842c88a1fae9be1e2", "sha256": "ca863f137583f143f1c2fe19292087218fd75148540540f83e902a4b94e989f9" }, "downloads": -1, "filename": "django-import-data-0.3.0.tar.gz", "has_sig": false, "md5_digest": "d310f2b197a2174842c88a1fae9be1e2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14676, "upload_time": "2016-02-07T16:26:32", "url": "https://files.pythonhosted.org/packages/fb/43/0ec209ab69cffcd9940d405ae66a61dec50d1d13f86d9587480b53e6edb9/django-import-data-0.3.0.tar.gz" } ], "0.3.1": [ { "comment_text": "", "digests": { "md5": "f4c37018d8d9f1aa54febde566dde3e7", "sha256": "0d5c60602f723c5034d4f8cdf741a1529cbe507d12d233516b70190d26542841" }, "downloads": -1, "filename": "django_import_data-0.3.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "f4c37018d8d9f1aa54febde566dde3e7", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 16906, "upload_time": "2016-02-07T16:40:37", "url": "https://files.pythonhosted.org/packages/3d/72/31f22b73af242c4fac69b3543827c495dc9f9424f5280a846e2311809dd3/django_import_data-0.3.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "3f344d0ec3cac38b6fad6f3fc02ef69c", "sha256": "b0a6bb2ee39531d23bf83c57e83e1ea55b262d0a6eada18d78fc977b23223a52" }, "downloads": -1, "filename": "django-import-data-0.3.1.tar.gz", "has_sig": false, "md5_digest": "3f344d0ec3cac38b6fad6f3fc02ef69c", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15011, "upload_time": "2016-02-07T16:40:45", "url": "https://files.pythonhosted.org/packages/3e/10/8e50ac7928e7fe14a4b750a994eda591f89eee7b1dcd8ec575eeddc07679/django-import-data-0.3.1.tar.gz" } ], "0.4.1": [ { "comment_text": "", "digests": { "md5": "7c2d72b776182e8bc3b773d3dd6d9ae7", "sha256": "9556243b9f6f3d50b6c54af816deef6feb08dae4b5a134874e599f727930bfb4" }, "downloads": -1, "filename": "django_import_data-0.4.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "7c2d72b776182e8bc3b773d3dd6d9ae7", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 58084, "upload_time": "2016-02-08T00:21:54", "url": "https://files.pythonhosted.org/packages/0f/02/452b7d4587c6d984720ad43e529d94cf3c5d4df76c5621682e13f604fdd8/django_import_data-0.4.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "af86c99ac6f934dbc7a2a38c49dc3929", "sha256": "03c28af0089adbcb59cdfb81e4ae8a31576bcf2fbb310bbd8ca69f96942b6405" }, "downloads": -1, "filename": "django-import-data-0.4.1.tar.gz", "has_sig": false, "md5_digest": "af86c99ac6f934dbc7a2a38c49dc3929", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 54647, "upload_time": "2016-02-08T00:22:02", "url": "https://files.pythonhosted.org/packages/84/c2/c308ad450baf261fa4968e2ca2c940a40fc656b5abf262d94492a6553f27/django-import-data-0.4.1.tar.gz" } ], "0.4.2": [ { "comment_text": "", "digests": { "md5": "cc04be9da3eed9debbe9bdc34f2ebea9", "sha256": "95a7f5b3b85748596ae6505db805209edeccfd0a216833c19f43b10b0b2ee2ec" }, "downloads": -1, "filename": "django_import_data-0.4.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "cc04be9da3eed9debbe9bdc34f2ebea9", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 67648, "upload_time": "2016-02-08T04:21:10", "url": "https://files.pythonhosted.org/packages/63/a6/f09d0bda43763eba4143414b1ca1d09dc22e99d69ba758c9792f843dc177/django_import_data-0.4.2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "1c07c4964b0b17a6f467db6354cf8087", "sha256": "2845adda59f4484d5d19caaa9fd9aa62bbcec0e28c3f4930f1c54dc9cef58f2f" }, "downloads": -1, "filename": "django-import-data-0.4.2.tar.gz", "has_sig": false, "md5_digest": "1c07c4964b0b17a6f467db6354cf8087", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 64201, "upload_time": "2016-02-08T04:21:15", "url": "https://files.pythonhosted.org/packages/0e/66/ada8c77936d4d26c54f45a9577b56357bfa20fd8042a798cb1c3284c8791/django-import-data-0.4.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "cc04be9da3eed9debbe9bdc34f2ebea9", "sha256": "95a7f5b3b85748596ae6505db805209edeccfd0a216833c19f43b10b0b2ee2ec" }, "downloads": -1, "filename": "django_import_data-0.4.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "cc04be9da3eed9debbe9bdc34f2ebea9", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 67648, "upload_time": "2016-02-08T04:21:10", "url": "https://files.pythonhosted.org/packages/63/a6/f09d0bda43763eba4143414b1ca1d09dc22e99d69ba758c9792f843dc177/django_import_data-0.4.2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "1c07c4964b0b17a6f467db6354cf8087", "sha256": "2845adda59f4484d5d19caaa9fd9aa62bbcec0e28c3f4930f1c54dc9cef58f2f" }, "downloads": -1, "filename": "django-import-data-0.4.2.tar.gz", "has_sig": false, "md5_digest": "1c07c4964b0b17a6f467db6354cf8087", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 64201, "upload_time": "2016-02-08T04:21:15", "url": "https://files.pythonhosted.org/packages/0e/66/ada8c77936d4d26c54f45a9577b56357bfa20fd8042a798cb1c3284c8791/django-import-data-0.4.2.tar.gz" } ] }