{ "info": { "author": "bung87", "author_email": "", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.2", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6" ], "description": "whatlangid\n==========\n\n`Build Status `__\n`PyPI `__\n\nThis project is build on top of\n`whatthelang `__ and\n`langid `__\n\nThe\ndependency(\\ `pyfasttext `__) of\n**whatthelang** is not facebook official release.\n\nThis project depends on the official release\nhttps://github.com/facebookresearch/fastText.git\n\nThe **fastText** for now is not release on pypi,so you need\n``--process-dependency-links`` arg when using pip.\n\nWhy this project exist?\n=======================\n\nsee `issue_lang.py `__\n\nDependencies\n------------\n\nThe dependencies can be installed using the requirements.txt file:\n\n.. code:: bash\n\n $ pip install -r requirements.txt\n\nInstall\n-------\n\nfrom github\n\n.. code:: bash\n\n $ pip install --extra-index-url=https://test.pypi.org/simple/ git+https://github.com/bung87/whatlangid \n\nfrom pypi\n\n.. code:: bash\n\n $ pip install --extra-index-url=https://test.pypi.org/simple/ whatlangid \n\nBasic Usage\n-----------\n\nPredicting Language using ``whatlangid``\n\n.. code:: python\n\n >>> from whatlangid import WhatLangId\n >>> wtl = WhatLangId()\n >>> wtl.predict_lang(\"Mother\")\n 'en'\n >>> wtl.predict_lang(\"\u0ba4\u0bbe\u0baf\u0bcd\")\n 'ta'\n >>> wtl.predict_lang(\"\u0d05\u0d2e\u0d4d\u0d2e\")\n 'ml'\n >>> wtl.predict_lang(\"\u092a\u093f\u0924\u093e\")\n 'hi'\n >>> wtl.predict_pro([\"English sentence\", \"\u0d05\u0d2e\u0d4d\u0d2e\"])\n [('en', 0.8848170638084412), ('ml', 0.9535570740699768)]\n\nBatch Prediction is also supported\n\n.. code:: python\n\n >>>wtl.predict_lang([\"\u0d05\u0d2e\u0d4d\u0d2e\",\"\u092a\u093f\u0924\u093e\",\"teacher\"])\n ['ml','hi','en']\n\nAdvanced usage\n--------------\n\n``wtl = WhatLangId(custom_model=abs_path)``\n\nuse bin version model which is faster and slightly more accurate, but\nhas a file size of 126MB\n\n``python -m whatlangid.use_bin``\n\nSupported Languages\n-------------------\n\nSupports 176 languages . The ISO codes for the corresponding languages\nare as below.\n\n::\n\n af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs bxr ca cbk\n ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es et eu fa fi fr frr fy ga\n gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia id ie ilo io is it ja jbo jv ka kk km\n kn ko krc ku kv kw ky la lb lez li lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms\n mt mwl my myv mzn nah nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu\n rm ro ru rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl tr\n tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh\n\nModel Training Details\n----------------------\n\nQuantized model built using Fasttext. More details present in the\nfasttext `blog `__\n\nReference\n---------\n\n``WhatLangId`` is powered by ``FastText`` and ``langid``\n\nEnriching Word Vectors with Subword Information\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n[1] P. Bojanowski*, E. Grave*, A. Joulin, T. Mikolov, `Enriching Word\nVectors with Subword Information `__\n\n::\n\n @article{bojanowski2016enriching,\n title={Enriching Word Vectors with Subword Information},\n author={Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas},\n journal={arXiv preprint arXiv:1607.04606},\n year={2016}\n }\n\nBag of Tricks for Efficient Text Classification\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n[2] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, `Bag of Tricks for\nEfficient Text Classification `__\n\n::\n\n @article{joulin2016bag,\n title={Bag of Tricks for Efficient Text Classification},\n author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},\n journal={arXiv preprint arXiv:1607.01759},\n year={2016}\n }\n\nFastText.zip: Compressing text classification models\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n[3] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. J\u00e9gou, T. Mikolov,\n`FastText.zip: Compressing text classification\nmodels `__\n\n::\n\n @article{joulin2016fasttext,\n title={FastText.zip: Compressing text classification models},\n author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Douze, Matthijs and J{\\'e}gou, H{\\'e}rve and Mikolov, Tomas},\n journal={arXiv preprint arXiv:1612.03651},\n year={2016}\n }", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/bung87/whatlangid", "keywords": "language detection library", "license": "Apache License, Version 2.0", "maintainer": "", "maintainer_email": "", "name": "whatlangid", "package_url": "https://pypi.org/project/whatlangid/", "platform": "", "project_url": "https://pypi.org/project/whatlangid/", "project_urls": { "Homepage": "https://github.com/bung87/whatlangid" }, "release_url": "https://pypi.org/project/whatlangid/1.0.9/", "requires_dist": null, "requires_python": ">=3", "summary": "Lightning Fast Language Prediction powered by FastText and langid.", "version": "1.0.9" }, "last_serial": 5224301, "releases": { "1.0.6": [ { "comment_text": "", "digests": { "md5": "4f7c64e05f977fb413dfcf27e3510bbe", "sha256": "ec0ad84ca4ab9b5d6866f8f139e95f47ef3782916e746c6e1f7eacc7e29cf9da" }, "downloads": -1, "filename": "whatlangid-1.0.6.tar.gz", "has_sig": false, "md5_digest": "4f7c64e05f977fb413dfcf27e3510bbe", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3", "size": 790135, "upload_time": "2018-05-30T15:16:38", "url": "https://files.pythonhosted.org/packages/bd/cc/7da9423a0a979628e56ebd7da02e99635e410be23d6c4adfa42200a9b998/whatlangid-1.0.6.tar.gz" } ], "1.0.7": [ { "comment_text": "", "digests": { "md5": "382acce63c351d0cdb31c02018e70cd9", "sha256": "bfae26202597d84e5c996d06cd26a129d27461c2f24b0e68fd0dae88a242f7c9" }, "downloads": -1, "filename": "whatlangid-1.0.7.tar.gz", "has_sig": false, "md5_digest": "382acce63c351d0cdb31c02018e70cd9", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3", "size": 791079, "upload_time": "2018-06-01T10:17:09", "url": "https://files.pythonhosted.org/packages/5e/f5/cbc1d8cd0be07c7e369b207e1379c298bf8a2cd1230b1566ee77372ac01b/whatlangid-1.0.7.tar.gz" } ], "1.0.8": [ { "comment_text": "", "digests": { "md5": "c2da9eee7c311e1ea5a06138e35e309f", "sha256": "615d96810d12feccff96efa9d3ed5ff4a5dca82188eb1b3cf037090d28ecfa05" }, "downloads": -1, "filename": "whatlangid-1.0.8.tar.gz", "has_sig": false, "md5_digest": "c2da9eee7c311e1ea5a06138e35e309f", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3", "size": 791092, "upload_time": "2019-02-27T23:43:11", "url": "https://files.pythonhosted.org/packages/6f/23/8a66cf652f4a1294b766d508acd8450ff790d710f2d56413da97b5afa362/whatlangid-1.0.8.tar.gz" } ], "1.0.9": [ { "comment_text": "", "digests": { "md5": "0ff1d6074689d583c3c7fe7cb64aa512", "sha256": "92536df9460fa8a6128fcbd70f0d5427bfab8a452957483d5ff1344deb8491db" }, "downloads": -1, "filename": "whatlangid-1.0.9.tar.gz", "has_sig": false, "md5_digest": "0ff1d6074689d583c3c7fe7cb64aa512", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3", "size": 790734, "upload_time": "2019-05-04T00:35:16", "url": "https://files.pythonhosted.org/packages/db/e4/4c3a1cbb1392663f16f3b761fb8bd61ec17727b04e81e2ed842c78dd6b03/whatlangid-1.0.9.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "0ff1d6074689d583c3c7fe7cb64aa512", "sha256": "92536df9460fa8a6128fcbd70f0d5427bfab8a452957483d5ff1344deb8491db" }, "downloads": -1, "filename": "whatlangid-1.0.9.tar.gz", "has_sig": false, "md5_digest": "0ff1d6074689d583c3c7fe7cb64aa512", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3", "size": 790734, "upload_time": "2019-05-04T00:35:16", "url": "https://files.pythonhosted.org/packages/db/e4/4c3a1cbb1392663f16f3b761fb8bd61ec17727b04e81e2ed842c78dd6b03/whatlangid-1.0.9.tar.gz" } ] }