{ "info": { "author": "", "author_email": "", "bugtrack_url": null, "classifiers": [], "description": "# OpenNMT-py: Open-Source Neural Machine Translation\n\n[![Build Status](https://travis-ci.org/OpenNMT/OpenNMT-py.svg?branch=master)](https://travis-ci.org/OpenNMT/OpenNMT-py)\n[![Run on FH](https://img.shields.io/badge/Run%20on-FloydHub-blue.svg)](https://floydhub.com/run?template=https://github.com/OpenNMT/OpenNMT-py)\n\nThis is a [Pytorch](https://github.com/pytorch/pytorch)\nport of [OpenNMT](https://github.com/OpenNMT/OpenNMT),\nan open-source (MIT) neural machine translation system. It is designed to be research friendly to try out new ideas in translation, summary, image-to-text, morphology, and many other domains. Some companies have proven the code to be production ready.\n\nWe love contributions. Please consult the Issues page for any [Contributions Welcome](https://github.com/OpenNMT/OpenNMT-py/issues?q=is%3Aissue+is%3Aopen+label%3A%22contributions+welcome%22) tagged post. \n\n
\n\nBefore raising an issue, make sure you read the requirements and the documentation examples.\n\nUnless there is a bug, please use the [Forum](http://forum.opennmt.net) or [Gitter](https://gitter.im/OpenNMT/OpenNMT-py) to ask questions.\n\n\nTable of Contents\n=================\n * [Full Documentation](http://opennmt.net/OpenNMT-py/)\n * [Requirements](#requirements)\n * [Features](#features)\n * [Quickstart](#quickstart)\n * [Run on FloydHub](#run-on-floydhub)\n * [Acknowledgements](#acknowledgements)\n * [Citation](#citation)\n\n## Requirements\n\nInstall `OpenNMT-py` from `pip`:\n```bash\npip install OpenNMT-py\n```\n\nor from the sources:\n```bash\ngit clone https://github.com/OpenNMT/OpenNMT-py.git\ncd OpenNMT-py\npython setup.py install\n```\n\nNote: If you have MemoryError in the install try to use `pip` with `--no-cache-dir`.\n\n*(Optionnal)* some advanced features (e.g. working audio, image or pretrained models) requires extra packages, you can install it with:\n```bash\npip install -r requirements.opt.txt\n```\n\nNote:\n\n- some features require Python 3.5 and after (eg: Distributed multigpu, entmax)\n- we currently only support PyTorch 1.2 (should work with 1.1)\n\n## Features\n\n- [Seq2Seq models (encoder-decoder) with multiple RNN cells (lstm/gru) and attention (dotprod/mlp) types](http://opennmt.net/OpenNMT-py/options/train.html#model-encoder-decoder)\n- [Transformer models\"](http://opennmt.net/OpenNMT-py/FAQ.html#how-do-i-use-the-transformer-model)\n- [Copy and Coverage Attention](http://opennmt.net/OpenNMT-py/options/train.html#model-attention)\n- [Pretrained Embeddings](http://opennmt.net/OpenNMT-py/FAQ.html#how-do-i-use-pretrained-embeddings-e-g-glove)\n- [Source word features](http://opennmt.net/OpenNMT-py/options/train.html#model-embeddings)\n- [Image-to-text processing](http://opennmt.net/OpenNMT-py/im2text.html)\n- [Speech-to-text processing](http://opennmt.net/OpenNMT-py/speech2text.html)\n- [TensorBoard logging](http://opennmt.net/OpenNMT-py/options/train.html#logging)\n- [Multi-GPU training](http://opennmt.net/OpenNMT-py/FAQ.html##do-you-support-multi-gpu)\n- [Data preprocessing](http://opennmt.net/OpenNMT-py/options/preprocess.html)\n- [Inference (translation) with batching and beam search](http://opennmt.net/OpenNMT-py/options/translate.html)\n- Inference time loss functions.\n- [Conv2Conv convolution model]\n- SRU \"RNNs faster than CNN\" paper\n- Mixed-precision training with [APEX](https://github.com/NVIDIA/apex), optimized on [Tensor Cores](https://developer.nvidia.com/tensor-cores)\n\n## Quickstart\n\n[Full Documentation](http://opennmt.net/OpenNMT-py/)\n\n\n### Step 1: Preprocess the data\n\n```bash\nonmt_preprocess -train_src data/src-train.txt -train_tgt data/tgt-train.txt -valid_src data/src-val.txt -valid_tgt data/tgt-val.txt -save_data data/demo\n```\n\nWe will be working with some example data in `data/` folder.\n\nThe data consists of parallel source (`src`) and target (`tgt`) data containing one sentence per line with tokens separated by a space:\n\n* `src-train.txt`\n* `tgt-train.txt`\n* `src-val.txt`\n* `tgt-val.txt`\n\nValidation files are required and used to evaluate the convergence of the training. It usually contains no more than 5000 sentences.\n\n\nAfter running the preprocessing, the following files are generated:\n\n* `demo.train.pt`: serialized PyTorch file containing training data\n* `demo.valid.pt`: serialized PyTorch file containing validation data\n* `demo.vocab.pt`: serialized PyTorch file containing vocabulary data\n\n\nInternally the system never touches the words themselves, but uses these indices.\n\n### Step 2: Train the model\n\n```bash\nonmt_train -data data/demo -save_model demo-model\n```\n\nThe main train command is quite simple. Minimally it takes a data file\nand a save file. This will run the default model, which consists of a\n2-layer LSTM with 500 hidden units on both the encoder/decoder.\nIf you want to train on GPU, you need to set, as an example:\nCUDA_VISIBLE_DEVICES=1,3\n`-world_size 2 -gpu_ranks 0 1` to use (say) GPU 1 and 3 on this node only.\nTo know more about distributed training on single or multi nodes, read the FAQ section.\n\n### Step 3: Translate\n\n```bash\nonmt_translate -model demo-model_acc_XX.XX_ppl_XXX.XX_eX.pt -src data/src-test.txt -output pred.txt -replace_unk -verbose\n```\n\nNow you have a model which you can use to predict on new data. We do this by running beam search. This will output predictions into `pred.txt`.\n\n!!! note \"Note\"\n The predictions are going to be quite terrible, as the demo dataset is small. Try running on some larger datasets! For example you can download millions of parallel sentences for [translation](http://www.statmt.org/wmt16/translation-task.html) or [summarization](https://github.com/harvardnlp/sent-summary).\n\n## Alternative: Run on FloydHub\n\n[![Run on FloydHub](https://static.floydhub.com/button/button.svg)](https://floydhub.com/run?template=https://github.com/OpenNMT/OpenNMT-py)\n\nClick this button to open a Workspace on [FloydHub](https://www.floydhub.com/?utm_medium=readme&utm_source=opennmt-py&utm_campaign=jul_2018) for training/testing your code.\n\n\n## Pretrained embeddings (e.g. GloVe)\n\nPlease see the FAQ: [How to use GloVe pre-trained embeddings in OpenNMT-py](http://opennmt.net/OpenNMT-py/FAQ.html#how-do-i-use-pretrained-embeddings-e-g-glove)\n\n## Pretrained Models\n\nThe following pretrained models can be downloaded and used with translate.py.\n\nhttp://opennmt.net/Models-py/\n\n## Acknowledgements\n\nOpenNMT-py is run as a collaborative open-source project.\nThe original code was written by [Adam Lerer](http://github.com/adamlerer) (NYC) to reproduce OpenNMT-Lua using Pytorch.\n\nMajor contributors are:\n[Sasha Rush](https://github.com/srush) (Cambridge, MA)\n[Vincent Nguyen](https://github.com/vince62s) (Ubiqus)\n[Ben Peters](http://github.com/bpopeters) (Lisbon)\n[Sebastian Gehrmann](https://github.com/sebastianGehrmann) (Harvard NLP)\n[Yuntian Deng](https://github.com/da03) (Harvard NLP)\n[Guillaume Klein](https://github.com/guillaumekln) (Systran)\n[Paul Tardy](https://github.com/pltrdy) (Ubiqus / Lium)\n[Fran\u00e7ois Hernandez](https://github.com/francoishernandez) (Ubiqus)\n[Jianyu Zhan](http://github.com/jianyuzhan) (Shanghai)\n[Dylan Flaute](http://github.com/flauted (University of Dayton)\nand more !\n\nOpentNMT-py belongs to the OpenNMT project along with OpenNMT-Lua and OpenNMT-tf.\n\n## Citation\n\n[OpenNMT: Neural Machine Translation Toolkit](https://arxiv.org/pdf/1805.11462)\n\n[OpenNMT technical report](https://doi.org/10.18653/v1/P17-4012)\n\n```\n@inproceedings{opennmt,\n author = {Guillaume Klein and\n Yoon Kim and\n Yuntian Deng and\n Jean Senellart and\n Alexander M. Rush},\n title = {Open{NMT}: Open-Source Toolkit for Neural Machine Translation},\n booktitle = {Proc. ACL},\n year = {2017},\n url = {https://doi.org/10.18653/v1/P17-4012},\n doi = {10.18653/v1/P17-4012}\n}\n```\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "", "keywords": "", "license": "", "maintainer": "", "maintainer_email": "", "name": "OpenNMT-py", "package_url": "https://pypi.org/project/OpenNMT-py/", "platform": "", "project_url": "https://pypi.org/project/OpenNMT-py/", "project_urls": { "Documentation": "http://opennmt.net/OpenNMT-py/", "Forum": "http://forum.opennmt.net/", "Gitter": "https://gitter.im/OpenNMT/OpenNMT-py", "Source": "https://github.com/OpenNMT/OpenNMT-py/" }, "release_url": "https://pypi.org/project/OpenNMT-py/1.0.0rc2/", "requires_dist": [ "six", "tqdm (~=4.30.0)", "torch (>=1.2)", "torchtext (==0.4.0)", "future", "configargparse", "tensorboard (>=1.14)", "flask", "pyonmttok (==1.*) ; platform_system == \"Linux\"" ], "requires_python": "", "summary": "A python implementation of OpenNMT", "version": "1.0.0rc2" }, "last_serial": 5971391, "releases": { "1.0.0rc1": [ { "comment_text": "", "digests": { "md5": "c5619c98d66317c03d3a5f9b2ca419ec", "sha256": "31f1488c23238911da2dc0c7d31faa8c296303d9130c81d16e15b9f8a3d7d152" }, "downloads": -1, "filename": "OpenNMT_py-1.0.0rc1-py3-none-any.whl", "has_sig": false, "md5_digest": "c5619c98d66317c03d3a5f9b2ca419ec", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 179658, "upload_time": "2019-10-02T08:16:45", "url": "https://files.pythonhosted.org/packages/d5/ab/0f5223cb441d33cc807fd021c60266d27054be9599d3464c1a016063381c/OpenNMT_py-1.0.0rc1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "679b6826d7ae91aef3eed6338aa7837b", "sha256": "1d73c746a171364f93d60cd02f328ef1046039ff8566cbafb8344ba610c2fd4a" }, "downloads": -1, "filename": "OpenNMT-py-1.0.0rc1.tar.gz", "has_sig": false, "md5_digest": "679b6826d7ae91aef3eed6338aa7837b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 143879, "upload_time": "2019-10-02T08:16:49", "url": "https://files.pythonhosted.org/packages/46/65/6dc4449f0218e43e9f06eebad35f29aa36d04108192ecf270b93858bd6f4/OpenNMT-py-1.0.0rc1.tar.gz" } ], "1.0.0rc2": [ { "comment_text": "", "digests": { "md5": "b9fae2f86075e60cc3b544eaaf251c36", "sha256": "ae712a8f348eaf499f0e85dde527782d82d704f8b4d8629e4764c0eddc0cd8da" }, "downloads": -1, "filename": "OpenNMT_py-1.0.0rc2-py3-none-any.whl", "has_sig": false, "md5_digest": "b9fae2f86075e60cc3b544eaaf251c36", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 179704, "upload_time": "2019-10-14T12:52:16", "url": "https://files.pythonhosted.org/packages/13/2c/48bf5505f676acc2b1cf2735c4f4efe41c40f5f562106795158db2a5b08b/OpenNMT_py-1.0.0rc2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "304b7c63d0a8297f02c3f8e82ac58ecc", "sha256": "4205e964ab6b88e6de611d582adef7d43d5b60e0dc0b5078f723b726b4068d98" }, "downloads": -1, "filename": "OpenNMT-py-1.0.0rc2.tar.gz", "has_sig": false, "md5_digest": "304b7c63d0a8297f02c3f8e82ac58ecc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 143888, "upload_time": "2019-10-14T12:52:18", "url": "https://files.pythonhosted.org/packages/9c/ec/7da121157f09be02c9fca4ada53da47c2954d8f5a9df46a2d4360fad0c3e/OpenNMT-py-1.0.0rc2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "b9fae2f86075e60cc3b544eaaf251c36", "sha256": "ae712a8f348eaf499f0e85dde527782d82d704f8b4d8629e4764c0eddc0cd8da" }, "downloads": -1, "filename": "OpenNMT_py-1.0.0rc2-py3-none-any.whl", "has_sig": false, "md5_digest": "b9fae2f86075e60cc3b544eaaf251c36", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 179704, "upload_time": "2019-10-14T12:52:16", "url": "https://files.pythonhosted.org/packages/13/2c/48bf5505f676acc2b1cf2735c4f4efe41c40f5f562106795158db2a5b08b/OpenNMT_py-1.0.0rc2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "304b7c63d0a8297f02c3f8e82ac58ecc", "sha256": "4205e964ab6b88e6de611d582adef7d43d5b60e0dc0b5078f723b726b4068d98" }, "downloads": -1, "filename": "OpenNMT-py-1.0.0rc2.tar.gz", "has_sig": false, "md5_digest": "304b7c63d0a8297f02c3f8e82ac58ecc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 143888, "upload_time": "2019-10-14T12:52:18", "url": "https://files.pythonhosted.org/packages/9c/ec/7da121157f09be02c9fca4ada53da47c2954d8f5a9df46a2d4360fad0c3e/OpenNMT-py-1.0.0rc2.tar.gz" } ] }