\n ![](\"https://abstrusegoose.com/strips/muggle_problems.png\")
\n
![](\"doc/cat-delft-small.jpg\")
\n\n[](https://travis-ci.org/kermitt2/delft)\n[](https://badge.fury.io/py/delft)\n[](http://www.apache.org/licenses/LICENSE-2.0.html)\n\n\n\n# DeLFT\n\n__Work in progress !__\n\n__DeLFT__ (**De**ep **L**earning **F**ramework for **T**ext) is a Keras framework for text processing, covering sequence labelling (e.g. named entity tagging) and text classification (e.g. comment classification). This library re-implements standard state-of-the-art Deep Learning architectures.\n\nFrom the observation that most of the open source implementations using Keras are toy examples, our motivation is to develop a framework that can be efficient, scalable and more usable in a production environment (with all the known limitations of Python of course for this purpose). The benefits of DeLFT are:\n\n* Re-implement a variety of state-of-the-art deep learning architectures for both sequence labelling and text classification problems, including the usage of the recent [ELMo](https://allennlp.org/elmo) and [BERT](https://github.com/google-research/bert) contextualised embeddings, which can all be used within the same environment. For instance, this allows to reproduce under similar conditions the performance of all recent NER systems, and even improve most of them.\n\n* Reduce model size, in particular by removing word embeddings from them. For instance, the model for the toxic comment classifier went down from a size of 230 MB with embeddings to 1.8 MB. In practice the size of all the models of DeLFT is less than 2 MB, except for Ontonotes 5.0 NER model which is 4.7 MB.\n\n* Use dynamic data generator so that the training data do not need to stand completely in memory.\n\n* Load and manage efficiently an unlimited volume of pre-trained embeddings: instead of loading pre-trained embeddings in memory - which is horribly slow in Python and limits the number of embeddings to be used simultaneously - the pre-trained embeddings are compiled the first time they are accessed and stored efficiently in a LMDB database. This permits to have the pre-trained embeddings immediately \"warm\" (no load time), to free memory and to use any number of embeddings with a very negligible impact on runtime when using SSD.\n\nThe medium term goal is then to provide good performance (accuracy, runtime, compactness) models also to productions stack such as Java/Scala and C++. A native Java integration of these deep learning models has been realized in [GROBID](https://github.com/kermitt2/grobid) via [JEP](https://github.com/ninia/jep).\n\nDeLFT has been tested with python 3.5, Keras 2.1 and Tensorflow 1.7+ as backend. At this stage, we do not guarantee that DeLFT will run with other different versions of these libraries or other Keras backend versions. As always, GPU(s) are required for decent training time: a GeForce GTX 1050 Ti for instance is absolutely OK without ELMo contextual embeddings. Using ELMo or BERT Base model is fine with a GeForce GTX 1080 Ti.\n\n## Install\n\nGet the github repo:\n\n```sh\ngit clone https://github.com/kermitt2/delft\ncd delft\n```\nIt is advised to setup first a virtual environment to avoid falling into one of these gloomy python dependency marshlands:\n\n```sh\nvirtualenv --system-site-packages -p python3 env\nsource env/bin/activate\n```\n\nInstall the dependencies:\n\n```sh\npip3 install -r requirements.txt\n```\n\nDeLFT uses tensorflow 1.7 as backend, and will exploit your available GPU with the condition that CUDA (>=8.0) is properly installed. \n\nYou need then to download some pre-trained word embeddings and notify their path into the embedding registry. We suggest for exploiting the provided models:\n\n* _glove Common Crawl_ (2.2M vocab., cased, 300 dim. vectors): [glove-840B](http://nlp.stanford.edu/data/glove.840B.300d.zip)\n\n* _fasttext Common Crawl_ (2M vocab., cased, 300 dim. vectors): [fasttext-crawl](https://s3-us-west-1.amazonaws.com/fasttext-vectors/crawl-300d-2M.vec.zip)\n\n* _word2vec GoogleNews_ (3M vocab., cased, 300 dim. vectors): [word2vec](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing)\n\n* _fasttext_wiki_fr_ (1.1M, NOT cased, 300 dim. vectors) for French: [wiki.fr](https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.fr.vec)\n\n* _ELMo_ trained on 5.5B word corpus (will produce 1024 dim. vectors) for English: [options](https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway_5.5B/elmo_2x4096_512_2048cnn_2xhighway_5.5B_options.json) and [weights](https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway_5.5B/elmo_2x4096_512_2048cnn_2xhighway_5.5B_weights.hdf5)\n\n* _BERT_ for English, we are using BERT-Base, Cased, 12-layer, 768-hidden, 12-heads , 110M parameters: available [here](https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip)\n\n\nThen edit the file `embedding-registry.json` and modify the value for `path` according to the path where you have saved the corresponding embeddings. The embedding files must be unzipped.\n\n```json\n{\n \"embeddings\": [\n {\n \"name\": \"glove-840B\",\n \"path\": \"/PATH/TO/THE/UNZIPPED/EMBEDDINGS/FILE/glove.840B.300d.txt\",\n \"type\": \"glove\",\n \"format\": \"vec\",\n \"lang\": \"en\",\n \"item\": \"word\"\n },\n ...\n ]\n}\n\n```\n\nYou're ready to use DeLFT.\n\n## Management of embeddings\n\nThe first time DeLFT starts and accesses pre-trained embeddings, these embeddings are serialised and stored in a LMDB database, a very efficient embedded database using memory page (already used in the Machine Learning world by Caffe and Torch for managing large training data). The next time these embeddings will be accessed, they will be immediately available.\n\nOur approach solves the bottleneck problem pointed for instance [here](https://spenai.org/bravepineapple/faster_em/) in a much better way than quantising+compression or pruning. After being compiled and stored at the first access, any volume of embeddings vectors can be used immediately without any loading, with a negligible usage of memory, without any accuracy loss and with a negligible impact on runtime when using SSD. In practice, we can exploit for instance embeddings for dozen languages simultaneously, without any memory and runtime issues - a requirement for any ambitious industrial deployment of a neural NLP system. \n\nFor instance, in a traditional approach `glove-840B` takes around 2 minutes to load and 4GB in memory. Managed with LMDB, after a first load time of around 4 minutes, `glove-840B` can be accessed immediately and takes only a couple MB in memory, for an impact on runtime negligible (around 1% slower) for any further command line calls.\n\nBy default, the LMDB databases are stored under the subdirectory `data/db`. The size of a database is roughly equivalent to the size of the original uncompressed embeddings file. To modify this path, edit the file `embedding-registry.json` and change the value of the attribute `embedding-lmdb-path`.\n\nTo get FastText .bin format support please uncomment the package `fasttextmirror==0.8.22` in `requirements.txt` or `requirements-gpu.txt` according to your system's configuration. Please note that the **.bin format is not supported on Windows platforms**. Installing the FastText .bin format support introduces the following additional dependencies:\n\n* (gcc-4.8 or newer) or (clang-3.3 or newer)\n* [Python](https://www.python.org/) version 2.7 or >=3.4\n* [pybind11](https://github.com/pybind/pybind11)\n\nWhile FastText .bin format are supported by DeLFT (including using ngrams for OOV words), this format will be loaded entirely in memory and does not take advantage of our memory-efficient management of embeddings.\n\n> I have plenty of memory on my machine, I don't care about load time because I need to grab a coffee, I only process one language at the time, so I am not interested in taking advantage of the LMDB emebedding management !\n\nOk, ok, then set the `embedding-lmdb-path` value to `\"None\"` in the file `embedding-registry.json`, the embeddings will be loaded in memory as immutable data, like in the usual Keras scripts.\n\n## Sequence Labelling\n\n### Available models\n\n* _BidLSTM-CRF_ with words and characters input following:\n\n [1] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, Chris Dyer. \"Neural Architectures for Named Entity Recognition\". Proceedings of NAACL 2016. https://arxiv.org/abs/1603.01360\n\n* _BidLSTM-CNN_ with words, characters and custom casing features input, see:\n\n [2] Jason P. C. Chiu, Eric Nichols. \"Named Entity Recognition with Bidirectional LSTM-CNNs\". 2016. https://arxiv.org/abs/1511.08308\n\n* _BidLSTM-CNN-CRF_ with words, characters and custom casing features input following:\n\n [3] Xuezhe Ma and Eduard Hovy. \"End-to-end Sequence Labelling via Bi-directional LSTM-CNNs-CRF\". 2016. https://arxiv.org/abs/1603.01354\n\n* _BidGRU-CRF_, similar to: \n\n [4] Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, Russell Power. \"Semi-supervised sequence tagging with bidirectional language models\". 2017. https://arxiv.org/pdf/1705.00108 \n\n* the current state of the art (92.22% F1 on CoNLL2003 NER dataset, averaged over five runs), _BidLSTM-CRF_ with [ELMo](https://allennlp.org/elmo) contextualised embeddings, see:\n\n [5] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer. \"Deep contextualized word representations\". 2018. https://arxiv.org/abs/1802.05365\n\n* Feature extraction to be used as contextual embeddings can also be obtained from BERT, as ELMo alternative, as explained in section 5.4 of: \n\n [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2018. https://arxiv.org/abs/1810.04805\n\nThe addition of BERT transformer architecture, as alternative to the above RNN architectures, is currently work in progress. \n\nNote that all our annotation data for sequence labelling follows the [IOB2](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)) scheme.\n\n### Examples\n\n#### NER\n\n##### Overview\n\nWe have reimplemented in DeLFT the main neural architectures for NER of the last two years and performed a reproducibility analysis of the these systems with comparable evaluation criterias. Unfortunaltely, in publications, systems are usually compared directly with reported results obtained in different settings, which can bias scores by more than 1.0 points and completely invalidate both comparison and interpretation of results. \n\nYou can read more about our reproducibility study of neural NER in this [blog article](http://science-miner.com/a-reproducibility-study-on-neural-ner/).\n\nAll reported scores bellow are __f-score__ for the CoNLL-2003 NER dataset. We report first the f-score averaged over 10 training runs, and second the best f-score over these 10 training runs. All the DeLFT trained models are included in this repository. \n\n| Architecture | Implementation | Glove only (avg / best)| Glove + valid. set (avg / best)| ELMo + Glove (avg / best)| ELMo + Glove + valid. set (avg / best)|\n| --- | --- | --- | --- | --- | --- |\n| BidLSTM-CRF | DeLFT | __90.75__ / __91.35__ | 91.13 / 91.60 | __92.47__ / __92.71__ | __92.69__ / __93.09__ | \n| | [(Lample and al., 2016)](https://arxiv.org/abs/1603.01360) | - / 90.94 | | | | \n| BidLSTM-CNN-CRF | DeLFT | 90.73 / 91.07| 91.01 / 91.26 | 92.30 / 92.57| 92.67 / 93.04 |\n| | [(Ma & Hovy, 2016)](https://arxiv.org/abs/1603.01354) | - / 91.21 | | | |\n| | [(Peters & al. 2018)](https://arxiv.org/abs/1802.05365) | | | 92.22** / - | |\n| BidLSTM-CNN | DeLFT | 89.23 / 89.47 | 89.35 / 89.87 | 91.66 / 92.00 | 92.01 / 92.16 |\n| | [(Chiu & Nichols, 2016)](https://arxiv.org/abs/1511.08308) || __90.88***__ / - | | |\n| BidGRU-CRF | DeLFT | 90.38 / 90.72 | 90.28 / 90.69 | 92.03 / 92.44 | 92.43 / 92.71 |\n| | [(Peters & al. 2017)](https://arxiv.org/abs/1705.00108) | | | | 91.93* / - |\n\n\n\n_*_ reported f-score using Senna word embeddings and not Glove.\n\n** f-score is averaged over 5 training runs. \n\n*** reported f-score with Senna word embeddings (Collobert 50d) averaged over 10 runs, including case features and not including lexical features. DeLFT implementation of the same architecture includes the capitalization features too, but uses the more efficient GloVe 300d embeddings.\n\n\n##### Command Line Interface\n\nDifferent datasets and languages are supported. They can be specified by the command line parameters. The general usage of the CLI is as follow: \n\n```\nusage: nerTagger.py [-h] [--fold-count FOLD_COUNT] [--lang LANG]\n [--dataset-type DATASET_TYPE]\n [--train-with-validation-set]\n [--architecture ARCHITECTURE] [--use-ELMo] [--use-BERT]\n [--data-path DATA_PATH] [--file-in FILE_IN]\n [--file-out FILE_OUT]\n action\n\nNeural Named Entity Recognizers\n\npositional arguments:\n action one of [train, train_eval, eval, tag]\n\noptional arguments:\n -h, --help show this help message and exit\n --fold-count FOLD_COUNT\n number of folds or re-runs to be used when training\n --lang LANG language of the model as ISO 639-1 code\n --dataset-type DATASET_TYPE\n dataset to be used for training the model\n --train-with-validation-set\n Use the validation set for training together with the\n training set\n --architecture ARCHITECTURE\n type of model architecture to be used, one of\n [BidLSTM_CRF, BidLSTM_CNN, BidLSTM_CNN_CRF, BidGRU-\n CRF]\n --use-ELMo Use ELMo contextual embeddings\n --use-BERT Use BERT extracted features (embeddings)\n --data-path DATA_PATH\n path to the corpus of documents for training (only use\n currently with Ontonotes corpus in orginal XML format)\n --file-in FILE_IN path to a text file to annotate\n --file-out FILE_OUT path for outputting the resulting JSON NER anotations\n```\n\nMore explanations and examples are presented in the following sections. \n\n##### CONLL 2003\n\nDeLFT comes with various pre-trained models with the CoNLL-2003 NER dataset.\n\nBy default, the BidLSTM-CRF architecture is used. With this available model, glove-840B word embeddings, and optimisation of hyperparameters, the current f1 score on CoNLL 2003 _testb_ set is __91.35__ (best run over 10 training, using _train_ set for training and _testa_ for validation), as compared to the 90.94 reported in [1], or __90.75__ when averaged over 10 training. Best model f1 score becomes __91.60__ when using both _train_ and _testa_ (validation set) for training (best run over 10 training), as it is done by (Chiu & Nichols, 2016) or some recent works like (Peters and al., 2017). \n\nUsing BidLSTM-CRF model with ELMo embeddings, following [5] and some parameter optimisations and [warm-up](https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md#notes-on-statefulness-and-non-determinism), make the predictions around 30 times slower but improve the f1 score on CoNLL 2003 currently to __92.47__ (averaged over 10 training, __92.71__ for best model, using _train_ set for training and _testa_ for validation), or __92.69__ (averaged over 10 training, __93.09__ best model) when training with the validation set (as in the paper Peters and al., 2017).\n\nFor re-training a model, the CoNLL-2003 NER dataset (`eng.train`, `eng.testa`, `eng.testb`) must be present under `data/sequenceLabelling/CoNLL-2003/` in IOB2 tagging sceheme (look [here](https://github.com/Franck-Dernoncourt/NeuroNER/tree/master/data/conll2003/en) for instance ;) and [here](https://github.com/kermitt2/delft/tree/master/utilities). The CONLL 2003 dataset (English) is the default dataset and English is the default language, but you can also indicate it explicitly as parameter with `--dataset-type conll2003` and specifying explicitly the language `--lang en`.\n\nFor training and evaluating following the traditional approach (training with the train set without validation set, and evaluating on test set), use:\n\n> python3 nerTagger.py --dataset-type conll2003 train_eval\n\nTo use ELMo contextual embeddings, add the parameter `--use-ELMo`. This will slow down considerably (30 times) the first epoch of the training, then the contextual embeddings will be cached and the rest of the training will be similar to usual embeddings in term of training time. Alternatively add `--use-BERT` to use BERT extracted features as contextual embeddings to the RNN architecture. \n\n> python3 nerTagger.py --dataset-type conll2003 --use-ELMo train_eval\n\nSome recent works like (Chiu & Nichols, 2016) and (Peters and al., 2017) also train with the validation set, leading obviously to a better accuracy (still they compare their scores with scores previously reported trained differently, which is arguably a bit unfair - this aspect is mentioned in (Ma & Hovy, 2016)). To train with both train and validation sets, use the parameter `--train-with-validation-set`:\n\n> python3 nerTagger.py --dataset-type conll2003 --train-with-validation-set train_eval\n\nNote that, by default, the BidLSTM-CRF model is used. (Documentation on selecting other models and setting hyperparameters to be included here !)\n\nFor evaluating against CoNLL 2003 testb set with the existing model:\n\n> python3 nerTagger.py --dataset-type conll2003 eval\n\n```text\n Evaluation on test set:\n f1 (micro): 91.35\n precision recall f1-score support\n\n ORG 0.8795 0.9007 0.8899 1661\n PER 0.9647 0.9623 0.9635 1617\n MISC 0.8261 0.8120 0.8190 702\n LOC 0.9260 0.9305 0.9282 1668\n\n avg / total 0.9109 0.9161 0.9135 5648\n\n```\n\nIf the model has been trained also with the validation set (`--train-with-validation-set`), similarly to (Chiu & Nichols, 2016) or (Peters and al., 2017), results are significantly better:\n\n```text\n Evaluation on test set:\n f1 (micro): 91.60\n precision recall f1-score support\n\n LOC 0.9219 0.9418 0.9318 1668\n MISC 0.8277 0.8077 0.8176 702\n PER 0.9594 0.9635 0.9614 1617\n ORG 0.9029 0.8904 0.8966 1661\n\n avg / total 0.9158 0.9163 0.9160 5648\n```\n\nUsing ELMo with the best model obtained over 10 training (not using the validation set for training, only for early stop):\n\n```text\n Evaluation on test set:\n f1 (micro): 92.71\n precision recall f1-score support\n\n PER 0.9787 0.9672 0.9729 1617\n LOC 0.9368 0.9418 0.9393 1668\n MISC 0.8237 0.8319 0.8278 702\n ORG 0.9072 0.9181 0.9126 1661\n\n all (micro avg.) 0.9257 0.9285 0.9271 5648\n\n```\n\nUsing ELMo and training with the validation set gives a f-score of 93.09 (best model), 92.69 averaged over 10 runs (the best model is provided under `data/models/sequenceLabelling/ner-en-conll2003-BidLSTM_CRF/with_validation_set/`).\n\nFor training with all the available data:\n\n> python3 nerTagger.py --dataset-type conll2003 train\n\nTo take into account the strong impact of random seed, you need to train multiple times with the n-folds options. The model will be trained n times with different seed values but with the same sets if the evaluation set is provided. The evaluation will then give the average scores over these n models (against test set) and for the best model which will be saved. For 10 times training for instance, use:\n\n> python3 nerTagger.py --dataset-type conll2003 --fold-count 10 train_eval\n\nAfter training a model, for tagging some text, for instance in a file `data/test/test.ner.en.txt` (), use the command:\n\n> python3 nerTagger.py --dataset-type conll2003 --file-in data/test/test.ner.en.txt tag\n\nNote that, currently, the input text file must contain one sentence per line, so the text must be presegmented into sentences. To obtain the JSON annotations in a text file instead than in the standard output, use the parameter `--file-out`. Predictions work at around 7400 tokens per second for the BidLSTM_CRF architecture with a GeForce GTX 1080 Ti. \n\nThis produces a JSON output with entities, scores and character offsets like this:\n\n```json\n{\n \"runtime\": 0.34,\n \"texts\": [\n {\n \"text\": \"The University of California has found that 40 percent of its students suffer food insecurity. At four state universities in Illinois, that number is 35 percent.\",\n \"entities\": [\n {\n \"text\": \"University of California\",\n \"endOffset\": 32,\n \"score\": 1.0,\n \"class\": \"ORG\",\n \"beginOffset\": 4\n },\n {\n \"text\": \"Illinois\",\n \"endOffset\": 134,\n \"score\": 1.0,\n \"class\": \"LOC\",\n \"beginOffset\": 125\n }\n ]\n },\n {\n \"text\": \"President Obama is not speaking anymore from the White House.\",\n \"entities\": [\n {\n \"text\": \"Obama\",\n \"endOffset\": 18,\n \"score\": 1.0,\n \"class\": \"PER\",\n \"beginOffset\": 10\n },\n {\n \"text\": \"White House\",\n \"endOffset\": 61,\n \"score\": 1.0,\n \"class\": \"LOC\",\n \"beginOffset\": 49\n }\n ]\n }\n ],\n \"software\": \"DeLFT\",\n \"date\": \"2018-05-02T12:24:55.529301\",\n \"model\": \"ner\"\n}\n\n```\n\nIf you have trained the model with ELMo, you need to indicate to use ELMo-based model when annotating with the parameter `--use-ELMo` (note that the runtime impact is important as compared to traditional embeddings): \n\n> python3 nerTagger.py --dataset-type conll2003 --use-ELMo --file-in data/test/test.ner.en.txt tag\n\n##### Ontonotes 5.0 CONLL 2012\n\nDeLFT comes with pre-trained models with the [Ontonotes 5.0 CoNLL-2012 NER dataset](http://cemantix.org/data/ontonotes.html). As dataset-type identifier, use `conll2012`. All the options valid for CoNLL-2003 NER dataset are usable for this dataset.\n\nWith the default BidLSTM-CRF architecture, FastText embeddings and without any parameter tuning, f1 score is __86.65__ averaged over these 10 trainings, with best run at __87.01__ (provided model) when trained with the train set strictly. \n\nWith ELMo, f-score is __88.66__ averaged over these 10 trainings, and with best best run at __89.01__.\n\nFor re-training, the assembled Ontonotes datasets following CoNLL-2012 must be available and converted into IOB2 tagging scheme, see [here](https://github.com/kermitt2/delft/tree/master/utilities) for more details. To train and evaluate following the traditional approach (training with the train set without validation set, and evaluating on test set), use:\n\n> python3 nerTagger.py --dataset-type conll2012 train_eval\n\n```text\nEvaluation on test set:\n\tf1 (micro): 87.01\n precision recall f1-score support\n\n DATE 0.8029 0.8695 0.8349 1602\n CARDINAL 0.8130 0.8139 0.8135 935\n PERSON 0.9061 0.9371 0.9214 1988\n GPE 0.9617 0.9411 0.9513 2240\n ORG 0.8799 0.8568 0.8682 1795\n MONEY 0.8903 0.8790 0.8846 314\n NORP 0.9226 0.9501 0.9361 841\n ORDINAL 0.7873 0.8923 0.8365 195\n TIME 0.5772 0.6698 0.6201 212\n WORK_OF_ART 0.6000 0.5060 0.5490 166\n LOC 0.7340 0.7709 0.7520 179\n EVENT 0.5000 0.5556 0.5263 63\n PRODUCT 0.6528 0.6184 0.6351 76\n PERCENT 0.8717 0.8567 0.8642 349\n QUANTITY 0.7155 0.7905 0.7511 105\n FAC 0.7167 0.6370 0.6745 135\n LANGUAGE 0.8462 0.5000 0.6286 22\n LAW 0.7308 0.4750 0.5758 40\n\nall (micro avg.) 0.8647 0.8755 0.8701 11257\n```\n\nWith ELMo embeddings (using the default hyper-parameters, except the batch size which is increased to better learn the less frequent classes):\n\n```text\nEvaluation on test set:\n f1 (micro): 89.01\n precision recall f1-score support\n\n LAW 0.7188 0.5750 0.6389 40\n PERCENT 0.8946 0.8997 0.8971 349\n EVENT 0.6212 0.6508 0.6357 63\n CARDINAL 0.8616 0.7722 0.8144 935\n QUANTITY 0.7838 0.8286 0.8056 105\n NORP 0.9232 0.9572 0.9399 841\n LOC 0.7459 0.7709 0.7582 179\n DATE 0.8629 0.8252 0.8437 1602\n LANGUAGE 0.8750 0.6364 0.7368 22\n GPE 0.9637 0.9607 0.9622 2240\n ORDINAL 0.8145 0.9231 0.8654 195\n ORG 0.9033 0.8903 0.8967 1795\n MONEY 0.8851 0.9076 0.8962 314\n FAC 0.8257 0.6667 0.7377 135\n TIME 0.6592 0.6934 0.6759 212\n PERSON 0.9350 0.9477 0.9413 1988\n WORK_OF_ART 0.6467 0.7169 0.6800 166\n PRODUCT 0.6867 0.7500 0.7170 76\n\nall (micro avg.) 0.8939 0.8864 0.8901 11257\n```\n\nFor ten model training with average, worst and best model with ELMo embeddings, use:\n\n> python3 nerTagger.py --dataset-type conll2012 --use-ELMo --fold-count 10 train_eval\n\n##### French model (based on Le Monde corpus)\n\nNote that Le Monde corpus is subject to copyrights and is limited to research usage only. This is the default French model, so it will be used by simply indicating the language as parameter: `--lang fr`, but you can also indicate explicitly the dataset with `--dataset-type lemonde`.\n\nSimilarly as before, for training and evaluating use:\n\n> python3 nerTagger.py --lang fr train_eval\n\nIn practice, we need to repeat training and evaluation several times to neutralise random seed effects and to average scores, here ten times:\n\n> python3 nerTagger.py --lang fr --fold-count 10 train_eval\n\nThe performance is as follow, with a f-score of __91.01__ averaged over 10 training:\n\n```text\naverage over 10 folds\n macro f1 = 0.9100881012386587\n macro precision = 0.9048633201198737\n macro recall = 0.9153907496012759 \n\n** Worst ** model scores - \n\n precision recall f1-score support\n\n \n \n