{
    "info": {
        "author": "Stanford Mobisocial Lab",
        "author_email": "mobisocial@lists.stanford.edu",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# Genie-parser\n\nA neural-network based semantic parser, designed to be used in conjuction with\n[genie-toolkit](https://github.com/Stanford-Mobisocial-IoT-Lab/genie-toolkit),\na set of tools to generate large scale semantic parsing datasets quickly.\n\nGenie was described in the paper:\n\n_Genie: A Generator of Natural Language Parsers for Compositional Virtual Assistants_  \nGiovanni Campagna (\\*), Silei Xu (\\*), Mehrad Moradshahi, and Monica S. Lam  \nTo appear in _Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation_ (PLDI 2019), Phoenix, AZ, June 2019.\n\nGenie-parser is part of Almond, a research project of the Mobile and Social\nComputing Lab at Stanford University.  You can find more\ninformation at <http://almond.stanford.edu/>.\n\n## Installation\n\ngenie-parser depends on numpy, Tensorflow 1.12 for Python 3, and a number of\nother python modules. To install the dependencies, use:\n\n    pip3 install -r requirements.txt\n\nIt's recommended to install numpy from distribution packages, not\npip because it's faster and more reliable.\n\nYou must also install `tensor2tensor` using our own fork, using the version\nindicated in requirements.txt.\nDo not install tensor2tensor from pypi, as that is not compatible.\n\nPlease see the Tensorflow documentation if you wish to use GPU\nacceleration. You'll need to install nvidia+cuda or amdgpu-pro+rocm\ndrivers, and then install \"tensorflow-gpu\" from pip, or build\nTensorflow from source.\n\ngenie-parser has been tested successfully on Fedora 25 to 28\nx86_64 with CPU and Nvidia GPU acceleration, as well as Ubuntu 16.04,\n18.04. On RHEL/CentOS 7, you will need to use python 3.6 from the `rh-python36`\nsoftware collection. On Fedora 29 and later, the default is python 3.7, which is not\ncompatible with Tensorflow, so you will need to install python 3.6 separately.\n\nIt is also possible to use the [pipenv](https://pipenv.readthedocs.io/en/latest/)\ntool to set up a virtualenv and install the dependencies inside it.\n\n## Training\n\nTo train a new model, you should do the follow:\n\n1. Unpack the dataset (*.tsv files) into a `dataset/` directory.\n\n2. Download the word embeddings. We recommend using Glove 42B 300-dimensional,\n   which can be downloaded from <https://nlp.stanford.edu/data/glove.42B.300d.zip>,\n   or from our mirror at <https://oval.cs.stanford.edu/data/glove/glove.42B.300d.zip>.\n   Set the `GLOVE` environment variable to the path of uncompressed text file.\n\n   You can skip this step, in which case, the `luinet-datagen` script will download\n   the recommended GloVe file automatically.\n\n3. Prepare the working directory:\n   ```\n   luinet-datagen --src_data_dir ./dataset --data_dir ./workdir --thingpedia_snapshot [SNAPSHOT]\n      --problem semparse_thingtalk_noquote\n   ```\n   This script computes the input dictionary, downloads a snapshot of Thingpedia,\n   and computes a subset of the word embedding matrix corresponding to the dictionary.\n\n   Use the optional SNAPSHOT argument to choose which Thingpedia snapshot to train\n   against, or pass -1 for the latest content of Thingpedia. Be aware that\n   using -1 might make the results impossible to reproduce.\n   This script will verify that the dataset is compatible with the\n   Thingpedia snapshot.\n\n   The `--problem` parameter should be used to chosen to match the target language.\n   The meaning is similar to that in the [tensor2tensor](https://github.com/Stanford-Mobisocial-IoT-Lab/tensor2tensor)\n   library. See [genieparser/tasks/__init__.py](genieparser/tasks/__init__.py) for a\n   list of available problems.\n\n4. Train:\n   ```\n   genie-trainer --data_dir ./workdir --output_dir ./workdir/model\n     --model genie_copy_seq2seq\n     --hparams_set 'lstm_genie'\n     --hparams_overrides ''\n     --decode_hparams 'beam_size=20,return_beams=true'\n     --problem 'semparse_thingtalk_noquote'\n     --eval_early_stopping_metric 'metrics-semparse_thingtalk_noquote/accuracy'\n     --noeval_early_stopping_metric_minimize\n   ```\n   See `luinet-trainer --help` for a description of each options. Available hparams sets are\n   in [genieparser/layers/hparams.py](genieparser/layers/hparams.py), and available models\n   are at [genieparser/models/__init__.py](genieparser/models/__init__.py).\n\n   During training, you can use [tensorboard](https://github.com/tensorflow/tensorboard) to visualize\n   progress:\n   ```\n   tensorboard --logdir ./workdir/model\n   ```\n\n   After training, you can extract the metrics of the best model on the validation set with:\n   ```\n   genie-print-metrics --output_dir ./workdir/model\n     --eval_early_stopping_metric 'metrics-semparse_thingtalk_noquote/accuracy'\n     --noeval_early_stopping_metric_minimize\n   ```\n\n5. Evaluate:\n   ```\n   genie-trainer --data_dir ./workdir --output_dir ./workdir/model\n     --model genie_copy_seq2seq\n     --hparams_set 'lstm_genie'\n     --hparams_overrides ''\n     --decode_hparams 'beam_size=20,return_beams=true'\n     --problem 'semparse_thingtalk_noquote'\n     --eval_early_stopping_metric 'metrics-semparse_thingtalk_noquote/accuracy'\n     --noeval_early_stopping_metric_minimize\n     --schedule evaluate\n   ```\n\n   Add `--eval_use_test_set` to use the test set instead of the validation set.\n\n   You can also evaluate on a specific saved model (such as the best model according to the metric\n   on the validation set) using the flag `--checkpoint_path ./workdir/model/export/best/.../variables/variables`\n\n6. To deploy the model, point the server to a saved model directory from `workdir/model/export/best`,\n   by writing a configuration file `server.conf` containing:\n   ```\n   [models]\n   en=<path-to-saved-model>\n   ```\n\n   Run the server with:\n   ```\n   genie-server --config-file <path-to-server.conf>\n   ```\n\n   By default, the server runs at port 8400. You can change that by editing the server.conf file.\n   An the example server.conf file is provided in the `data` folder, which describes all available\n   options, including SSL and privilege separation.\n\n   The server expects to connect to a TokenizerService (provided by [Almond Tokenizer](https://github.com/Stanford-Mobisocial-IoT-Lab/almond-tokenizer)) on\n   localhost, port 8888.\n\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/Stanford-Mobisocial-IoT-Lab/genie-parser",
        "keywords": "",
        "license": "GPL-3.0+",
        "maintainer": "",
        "maintainer_email": "",
        "name": "genie-parser",
        "package_url": "https://pypi.org/project/genie-parser/",
        "platform": "",
        "project_url": "https://pypi.org/project/genie-parser/",
        "project_urls": {
            "Homepage": "https://github.com/Stanford-Mobisocial-IoT-Lab/genie-parser"
        },
        "release_url": "https://pypi.org/project/genie-parser/0.1.0/",
        "requires_dist": [
            "tensorflow (>=1.12.0)",
            "tensor2tensor",
            "orderedset",
            "numpytornadoSQLAlchemysemver"
        ],
        "requires_python": ">=3.6",
        "summary": "",
        "version": "0.1.0"
    },
    "last_serial": 4856265,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "196e0b81662d9aee68c13ab818919b6f",
                    "sha256": "cc12bcf44fa9d0448a2ce6b9b5394398a0b4034025e00755da74f144b04af575"
                },
                "downloads": -1,
                "filename": "genie_parser-0.1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "196e0b81662d9aee68c13ab818919b6f",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 144202,
                "upload_time": "2019-02-22T22:54:29",
                "url": "https://files.pythonhosted.org/packages/ba/b6/8b85a48f0cc75d370073b9ac46bb86d8c9dbcfa12bc2d5e9ed18d99a3869/genie_parser-0.1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "9b2e3bae14958ae767035b14ef7c1d9b",
                    "sha256": "8fd0d7a431599b5905eda843c0582bbd13c463f69d4423cbed3b595b8821fff5"
                },
                "downloads": -1,
                "filename": "genie-parser-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "9b2e3bae14958ae767035b14ef7c1d9b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 85042,
                "upload_time": "2019-02-22T22:54:31",
                "url": "https://files.pythonhosted.org/packages/86/c3/7de390b2205063eced29ff548186cb2828b850a4ff63ff8540f43be177c4/genie-parser-0.1.0.tar.gz"
            }
        ],
        "0.1.dev0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bb9de6644d3572bb132da0dfd4e30b2e",
                    "sha256": "ff3d506264f7f51795a621ff2e824f3d1bb04dcda9b4fed3e2c8d9048133f312"
                },
                "downloads": -1,
                "filename": "genie_parser-0.1.dev0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "bb9de6644d3572bb132da0dfd4e30b2e",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 144282,
                "upload_time": "2019-02-22T22:52:55",
                "url": "https://files.pythonhosted.org/packages/e4/b2/b76a4db085a30f8c362390fb42087464f07c276bbc6cf53c997770187ea0/genie_parser-0.1.dev0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "447ebd6a7d1ac8a1a6f3e0efcd7edb08",
                    "sha256": "dd0339111f15a4721b23b31eba6ecc3118d3368f5b3a9ac83495463658fe9de8"
                },
                "downloads": -1,
                "filename": "genie-parser-0.1.dev0.tar.gz",
                "has_sig": false,
                "md5_digest": "447ebd6a7d1ac8a1a6f3e0efcd7edb08",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 85056,
                "upload_time": "2019-02-22T22:52:58",
                "url": "https://files.pythonhosted.org/packages/06/d3/d8ac79031931b5c7b19036861c6f6b8339ea25ff977fa6ab6516ede00cd9/genie-parser-0.1.dev0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "196e0b81662d9aee68c13ab818919b6f",
                "sha256": "cc12bcf44fa9d0448a2ce6b9b5394398a0b4034025e00755da74f144b04af575"
            },
            "downloads": -1,
            "filename": "genie_parser-0.1.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "196e0b81662d9aee68c13ab818919b6f",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.6",
            "size": 144202,
            "upload_time": "2019-02-22T22:54:29",
            "url": "https://files.pythonhosted.org/packages/ba/b6/8b85a48f0cc75d370073b9ac46bb86d8c9dbcfa12bc2d5e9ed18d99a3869/genie_parser-0.1.0-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "9b2e3bae14958ae767035b14ef7c1d9b",
                "sha256": "8fd0d7a431599b5905eda843c0582bbd13c463f69d4423cbed3b595b8821fff5"
            },
            "downloads": -1,
            "filename": "genie-parser-0.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "9b2e3bae14958ae767035b14ef7c1d9b",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6",
            "size": 85042,
            "upload_time": "2019-02-22T22:54:31",
            "url": "https://files.pythonhosted.org/packages/86/c3/7de390b2205063eced29ff548186cb2828b850a4ff63ff8540f43be177c4/genie-parser-0.1.0.tar.gz"
        }
    ]
}