{
    "info": {
        "author": "Lnk2past",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# contest\n[![Build Status](https://github.com/Lnk2past/contest/workflows/Build%20Examples/badge.svg)](https://github.com/Lnk2past/contest/actions) [![Travis (.com)](https://img.shields.io/travis/com/Lnk2past/contest?label=Build%20Examples&logo=travis)](https://travis-ci.org/Lnk2past/contest)\n\nA `CON`sole application `TEST`er\n\n## Introduction\n`contest` is a testing application that exercises a program with configured input and then checks the output with some expected content. Simply put, `contest` is a driver that facilitates supplying input to and validating output from a specific executable. `contest` itself is not obscenely robust, but lets the developer implement what is needed as they need it.\n\n## Installation\n```\npip install contest\n```\n\nYou may install from this repo, clone and simply:\n```\npython setup.py install\n```\n\n## Motivation\nI wrote this for a class that I teach to help with grading. Given the configuration driven nature of it (how else should a testing framework/tool work?) `contest` lets me define multiple test cases for particular programs (assignments) so that not only is grading easier for me, but I can integrate assignments into a grading pipeline so that I can do as little work as possible and my students can get immediate feedback on submissions.\n\n## How It Works\n### Overview\nYAML is the preferred choice of input for `contest` for a few reasons, but most notably for its easy to follow syntax and allowance of multiline strings (sorry JSON). `contest` consumes an input file that specifies at least one executable and then whatever additional information is provided. Check out the test skeleton below to see what can be specified; the main ones though are the input and output streams `stdout`, `stderr`, and `stderr` as well and CLI inputs `argv`. You can specify newly generated files that you expect to be created and even go further and specify custom tests (as `Python` files) that cover the things simple I/O comparisons do not.\n\n#### Test Skeleton\n```python\nexecutable:\ntest-cases:\n    standard:\n        executable:\n        returncode:\n        argv: []\n        stdin: |\n        stdout: |\n        stderr: |\n        ofstreams:\n            - base-file:\n              test-file:\n        extra-tests: []\n```\n\n## Development Environment\nI am coding this to work with latest Python. I have ~~absolutely no~~ little interest in backwards compatibility. While earlier versions and standards may work right now, I do not guarantee any of that moving forward. I will not hinder development for the sake of supporting something older. Given the range of development environments currently at my disposal there will be some compatibility for a bit.\n\nI have a few primary development environments at the moment and so you can for expect support for at least the following:\n- **Python 3.5.3** and GCC 6.3.0 (Raspbian)\n- **Python 3.6.7** and GCC 7.3.0 (Ubuntu 18.04.2)\n- **Python 3.6.8** and MSVC v1916 (Windows 10, Visual C++ 2017 (15.9))\n- **Python 3.7.3** and MSVC v1916 (Windows 10, Visual C++ 2017 (15.9))\n\n## Basic Usage\nGiven some configuration you can run `contest` using the following:\n```\ncontest <path to configuration file>\n```\n\nThis will parse the configuration and run the specified test cass(s). In the configuration file each test case is defined under the `test-cases` node in the recipe; simply add a new section as desired. You will just need to make sure each test is named uniquely. Here is an example of a test recipe (taken from `exampels/native_console_app`):\n\n```\nexecutable: hello_world.exe\ntest-cases:\n    standard:\n        stdin: |\n            Lnk2past\n        stdout: |\n            Hello! What is your name?\n            Welcome to the world, Lnk2past!\n```\n\nLet us break down what this is specifying:\n\n1. The name of the `executable` to run is\n    ```\n    hello_world.exe\n    ```\n2. We have a single test-case named\n    ```\n    standard\n    ```\n3. We define the input to `stdin`, which is a single entry:\n    ```\n    Lnk2past\n    ```\n4. We define the output to `stdout`, which is:\n    ```\n    Hello! What is your name?\n    Welcome to the world, Lnk2past!\n    ```\n\nThis is really the equivalent of the following in some shell environment:\n```\n~/project> ./hello_world.exe\nHello! What is your name?\nLnk2past\nWelcome to the world, Lnk2past!\n~/project>\n```\n\nThis means that when running the executable `hello_world.exe` we can expect the input in step 3 to yield the output in step 4. `contest` does this comparison for you! This allows you to write tests that would reflect actual use cases of your executable. Add as many tests as you like to cover various pathways through your program and to cover the various errors your program may encounter.\n\nCheck out the other examples under the `examples` directory.\n\n### Test Directories\n\n`contest` will run each test-case in a separate directory, and will create those directories in the same directory containing the test recipe. This ensures minimal conflict between test cases. For example, if your test recipe contains tests \"foo\" and \"bar\" and is located in \"C:\\Users\\Lnk2past\\MyProject\", then you can expact the following directory structure:\n\n```\nC:\\Users\\Lnk2past\\MyProject\\\n|---src\\...\n|---include\\...\n|---contest_recipe.yaml\n|---test_output\\\n    |---foo\\...\n    |---bar\\...\n```\n\nEven if your test-case produces no output on disk, the test-output directory will be created.\n\n### Filtering Tests\n\nYou can filter your test-recipes to only run a select few. This may be useful during debugging to only run your new test without needing to run the entire test recipe. You can do this via the `--filter` option. This expects some `regular expression` to filter on. e.g. we can test only those tests that are marked with a specific keyword in their names, say \"tracking\", by doing the following:\n\n```\npython contest.py test_recipe.yaml --filter \"tracking\"\n```\n\nLikewise, you can exclude specific tests in order if they are problematic or if you are focusing on other tests. simply use the `--exclude-filters` or `--exclude` for short. So long as you know `regex` you can do whatever you like for filtering your tests!\n\n## TODO\n- add configuration option for file output\n- improve testing\n    - add more test cases\n    - test custom test examples\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/Lnk2past/contest",
        "keywords": "",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "contest",
        "package_url": "https://pypi.org/project/contest/",
        "platform": "",
        "project_url": "https://pypi.org/project/contest/",
        "project_urls": {
            "Homepage": "https://github.com/Lnk2past/contest"
        },
        "release_url": "https://pypi.org/project/contest/2019.10.2/",
        "requires_dist": [
            "PyYAML"
        ],
        "requires_python": "",
        "summary": "A console application tester.",
        "version": "2019.10.2"
    },
    "last_serial": 5979101,
    "releases": {
        "2019.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8c5f5c63287435a0df7e9f2dc3cd84a3",
                    "sha256": "0113bbfe818db9026e6934d102f15f478834b6157d7121575b453f7c077773ec"
                },
                "downloads": -1,
                "filename": "contest-2019.1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "8c5f5c63287435a0df7e9f2dc3cd84a3",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 9092,
                "upload_time": "2019-08-11T03:55:40",
                "url": "https://files.pythonhosted.org/packages/59/1b/b52f47786e3deb699b27ea6989c94b58bbcc89cc26a58445cc1e023ac44c/contest-2019.1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ba6d7b8b8fde5c3dd7b62d7fe5fdf3d7",
                    "sha256": "a5276c4bf77080ec9f528e5fe392e0868e12a12df3fe8f8efbbc1c3c8dd2a032"
                },
                "downloads": -1,
                "filename": "contest-2019.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "ba6d7b8b8fde5c3dd7b62d7fe5fdf3d7",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7672,
                "upload_time": "2019-08-11T03:55:42",
                "url": "https://files.pythonhosted.org/packages/1d/7d/2752339a747c258e19fade2e82c8296eb3ddf8b1fad8371e269cd5c1c975/contest-2019.1.0.tar.gz"
            }
        ],
        "2019.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4546dadadcb671ff431e5cbef3420fc9",
                    "sha256": "4eb049f0543f319f97b42789ea584b25675153fb30e57ee11f920ed5081cf969"
                },
                "downloads": -1,
                "filename": "contest-2019.1.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "4546dadadcb671ff431e5cbef3420fc9",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 9096,
                "upload_time": "2019-08-11T14:41:09",
                "url": "https://files.pythonhosted.org/packages/a0/ec/dca240523228eb12a988d9c8e5f24d012ce6e29b51cfee80ef8546e27e12/contest-2019.1.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "81b7cd8f3f2639e8d4ef7ba627630a7c",
                    "sha256": "9927e9435fad323716beae409b1fa0d84a8842c64db54b51a5addf9552472dcf"
                },
                "downloads": -1,
                "filename": "contest-2019.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "81b7cd8f3f2639e8d4ef7ba627630a7c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7680,
                "upload_time": "2019-08-11T14:41:10",
                "url": "https://files.pythonhosted.org/packages/7a/66/621007659fe51f296b3d49a54b90e0683eca851ffd403bf6001fb3f1d893/contest-2019.1.1.tar.gz"
            }
        ],
        "2019.10.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "060554c49eafb5a862fa460e7835902b",
                    "sha256": "1a246fd9cca657afeae9224f0bec58171f48cc2019eb67dc0e786628f33da79b"
                },
                "downloads": -1,
                "filename": "contest-2019.10.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "060554c49eafb5a862fa460e7835902b",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 11035,
                "upload_time": "2019-10-15T19:45:18",
                "url": "https://files.pythonhosted.org/packages/ed/24/ac0293e4791850495dcecff75b1a60e4e5900ceb32ef8dbbac218fd8cfcb/contest-2019.10.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ea992743f20a20cab6351856e517dbe6",
                    "sha256": "94854a6474aa657026204586218632cf77d0da5d5659b0fb1b5977ddf0658523"
                },
                "downloads": -1,
                "filename": "contest-2019.10.2.tar.gz",
                "has_sig": false,
                "md5_digest": "ea992743f20a20cab6351856e517dbe6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9526,
                "upload_time": "2019-10-15T19:45:20",
                "url": "https://files.pythonhosted.org/packages/4f/ff/a313b58aa5ea9d0a2d2d8f14f3779190bad53d1946384d43804f22c44c0e/contest-2019.10.2.tar.gz"
            }
        ],
        "2019.9.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0bb5bf0ba85e4f0c4500ad6332b2b9d3",
                    "sha256": "667743e33b8b527c201578526424c4943c1cefa401c92fa52e43ad56e96c7707"
                },
                "downloads": -1,
                "filename": "contest-2019.9.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "0bb5bf0ba85e4f0c4500ad6332b2b9d3",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10887,
                "upload_time": "2019-09-03T01:40:07",
                "url": "https://files.pythonhosted.org/packages/76/02/6f4c05c4bd07c6833d8ab7cfcbf3acf46b6c0cd0c3f7eff34f4526f30c4e/contest-2019.9.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "8d860ff317fb7962be495bd4274bca11",
                    "sha256": "d2efdcb67bb79ee0023aa14478076489b86342ae2002785a5a0ce95451a047e8"
                },
                "downloads": -1,
                "filename": "contest-2019.9.0.tar.gz",
                "has_sig": false,
                "md5_digest": "8d860ff317fb7962be495bd4274bca11",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8621,
                "upload_time": "2019-09-03T01:40:08",
                "url": "https://files.pythonhosted.org/packages/47/d8/b6b0ce2f4649a21dce5258b2538f847098b043d6efdc52af636ec1e0d1c1/contest-2019.9.0.tar.gz"
            }
        ],
        "2019.9.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "53c18ad1ea01a6622cc6d44418fc33e8",
                    "sha256": "00dcf47eae3b1e4967b1529539f23465d72506a509ff59f6d1307097ffa2a933"
                },
                "downloads": -1,
                "filename": "contest-2019.9.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "53c18ad1ea01a6622cc6d44418fc33e8",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10962,
                "upload_time": "2019-09-03T17:45:28",
                "url": "https://files.pythonhosted.org/packages/a9/69/7ebdb27a08ab8db42d3a89da70f99b4cafa6274d651d118a4f66e711905b/contest-2019.9.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "dc02eadc3ebed11090031e0411c225c4",
                    "sha256": "6ba728c58d87e280c2ceed2f9de51a1cc580615ae2ff2d795964f7b405aab207"
                },
                "downloads": -1,
                "filename": "contest-2019.9.1.tar.gz",
                "has_sig": false,
                "md5_digest": "dc02eadc3ebed11090031e0411c225c4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8670,
                "upload_time": "2019-09-03T17:45:29",
                "url": "https://files.pythonhosted.org/packages/d5/4e/029199c7a4c3e151e796e8cbfcead91887887900384078db3bffec80363a/contest-2019.9.1.tar.gz"
            }
        ],
        "2019.9.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "028628dd984c0c19206c4a4db3e90396",
                    "sha256": "d5e5f03de358aed183ca484f004c4014e4c4eb35a6a8e56adf930288c8d6738e"
                },
                "downloads": -1,
                "filename": "contest-2019.9.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "028628dd984c0c19206c4a4db3e90396",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10963,
                "upload_time": "2019-09-03T18:04:17",
                "url": "https://files.pythonhosted.org/packages/92/92/bf4a386dbe88d7040a22af0850f668b3348b0b472bd519f5b2a87da26e34/contest-2019.9.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e488595ee96d003b48ef293ce0e92215",
                    "sha256": "65b55d87bd2f434ee028eb14df41572f43d989e44926ca242863c6987e4d73dc"
                },
                "downloads": -1,
                "filename": "contest-2019.9.2.tar.gz",
                "has_sig": false,
                "md5_digest": "e488595ee96d003b48ef293ce0e92215",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9467,
                "upload_time": "2019-09-03T18:04:19",
                "url": "https://files.pythonhosted.org/packages/32/b5/4e905402d4bcb169df0ff8bdbffb202936e1ae852123ac2f0a51d425daf2/contest-2019.9.2.tar.gz"
            }
        ],
        "2019.9.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f2a13544168bc03ecd59ea7eb23dc2f6",
                    "sha256": "e85062f3cb7c72685e9b775e1d40aeb91035778a3d9b1c66a9bd15c7747a7891"
                },
                "downloads": -1,
                "filename": "contest-2019.9.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "f2a13544168bc03ecd59ea7eb23dc2f6",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10951,
                "upload_time": "2019-09-03T18:35:43",
                "url": "https://files.pythonhosted.org/packages/74/6f/c5024e6b40b077d2be676c0aaadcf5bb79b4c0a118f884fd1012166acda9/contest-2019.9.3-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ad4dff8b7970fc846bf19d3e8fa1cb64",
                    "sha256": "deb4c9d2a0ab83f56430013e3fdcc77354ae2cff27160b1263ff754dcad835c8"
                },
                "downloads": -1,
                "filename": "contest-2019.9.3.tar.gz",
                "has_sig": false,
                "md5_digest": "ad4dff8b7970fc846bf19d3e8fa1cb64",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9448,
                "upload_time": "2019-09-03T18:35:45",
                "url": "https://files.pythonhosted.org/packages/af/26/0225ef4cac38cb71885788f5b0c71a8d7ef70b4e5f36d92cfd3412c4c3c2/contest-2019.9.3.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "060554c49eafb5a862fa460e7835902b",
                "sha256": "1a246fd9cca657afeae9224f0bec58171f48cc2019eb67dc0e786628f33da79b"
            },
            "downloads": -1,
            "filename": "contest-2019.10.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "060554c49eafb5a862fa460e7835902b",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 11035,
            "upload_time": "2019-10-15T19:45:18",
            "url": "https://files.pythonhosted.org/packages/ed/24/ac0293e4791850495dcecff75b1a60e4e5900ceb32ef8dbbac218fd8cfcb/contest-2019.10.2-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "ea992743f20a20cab6351856e517dbe6",
                "sha256": "94854a6474aa657026204586218632cf77d0da5d5659b0fb1b5977ddf0658523"
            },
            "downloads": -1,
            "filename": "contest-2019.10.2.tar.gz",
            "has_sig": false,
            "md5_digest": "ea992743f20a20cab6351856e517dbe6",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 9526,
            "upload_time": "2019-10-15T19:45:20",
            "url": "https://files.pythonhosted.org/packages/4f/ff/a313b58aa5ea9d0a2d2d8f14f3779190bad53d1946384d43804f22c44c0e/contest-2019.10.2.tar.gz"
        }
    ]
}