{
    "info": {
        "author": "I. Chuang and G. Lopez",
        "author_email": "ichuang@mit.edu",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "edxcut\n======\n\nedxcut = edX Course Unit Tests\n\nHave you ever had your open edX course unexpectedly break, due to some\nchange in how the edX platform does grading, or due to some code\nchange in a custom response grading library?\n\nWould you like to be able to automatically verify that *all* the\nauto-graded problems in your edX course are functioning as expected,\nwithout having to manually click through all the problems and remember\ntest cases to enter?\n\n__edxcut__ is an open source package for performing automated unit\ntests of answer box grading correctness, across all problems, in a\nlive, open edX course instance.\n\nedxcut accepts a course unit test specification file (in YAML format),\nand interacts with the edX course instance, mimicing a live learner,\nvia direct calls to the edX xblock APIs for problem checking and the\ninstructor dashboard, for resetting problem attempts.  Test cases\nspecify inputs, and whether the expected graded return should be\ncorrect or incorrect, for each case.\n\nThe course unit tests file can be produced manually, or by digesting\nthe course XML, or automatically, during compilation using\n[latex2edx](https://github.com/mitocw/latex2edx).  When using\nlatex2edx, you can specify multiple test cases within the `\\edXabox`\nmacro, including both expected correct and incorrect cases.\n\nExample tests file\n------------------\n\nBelow is an [example tests YAML file](https://github.com/mitodl/edxcut/blob/master/test_data/test_demo_course.yaml)\nwhich can be run to test some problems in the demo course provided with the [edX fullstack](https://openedx.atlassian.net/wiki/display/OpenOPS/Running+Fullstack) (dogwood release) virtualbox VM:\n\n```YAML\nconfig:\n  course_id: course-v1:edX+DemoX+Demo_Course\n  site_base_url: http://192.168.33.10\n  username: staff@example.com\n  password: edx\n\ntests:\n  - url_name: 75f9562c77bc4858b61f907bb810d974\n    responses: [ 43.141, 4500, 5 ]\n    expected: [incorrect, correct, correct]\n  - url_name: 75f9562c77bc4858b61f907bb810d974\n    responses: [ 43.141, 4500, 9 ]\n    expected: [incorrect, correct, incorrect]\n  - url_name: 75f9562c77bc4858b61f907bb810d974\n    responses: [ 43.141, 4500, 9 ]\n    expected: incorrect\n  - url_name: Sample_Algebraic_Problem\n    responses: [A*x^2 + sqrt(y)]\n    expected: correct\n  - url_name: Sample_ChemFormula_Problem\n    responses: [H2SO4  -> H^+ + 2 HSO4^-]\n    expected: incorrect\n  - url_name: Sample_ChemFormula_Problem\n    responses: [H2SO4  -> H^+ + HSO4^-]\n    expected: correct\n```\n\nusing this command line:\n\n    edxcut test test_data/test_demo_course.yaml\n\nto get results like this:\n\n    ======================================================================\n    Running tests from test_data/test_demo_course.yaml\n    [CourseUnitTestSet] Loaded 6 answer box unit tests from test_data/test_demo_course.yaml\n    ============================================================ Running 6 tests\n    Test 1: OK\n    Test 2: OK\n    Test 3: OK\n    Test 4: OK\n    Test 5: OK\n    Test 6: OK\n    ======================================== Tests done\n    6 total tests, 6 passed, 0 failed\n\nNote that you may need to change the `url_name` for the first three\ncases, which have a edx-studio-specific hexstring, if using a different VM\ninstance.\n\nUsage\n-----\n\nHere's how to run unit tests on a course on edx.org:\n\n1. Generate course unit tests YAML file.  For example, with latex2edx:\n\n    latex2edx -d course_directory --output-course-unit-tests my_tests.yaml -m the_course.tex\n    \n2. Run edxcut, adding specifications for the course ID, username, and login password:\n\n```\n    edxcut  -s https://courses.edx.org \\\n    \t    -u my-course-tester@myorg.org \\\n            -p my-password \\\n            -c course-v1:MYx+NUM+SEM \\\n            test my_tests.yaml\n```\n\nCourse Unit Tests File\n----------------------\n\nThe course unit tests file should be in [YAML\nformat](https://en.wikipedia.org/wiki/YAML).  It may specify `config`\nparameters, for the `course_id`, `site_base_url`, `username`, and `password`, e.g.:\n\n```YAML\nconfig:\n  course_id: course-v1:edX+DemoX+Demo_Course\n  site_base_url: http://192.168.33.10\n  username: staff@example.com\n  password: edx\n```\n\nThe tests file should also specify one or more `tests`.  Each test\nshould give at least the `url_name`, `responses`, and `expected`\ngrader output.  For example:\n\n```YAML\n  - url_name: Sample_Algebraic_Problem\n    responses: [A*x^2 + sqrt(y)]\n    expected: correct\n```\n\nNote that `responses` should be a list.  `expected` should either be a\nstring, `correct` or `incorrect`, or it may be a list (of the same\nlength as `responses`, of those two strings.\n\nEach test may also specify a `name`.\n\nAnd each test may also specify `box_indexes`, which are pairs of (x,y)\ncoordinates for input boxes.  This is useful when there is more than\none question (ie answer box) for a given problem.  The coordinates are\nused to construct the input box IDs, which is of the form\ninput_<url_name>_<x>_<y>, where <x> indexes which `\\abox` (aka\n`<*response>`) the input is, and <y> indexes which input element it\nis, within a given abox (for aboxes with multiple input boxes).  This\nlist should have the same length as `responses`.  For example:\n\n```YAML\ntests:\n- box_indexes:\n  - [0, 0]\n  - [1, 0]\n  - [2, 0]\n  - [3, 0]\n  - [4, 0]\n  expected: [incorrect, incorrect, incorrect, incorrect, incorrect]\n  name: (Simple quantum gate identities) s12-wk1-gates/test_1\n  responses: [Z, I, Y, Z, I]\n  url_name: s12-wk1-gates\n```\n\nGenerating Tests with latex2edx\n-------------------------------\n\nHere are some example `\\edXabox` statements which may be used with\n[latex2edx](https://github.com/mitocw/latex2edx), to specify answer\nbox unit tests:\n\n```tex\n\\edXabox{type='custom' size=10 expect=\"I\" cfn=check_paulis inline=\"1\" test_fail=\"Z\"}\n\n\\edXabox{type=option options=\"true\",\"false\" expect=\"false\" inline=\"1\"}\n\n\\edXabox{type=symbolic size=10 expect=\"7\"  inline=\"1\"}\n\n\\edXabox{type=symbolic size=10 expect=\"7\"  inline=\"1\" test_pass=\"7\"}\n\n\\edXabox{type=symbolic size=10 expect=\"7\"  inline=\"1\" test_pass=\"7\" test_fail=\"9\"}\n```\n\nThis is an more complex example, where the grader is a custom python\nscript (not shown), that knows the expected answer; here, `test_pass`\nis necessary since `expect` does not provide an answer useful for the\nunit test:\n\n```tex\n\\edXabox{type=\"custom\" \n  size=30 \n  expect=\"See solutions\"  \n  options=\"expect=(Qubit('00')+Qubit('01'))/2\"\n  cfn=check_qcircuit_output \n  hints='myhints'\n  test_pass=\"H(0),H(1)\"\n  inline=\"1\"\n}\n```\n\nNote that the latex2edx correctly handles the case when there are\nmultiple answer boxes in a single problem, e.g.:\n\n```tex\n\\begin{edXproblem}{A problem with two aboxes}{url_name=\"a_problem\"}\n\n\\edXabox{type=\"option\" options=\"red\",\"green\",\"blue\" expect=\"red\"}\n\n\\edXabox{type=\"custom\" expect=\"42\" cfn=\"mytest\"}\n\\end{edXproblem}\n```\n\nSuch multi-box problems properly generate test cases with\n`box_indexes` set to specify the (x,y) coordinates of the input boxes.\n\nInstallation\n------------\n\n    pip install -e git+https://github.com/mitodl/edxcut.git#egg=edxcut\n\nUnit tests\n----------\n\nThis package includes unit tests for build testing.\n\nVersions\n--------\n\n0.1 - original\n0.2 - grades download added to edxapi.py (sdotglenn)\n",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://pypi.python.org/pypi/edxcut/",
        "keywords": "",
        "license": "LICENSE.txt",
        "maintainer": "",
        "maintainer_email": "",
        "name": "edxcut",
        "package_url": "https://pypi.org/project/edxcut/",
        "platform": "",
        "project_url": "https://pypi.org/project/edxcut/",
        "project_urls": {
            "Homepage": "http://pypi.python.org/pypi/edxcut/"
        },
        "release_url": "https://pypi.org/project/edxcut/0.3/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "edX course unit tester",
        "version": "0.3"
    },
    "last_serial": 2811118,
    "releases": {
        "0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "340a3b4b55ea9721d90a562ef48c87b0",
                    "sha256": "41febb76e4efafb4825785a5f2c7420a4d10c2ea85a1888188f47ee98a67bf1e"
                },
                "downloads": -1,
                "filename": "edxcut-0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "340a3b4b55ea9721d90a562ef48c87b0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10276,
                "upload_time": "2016-09-12T02:20:59",
                "url": "https://files.pythonhosted.org/packages/85/7d/45851a26eadc13e34f9834d719a1494d4acbb8000fe457997f4b0cc5d9d9/edxcut-0.1.tar.gz"
            }
        ],
        "0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "df59e751c1641166bab6b24d43af9773",
                    "sha256": "2cae41abf93faff897ae77c6b8d618781a2dbf1c1579e1d0827c7eaae299d27e"
                },
                "downloads": -1,
                "filename": "edxcut-0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "df59e751c1641166bab6b24d43af9773",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15194,
                "upload_time": "2017-03-30T19:59:01",
                "url": "https://files.pythonhosted.org/packages/1b/a3/ae21f9f9d0f1b720e1e80b87b9f396e7adc51115136f7c39d5aa79b663ae/edxcut-0.2.tar.gz"
            }
        ],
        "0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ba7d1b4e41a4d180916f55f2d14c9c2a",
                    "sha256": "5965e9921c37eacc738c1d51fc1ba8261e2d236f3afce3f5627db068900259e7"
                },
                "downloads": -1,
                "filename": "edxcut-0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "ba7d1b4e41a4d180916f55f2d14c9c2a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 18686,
                "upload_time": "2017-04-18T13:09:59",
                "url": "https://files.pythonhosted.org/packages/2e/99/73fc6ddd4559cc30efd39c94b01f8bb79ed0cb77d7a75521702937d07cf8/edxcut-0.3.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "ba7d1b4e41a4d180916f55f2d14c9c2a",
                "sha256": "5965e9921c37eacc738c1d51fc1ba8261e2d236f3afce3f5627db068900259e7"
            },
            "downloads": -1,
            "filename": "edxcut-0.3.tar.gz",
            "has_sig": false,
            "md5_digest": "ba7d1b4e41a4d180916f55f2d14c9c2a",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 18686,
            "upload_time": "2017-04-18T13:09:59",
            "url": "https://files.pythonhosted.org/packages/2e/99/73fc6ddd4559cc30efd39c94b01f8bb79ed0cb77d7a75521702937d07cf8/edxcut-0.3.tar.gz"
        }
    ]
}