{
    "info": {
        "author": "",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# Codalab Yaml Validator\nCodalab Yaml Validator is a command line tool made to be used in conjunction with Codalab V2. It validates competition\nbundles locally without having to upload to the server first. It can also be used to compare one bundle to another, and\nshow the differences between them. Functionally, this is aimed at comparing a bundle used to upload a competition with\na dump of that competition created by the server, used to point out any differences between the two. This can be used\nboth to validate that changes _have not_ been made during the upload process, and also to validate that any changes\nmade in the editor on the server _have_ been accounted for in the dumps file.\n\n## Installation\nUsing `pip`\n```bash\npip install codalab_yaml_validator\n```\n\n## Usage\n### Single Directory Validation\nThis can be used to validate a folder or a .zip file.\n```bash\n# Validating a folder\nvalidate_bundle /path/to/folder/\n\n# Validating a zip file\nvalidate_bundle /path/to/file.zip\n```\n\n#### Output\nFirst, the yaml file `competition.yaml` is run through an initial formatting validation. This is done using the expected\nschema (provided below). If there are errors on this level, a `ValueError` is raised and the validation process stops.\n##### Example error message\n```\nTraceback\n...\nValueError: \nError validating data /.../competition.yaml with schema /.../site-packages/codalab_yaml_validator/schema.yaml\n\ttasks.0.index: Required field missing\n```\nIf the first validation process is passed, `Yaml file passed initial formatting tests` is printed and a deeper validation\nprocess begins. This verifies things like the same index is not used on multiple phases, or that the files provided for\nthis like images and scoring programs actually exist at the provided file path. In this process there are both `Errors`\nand `Warnings`. Errors will prevent a bundle from being valid, and thus cannot be uploaded to Codalab, while warnings\nare not invalid bundles, but uploading the bundle may not produce the desired competition.\n##### Example\n```\nWARNINGS:\n- Task with index 0: If specifying a key, all other fields will be ignored on upload\nERRORS:\n- Duplicate task index(es): [0]\n- Task index: \"1\" on phase: \"Example Phase Name\" not present in tasks\n- File for scoring_program - (path/to/scoring_program.zip) - not found\n```\n\nIf there are no errors `Yaml bundle is valid` will be printed\n\n### Bundle to Bundle Comparison\n```bash\nvalidate_bundle /path/to/bundle/one /path/to/bundle/two\n```\nJust as before, both directories and zip files are acceptable, and one can be compared to the other, i.e.,\n```bash\nvalidate_bundle /path/to/zip.zip /path/to/folder\n```\n\nBundles are each run through the single bundle validation before comparisons are made. If either bundle is invalid,\nthe comparisons will not be made and errors must be addressed. If no errors are present, comparison will begin.\n<br>\nNote: This validation is run silently, so warnings will not be printed, nor will validity affirmations. The only\nfeedback that will be printed are errors to be addressed.\n\nIf both bundles are valid, comparisons will be made. Because the competition editor on Codalab allows for changing every\nvalue present in an upload bundle, and the dumps process may print things like Tasks in different orders than they were\nuploaded in, there is no definitive way to _know_ which Task originated with which. This comparison process examines all\npossible options and compares the ones that match the closest. \n\nFor example, if the upload bundle looks like:\n```yaml\n# ...\nphases:\n- index: 0\n  name: Fast Phase\n  description: Computing Pi Faster\n  start: 02-01-2019\n  end: 09-01-2019\n  tasks:\n  - 1\n- index: 1\n  name: Slower Phase\n  description: Computing Pi\n  start: 08-01-2018\n  end: 02-01-2019\n  tasks:\n  - 0\n# ...\n```\nAnd the dump bundle looks something like:\n```yaml\n# ...\nphases:\n- index: 0\n  name: Slow Phase\n  description: Computing Pi\n  start: 08-01-2018\n  end: 02-01-2019\n  tasks:\n  - 0\n- index: 1\n  name: Fast Phase\n  description: Computing Pi Quickly\n  start: 02-01-2019\n  end: 09-01-2019\n  tasks:\n  - 1\n# ...\n```\n\nThe Comparison process can intelligently determine that index 0 in the upload bundle should be compared to index 1 in the\ndump bundle, so that the most accurate account of differences can be given. This does have some limitations, especially \nas the number of changes made in the editor increase, but it should seek to minimize the number of differences when\nmaking comparisons. This process is the same for comparing tasks, solutions, leaderboards, and columns.\n\n##### Example Output\nIn the case of the above yamls:\n```\n$ validate_bundle /path/to/Archive/ /path/to/Dump.zip\nDifferences:\n\n- Values on Phases index:1 in Archive and index:0 in Dump.zip do not match for key: name.\n  - Archive = Slower Phase\n  - Dump.zip = Slow Phase\n\n- Values on Phases index:0 in Archive and index:1 in Dump.zip do not match for key: description.\n  - Archive = Computing Pi Faster\n  - Dump.zip = Computing Pi Quickly\n```\n\n##### Limitations\nCodalab allows uploading things like scoring programs in unzipped directories and zips them itself during the\nupload process. When a dump is created, these zipped directories are returned. Hashes are used to compare files like this\nso the folder must be compressed and then hashed. The compression of this directory yields a different hash than its\nalready compressed counterpart, so these files must be validated manually.\n\nWhile a bundle using the same hierarchy as Codalab v1.5 is currently acceptable to upload to Codalab v2, its validation\nis outside the scope of this tool. Bundle must be of the \"Task Solution\" style to be validated properly.\n\nEOFs on pages are changed in the process of storing their content as text on the server, so when a dump of that content\nis created, even if the characters are the same, the contents of the file differ slightly so verifying these with a\nhash is impossible. Again, these files will need to be checked manually. Because every page would be flagged as different,\npages are not checked for differences at all.\n\n## Schema\nUsed in conjunction with [Yamale](https://pypi.org/project/yamale/) to validate the yaml formatting.\n```yaml\ntitle: str(min=8, max=255)\nimage: str(max=1024)\ntasks: list(include('task'))\nsolutions: list(include('solution'), required=False)\npages: list(include('page'))\nphases: list(include('phase'))\nleaderboards: list(include('leaderboard'))\n\n---\n\npage:\n  title: str(max=32)\n  file: str(max=1024)\n\nphase:\n  name: str(max=128)\n  description: str(max=1024)\n  index: int(max=99, required=False)\n  max_submissions: int(required=False)\n  max_submissions_per_day: int(required=False)\n  execution_time_limit_ms: int(max=5184000, required=False)\n  start: date()\n  end: date(required=False)\n\n  tasks: list(int())\n  solutions: list(int(), required=False)\n\nfile:\n  name: str(max=128)\n  description: str(max=1024)\n  path: str(max=1024)\n\ntask:  # key or scoring_program is required\n  index: int()\n  name: str(max=256, required=False)\n  description: str(max=1024, required=False)\n  ingestion_program: str(max=1024, required=False)\n  ingestion_only_during_scoring: bool(required=False)\n  input_data: str(max=1024, required=False)\n  scoring_program: str(max=1024, required=False)\n  reference_data: str(max=1024, required=False)\n  key: regex(r'[0-9a-fA-F]{8}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{12}', required=False)\n\nsolution:\n  index: int()\n  name: str(max=256)\n  description: str(max=1024, required=False)\n  path: str(max=1024, required=False)\n  key: regex(r'[0-9a-fA-F]{8}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{4}\\-[0-9a-fA-F]{12}', required=False)\n\nleaderboard:\n  title: str(max=128)\n  key: str(max=128)\n  index: int()\n  columns: list(include('leaderboard_column'))\n  force_submission_to_leaderboard: bool(required=False)\n  force_best_submission_to_leaderboard: bool(required=False)\n  disallow_leaderboard_modifying: bool(required=False)\n\nleaderboard_column:\n  title: str(max=128)\n  key: str(max=128)\n  index: int()\n  computation: enum('avg', required=False)\n  computation_indexes: str(required=False)\n  sorting: enum('asc', 'desc', required=False)\n  decimal_count: int(required=False)\n```",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/codalab/codalab-bundle-validator",
        "keywords": "",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "codalab-yaml-validator",
        "package_url": "https://pypi.org/project/codalab-yaml-validator/",
        "platform": "",
        "project_url": "https://pypi.org/project/codalab-yaml-validator/",
        "project_urls": {
            "Homepage": "https://github.com/codalab/codalab-bundle-validator"
        },
        "release_url": "https://pypi.org/project/codalab-yaml-validator/0.0.10/",
        "requires_dist": null,
        "requires_python": ">=3",
        "summary": "",
        "version": "0.0.10"
    },
    "last_serial": 5189172,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4554987cac5e962949bcfd88414d8dfa",
                    "sha256": "6d88321f70d99d3d45987b5fd90dcd57da20eaa869b20dc8a0ef9eaf5566f44a"
                },
                "downloads": -1,
                "filename": "codalab_yaml_validator-0.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "4554987cac5e962949bcfd88414d8dfa",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3169,
                "upload_time": "2019-04-15T21:41:17",
                "url": "https://files.pythonhosted.org/packages/3f/3e/154e5607df85baab5b0a5d215b5a3ef47d3a1673fc4577f0c55b7145f4dd/codalab_yaml_validator-0.0.1.tar.gz"
            }
        ],
        "0.0.10": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "69a8631f071730e8d41dfe3668ffc90f",
                    "sha256": "cb6f3fa6c512fb61a6931dfbb69fbb50b3086cb119ff2bc792156eda4ae90f65"
                },
                "downloads": -1,
                "filename": "codalab_yaml_validator-0.0.10.tar.gz",
                "has_sig": false,
                "md5_digest": "69a8631f071730e8d41dfe3668ffc90f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3",
                "size": 11662,
                "upload_time": "2019-04-25T17:48:39",
                "url": "https://files.pythonhosted.org/packages/5c/6a/aef8f6af7fc78fed5758f83aa07b554c39497c03ea2e0e815c07e16075ba/codalab_yaml_validator-0.0.10.tar.gz"
            }
        ],
        "0.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0d5807bdfa987e00a9e93bb21c58c4e0",
                    "sha256": "1ec8355eecdbdda348d768474fd2bcfb70941f9c168593f84f2aeee067af6659"
                },
                "downloads": -1,
                "filename": "codalab_yaml_validator-0.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "0d5807bdfa987e00a9e93bb21c58c4e0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5307,
                "upload_time": "2019-04-23T00:16:22",
                "url": "https://files.pythonhosted.org/packages/db/b8/395deb3c824bb08fca3de47eea039e42529039526a8a15247aa56020af6a/codalab_yaml_validator-0.0.2.tar.gz"
            }
        ],
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0573a8979a30aec921aa23ab14988e9a",
                    "sha256": "abfe79fe873e884f139fe5a023e5f16029df13ecc071d75482a03aa8430ccabf"
                },
                "downloads": -1,
                "filename": "codalab_yaml_validator-0.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "0573a8979a30aec921aa23ab14988e9a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5886,
                "upload_time": "2019-04-23T00:25:18",
                "url": "https://files.pythonhosted.org/packages/2b/44/67d0ea8b79a70ce18b7e13099988194522cbbd3549ebaaddd80513554765/codalab_yaml_validator-0.0.3.tar.gz"
            }
        ],
        "0.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "523d6f3901d9a786eb493921cd49d05b",
                    "sha256": "6808c4bcc42c5744a498318db9f8614773cbe83f2fbb21f123b343de22b00491"
                },
                "downloads": -1,
                "filename": "codalab_yaml_validator-0.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "523d6f3901d9a786eb493921cd49d05b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5886,
                "upload_time": "2019-04-23T00:28:15",
                "url": "https://files.pythonhosted.org/packages/ef/01/2b40dbd8546db3da5c313f7c536ba2eeb93f03ab55d1200149a145a04f84/codalab_yaml_validator-0.0.4.tar.gz"
            }
        ],
        "0.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4fb8cfb56a5856ae4b4a8ce987ddd393",
                    "sha256": "743a067789aaf05ad01ab1ea6278706fae65e5cc938dea7b63b44a903158292a"
                },
                "downloads": -1,
                "filename": "codalab_yaml_validator-0.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "4fb8cfb56a5856ae4b4a8ce987ddd393",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9905,
                "upload_time": "2019-04-23T19:25:41",
                "url": "https://files.pythonhosted.org/packages/86/77/0e6bcffdf066a08e6bda7d575f0039ebdffeb252f3651cd7fe9ab0854f17/codalab_yaml_validator-0.0.5.tar.gz"
            }
        ],
        "0.0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e74d3ab38cacef37baae8103fc6d3e47",
                    "sha256": "eeb33dfb5e1aa6afb3ef85d96e00023eb3bb4a01fc60a2f0ed00850518b7fd7e"
                },
                "downloads": -1,
                "filename": "codalab_yaml_validator-0.0.6.tar.gz",
                "has_sig": false,
                "md5_digest": "e74d3ab38cacef37baae8103fc6d3e47",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9904,
                "upload_time": "2019-04-23T19:31:19",
                "url": "https://files.pythonhosted.org/packages/14/a6/01e1ab516dd55e6598cd668e53de9362b2d3d5e00ecefc5f07d90a1fe88d/codalab_yaml_validator-0.0.6.tar.gz"
            }
        ],
        "0.0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9c69cecf04f7d267860463f1e7cdf084",
                    "sha256": "cea399d2b94020958d3b42e90e83ccf10185e6f1a9d3ab43aaed76d7b0ab2b49"
                },
                "downloads": -1,
                "filename": "codalab_yaml_validator-0.0.7.tar.gz",
                "has_sig": false,
                "md5_digest": "9c69cecf04f7d267860463f1e7cdf084",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3",
                "size": 9940,
                "upload_time": "2019-04-23T19:51:43",
                "url": "https://files.pythonhosted.org/packages/9d/75/24a310da5320e1f46bb4416b32df033f2351e4ec56e59543a4e2862c5d2c/codalab_yaml_validator-0.0.7.tar.gz"
            }
        ],
        "0.0.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e934f1b2efe950ad2ceb1219a376d6eb",
                    "sha256": "25f78dc89f9f55111247aa9f4fe8bb71268c9a45a71b84ce3e4d4c6e016cd6ba"
                },
                "downloads": -1,
                "filename": "codalab_yaml_validator-0.0.8.tar.gz",
                "has_sig": false,
                "md5_digest": "e934f1b2efe950ad2ceb1219a376d6eb",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3",
                "size": 11006,
                "upload_time": "2019-04-23T20:42:54",
                "url": "https://files.pythonhosted.org/packages/87/8b/a9a41ab2921aa312c17bc3808a55ecef4f365eb78c2f741e697334562365/codalab_yaml_validator-0.0.8.tar.gz"
            }
        ],
        "0.0.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "91d0e64770f25ecf482cd1737d79af28",
                    "sha256": "223ab91b090ab72408cb94e65721cd6985f0347e1bc28e5a7f2c312e6b905e7e"
                },
                "downloads": -1,
                "filename": "codalab_yaml_validator-0.0.9.tar.gz",
                "has_sig": false,
                "md5_digest": "91d0e64770f25ecf482cd1737d79af28",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3",
                "size": 11657,
                "upload_time": "2019-04-23T20:50:36",
                "url": "https://files.pythonhosted.org/packages/63/ee/0661734607a18d668f2b128c006d5311fc231aa9905b43cc5a69a3454b9b/codalab_yaml_validator-0.0.9.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "69a8631f071730e8d41dfe3668ffc90f",
                "sha256": "cb6f3fa6c512fb61a6931dfbb69fbb50b3086cb119ff2bc792156eda4ae90f65"
            },
            "downloads": -1,
            "filename": "codalab_yaml_validator-0.0.10.tar.gz",
            "has_sig": false,
            "md5_digest": "69a8631f071730e8d41dfe3668ffc90f",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3",
            "size": 11662,
            "upload_time": "2019-04-25T17:48:39",
            "url": "https://files.pythonhosted.org/packages/5c/6a/aef8f6af7fc78fed5758f83aa07b554c39497c03ea2e0e815c07e16075ba/codalab_yaml_validator-0.0.10.tar.gz"
        }
    ]
}