{ "info": { "author": "Kirill Simonov (Prometheus Research, LLC)", "author_email": "xi@resolvent.net", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Environment :: Console", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Operating System :: POSIX :: Linux", "Programming Language :: Python", "Programming Language :: Python :: 2", "Programming Language :: Python :: 3", "Topic :: Utilities" ], "description": "***********************************************\n PBBT -- Pluggable Black-Box Testing toolkit\n***********************************************\n\n.. contents:: Table of Contents\n\n\nOverview\n========\n\nPBBT is a regression test harness for *black-box testing*. It is\nsuitable for testing complex software components with well-defined input\nand output interfaces.\n\n::\n\n input +----------+ output\n o--------> | Software | --------->o\n +----------+\n\nIn black-box testing, a *test case* is a combination of *input* and\nexpected *output* data. The test harness executes the software with the\ngiven input and verifies that the produced output coincides with the\nexpected output.\n\nBlack-box testing could be implemented for many different types of\nsoftware. For example,\n\n* *a database system:* the input is a SQL statement, the output is a set\n of records;\n* *a web service:* the input is an HTTP request, the output is an HTTP\n response;\n* *a command-line utility:* the input is a sequence of command-line\n parameters and ``stdin``, the output is ``stdout``;\n* *a GUI application:* different approaches are possible; for instance,\n the input could be a sequence of user actions, and the output could be\n the state of a particular widget.\n\nPBBT is a Python library and an application which allows you to:\n\n* use built-in test types for testing command-line scripts and Python\n code;\n* register custom test types;\n* prepare test input in a succinct YAML_ format;\n* in the *train* mode, run the test cases and record expected output;\n* in the *check* mode, run the test cases and verify that the produced\n output coincides with the pre-recorded expected output.\n\nPBBT is a free software released under MIT license. PBBT is created by\nClark C. Evans and Kirill Simonov from `Prometheus Research, LLC`_.\n\n\nUsing PBBT\n==========\n\nTo install PBBT, you can use pip_ package manager::\n\n # pip install pbbt\n\nThis command downloads and installs the latest version of PBBT from\n`Python Package Index`_. After successful installation, you should be\nable to import ``pbbt`` Python module and run ``pbbt`` command-line\nutility.\n\nTo start using PBBT, you need to create a file with input data. For\nexample, create ``input.yaml`` with the following content::\n\n py: |\n print \"Hello, World!\"\n\nThis file is in YAML_ format, which is a data serialization language\nsimilar to JSON_, and, in fact, a superset of JSON. The file above\ncould be represented in JSON as::\n\n { \"py\": \"print \\\"Hello, World!\\\"\\n\" }\n\nFor description of YAML syntax and semantics, see http://yaml.org/.\n\nNext, execute PBBT in *training* mode to generate expected output data.\nRun::\n\n $ pbbt input.yaml output.yaml --train\n\nand accept new output when asked. PBBT will write output data to\n``output.yaml``::\n\n py: print-hello-world\n stdout: |\n Hello, World!\n\nNow you can start PBBT in *checking* mode, in which it executes test\ncases and verifies that expected and actual output data coincide::\n\n $ pbbt input.yaml output.yaml\n\nTo add more test cases to ``input.yaml``, you need to convert it to a\n*test suite*::\n\n title: My Tests\n tests:\n - py: |\n print \"Hello, World!\"\n - sh: echo Hello, World!\n\nThe file now contains a test suite *My Tests* with two test cases: one\nas in the previous example, and another that executes a shell command\n``echo Hello, World!``::\n\n sh: echo Hello, World!\n\nThe output of this test case is ``stdout`` produced by the shell\ncommand. To record expected output, run ``pbbt`` in training mode\nagain.\n\n\nBuilt-in Test Types\n===================\n\nOut of the box, PBBT supports a small set of predefined test types:\n\n* test Python code;\n* test a shell command;\n* file manipulation tests.\n\nAlso available are special test types:\n\n* test suite;\n* include;\n* conditional variables.\n* gateway to other test systems.\n\nEach test type defines the structure of input and output records, that\nis, the set of mandatory and optional fields and the type of field\nvalues. In this section, we list all available test types and describe\ntheir input fields.\n\nCommon Fields\n-------------\n\nThe following optional fields are available for all test types where\nthey make sense:\n\n``skip``: ``true`` or ``false``\n On ``true``, skip this test case.\n\n``if``: variable, list of variables or Python expression\n On a *variable name*, run this test case only if the variable is\n defined and not false.\n\n On a *list of variables*, run this test case only if at least one\n variable is defined and not false.\n\n On a *Python expression*, run this test case if the expression\n evaluates to true. You can use any conditional variables in the\n expression.\n\n``unless``: variable, list of variables or Python expression\n On a *variable name*, skip this test case if the variable is defined\n and not false.\n\n On a *list of variables*, skip this test case if at least one\n variable is defined and not false.\n\n On a *Python expression*, skip this test case if the expression\n evaluates to true. You can use any conditional variables in the\n expression.\n\n``ignore``: ``true``, ``false`` or regular expression\n On ``true``, permit the actual and expected output to be unequal.\n The test case must still execute without any errors.\n\n On a *regular expression*, pre-process the actual and expected\n output before comparing them:\n\n 1. Run the regular expression against the output data and find\n all matches.\n 2. If the regular expression does not contain ``()`` subgroups,\n erase all the matches from the output.\n 3. If the regular expression contains one or more ``()`` subgroups,\n erase the content of the subgroups from the output.\n\n The regular expression is compiled with ``MULTILINE`` and\n ``VERBOSE`` flags.\n\nExample::\n\n title: Integration with MySQL\n if: has_mysql\n tests:\n - set:\n MYSQL_HOST: localhost\n MYSQL_PORT: 3306\n unless: [MYSQL_HOST, MYSQL_PORT]\n - read: /etc/mysql/my.cnf\n if: MYSQL_HOST == 'localhost'\n - py: test-scalar-types.py\n ignore: |\n ^Today:.(\\d\\d\\d\\d)-(\\d\\d)-(\\d\\d)$\n - py: test-array-types.py\n skip: true # No array type in MySQL\n\nTest Suite\n----------\n\nA test suite is a collection of test cases.\n\nA suite may contain other suites and thus all test suites form a tree\nstructure. A *path* formed from suite identifiers can uniquely locate\nany suite. We use file-system notation for suite paths (e.g.\n``/path/to/suite``).\n\nFields:\n\n``title``: text\n The title of the suite.\n\n``suite``: identifier (optional)\n The identifier of the suite. If not set, generated from the title.\n\n``tests``: list of input records\n The content of the suite.\n\n``output``: path (optional)\n If set, the expected output of the suite is loaded from the given\n file.\n\nExample::\n\n title: All Tests\n suite: all\n output: output.yaml\n tests:\n - py: ./test/core.py\n - py: ./test/ext.py\n - title: Database Tests\n tests:\n - py: ./test/sqlite.py\n - py: ./test/pgsql.py\n - py: ./test/mysql.py\n\nIn this example, the path to the *Database Tests* suite is\n``/all/database-tests``.\n\nConditional Variables\n---------------------\n\nThis test case defines a conditional variable.\n\nVariables could be used in ``if`` and ``unless`` clauses to\nconditionally enable or disable a test case. Variables could also be\nset or read in Python tests via a global dictionary ``__pbbt__``.\n\nConditional variables could also be set from command line using ``-D``\noption.\n\nSetting a conditional variable affects all subsequent test cases within\nthe same test suite. Variable values are reset on exit from the suite.\n\nFields:\n\n``set``: variable or dictionary of variables\n On a *variable name*, set the value of the given variable to\n ``True``.\n\n On a *dictionary*, set the values of the given variables.\n\nExample::\n\n title: MySQL Tests\n tests:\n - set: MYSQL\n - set:\n MYSQL_HOST: localhost\n MYSQL_PORT: 3306\n unless: [MYSQL_HOST, MYSQL_PORT]\n - py: |\n # Determine the version of the MySQL server\n import MySQLdb\n connection = MySQLdb.connect(host=__pbbt__['MYSQL_HOST'],\n port=int(__pbbt__['MYSQL_PORT']),\n db='mysql')\n cursor = connection.cursor()\n cursor.execute(\"\"\"SELECT VERSION()\"\"\")\n version_string = cursor.fetchone()[0]\n version = tuple(map(int, version_string.split('-')[0].split('.')))\n __pbbt__['MYSQL_VERSION'] = version\n - py: test-ddl.py\n - py: test-dml.py\n - py: test-select.py\n - py: test-new-features.py\n if: MYSQL_VERSION >= (5, 5)\n\nInclude Test\n------------\n\nThis test case loads and executes a test case from a file.\n\nFields:\n\n``include``: path\n The file to load. The file should contain an input test record in\n YAML format.\n\nExample::\n\n title: All Tests\n tests:\n - include: test/core.yaml\n - include: test/ext.yaml\n - include: test/sqlite.yaml\n - include: test/pgsql.yaml\n - include: test/mysql.yaml\n\nPython Code\n-----------\n\nThis test case executes Python code and produces ``stdout``.\n\nFields:\n\n``py``: path or Python code\n On *Python code*, the source code to execute.\n\n On a *file name*, the file which contains source code to execute.\n\n``stdin``: text (optional)\n Content of the standard input.\n\n``except``: exception type (optional)\n If set, indicates that the code is expected to raise an exception of\n the given type.\n\nExample::\n\n title: Python tests\n tests:\n - py: hello.py\n - py: &sum |\n # Sum of two numbers\n import sys\n a = int(sys.stdin.readline())\n b = int(sys.stdin.readline())\n c = a+b\n sys.stdout.write(\"%s\\n\" % c)\n stdin: |\n 2\n 2\n - py: *sum\n stdin: |\n 1\n -5\n - py: *sum\n stdin: |\n one\n three\n except: ValueError\n\nNote that we use a YAML anchor (denoted by ``&sum``) and aliases\n(denoted by ``*sum``) to use the same piece of code in several tests.\n\nShell Command\n-------------\n\nThis test case executes a shell command and produces ``stdout``.\n\nFields:\n\n``sh``: command or executable with a list of parameters\n The shell command to execute.\n\n``stdin``: text (optional)\n Content of the standard input.\n\n``cd``: path (optional)\n Change the current working directory to the given path before\n executing the command.\n\n``environ``: dictionary of variables (optional)\n Add the given variables to the command environment.\n\n``exit``: integer (optional)\n The expected exit code; ``0`` by default.\n\nExample::\n\n title: Shell tests\n tests:\n - sh: echo Hello, World!\n - sh: cat\n stdin: |\n Hello, World!\n - sh: [cat, /etc/shadow]\n exit: 1 # Permission denied\n\nWrite to File\n-------------\n\nThis test case creates a file with the given content.\n\nFields:\n\n``write``: path\n The file to create.\n\n``data``: text\n The file content.\n\nExample::\n\n write: test/tmp/data.txt\n data: |\n Hello, World!\n\nRead from File\n--------------\n\nThe output of this test is the content of a file.\n\nFields:\n\n``read``: path\n The file to read.\n\nExample::\n\n read: test/tmp/data.txt\n\nRemove File\n-----------\n\nThis test case removes a file. It is not an error if the file does not\nexist.\n\nFields:\n\n``rm``: path or list of paths\n File(s) to remove.\n\nExample::\n\n rm: test/tmp/data.txt\n\nMake Directory\n--------------\n\nThis test case creates a directory.\n\nParent directories are also created if necessary. It is not an error if\nthe directory already exists.\n\nFields:\n\n``mkdir``: path\n The directory to create.\n\nExample::\n\n mkdir: test/tmp\n\nRemove Directory\n----------------\n\nThis test case removes a directory with all its content.\n\nIt is not an error if the directory does not exist.\n\nFields:\n\n``rmdir``: path\n The directory to delete.\n\nExample::\n\n rmdir: test/tmp\n\nDoctest\n-------\n\nThis test case executes ``doctest`` on a set of files.\n\nFields:\n\n``doctest``: path pattern\n Files with doctest sessions.\n\nExample::\n\n doctest: test/test_*.rst\n\nUnittest\n--------\n\nThis test case executes ``unittest`` test suite.\n\nFields:\n\n``unittest``: path pattern\n Files with unittest tests.\n\nExample::\n\n unittest: test/test_*.py\n\n\nPytest\n------\n\nThis test case executes ``py.test`` test suite.\n\nPackage ``pytest`` from http://pytest.org/ must be installed.\n\nFields:\n\n``pytest``: path pattern\n Files with py.test tests.\n\nExample::\n\n pytest: test/test_*.py\n\nCoverage\n--------\n\nThis test case starts coverage of Python code.\n\nPackage ``coverage`` from http://nedbatchelder.com/code/coverage/ must\nbe installed.\n\nFields:\n\n``coverage``: file name or ``None``\n Path to the configuration file.\n\n``data_file``: file name\n Where to save coverage data.\n\n``timid``: ``false`` or ``true`` (optional)\n Use a simpler trace function.\n\n``branch``: ``false`` or ``true`` (optional)\n Enable branch coverage.\n\n``source``: file paths of package names (optional)\n Source files or packages to measure.\n\n``include``: file patterns\n Files to measure.\n\n``omit``: file patterns\n Files to omit.\n\nExample::\n\n coverage:\n source: pbbt\n branch: true\n\nCoverage check\n--------------\n\nThis test case stops coverage and reports measurement summary.\n\nFields:\n\n``coverage-check``: float\n Expected coverage percentage.\n\nExample::\n\n coverage-check: 99.0\n\nCoverage report\n---------------\n\nThis test case stops coverage and saves coverage report to a file.\n\nFields:\n\n``coverage-report``: directory\n Where to save the report.\n\nExample::\n\n coverage-report: coverage\n\n\nCustom Test Types\n=================\n\nIn this section, we discuss how to add custom test types to PBBT.\n\nSuppose we want to test a SQL database by running a series of SQL\nqueries and validating that we get expected output. To implement this\ntesting scheme, we need a way to:\n\n* open a connection to the database;\n* execute a SQL statement and produce a sequence of records.\n\nThe input file may look like this::\n\n title: Database tests\n tests:\n # Remove the database file left after the last testing session.\n - rm: test.sqlite\n # Create a new SQLite database.\n - connect: test.sqlite\n # Run a series of SQL commands.\n - sql: |\n SELECT 'Hello, World!';\n - sql: |\n CREATE TABLE student (\n id INTEGER PRIMARY KEY,\n name TEXT NOT NULL,\n gender CHAR(1) NOT NULL\n CHECK (gender in ('f', 'i', 'm')),\n dob DATE NOT NULL\n );\n - sql: |\n INSERT INTO student (id, name, gender, dob)\n VALUES (1001, 'Linda Wright', 'f', '1988-10-03'),\n (1002, 'Beth Thompson', 'f', '1988-01-24'),\n (1003, 'Mark Melton', 'm', '1984-06-05'),\n (1004, 'Judy Flynn', 'f', '1986-09-02'),\n (1005, 'Jonathan Bouchard', 'm', '1982-02-12');\n - sql: |\n SELECT *\n FROM student\n ORDER BY dob;\n - sql: |\n SELECT name\n FROM student\n WHERE id = 1003;\n\nIn this input file, we use two new types of test cases:\n\n``connect``\n Specifies the connection to a SQLite database.\n``sql``\n Specifies a SQL statement to execute.\n\nWe will write a PBBT extension implementing these test types.\n\nCreate a file ``sql.py`` with the following content::\n\n from pbbt import Test, Field, BaseCase, MatchCase, listof\n import sqlite3, traceback, csv, StringIO\n\n @Test\n class ConnectCase(BaseCase):\n\n class Input:\n connect = Field(str)\n\n def check(self):\n self.state['connect'] = None\n try:\n self.state['connect'] = sqlite3.connect(self.input.connect)\n except:\n self.ui.literal(traceback.format_exc())\n self.ui.warning(\"exception occurred while\"\n \" connecting to the database\")\n self.ctl.failed()\n else:\n self.ctl.passed()\n\n @Test\n class SQLCase(MatchCase):\n\n class Input:\n sql = Field(str)\n\n class Output:\n sql = Field(str)\n rows = Field(listof(listof(object)))\n\n def render(self, output):\n stream = StringIO.StringIO()\n writer = csv.writer(stream, lineterminator='\\n')\n writer.writerows(output.rows)\n return stream.getvalue()\n\n def run(self):\n connection = self.state.get('connect')\n if not connection:\n self.ui.warning(\"database connection is not defined\")\n return\n rows = []\n try:\n cursor = connection.cursor()\n cursor.execute(self.input.sql)\n for row in cursor.fetchall():\n rows.append(list(row))\n except:\n self.ui.literal(traceback.format_exc())\n self.ui.warning(\"exception occurred while\"\n \" executing a SQL query\")\n connection.rollback()\n new_output = None\n else:\n connection.commit()\n new_output = self.Output(sql=self.input.sql, rows=rows)\n return new_output\n\nTo use this extension, add parameter ``-E sql.py`` to all PBBT\ninvocations. For example::\n\n $ pbbt -E sql.py input.yaml output.yaml --train\n\nNow we will explain the content of ``sql.py`` line by line.\n\nThe first line imports some classes and decorators we will use::\n\n from pbbt import Test, Field, BaseCase, MatchCase\n\nTo register a test type, use ``@Test`` decorator. Here is a most\ngeneral template::\n\n @Test\n class CustomCase(object):\n\n class Input:\n some_field = Field(...)\n [...]\n\n class Output:\n some_field = Field(...)\n [...]\n\n def __init__(self, ctl, input, output):\n self.ctl = ctl\n self.input = input\n self.output = output\n\n def __call__(self):\n [...]\n return new_output\n\nThe argument of the decorator must be a class that adheres the following\nrules:\n\n* The structure of input and output records is described with nested\n classes ``Input`` and ``Output``. Record fields are specified using\n ``Field`` descriptor.\n\n* To create a test case, the test harness makes an instance of the class.\n The class constructor accepts three arguments:\n\n ``ctl``\n Test harness controller. It is used for user interaction,\n reporting test success or failure, and as a storage for\n conditional variables.\n ``input``\n The input record.\n ``output``\n The expected output record or ``None``.\n\n* The test case is executed by calling its ``__call__()`` method. This\n method must run the test case, generate a new output record, and compare\n it with the given expected output record.\n\n If the expected and actual output record do not coincide:\n\n * In the *check* mode, the method must report a failure.\n * In the *train* mode, the method may ask the user for permission to\n update expected output. If expected output is to be updated, the\n method should return the new output record.\n\nPBBT also provides two mixin classes: ``BaseCase`` and ``MatchCase``.\nThese classes implement most of the necessary functionality for common\ntypes of tests.\n\nLet's review ``connect`` test type, which is implemented by\n``ConnectCase`` class::\n\n @Test\n class ConnectCase(BaseCase):\n\n class Input:\n connect = Field(str)\n\n def check(self):\n [...]\n\n``ConnectCase`` is inherited from ``BaseCase``, which is suitable for\ntest which produce no output data and are executed for their side\neffects. The nested ``Input`` class definition is used to declare the\nfields of the input record. In this case, the input record has just one\ntext field ``connect``, which contains the name of the database.\n\nTest types inherited from ``BaseCase`` must override abstract method\n``check()``::\n\n import sqlite3, traceback\n [...]\n\n @Test\n class ConnectCase(BaseCase):\n [...]\n\n def check(self):\n self.state['connect'] = None\n try:\n self.state['connect'] = sqlite3.connect(self.input.connect)\n except:\n self.ui.literal(traceback.format_exc())\n self.ui.warning(\"exception occurred while\"\n \" connecting to the database\")\n self.ctl.failed()\n else:\n self.ctl.passed()\n\nThis code attempts to create a new database connection and store it as a\nconditional variable ``connect``. If the attempt fails, it calls\n``ui.literal()`` to display the exception traceback and ``ctl.failed()``\nto report test failure. Otherwise, ``ctl.passed()`` is called to\nindicate that the test succeeded.\n\nNext, let's review ``sql`` test type::\n\n @Test\n class SQLCase(MatchCase):\n\n class Input:\n sql = Field(str)\n\n class Output:\n sql = Field(str)\n rows = Field(listof(listof(object)))\n\n def run(self):\n [...]\n\n def render(self, output):\n [...]\n\nThis test type has both input and output records, which are described\nwith ``Input`` and ``Output`` nested classes. The input record contains\none field ``sql``, a SQL query to execute. The output record contains\ntwo fields: ``sql`` and ``rows``. Field ``sql`` contains the same SQL\nquery and is used for matching the output record with the complementary\ninput record. Field ``rows`` contains a list of output rows generated\nby the SQL query.\n\n``SQLCase`` is inherited from ``MatchCase``, which is a mixin class\nfor test types that produce text output. Test types inherited from\n``MatchCase`` must override two methods:\n\n``run()``\n Executes the test case and returns the produced output record.\n``render(output)``\n Convert an output record to printable form.\n\nIn ``SQLCase``, ``render()`` is implemented by converting output rows to\nCSV format::\n\n import csv, StringIO\n [...]\n\n @Test\n class SQLCase(MatchCase):\n [...]\n\n def render(self, output):\n stream = StringIO.StringIO()\n writer = csv.writer(stream, lineterminator='\\n')\n writer.writerows(output.rows)\n return stream.getvalue()\n\nMethod ``run()`` is implemented as follows::\n\n @Test\n class SQLCase(MatchCase):\n [...]\n\n def run(self):\n connection = self.state.get('connect')\n if not connection:\n self.ui.warning(\"database connection is not defined\")\n return\n rows = []\n try:\n cursor = connection.cursor()\n cursor.execute(self.input.sql)\n for row in cursor.fetchall():\n rows.append(list(row))\n except:\n self.ui.literal(traceback.format_exc())\n self.ui.warning(\"exception occurred while\"\n \" executing a SQL query\")\n connection.rollback()\n new_output = None\n else:\n connection.commit()\n new_output = self.Output(sql=self.input.sql, rows=rows)\n return new_output\n\n\nCommand-line Interface\n======================\n\nUsage::\n\n pbbt [] INPUT [OUTPUT]\n\nHere, ``INPUT`` and ``OUTPUT`` are files which contain input and output\ntest data respectively.\n\nThe following options are available:\n\n``-h``, ``--help``\n Display usage information and exit.\n\n``-q``, ``--quiet``\n Show only warnings and errors.\n\n``-T``, ``--train``\n Run tests in the training mode.\n\n``-P``, ``--purge``\n Purge stale output data.\n\n``-M N``, ``--max-errors N``\n Halt after ``N`` tests failed; ``0`` means \"never\".\n\n``-D VAR``, ``-D VAR=VALUE``, ``--define VAR``, ``--define VAR=VALUE``\n Set a conditional variable.\n\n``-E FILE``, ``-E MODULE``, ``--extend FILE``, ``--extend MODULE``\n Load a PBBT extension from a file or a Python module.\n\n``-S ID``, ``--suite ID``\n Run a specific test suite.\n\nPBBT can also read configuration from the following files:\n\n``setup.cfg``\n This file is in INI format with PBBT settings defined in section\n ``[pbbt]``. The following parameters are recognized: ``extend``,\n ``input``, ``output``, ``define``, ``suite``, ``train``, ``purge``,\n ``max_errors``, ``quiet``.\n\n``pbbt.yaml``\n This file must be a YAML file with the following keys: ``extend``,\n ``input``, ``output``, ``define``, ``suite``, ``train``, ``purge``,\n ``max-errors``, ``quiet``.\n\nPBBT can also be executed as a Distutils command::\n\n python setup.py pbbt\n\nIn this case, PBBT configuration could be specified in ``setup.cfg``\nor via command-line parameters.\n\n\nAPI Reference\n=============\n\n``pbbt.maybe(T)``\n Pseudo-type for ``isinstance(X, ...)``: checks if ``X`` is an\n instance of ``T`` or equal to ``None``.\n\n``pbbt.oneof(T1, T2, ...)``\n Pseudo-type for ``isinstance(X, ...)``: checks if ``X`` is an\n instance of ``T1`` or an instance of ``T2``, etc.\n\n``pbbt.choiceof(values)``\n Pseudo-type for ``isinstance(X, ...)``: checks if ``X`` is equal to\n one of the given values.\n\n``pbbt.listof(T, length=None)``\n Pseudo-type for ``isinstance(X, ...)``: checks if ``X`` is a list of\n elements of ``T``.\n\n``pbbt.tupleof(T1, T2, ...)``\n Pseudo-type for ``isinstance(X, ...)``: checks if ``X`` is a tuple\n with fields of types ``T1``, ``T2``, etc.\n\n``pbbt.dictof(T1, T2)``\n Pseudo-type for ``isinstance(X, ...)``: checks if ``X`` is a\n dictionary with keys of type ``T1`` and values of type ``T2``.\n\n``pbbt.raises(E)``\n Use with ``with`` clause to verify that the nested block raises an\n exception of the given type.\n\n``pbbt.raises(E, fn, *args, **kwds)``\n Verifies that the function call ``fn(*args, **kwds)`` raises an\n exception of the given type.\n\n``pbbt.Test(CaseType)``\n Registers the given class as a test type.\n\n``pbbt.Field(check=None, default=REQUIRED, order=None, hint=None)``\n Describes a field of an input or an output record.\n\n ``check``\n The type of the field value.\n\n ``default``\n Default value if the field is missing. If not set, the\n field is mandatory.\n\n ``order``\n If set, allows you to override the default field order.\n\n ``hint``\n Short description of the field.\n\n``pbbt.Record(*p_fields, **kv_fields)``\n Base class for all input and output records.\n\n The ``Test`` decorator converts nested ``Input`` and ``Output``\n classes to ``Record`` subclasses.\n\n The following methods could be used or overridden:\n\n ``classmethod __recognizes__(keys)``\n Checks if the set of keys is compatible with the record type.\n\n The default implementation checks if the set of keys contains\n all mandatory record fields.\n\n ``classmethod __load__(mapping)``\n Generates a record instance from a mapping of record keys and\n values.\n\n ``__dump__()``\n Generates a list of field keys and values.\n\n ``__complements__(other)``\n Checks if two records are complementary input and output records\n for the same test case.\n\n ``__clone__(**kv_fields)``\n Makes a copy of the record with new values for the given fields.\n\n ``__str__()``\n Generates a printable representation.\n\n``pbbt.Control(...)``\n Test harness.\n\n The test harness object is passed to test cases as the first\n argument of the constructor. The following methods and attributes\n could be used by test case objects.\n\n Attributes:\n\n ``ui``\n Provides user interaction services.\n\n ``training``\n If set, the harness is in training mode.\n\n ``purging``\n If set, the harness is in purging mode.\n\n ``quiet``\n If set, only warnings and errors are displayed.\n\n ``halted``\n If set, the test process has been halted.\n\n ``state``\n Conditional variables.\n\n ``selection``\n Selected suites.\n\n Methods:\n\n ``passed(text=None)``\n Attests that the current test case has passed.\n\n ``failed(text=None)``\n Attests that the current test case has failed.\n\n ``updated(text=None)``\n Attests that output data for the current test case has been\n updated.\n\n ``halt(text=None)``\n Halts testing process.\n\n ``load_input(path)``\n Loads input test data from the given file.\n\n ``load_output(path)``\n Loads output test data from the given file.\n\n ``dump_output(path, data)``\n Saves output test data to the given file.\n\n ``run(case)``\n Executes a test case.\n\n ``__call__(input_path, output_path)``\n Runs testing process with the given input and output.\n\n``pbbt.locate(record)``\n Finds the location of the given record.\n\n``pbbt.Location(filename, line)``\n Position of an input or output record in the YAML document.\n\n``pbbt.run(input_path, output_path=None, **configuration)``\n Loads test data from the given files and runs the tests.\n\n Configuration:\n\n ``ui``\n User interaction controller.\n ``variables``\n Predefined conditional variables.\n ``targets``\n Selected suites.\n ``training``\n Set the harness into training mode.\n ``purging``\n Purge stale output data.\n ``max_errors``\n Maximum permitted number of failures before the harness halts.\n ``quiet``\n Display only warnings and errors.\n\n``pbbt.main()``\n Implements the ``pbbt`` command-line utility.\n\n``pbbt.BaseCase``\n Template class suitable for most test types.\n\n Subclasses need to override the following methods:\n\n ``check()``\n Runs the test case in check mode.\n\n ``train()``\n Runs the test case in train mode. The default implementation\n simply calls ``check()``.\n\n``pbbt.MatchCase``\n Template class for test types which produce output.\n\n Subclasses need to override the following methods:\n\n ``run()``\n Executes the case; returns produced output.\n\n ``render(output)``\n Converts output record to text.\n\n``pbbt.UI``\n Abstract class for user interaction services.\n\n Methods:\n\n ``part()``\n Starts a new section.\n\n ``section()``\n Starts a subsection.\n\n ``header(text)``\n Shows a section header.\n\n ``notice(text)``\n Shows a notice.\n\n ``warning(text)``\n Shows a warning.\n\n ``error(text)``\n Shows an error.\n\n ``literal(text)``\n Shows a literal text.\n\n ``choice(text, *choices)``\n Asks a single-choice question.\n\n``pbbt.ConsoleUI(stdin=None, stdout=None, stderr=None)``\n Implements ``UI`` for console input/output.\n\n``pbbt.SilentUI(backend)``\n Implements ``UI`` for use with ``--quiet`` option.\n\n\n.. _YAML: http://yaml.org/\n.. _Prometheus Research, LLC: http://prometheusresearch.com/\n.. _pip: http://pip-installer.org/\n.. _Python Package Index: http://pypi.python.org/pypi/pbbt\n.. _JSON: http://json.org/\n\n.. vim: set spell spelllang=en textwidth=72:\n", "description_content_type": "", "docs_url": null, "download_url": "http://pypi.python.org/pypi/pbbt", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://bitbucket.org/prometheus/pbbt", "keywords": "", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "pbbt", "package_url": "https://pypi.org/project/pbbt/", "platform": "", "project_url": "https://pypi.org/project/pbbt/", "project_urls": { "Download": "http://pypi.python.org/pypi/pbbt", "Homepage": "http://bitbucket.org/prometheus/pbbt" }, "release_url": "https://pypi.org/project/pbbt/0.1.6/", "requires_dist": null, "requires_python": "", "summary": "Pluggable Black-Box Testing toolkit", "version": "0.1.6" }, "last_serial": 4220211, "releases": { "0.1.1": [ { "comment_text": "", "digests": { "md5": "4f674a3b9865784fe9db4304c05cdf9f", "sha256": "c6948306b4b44a8e1cad9a63e609f59122c58b6f7a1c235ddf44bcda9007b120" }, "downloads": -1, "filename": "pbbt-0.1.1.tar.gz", "has_sig": false, "md5_digest": "4f674a3b9865784fe9db4304c05cdf9f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 39125, "upload_time": "2013-05-03T19:52:30", "url": "https://files.pythonhosted.org/packages/99/41/0f7d7ca301bca1e03bb5d5a936f1d7c33e8dd307b6f9cae4f861baad9b5c/pbbt-0.1.1.tar.gz" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "a51389a8e9f6ddaad905a5b705f5d20d", "sha256": "7522c03327221381d2611cd88651798e80ce275fc5ea525a153f0f6af944a78d" }, "downloads": -1, "filename": "pbbt-0.1.2.tar.gz", "has_sig": false, "md5_digest": "a51389a8e9f6ddaad905a5b705f5d20d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 46346, "upload_time": "2013-10-11T18:38:55", "url": "https://files.pythonhosted.org/packages/b7/ff/a576e53c8ea578b61c06561c09aa605eb79e0fb715a25bb29f91002f548e/pbbt-0.1.2.tar.gz" } ], "0.1.3": [ { "comment_text": "", "digests": { "md5": "d420ab2c60c15436ed1f31c3b19b51fa", "sha256": "8dd3b7b62562e86483fc9c0582b0c721894c9a1c2c6e904a1a87b5af6135aa92" }, "downloads": -1, "filename": "pbbt-0.1.3.tar.gz", "has_sig": false, "md5_digest": "d420ab2c60c15436ed1f31c3b19b51fa", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 46385, "upload_time": "2015-01-18T01:38:55", "url": "https://files.pythonhosted.org/packages/e0/ce/6bf808f82e5c213c9a7335b5c9554c79c69ea81b2c8cf1fdfd8e1ed3c106/pbbt-0.1.3.tar.gz" } ], "0.1.4": [ { "comment_text": "", "digests": { "md5": "e2e5b00416600f593b7db4b86f5a2618", "sha256": "292d1e7bb3e8f41f9806ab75e24ddb05035e6e35c5a5d269c5d21485f69d912b" }, "downloads": -1, "filename": "pbbt-0.1.4.tar.gz", "has_sig": false, "md5_digest": "e2e5b00416600f593b7db4b86f5a2618", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 46396, "upload_time": "2015-09-23T14:08:55", "url": "https://files.pythonhosted.org/packages/97/bf/07778f2ba204e7d7db492d28f9ef91712e113cfd5858be409e47020c92bc/pbbt-0.1.4.tar.gz" } ], "0.1.5": [ { "comment_text": "", "digests": { "md5": "5ebe6fe53fe5a5194ed22b6a1027899a", "sha256": "62faae1483dc6255851e2cae7762bca4e9a4c09263ff2f80320fab296fb26654" }, "downloads": -1, "filename": "pbbt-0.1.5.tar.gz", "has_sig": false, "md5_digest": "5ebe6fe53fe5a5194ed22b6a1027899a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 46417, "upload_time": "2017-01-10T16:29:55", "url": "https://files.pythonhosted.org/packages/4b/8d/aa334188bc5e5d4093caf6c0181e6d148bc1d6b085ddd3da90a0023e6b0a/pbbt-0.1.5.tar.gz" } ], "0.1.6": [ { "comment_text": "", "digests": { "md5": "df15f06751b69a470a912485aca39d05", "sha256": "643f34429e2e17bba814b2cb2cab6785ffe1f07f662303861f0005b108e69715" }, "downloads": -1, "filename": "pbbt-0.1.6.tar.gz", "has_sig": false, "md5_digest": "df15f06751b69a470a912485aca39d05", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 46915, "upload_time": "2018-08-29T20:05:47", "url": "https://files.pythonhosted.org/packages/32/5f/3bfe6181c381fd54c7cbe374a5786ac2b10f0b802c63180bc7b98a6122ea/pbbt-0.1.6.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "df15f06751b69a470a912485aca39d05", "sha256": "643f34429e2e17bba814b2cb2cab6785ffe1f07f662303861f0005b108e69715" }, "downloads": -1, "filename": "pbbt-0.1.6.tar.gz", "has_sig": false, "md5_digest": "df15f06751b69a470a912485aca39d05", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 46915, "upload_time": "2018-08-29T20:05:47", "url": "https://files.pythonhosted.org/packages/32/5f/3bfe6181c381fd54c7cbe374a5786ac2b10f0b802c63180bc7b98a6122ea/pbbt-0.1.6.tar.gz" } ] }