{
    "info": {
        "author": "Inktank Storage, Inc.",
        "author_email": "ceph-qa@ceph.com",
        "bugtrack_url": null,
        "classifiers": [
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Natural Language :: English",
            "Operating System :: POSIX :: Linux",
            "Programming Language :: Python :: 2.7",
            "Topic :: Software Development :: Quality Assurance",
            "Topic :: Software Development :: Testing",
            "Topic :: System :: Distributed Computing",
            "Topic :: System :: Filesystems"
        ],
        "description": "==================================================\n `Teuthology` -- The Ceph integration test runner\n==================================================\n\nThe Ceph project needs automated tests. Because Ceph is a highly\ndistributed system, and has active kernel development, its testing\nrequirements are quite different from e.g. typical LAMP web\napplications. Nothing out there seemed to handle our requirements,\nso we wrote our own framework, called `Teuthology`.\n\n\nOverview\n========\n\nTeuthology runs a given set of Python functions (`tasks`), with an SSH\nconnection to every host participating in the test. The SSH connection\nuses `Paramiko <http://www.lag.net/paramiko/>`__, a native Python\nclient for the SSH2 protocol, and this allows us to e.g. run multiple\ncommands inside a single SSH connection, to speed up test\nexecution. Tests can use `gevent <http://www.gevent.org/>`__ to\nperform actions concurrently or in the background.\n\n\nBuild\n=====\nTeuthology is not meant to be distributed as a library, therefore we depend\non the pinned dependencies listed in ``requirements.txt``, the ``setup.py``\nwill not list any and will only be there to install the package entry points\n(a.k.a teuthology's scripts).\n\n    git clone https://github.com/ceph/teuthology.git\n    cd teuthology\n\n\nBootstrap for Ubuntu Systems\n----------------------------\nA ``boostrap`` script is provided for automated builds/execution of teuthology\nitself. You can run it directly **only if you are using Ubuntu**.\n\nTeuthology uses several Python packages that are not in the standard\nlibrary. To make the dependencies easier to get right, we use a\n`virtualenv` to manage them. To get started, ensure you have the\n``virtualenv`` and ``pip`` programs installed; e.g. on Debian/Ubuntu::\n\n    sudo apt-get install python-dev python-virtualenv python-pip libevent-dev libmysqlclient-dev python-libvirt\n\nand then run::\n\n    ./bootstrap\n\n\nMacOS X\n-------\n\n.. note:: These instructions assume you are using `homebrew <http://brew.sh/>`_\n\nAs always, create a ``virtualenv`` specific to teuthology and make sure it\nis activated before proceeding (location doesn't matter, we use an example\nlocation)::\n\n    mkdir ~/.virtualenvs\n    virtualenv --system-site-packages ~/.virtualenvs/teuthology\n    source ~/.virtualenvs/teuthology/bin/activate\n\nInstall the system dependencies::\n\n    brew install libvirt mysql libevent\n\nMake sure you are able to import ``libvirt`` without error::\n\n    python -c \"import libvirt\"\n\nIf python can't find libvirt yet, you may need to do the following:\n\n    cd /Library/Python/{pyversion}/site-packages\n    sudo ln -s /usr/local/Cellar/libvirt/{version}/lib/python{pyversion}/site-packages/* .\n\nFinally, install the teuthology package and ``requirements.txt``::\n\n    python setup.py develop\n    pip install -r requirements.txt\n\n\nGeneric install\n---------------\nThese instructions should help get teuthology installed properly in\na system that is not OSX or Debian-based.\n\nInstall all the system dependencies needed:\n\n* mysql client\n* libevent\n* libvirt (with the Python bindings)\n\nInstall Python packaging tools:\n\n* pip\n* virtualenv\n\nIn some cases, depending on the OS, you will need a python development package\nwith some build helpers that are required to build packages. In Ubuntu, this is\nthe ``python-dev`` package.\n\nWith a dedicated ``virtualenv`` activated, install the teuthology package and\n``requirements.txt``::\n\n    python setup.py develop\n    pip install -r requirements.txt\n\n\nTest configuration\n==================\n\nAn integration test run takes three items of configuration:\n\n- ``targets``: what hosts to run on; this is a dictionary mapping\n  hosts to ssh host keys, like:\n  \"username@hostname.example.com: ssh-rsa long_hostkey_here\"\n- ``roles``: how to use the hosts; this is a list of lists, where each\n  entry lists all the roles to be run on a single host; for example, a\n  single entry might say ``[mon.1, osd.1]``\n- ``tasks``: how to set up the cluster and what tests to run on it;\n  see below for examples\n\nThe format for this configuration is `YAML <http://yaml.org/>`__, a\nstructured data format that is still human-readable and editable.\n\nFor example, a full config for a test run that sets up a three-machine\ncluster, mounts Ceph via ``ceph-fuse``, and leaves you at an interactive\nPython prompt for manual exploration (and enabling you to SSH in to\nthe nodes & use the live cluster ad hoc), might look like this::\n\n    roles:\n    - [mon.0, mds.0, osd.0]\n    - [mon.1, osd.1]\n    - [mon.2, client.0]\n    targets:\n        ubuntu@host07.example.com: ssh-rsa host07_ssh_key\n        ubuntu@host08.example.com: ssh-rsa host08_ssh_key\n        ubuntu@host09.example.com: ssh-rsa host09_ssh_key\n    tasks:\n    - install:\n    - ceph:\n    - ceph-fuse: [client.0]\n    - interactive:\n\nThe number of entries under ``roles`` and ``targets`` must match.\n\nNote the colon after every task name in the ``tasks`` section.\n\nThe ``install`` task needs to precede all other tasks.\n\nThe listed targets need resolvable hostnames. If you do not have a DNS server\nrunning, you can add entries to ``/etc/hosts``. You also need to be able to SSH\nin to the listed targets without passphrases, and the remote user needs to have\npasswordless `sudo` access. Note that the ssh keys at the end of the\n``targets`` entries are the public ssh keys for the hosts.  On Ubuntu, these\nare located at /etc/ssh/ssh_host_rsa_key.pub\n\nIf you'd save the above file as ``example.yaml``, you could run\nteuthology on it by saying::\n\n    ./virtualenv/bin/teuthology example.yaml\n\nYou can also pass the ``-v`` option, for more verbose execution. See\n``teuthology --help`` for more.\n\n\nMultiple config files\n---------------------\n\nYou can pass multiple files as arguments to teuthology. Each one\nwill be read as a config file, and their contents will be merged. This\nallows you to e.g. share definitions of what a \"simple 3 node cluster\"\nis. The source tree comes with ``roles/3-simple.yaml``, so we could\nskip the ``roles`` section in the above ``example.yaml`` and then\nrun::\n\n    ./virtualenv/bin/teuthology roles/3-simple.yaml example.yaml\n\n\nReserving target machines\n-------------------------\n\nBefore locking machines will work, you must create a .teuthology.yaml\nfile in your home directory that sets a lock_server, i.e.::\n\n    lock_server: http://host.example.com:8080/lock\n\nTeuthology automatically locks nodes for you if you specify the\n``--lock`` option. Without this option, you must specify machines to\nrun on in a ``targets.yaml`` file, and lock them using\nteuthology-lock.\n\nNote that the default owner of a machine is ``USER@HOST``.\nYou can override this with the ``--owner`` option when running\nteuthology or teuthology-lock.\n\nWith teuthology-lock, you can also add a description, so you can\nremember which tests you were running on them. This can be done when\nlocking or unlocking machines, or as a separate action with the\n``--update`` option. To lock 3 machines and set a description, run::\n\n    ./virtualenv/bin/teuthology-lock --lock-many 3 --desc 'test foo'\n\nIf machines become unusable for some reason, you can mark them down::\n\n    ./virtualenv/bin/teuthology-lock --update --status down machine1 machine2\n\nTo see the status of all machines, use the ``--list`` option. This can\nbe restricted to particular machines as well::\n\n    ./virtualenv/bin/teuthology-lock --list machine1 machine2\n\n\nTasks\n=====\n\nA task is a Python module in the ``teuthology.task`` package, with a\ncallable named ``task``. It gets the following arguments:\n\n- ``ctx``: a context that is available through the lifetime of the\n  test run, and has useful attributes such as ``cluster``, letting the\n  task access the remote hosts. Tasks can also store their internal\n  state here. (TODO beware namespace collisions.)\n- ``config``: the data structure after the colon in the config file,\n  e.g. for the above ``ceph-fuse`` example, it would be a list like\n  ``[\"client.0\"]``.\n\nTasks can be simple functions, called once in the order they are\nlisted in ``tasks``. But sometimes, it makes sense for a task to be\nable to clean up after itself; for example, unmounting the filesystem\nafter a test run. A task callable that returns a Python `context\nmanager\n<http://docs.python.org/library/stdtypes.html#typecontextmanager>`__\nwill have the manager added to a stack, and the stack will be unwound\nat the end of the run. This means the cleanup actions are run in\nreverse order, both on success and failure. A nice way of writing\ncontext managers is the ``contextlib.contextmanager`` decorator; look\nfor that string in the existing tasks to see examples, and note where\nthey use ``yield``.\n\nFurther details on some of the more complex tasks such as install or workunit\ncan be obtained via python help. For example::\n\n    >>> import teuthology.task.workunit\n    >>> help(teuthology.task.workunit)\n\ndisplays a page of more documentation and more concrete examples.\n\nSome of the more important / commonly used tasks include:\n\n* ``chef``: Run the chef task.\n* ``install``: by default, the install task goes to gitbuilder and installs the\n  results of the latest build. You can, however, add additional parameters to\n  the test configuration to cause it to install any branch, SHA, archive or\n  URL. The following are valid parameters.\n\n- ``branch``: specify a branch (bobtail, cuttlefish...)\n- ``flavor``: specify a flavor (next, unstable...). Flavors can be thought of\n  as subsets of branches.  Sometimes (unstable, for example) they may have a\n  predefined meaning.\n- ``project``: specify a project (ceph, samba...)\n- ``sha1``: install the build with this sha1 value.\n- ``tag``: specify a tag/identifying text for this build (v47.2, v48.1...)\n* ``ceph``: Bring up Ceph\n\n* ``overrides``: override behavior. Typically, this includes sub-tasks being\n  overridden. Sub-tasks can nest further information.  For example, overrides\n  of install tasks are project specific, so the following section of a yaml\n  file would cause all ceph installation to default into using the cuttlefish\n  branch::\n\n    overrides:\n      install:\n        ceph:\n          branch: cuttlefish\n\n* ``workunit``: workunits are a way of grouping tasks and behavior on targets.\n* ``sequential``: group the sub-tasks into a unit where the sub-tasks run\n  sequentially as listed.\n* ``parallel``: group the sub-tasks into a unit where the sub-task all run in\n  parallel.\n\nSequential and parallel tasks can be nested.  Tasks run sequentially if not\nspecified.\n\nThe above list is a very incomplete description of the tasks available on\nteuthology. The teuthology/task subdirectory contains all the python files\nthat implement tasks.\nMany of these tasks are used to run shell scripts that are defined in the\nceph/ceph-qa-suite.\n\nIf machines were locked as part of the run (with the --lock switch), \nteuthology normally leaves them locked when there is any task failure\nfor investigation of the machine state.  When developing new teuthology\ntasks, sometimes this behavior is not useful.  The ``unlock_on_failure``\nglobal option can be set to True to make the unlocking happen unconditionally.\n\nTroubleshooting\n===============\n\nSometimes when a bug triggers, instead of automatic cleanup, you want\nto explore the system as is. Adding a top-level::\n\n    interactive-on-error: true\n\nas a config file for teuthology will make that possible. With that\noption, any *task* that fails, will have the ``interactive`` task\ncalled after it. This means that before any cleanup happens, you get a\nchance to inspect the system -- both through Teuthology and via extra\nSSH connections -- and the cleanup completes only when you choose so.\nJust exit the interactive Python session to continue the cleanup.\n\nNote that this only catches exceptions *between* the tasks. If a task\ncalls multiple subtasks, e.g. with ``contextutil.nested``, those\ncleanups *will* be performed. Later on, we can let tasks communicate\nthe subtasks they wish to invoke to the top-level runner, avoiding\nthis issue.\n\nInteractive task facilities\n===========================\nThe ``interactive`` task presents a prompt for you to interact with the\nteuthology configuration.  The ``ctx`` variable is available to explore,\nand a ``pprint.PrettyPrinter().pprint`` object is added for convenience as\n'pp', so you can do things like pp(dict-of-interest) to see a formatted\nview of the dict.\n\nTest Sandbox Directory\n======================\n\nTeuthology currently places most test files and mount points in a\nsandbox directory, defaulting to ``/home/$USER/cephtest``.  To change\nthe location of the sandbox directory, the following option can be\nspecified in ``$HOME/.teuthology.yaml``::\n\n    test_path: <directory>\n\n\nVIRTUAL MACHINE SUPPORT\n=======================\n\nTeuthology also supports virtual machines, which can function like\nphysical machines but differ in the following ways:\n\nVPSHOST:\n--------\n\nA new entry, vpshost, has been added to the teuthology database of\navailable machines.  For physical machines, this value is null. For\nvirtual machines, this entry is the name of the physical machine that\nthat virtual machine resides on.\n\nThere are fixed \"slots\" for virtual machines that appear in the teuthology\ndatabase.  These slots have a machine type of vps and can be locked like\nany other machine.  The existence of a vpshost field is how teuthology\nknows whether or not a database entry represents a physical or a virtual\nmachine.\n\nThe following needs to be set in ~/.config/libvirt/libvirt.conf in order to get the\nright virtual machine associations for the Inktank lab::\n\n    uri_aliases = [\n        'mira001=qemu+ssh://ubuntu@mira001.front.sepia.ceph.com/system?no_tty=1',\n        'mira003=qemu+ssh://ubuntu@mira003.front.sepia.ceph.com/system?no_tty=1',\n        'mira004=qemu+ssh://ubuntu@mira004.front.sepia.ceph.com/system?no_tty=1',\n        'mira006=qemu+ssh://ubuntu@mira006.front.sepia.ceph.com/system?no_tty=1',\n        'mira007=qemu+ssh://ubuntu@mira007.front.sepia.ceph.com/system?no_tty=1',\n        'mira008=qemu+ssh://ubuntu@mira008.front.sepia.ceph.com/system?no_tty=1',\n        'mira009=qemu+ssh://ubuntu@mira009.front.sepia.ceph.com/system?no_tty=1',\n        'mira010=qemu+ssh://ubuntu@mira010.front.sepia.ceph.com/system?no_tty=1',\n        'mira011=qemu+ssh://ubuntu@mira011.front.sepia.ceph.com/system?no_tty=1',\n        'mira013=qemu+ssh://ubuntu@mira013.front.sepia.ceph.com/system?no_tty=1',\n        'mira014=qemu+ssh://ubuntu@mira014.front.sepia.ceph.com/system?no_tty=1',\n        'mira015=qemu+ssh://ubuntu@mira015.front.sepia.ceph.com/system?no_tty=1',\n        'mira017=qemu+ssh://ubuntu@mira017.front.sepia.ceph.com/system?no_tty=1',\n        'mira018=qemu+ssh://ubuntu@mira018.front.sepia.ceph.com/system?no_tty=1',\n        'mira020=qemu+ssh://ubuntu@mira020.front.sepia.ceph.com/system?no_tty=1',\n        'vercoi01=qemu+ssh://ubuntu@vercoi01.front.sepia.ceph.com/system?no_tty=1',\n        'vercoi02=qemu+ssh://ubuntu@vercoi02.front.sepia.ceph.com/system?no_tty=1',\n        'vercoi03=qemu+ssh://ubuntu@vercoi03.front.sepia.ceph.com/system?no_tty=1',\n        'vercoi04=qemu+ssh://ubuntu@vercoi04.front.sepia.ceph.com/system?no_tty=1',\n        'vercoi05=qemu+ssh://ubuntu@vercoi05.front.sepia.ceph.com/system?no_tty=1',\n        'vercoi06=qemu+ssh://ubuntu@vercoi06.front.sepia.ceph.com/system?no_tty=1',\n        'vercoi07=qemu+ssh://ubuntu@vercoi07.front.sepia.ceph.com/system?no_tty=1',\n        'vercoi08=qemu+ssh://ubuntu@vercoi08.front.sepia.ceph.com/system?no_tty=1',\n        'senta01=qemu+ssh://ubuntu@senta01.front.sepia.ceph.com/system?no_tty=1',\n        'senta02=qemu+ssh://ubuntu@senta02.front.sepia.ceph.com/system?no_tty=1',\n        'senta03=qemu+ssh://ubuntu@senta03.front.sepia.ceph.com/system?no_tty=1',\n        'senta04=qemu+ssh://ubuntu@senta04.front.sepia.ceph.com/system?no_tty=1',\n    ]\n\nDOWNBURST:\n----------\n\nWhen a virtual machine is locked, downburst is run on that machine to install a\nnew image.  This allows the user to set different virtual OSes to be installed\non the newly created virtual machine.  Currently the default virtual machine is\nubuntu (precise).  A different vm installation can be set using the\n``--os-type`` option in ``teuthology.lock``.\n\nWhen a virtual machine is unlocked, downburst destroys the image on the\nmachine.\n\nTemporary yaml files are used to downburst a virtual machine.  A typical\nyaml file will look like this::\n\n    downburst:\n      cpus: 1\n      disk-size: 30G\n      distro: centos\n      networks:\n      - {source: front}\n      ram: 4G\n\nThese values are used by downburst to create the virtual machine.\n\nWhen locking a file, a downburst meta-data yaml file can be specified by using\nthe downburst-conf parameter on the command line.\n\nTo find the downburst executable, teuthology first checks the PATH environment\nvariable.  If not defined, teuthology next checks for\nsrc/downburst/virtualenv/bin/downburst executables in the user's home\ndirectory, /home/ubuntu, and /home/teuthology.  This can all be overridden if\nthe user specifies a downburst field in the user's .teuthology.yaml file.\n\nHOST KEYS:\n----------\n\nBecause teuthology reinstalls a new machine, a new hostkey is generated.  After\nlocking, once a connection is established to the new machine,\n``teuthology-lock`` with the ``--list`` or ``--list-targets`` options will\ndisplay the new keys.  When vps machines are locked using the ``--lock-many``\noption, a message is displayed indicating that ``--list-targets`` should be run\nlater.\n\nCEPH-QA-CHEF:\n-------------\n\nOnce teuthology starts after a new vm is installed, teuthology\nchecks for the existence of ``/ceph-qa-ready``.  If this file is not\npresent, ``ceph-qa-chef`` is run when teuthology first comes up.\n\nASSUMPTIONS:\n------------\n\nIt is assumed that downburst is on the user's ``$PATH``.\n\n\nTest Suites\n===========\n\nMost of the current teuthology test suite execution scripts automatically\ndownload their tests from the master branch of the appropriate github\nrepository.  People who want to run experimental test suites usually modify the\ndownload method in the ``teuthology/task`` script to use some other branch or\nrepository. This should be generalized in later teuthology releases.\nTeuthology QA suites can be found in ``src/ceph-qa-suite``. Make sure that this\ndirectory exists in your source tree before running the test suites.\n\nEach suite name is determined by the name of the directory in ``ceph-qa-suite``\nthat contains that suite. The directory contains subdirectories and yaml files,\nwhich, when assembled, produce valid tests that can be run. The test suite\napplication generates combinations of these files and thus ends up running a\nset of tests based off the data in the directory for the suite.\n\nTo run a suite, enter::\n\n    teuthology-suite -s <suite> [-c <ceph>] [-k <kernel>] [-t <teuth>] [-m <mtype>]\n\nwhere:\n\n* ``suite``: the name of the suite (the directory in ceph-qa-suite).\n* ``ceph``: ceph branch to be used.\n* ``kernel``: version of the kernel to be used.\n* ``email``: email address to send the results to.\n* ``flavor``: flavor of the test\n* ``teuth``: version of teuthology to run\n* ``mtype``: machine type of the run\n* ``templates``: template file used for further modifying the suite (optional)\n\nFor example, consider::\n\n     schedule_suite.sh rbd wip-fix cuttlefish bob.smith@foo.com master cuttlefish plana\n\nThe above command runs the rbd suite using wip-fix as the ceph branch, a\nstraight cuttlefish kernel, and the master flavor of cuttlefish teuthology.  It\nwill run on plana machines.\n\nIn order for a queued task to be run, a teuthworker thread on\n``teuthology.front.sepia.ceph.com`` needs to remove the task from the queue.\nOn ``teuthology.front.sepia.ceph.com``, run ``ps aux | grep teuthology-worker``\nto view currently running tasks. If no processes are reading from the test\nversion that you are running, additonal teuthworker tasks need to be started.\nTo start these tasks:\n\n* copy your build tree to ``/home/teuthworker`` on ``teuthology.front.sepia.ceph.com``.\n* Give it a unique name (in this example, xxx)\n* start up some number of worker threads (as many as machines you are testing with, there are 60 running for the default queue)::\n\n    /home/virtualenv/bin/python\n    /var/lib/teuthworker/xxx/virtualenv/bin/teuthworker\n    /var/lib/teuthworker/archive --tube xxx\n    --log-dir /var/lib/teuthworker/archive/worker_logs\n\n    Note: The threads on teuthology.front.sepia.ceph.com are started via\n    ~/teuthworker/start.sh.  You can use that file as a model for your\n    own threads, or add to this file if you want your threads to be\n    more permanent.\n\nOnce the suite completes, an email message is sent to the users specified, and\na large amount of information is left on ``teuthology.front.sepia.ceph.com`` in\n``/var/lib/teuthworker/archive``.\n\nThis is symbolically linked to /a for convenience. A new directory is created\nwhose name consists of a concatenation of the date and time that the suite was\nstarted, the name of the suite, the ceph branch tested, the kernel used, and\nthe flavor. For every test run there is a directory whose name is the pid\nnumber of the pid of that test.  Each of these directory contains a copy of the\n``teuthology.log`` for that process.  Other information from the suite is\nstored in files in the directory, and task-specific yaml files and other logs\nare saved in the subdirectories.\n\nThese logs are also publically available at\n``http://qa-proxy.ceph.com/teuthology/``.",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/ceph/teuthology",
        "keywords": "teuthology test ceph cluster",
        "license": "MIT",
        "maintainer": null,
        "maintainer_email": null,
        "name": "teuthology",
        "package_url": "https://pypi.org/project/teuthology/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/teuthology/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "https://github.com/ceph/teuthology"
        },
        "release_url": "https://pypi.org/project/teuthology/0.1.0/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "Ceph test framework",
        "version": "0.1.0"
    },
    "last_serial": 1205882,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a4c56a35fe47defc01ac03dc40d6da5d",
                    "sha256": "e16ee9d3e9fd537d103f84b53d724fee5ad1a916a002d70de5c56b325cbda13a"
                },
                "downloads": -1,
                "filename": "teuthology-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "a4c56a35fe47defc01ac03dc40d6da5d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 144715,
                "upload_time": "2014-08-28T22:54:16",
                "url": "https://files.pythonhosted.org/packages/80/d2/cce7da04103781bdebfb5eb2c69d0ffcb2c7cf44fdc8a591b7847b9f9174/teuthology-0.1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "a4c56a35fe47defc01ac03dc40d6da5d",
                "sha256": "e16ee9d3e9fd537d103f84b53d724fee5ad1a916a002d70de5c56b325cbda13a"
            },
            "downloads": -1,
            "filename": "teuthology-0.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "a4c56a35fe47defc01ac03dc40d6da5d",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 144715,
            "upload_time": "2014-08-28T22:54:16",
            "url": "https://files.pythonhosted.org/packages/80/d2/cce7da04103781bdebfb5eb2c69d0ffcb2c7cf44fdc8a591b7847b9f9174/teuthology-0.1.0.tar.gz"
        }
    ]
}