PART I: Basic usage
===================

Installation
------------

To install the latest stable version of the utah client, let's add the UTAH
stable PPA to our sources, and install the ``utah-client`` package:

.. code-block:: console

    $ sudo add-apt-repository -y ppa:utah/stable
    $ sudo apt-get update
    $ sudo atp-get install utah-client

The binary used to run the test cases is ``utah``. We can take a look at all
the available arguments using the ``-h``/``--help`` option:

.. code-block:: console

    $ utah -h

.. note:: ``utah`` is installed as part of the ``utah-client`` package.

In this example, we're interested just in the ``-r``/``--runlist`` argument
which is used to tell the client which test suites should executed in a single
run.

Writing tests
-------------

Test suite
~~~~~~~~~~

To create a test suite and a test case from scratch, we'll use the ``phoenix``
command installed as part of the ``utah-client`` package:

.. code-block:: console

    $ cd /tmp
    $ phoenix utah_howto test_one

This will create a new test suite under a directory called ``utah_howto`` with
some files in it:

- ``master.run``: main run list expected to be passed to ``utah`` in the
  ``-r``/``--runlist`` argument. As explained above, it contains a list of all
  the test suites to be executed in a single run.

  .. note::

    In the general case, the run list will be in a different location, not in
    the same directory as the test suite.

- ``tslist.run``: test suite list with a description of the test cases to be
  executed.

  .. note::

    Test cases created by ``phoenix`` will be automatically added to the test
    suite list. In particular, note that ``test_one`` is already in the file.

- ``ts_control``: test suite metadata file with additional information needed
  to set the environment to execute the test suite properly.

- ``test_one/tc_control``: test case metdata file with specific information
  needed to run a particular test case.

.. note::

    All the files above use yaml syntax, take advantage of the syntax
    highlighting feature of your preferred editor.

Test case
~~~~~~~~~

Let's edit ``test_one/tc_control`` to write a simple test case that verifies
that ``/bin/true`` works as expected. The final result should be as follows:

.. code-block:: yaml

    description: System sanity check
    dependencies: coreutils
    action: |
     1. Run /bin/true
    expected_results: |
     1. /bin/true exits with status 0
    type: userland
    timeout: 60
    command: /bin/true
    run_as: utah

where:

- ``command``: is what will be executed to run the test case

    .. note::

        the return code from the command is used by utah to determine whether
        the test case passed or not using the unix convention.

- ``run_as``: is the user that will executed the command

.. note::

    ``dependencies``, ``action`` and ``expected_results`` are there for
    description purposes only. The utah client doesn't parse/use them for now,
    but that might change in the future.

Run list
~~~~~~~~

Once we have a test suite and a test case, we need to edit the run list to be
able to execute them:

.. code-block:: yaml
    :emphasize-lines: 4-5

    ---
    testsuites:
      - name: utah_howto
        fetch_method: dev
        fetch_location: /tmp/utah_howto

where:

- ``fetch_method``: tells the utah client how to get the test suite
- ``fetch_location``: tells the utah client where to get the test suite from

.. note:: By default all test cases in the test suite are executed

Executing tests
---------------

Once the test suite and cases have been writen and the run list is ready, the
utah client can be used to run the test cases as follows:

.. code-block:: console

    $ sudo utah -r master.run > report.yaml
    $ vim report.yaml

.. note::

    ``utah`` must be executed as ``root`` for now to make it possible to
    execute commands as a different user easily. In the future this might be
    improved to avoid the this.

The contents of the test execution report should be similar to the one below:

.. code-block:: yaml
    :linenos:
    :emphasize-lines: 5-6,22-23,45

    ---
    arch: amd64
    build_number: '20121017.5'
    commands:
    - cmd_type: testsuite_fetch
      command: cp -r /tmp/utah_howto utah_howto
      returncode: 0
      start_time: '2012-11-08 14:08:21.972824'
      stderr: ''
      stdout: ''
      time_delta: '0:00:00.003381'
      user: root
    - cmd_type: testsuite_fetch
      command: echo 'DEVELOPMENT'
      returncode: 0
      start_time: '2012-11-08 14:08:21.976431'
      stderr: ''
      stdout: |-
        DEVELOPMENT
      time_delta: '0:00:00.001907'
      user: root
    - cmd_type: testcase_test
      command: /bin/true
      extra_info:
        action: |-
          1. Run /bin/true
        dependencies: coreutils
        description: System sanity check
        expected_results: |-
          1. /bin/true exits with status 0
      returncode: 0
      start_time: '2012-11-08 14:08:22.004614'
      stderr: ''
      stdout: ''
      testcase: test_one
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.029548'
      user: utah
    errors: 0
    failures: 0
    fetch_errors: 0
    install_type: desktop
    media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
    name: unnamed
    passes: 1
    ran_at: '2012-11-08 14:08:21.972824'
    release: quantal
    runlist: /tmp/utah_howto/master.run
    uname:
    - Linux
    - xps8300
    - 3.5.0-18-generic
    - '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
    - x86_64
    - x86_64

The more important things to note for now are:

- lines 5-6: the test suite is fetched from its location.
- lines 22-23: the test case is executed
- line 45: the test case passed successfully

Including/excluding test cases
------------------------------

Let's continue the example by adding a new test case to the test suite we've
already created:

.. code-block:: console

    $ phoenix . test_two

.. note:: ``phoenix`` will add ``test_two`` to ``tslist.run`` automatically

After that, let's edit ``test_two/tc_control`` and set the following contents:

.. code-block:: yaml

    description: Test FAIL protocol
    dependencies: wget
    action: |
     1. Use fail protocol to retrieve example.com
    expected_results: |
     1. example.com retrieved
    type: userland
    timeout: 60
    command: wget fail://example.com
    run_as: utah

As it can be seen, the call to ``wget`` will fail because the protocol in the
URL is invalid.

.. warning::

    there's a bug and utah that will cause problems when trying this example
    depending on the locale configuration.

When we're done editing the test case metadata, the utah client can be executed
again:

.. code-block:: console

    $ sudo utah -r master.run > report.yaml
    $ vim report.yaml

Looking at the test execution report, the part about the new test case command
is as follows:

.. code-block:: yaml
    :linenos:
    :emphasize-lines: 48,50-51,58

    ---
    arch: amd64
    build_number: '20121017.5'
    commands:
    - cmd_type: testsuite_fetch
      command: cp -r /tmp/utah_howto utah_howto
      returncode: 0
      start_time: '2012-11-08 15:02:46.993684'
      stderr: ''
      stdout: ''
      time_delta: '0:00:00.003440'
      user: root
    - cmd_type: testsuite_fetch
      command: echo 'DEVELOPMENT'
      returncode: 0
      start_time: '2012-11-08 15:02:46.997347'
      stderr: ''
      stdout: |-
        DEVELOPMENT
      time_delta: '0:00:00.001918'
      user: root
    - cmd_type: testcase_test
      command: /bin/true
      extra_info:
        action: |-
          1. Run /bin/true
        dependencies: coreutils
        description: System sanity check
        expected_results: |-
          1. /bin/true exits with status 0
      returncode: 0
      start_time: '2012-11-08 15:02:47.024652'
      stderr: ''
      stdout: ''
      testcase: test_one
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.010179'
      user: utah
    - cmd_type: testcase_test
      command: wget fail://example.com
      extra_info:
        action: |-
          1. Use fail protocol to retrieve example.com
        dependencies: wget
        description: Test FAIL protocol
        expected_results: |-
          1. example.com retrieved
      returncode: 1
      start_time: '2012-11-08 15:02:47.064155'
      stderr: |-
        fail://example.com: Unsupported scheme `fail'.
      stdout: ''
      testcase: test_two
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.049322'
      user: utah
    errors: 0
    failures: 1
    fetch_errors: 0
    install_type: desktop
    media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
    name: unnamed
    passes: 1
    ran_at: '2012-11-08 15:02:46.993684'
    release: quantal
    runlist: /tmp/utah_howto/master.run
    uname:
    - Linux
    - xps8300
    - 3.5.0-18-generic
    - '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
    - x86_64
    - x86_64

where it can be seen that:

- line 48: the test case command failed
- lines 50-51: the problem was indeed using an invalid protocol in the url
- line 58: the command failure was considered a test case failure

Let's say that we know the test case has a problem, but we don't have time to
fix it now. Instead, what we want to do is skip it until it's fixed in the
future.

To do that, edit ``master.run`` and specify that ``test_two`` must be excluded:

.. code-block:: yaml
    :emphasize-lines: 6-7

    ---
    testsuites:
      - name: utah_howto
        fetch_method: dev
        fetch_location: /tmp/utah_howto
        exclude_tests:
         - test_two

After this change, if the utah client is executed again:

.. code-block:: console

    $ sudo utah -r master.run > report.yaml
    $ vim report.yaml

The report only shows a test case executed and no errors:

.. code-block:: yaml
    :linenos:
    :emphasize-lines: 39,45

    ---
    arch: amd64
    build_number: '20121017.5'
    commands:
    - cmd_type: testsuite_fetch
      command: cp -r /tmp/utah_howto utah_howto
      returncode: 0
      start_time: '2012-11-08 15:34:58.501273'
      stderr: ''
      stdout: ''
      time_delta: '0:00:00.003482'
      user: root
    - cmd_type: testsuite_fetch
      command: echo 'DEVELOPMENT'
      returncode: 0
      start_time: '2012-11-08 15:34:58.504980'
      stderr: ''
      stdout: |-
        DEVELOPMENT
      time_delta: '0:00:00.001902'
      user: root
    - cmd_type: testcase_test
      command: /bin/true
      extra_info:
        action: |-
          1. Run /bin/true
        dependencies: coreutils
        description: System sanity check
        expected_results: |-
          1. /bin/true exits with status 0
      returncode: 0
      start_time: '2012-11-08 15:34:58.526534'
      stderr: ''
      stdout: ''
      testcase: test_one
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.010364'
      user: utah
    errors: 0
    failures: 0
    fetch_errors: 0
    install_type: desktop
    media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
    name: unnamed
    passes: 1
    ran_at: '2012-11-08 15:34:58.501273'
    release: quantal
    runlist: /tmp/utah_howto/master.run
    uname:
    - Linux
    - xps8300
    - 3.5.0-18-generic
    - '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
    - x86_64
    - x86_64

Build/setup/cleanup
-------------------

Sometimes, it might happen that a test case is written in a compiled language
or that it requires a special configuration to be in place before it's
executed. To handle those test cases, there's a special metadata that can be
added to the test case.

Let's create another test case in our test suite:

.. code-block:: console

    $ phoenix . test_three

To simulate a test case that requires a build step, let's write a ``Makefile``
under the ``test_three`` directory that generates the a script we want to
execute later in the test case:

.. code-block:: make

    test_three.sh:
        echo 'test -f /tmp/foo' > test_three.sh
        chmod +x test_three.sh

After that, let's edit ``test_three/tc_control`` to make sure that the ``make``
command is used in a build step before running the test case:

.. code-block:: yaml

    description: Test that /tmp/foo exists
    dependencies: make
    action: |
      1. Test that /tmp/foo exists
    expected_results: |
      1. /tmp/foo indeed exists
    type: userland
    timeout: 60
    command: ./test_three.sh
    run_as: utah
    build_cmd: make

At this point, we can run the utah client:

.. code-block:: console

    $ sudo utah -r master.run > report.yaml
    $ vim report.yaml

and check the test execution report:

.. code-block:: yaml
    :linenos:
    :emphasize-lines: 39-40, 60

    ---
    arch: amd64
    build_number: '20121017.5'
    commands:
    - cmd_type: testsuite_fetch
      command: cp -r /tmp/utah_howto utah_howto
      returncode: 0
      start_time: '2012-11-08 16:23:18.733182'
      stderr: ''
      stdout: ''
      time_delta: '0:00:00.003528'
      user: root
    - cmd_type: testsuite_fetch
      command: echo 'DEVELOPMENT'
      returncode: 0
      start_time: '2012-11-08 16:23:18.736933'
      stderr: ''
      stdout: |-
        DEVELOPMENT
      time_delta: '0:00:00.001879'
      user: root
    - cmd_type: testcase_test
      command: /bin/true
      extra_info:
        action: |-
          1. Run /bin/true
        dependencies: coreutils
        description: System sanity check
        expected_results: |-
          1. /bin/true exits with status 0
      returncode: 0
      start_time: '2012-11-08 16:23:18.765374'
      stderr: ''
      stdout: ''
      testcase: test_one
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.010362'
      user: utah
    - cmd_type: testcase_build
      command: make
      returncode: 0
      start_time: '2012-11-08 16:23:18.796690'
      stderr: ''
      stdout: |-
        echo 'test -f /tmp/foo' > test_three.sh
        chmod +x test_three.sh
      testcase: test_three
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.005390'
      user: root
    - cmd_type: testcase_test
      command: ./test_three.sh
      extra_info:
        action: |-
          1. Test that /tmp/foo exists
        dependencies: make
        description: Test that /tmp/foo exists
        expected_results: |-
          1. /tmp/foo indeed exists
      returncode: 1
      start_time: '2012-11-08 16:23:18.817905'
      stderr: ''
      stdout: ''
      testcase: test_three
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.010506'
      user: utah
    errors: 0
    failures: 1
    fetch_errors: 0
    install_type: desktop
    media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
    name: unnamed
    passes: 1
    ran_at: '2012-11-08 16:23:18.733182'
    release: quantal
    runlist: /tmp/utah_howto/master.run
    uname:
    - Linux
    - xps8300
    - 3.5.0-18-generic
    - '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
    - x86_64
    - x86_64

What we see here is that:

- lines 39-40: there's a new build command that generates the
  files needed to run the test case.
- line 60: the test case failed because the ``/tmp/foo`` doesn't exist.

Hence, we managed to generate the file needed to run the test case, but
failed to configure the environment properly, that is, have
the ``/tmp/foo`` file in place.

To address that issue, let's edit again
``test_three/tc_control`` as follows:

.. code-block:: yaml

    description: Test that /tmp/foo exists
    dependencies: make
    action: |
      1. Test that /tmp/foo exists
    expected_results: |
      1. /tmp/foo indeed exists
    type: userland
    timeout: 60
    command: ./test_three.sh
    run_as: utah
    build_cmd: make
    tc_setup: touch /tmp/foo
    tc_cleanup: rm /tmp/foo

where:

- ``tc_setup`` is a command that is executed to take care of
  all the configuration needed for the test case to work
  correctly.

- ``tc_cleanup`` is a command that is executed to undo
  whatever the setup command did and set the environment as it
  was before executing the test case.

Now if we run agan the utah client,

.. code-block:: console

    $ sudo utah -r master.run > report.yaml
    $ vim report.yaml

we see the following test execution report:

.. code-block:: yaml
    :linenos:
    :emphasize-lines: 51-52, 78-79, 94

    ---
    arch: amd64
    build_number: '20121017.5'
    commands:
    - cmd_type: testsuite_fetch
      command: cp -r /tmp/utah_howto utah_howto
      returncode: 0
      start_time: '2012-11-08 16:37:12.817425'
      stderr: ''
      stdout: ''
      time_delta: '0:00:00.003487'
      user: root
    - cmd_type: testsuite_fetch
      command: echo 'DEVELOPMENT'
      returncode: 0
      start_time: '2012-11-08 16:37:12.821134'
      stderr: ''
      stdout: |-
        DEVELOPMENT
      time_delta: '0:00:00.001893'
      user: root
    - cmd_type: testcase_test
      command: /bin/true
      extra_info:
        action: |-
          1. Run /bin/true
        dependencies: coreutils
        description: System sanity check
        expected_results: |-
          1. /bin/true exits with status 0
      returncode: 0
      start_time: '2012-11-08 16:37:12.849988'
      stderr: ''
      stdout: ''
      testcase: test_one
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.010241'
      user: utah
    - cmd_type: testcase_build
      command: make
      returncode: 0
      start_time: '2012-11-08 16:37:12.874301'
      stderr: ''
      stdout: |-
        echo 'test -f /tmp/foo' > test_three.sh
        chmod +x test_three.sh
      testcase: test_three
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.005345'
      user: root
    - cmd_type: testcase_setup
      command: touch /tmp/foo
      returncode: 0
      start_time: '2012-11-08 16:37:12.889391'
      stderr: ''
      stdout: ''
      testcase: test_three
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.002992'
      user: root
    - cmd_type: testcase_test
      command: ./test_three.sh
      extra_info:
        action: |-
          1. Test that /tmp/foo exists
        dependencies: make
        description: Test that /tmp/foo exists
        expected_results: |-
          1. /tmp/foo indeed exists
      returncode: 0
      start_time: '2012-11-08 16:37:12.900769'
      stderr: ''
      stdout: ''
      testcase: test_three
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.010171'
      user: utah
    - cmd_type: testcase_cleanup
      command: rm /tmp/foo
      returncode: 0
      start_time: '2012-11-08 16:37:12.919748'
      stderr: ''
      stdout: ''
      testcase: test_three
      testsuite: /var/lib/utah/testsuites/utah_howto
      time_delta: '0:00:00.002942'
      user: root
    errors: 0
    failures: 0
    fetch_errors: 0
    install_type: desktop
    media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
    name: unnamed
    passes: 2
    ran_at: '2012-11-08 16:37:12.817425'
    release: quantal
    runlist: /tmp/utah_howto/master.run
    uname:
    - Linux
    - xps8300
    - 3.5.0-18-generic
    - '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
    - x86_64
    - x86_64

where:

- lines 51-52: there's a new setup step before executing the
  test case.
- lines 78-79: there's a new cleanup step after execution the
  test case.
- line 94: all test cases now pass.

.. todo::

    Move the setup/cleanup code to the test suite to give an
    example about how to do the same thing at the suite level
    (useful when multiple test cases need the same
    configuration).

Timeout
-------

.. todo::

    Fix the formatting and provide more information about the
    example.

.. code-block:: console

    $ phoenix . test_four

Edit tc_control file:

.. code-block:: yaml

    description: Sleep test
    despendencies: sleep
    action: |
     1. Sleep for 10 seconds
     expected_results: |
      system waits and returns 0
     command: sleep 10
     timeout: 5

 - Run again
 There's one failure because of the timeout

.. note::

    Timeout returncode is -9 (process is killed). This is documented, but the
    yaml output file might explain this better in the future.

Different architectures? Override the timeout value in the master.run, so that
the timeout value adjust to the target hardware.

- Edit master.run:

.. code-block:: yaml

    timeout: 15

(top level setting, not per test suite)

- Run again

Now we have three passes