PART I: Basic usage

Installation

To install the latest stable version of the utah client, let’s add the UTAH stable PPA to our sources, and install the utah-client package:

$ sudo add-apt-repository -y ppa:utah/stable
$ sudo apt-get update
$ sudo atp-get install utah-client

The binary used to run the test cases is utah. We can take a look at all the available arguments using the -h/--help option:

$ utah -h

Note

utah is installed as part of the utah-client package.

In this example, we’re interested just in the -r/--runlist argument which is used to tell the client which test suites should executed in a single run.

Writing tests

Test suite

To create a test suite and a test case from scratch, we’ll use the phoenix command installed as part of the utah-client package:

$ cd /tmp
$ phoenix utah_howto test_one

This will create a new test suite under a directory called utah_howto with some files in it:

  • master.run: main run list expected to be passed to utah in the -r/--runlist argument. As explained above, it contains a list of all the test suites to be executed in a single run.

    Note

    In the general case, the run list will be in a different location, not in the same directory as the test suite.

  • tslist.run: test suite list with a description of the test cases to be executed.

    Note

    Test cases created by phoenix will be automatically added to the test suite list. In particular, note that test_one is already in the file.

  • ts_control: test suite metadata file with additional information needed to set the environment to execute the test suite properly.

  • test_one/tc_control: test case metdata file with specific information needed to run a particular test case.

Note

All the files above use yaml syntax, take advantage of the syntax highlighting feature of your preferred editor.

Test case

Let’s edit test_one/tc_control to write a simple test case that verifies that /bin/true works as expected. The final result should be as follows:

description: System sanity check
dependencies: coreutils
action: |
 1. Run /bin/true
expected_results: |
 1. /bin/true exits with status 0
type: userland
timeout: 60
command: /bin/true
run_as: utah

where:

  • command: is what will be executed to run the test case

    Note

    the return code from the command is used by utah to determine whether the test case passed or not using the unix convention.

  • run_as: is the user that will executed the command

Note

dependencies, action and expected_results are there for description purposes only. The utah client doesn’t parse/use them for now, but that might change in the future.

Run list

Once we have a test suite and a test case, we need to edit the run list to be able to execute them:

---
testsuites:
  - name: utah_howto
    fetch_method: dev
    fetch_location: /tmp/utah_howto

where:

  • fetch_method: tells the utah client how to get the test suite
  • fetch_location: tells the utah client where to get the test suite from

Note

By default all test cases in the test suite are executed

Executing tests

Once the test suite and cases have been writen and the run list is ready, the utah client can be used to run the test cases as follows:

$ sudo utah -r master.run > report.yaml
$ vim report.yaml

Note

utah must be executed as root for now to make it possible to execute commands as a different user easily. In the future this might be improved to avoid the this.

The contents of the test execution report should be similar to the one below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
---
arch: amd64
build_number: '20121017.5'
commands:
- cmd_type: testsuite_fetch
  command: cp -r /tmp/utah_howto utah_howto
  returncode: 0
  start_time: '2012-11-08 14:08:21.972824'
  stderr: ''
  stdout: ''
  time_delta: '0:00:00.003381'
  user: root
- cmd_type: testsuite_fetch
  command: echo 'DEVELOPMENT'
  returncode: 0
  start_time: '2012-11-08 14:08:21.976431'
  stderr: ''
  stdout: |-
    DEVELOPMENT
  time_delta: '0:00:00.001907'
  user: root
- cmd_type: testcase_test
  command: /bin/true
  extra_info:
    action: |-
      1. Run /bin/true
    dependencies: coreutils
    description: System sanity check
    expected_results: |-
      1. /bin/true exits with status 0
  returncode: 0
  start_time: '2012-11-08 14:08:22.004614'
  stderr: ''
  stdout: ''
  testcase: test_one
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.029548'
  user: utah
errors: 0
failures: 0
fetch_errors: 0
install_type: desktop
media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
name: unnamed
passes: 1
ran_at: '2012-11-08 14:08:21.972824'
release: quantal
runlist: /tmp/utah_howto/master.run
uname:
- Linux
- xps8300
- 3.5.0-18-generic
- '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
- x86_64
- x86_64

The more important things to note for now are:

  • lines 5-6: the test suite is fetched from its location.
  • lines 22-23: the test case is executed
  • line 45: the test case passed successfully

Including/excluding test cases

Let’s continue the example by adding a new test case to the test suite we’ve already created:

$ phoenix . test_two

Note

phoenix will add test_two to tslist.run automatically

After that, let’s edit test_two/tc_control and set the following contents:

description: Test FAIL protocol
dependencies: wget
action: |
 1. Use fail protocol to retrieve example.com
expected_results: |
 1. example.com retrieved
type: userland
timeout: 60
command: wget fail://example.com
run_as: utah

As it can be seen, the call to wget will fail because the protocol in the URL is invalid.

Warning

there’s a bug and utah that will cause problems when trying this example depending on the locale configuration.

When we’re done editing the test case metadata, the utah client can be executed again:

$ sudo utah -r master.run > report.yaml
$ vim report.yaml

Looking at the test execution report, the part about the new test case command is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
---
arch: amd64
build_number: '20121017.5'
commands:
- cmd_type: testsuite_fetch
  command: cp -r /tmp/utah_howto utah_howto
  returncode: 0
  start_time: '2012-11-08 15:02:46.993684'
  stderr: ''
  stdout: ''
  time_delta: '0:00:00.003440'
  user: root
- cmd_type: testsuite_fetch
  command: echo 'DEVELOPMENT'
  returncode: 0
  start_time: '2012-11-08 15:02:46.997347'
  stderr: ''
  stdout: |-
    DEVELOPMENT
  time_delta: '0:00:00.001918'
  user: root
- cmd_type: testcase_test
  command: /bin/true
  extra_info:
    action: |-
      1. Run /bin/true
    dependencies: coreutils
    description: System sanity check
    expected_results: |-
      1. /bin/true exits with status 0
  returncode: 0
  start_time: '2012-11-08 15:02:47.024652'
  stderr: ''
  stdout: ''
  testcase: test_one
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.010179'
  user: utah
- cmd_type: testcase_test
  command: wget fail://example.com
  extra_info:
    action: |-
      1. Use fail protocol to retrieve example.com
    dependencies: wget
    description: Test FAIL protocol
    expected_results: |-
      1. example.com retrieved
  returncode: 1
  start_time: '2012-11-08 15:02:47.064155'
  stderr: |-
    fail://example.com: Unsupported scheme `fail'.
  stdout: ''
  testcase: test_two
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.049322'
  user: utah
errors: 0
failures: 1
fetch_errors: 0
install_type: desktop
media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
name: unnamed
passes: 1
ran_at: '2012-11-08 15:02:46.993684'
release: quantal
runlist: /tmp/utah_howto/master.run
uname:
- Linux
- xps8300
- 3.5.0-18-generic
- '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
- x86_64
- x86_64

where it can be seen that:

  • line 48: the test case command failed
  • lines 50-51: the problem was indeed using an invalid protocol in the url
  • line 58: the command failure was considered a test case failure

Let’s say that we know the test case has a problem, but we don’t have time to fix it now. Instead, what we want to do is skip it until it’s fixed in the future.

To do that, edit master.run and specify that test_two must be excluded:

---
testsuites:
  - name: utah_howto
    fetch_method: dev
    fetch_location: /tmp/utah_howto
    exclude_tests:
     - test_two

After this change, if the utah client is executed again:

$ sudo utah -r master.run > report.yaml
$ vim report.yaml

The report only shows a test case executed and no errors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
---
arch: amd64
build_number: '20121017.5'
commands:
- cmd_type: testsuite_fetch
  command: cp -r /tmp/utah_howto utah_howto
  returncode: 0
  start_time: '2012-11-08 15:34:58.501273'
  stderr: ''
  stdout: ''
  time_delta: '0:00:00.003482'
  user: root
- cmd_type: testsuite_fetch
  command: echo 'DEVELOPMENT'
  returncode: 0
  start_time: '2012-11-08 15:34:58.504980'
  stderr: ''
  stdout: |-
    DEVELOPMENT
  time_delta: '0:00:00.001902'
  user: root
- cmd_type: testcase_test
  command: /bin/true
  extra_info:
    action: |-
      1. Run /bin/true
    dependencies: coreutils
    description: System sanity check
    expected_results: |-
      1. /bin/true exits with status 0
  returncode: 0
  start_time: '2012-11-08 15:34:58.526534'
  stderr: ''
  stdout: ''
  testcase: test_one
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.010364'
  user: utah
errors: 0
failures: 0
fetch_errors: 0
install_type: desktop
media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
name: unnamed
passes: 1
ran_at: '2012-11-08 15:34:58.501273'
release: quantal
runlist: /tmp/utah_howto/master.run
uname:
- Linux
- xps8300
- 3.5.0-18-generic
- '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
- x86_64
- x86_64

Build/setup/cleanup

Sometimes, it might happen that a test case is written in a compiled language or that it requires a special configuration to be in place before it’s executed. To handle those test cases, there’s a special metadata that can be added to the test case.

Let’s create another test case in our test suite:

$ phoenix . test_three

To simulate a test case that requires a build step, let’s write a Makefile under the test_three directory that generates the a script we want to execute later in the test case:

test_three.sh:
    echo 'test -f /tmp/foo' > test_three.sh
    chmod +x test_three.sh

After that, let’s edit test_three/tc_control to make sure that the make command is used in a build step before running the test case:

description: Test that /tmp/foo exists
dependencies: make
action: |
  1. Test that /tmp/foo exists
expected_results: |
  1. /tmp/foo indeed exists
type: userland
timeout: 60
command: ./test_three.sh
run_as: utah
build_cmd: make

At this point, we can run the utah client:

$ sudo utah -r master.run > report.yaml
$ vim report.yaml

and check the test execution report:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
---
arch: amd64
build_number: '20121017.5'
commands:
- cmd_type: testsuite_fetch
  command: cp -r /tmp/utah_howto utah_howto
  returncode: 0
  start_time: '2012-11-08 16:23:18.733182'
  stderr: ''
  stdout: ''
  time_delta: '0:00:00.003528'
  user: root
- cmd_type: testsuite_fetch
  command: echo 'DEVELOPMENT'
  returncode: 0
  start_time: '2012-11-08 16:23:18.736933'
  stderr: ''
  stdout: |-
    DEVELOPMENT
  time_delta: '0:00:00.001879'
  user: root
- cmd_type: testcase_test
  command: /bin/true
  extra_info:
    action: |-
      1. Run /bin/true
    dependencies: coreutils
    description: System sanity check
    expected_results: |-
      1. /bin/true exits with status 0
  returncode: 0
  start_time: '2012-11-08 16:23:18.765374'
  stderr: ''
  stdout: ''
  testcase: test_one
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.010362'
  user: utah
- cmd_type: testcase_build
  command: make
  returncode: 0
  start_time: '2012-11-08 16:23:18.796690'
  stderr: ''
  stdout: |-
    echo 'test -f /tmp/foo' > test_three.sh
    chmod +x test_three.sh
  testcase: test_three
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.005390'
  user: root
- cmd_type: testcase_test
  command: ./test_three.sh
  extra_info:
    action: |-
      1. Test that /tmp/foo exists
    dependencies: make
    description: Test that /tmp/foo exists
    expected_results: |-
      1. /tmp/foo indeed exists
  returncode: 1
  start_time: '2012-11-08 16:23:18.817905'
  stderr: ''
  stdout: ''
  testcase: test_three
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.010506'
  user: utah
errors: 0
failures: 1
fetch_errors: 0
install_type: desktop
media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
name: unnamed
passes: 1
ran_at: '2012-11-08 16:23:18.733182'
release: quantal
runlist: /tmp/utah_howto/master.run
uname:
- Linux
- xps8300
- 3.5.0-18-generic
- '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
- x86_64
- x86_64

What we see here is that:

  • lines 39-40: there’s a new build command that generates the files needed to run the test case.
  • line 60: the test case failed because the /tmp/foo doesn’t exist.

Hence, we managed to generate the file needed to run the test case, but failed to configure the environment properly, that is, have the /tmp/foo file in place.

To address that issue, let’s edit again test_three/tc_control as follows:

description: Test that /tmp/foo exists
dependencies: make
action: |
  1. Test that /tmp/foo exists
expected_results: |
  1. /tmp/foo indeed exists
type: userland
timeout: 60
command: ./test_three.sh
run_as: utah
build_cmd: make
tc_setup: touch /tmp/foo
tc_cleanup: rm /tmp/foo

where:

  • tc_setup is a command that is executed to take care of all the configuration needed for the test case to work correctly.
  • tc_cleanup is a command that is executed to undo whatever the setup command did and set the environment as it was before executing the test case.

Now if we run agan the utah client,

$ sudo utah -r master.run > report.yaml
$ vim report.yaml

we see the following test execution report:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
---
arch: amd64
build_number: '20121017.5'
commands:
- cmd_type: testsuite_fetch
  command: cp -r /tmp/utah_howto utah_howto
  returncode: 0
  start_time: '2012-11-08 16:37:12.817425'
  stderr: ''
  stdout: ''
  time_delta: '0:00:00.003487'
  user: root
- cmd_type: testsuite_fetch
  command: echo 'DEVELOPMENT'
  returncode: 0
  start_time: '2012-11-08 16:37:12.821134'
  stderr: ''
  stdout: |-
    DEVELOPMENT
  time_delta: '0:00:00.001893'
  user: root
- cmd_type: testcase_test
  command: /bin/true
  extra_info:
    action: |-
      1. Run /bin/true
    dependencies: coreutils
    description: System sanity check
    expected_results: |-
      1. /bin/true exits with status 0
  returncode: 0
  start_time: '2012-11-08 16:37:12.849988'
  stderr: ''
  stdout: ''
  testcase: test_one
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.010241'
  user: utah
- cmd_type: testcase_build
  command: make
  returncode: 0
  start_time: '2012-11-08 16:37:12.874301'
  stderr: ''
  stdout: |-
    echo 'test -f /tmp/foo' > test_three.sh
    chmod +x test_three.sh
  testcase: test_three
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.005345'
  user: root
- cmd_type: testcase_setup
  command: touch /tmp/foo
  returncode: 0
  start_time: '2012-11-08 16:37:12.889391'
  stderr: ''
  stdout: ''
  testcase: test_three
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.002992'
  user: root
- cmd_type: testcase_test
  command: ./test_three.sh
  extra_info:
    action: |-
      1. Test that /tmp/foo exists
    dependencies: make
    description: Test that /tmp/foo exists
    expected_results: |-
      1. /tmp/foo indeed exists
  returncode: 0
  start_time: '2012-11-08 16:37:12.900769'
  stderr: ''
  stdout: ''
  testcase: test_three
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.010171'
  user: utah
- cmd_type: testcase_cleanup
  command: rm /tmp/foo
  returncode: 0
  start_time: '2012-11-08 16:37:12.919748'
  stderr: ''
  stdout: ''
  testcase: test_three
  testsuite: /var/lib/utah/testsuites/utah_howto
  time_delta: '0:00:00.002942'
  user: root
errors: 0
failures: 0
fetch_errors: 0
install_type: desktop
media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
name: unnamed
passes: 2
ran_at: '2012-11-08 16:37:12.817425'
release: quantal
runlist: /tmp/utah_howto/master.run
uname:
- Linux
- xps8300
- 3.5.0-18-generic
- '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
- x86_64
- x86_64

where:

  • lines 51-52: there’s a new setup step before executing the test case.
  • lines 78-79: there’s a new cleanup step after execution the test case.
  • line 94: all test cases now pass.

Todo

Move the setup/cleanup code to the test suite to give an example about how to do the same thing at the suite level (useful when multiple test cases need the same configuration).

Timeout

Todo

Fix the formatting and provide more information about the example.

$ phoenix . test_four

Edit tc_control file:

   description: Sleep test
   despendencies: sleep
   action: |
    1. Sleep for 10 seconds
    expected_results: |
     system waits and returns 0
    command: sleep 10
    timeout: 5

- Run again
There's one failure because of the timeout

Note

Timeout returncode is -9 (process is killed). This is documented, but the yaml output file might explain this better in the future.

Different architectures? Override the timeout value in the master.run, so that the timeout value adjust to the target hardware.

  • Edit master.run:
timeout: 15

(top level setting, not per test suite)

  • Run again

Now we have three passes