PART I: Basic usage¶
Installation¶
To install the latest stable version of the utah client, let’s add the UTAH
stable PPA to our sources, and install the utah-client
package:
$ sudo add-apt-repository -y ppa:utah/stable
$ sudo apt-get update
$ sudo atp-get install utah-client
The binary used to run the test cases is utah
. We can take a look at all
the available arguments using the -h
/--help
option:
$ utah -h
Note
utah
is installed as part of the utah-client
package.
In this example, we’re interested just in the -r
/--runlist
argument
which is used to tell the client which test suites should executed in a single
run.
Writing tests¶
Test suite¶
To create a test suite and a test case from scratch, we’ll use the phoenix
command installed as part of the utah-client
package:
$ cd /tmp
$ phoenix utah_howto test_one
This will create a new test suite under a directory called utah_howto
with
some files in it:
master.run
: main run list expected to be passed toutah
in the-r
/--runlist
argument. As explained above, it contains a list of all the test suites to be executed in a single run.Note
In the general case, the run list will be in a different location, not in the same directory as the test suite.
tslist.run
: test suite list with a description of the test cases to be executed.Note
Test cases created by
phoenix
will be automatically added to the test suite list. In particular, note thattest_one
is already in the file.ts_control
: test suite metadata file with additional information needed to set the environment to execute the test suite properly.test_one/tc_control
: test case metdata file with specific information needed to run a particular test case.
Note
All the files above use yaml syntax, take advantage of the syntax highlighting feature of your preferred editor.
Test case¶
Let’s edit test_one/tc_control
to write a simple test case that verifies
that /bin/true
works as expected. The final result should be as follows:
description: System sanity check
dependencies: coreutils
action: |
1. Run /bin/true
expected_results: |
1. /bin/true exits with status 0
type: userland
timeout: 60
command: /bin/true
run_as: utah
where:
command
: is what will be executed to run the test caseNote
the return code from the command is used by utah to determine whether the test case passed or not using the unix convention.
run_as
: is the user that will executed the command
Note
dependencies
, action
and expected_results
are there for
description purposes only. The utah client doesn’t parse/use them for now,
but that might change in the future.
Run list¶
Once we have a test suite and a test case, we need to edit the run list to be able to execute them:
---
testsuites:
- name: utah_howto
fetch_method: dev
fetch_location: /tmp/utah_howto
where:
fetch_method
: tells the utah client how to get the test suitefetch_location
: tells the utah client where to get the test suite from
Note
By default all test cases in the test suite are executed
Executing tests¶
Once the test suite and cases have been writen and the run list is ready, the utah client can be used to run the test cases as follows:
$ sudo utah -r master.run > report.yaml
$ vim report.yaml
Note
utah
must be executed as root
for now to make it possible to
execute commands as a different user easily. In the future this might be
improved to avoid the this.
The contents of the test execution report should be similar to the one below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | ---
arch: amd64
build_number: '20121017.5'
commands:
- cmd_type: testsuite_fetch
command: cp -r /tmp/utah_howto utah_howto
returncode: 0
start_time: '2012-11-08 14:08:21.972824'
stderr: ''
stdout: ''
time_delta: '0:00:00.003381'
user: root
- cmd_type: testsuite_fetch
command: echo 'DEVELOPMENT'
returncode: 0
start_time: '2012-11-08 14:08:21.976431'
stderr: ''
stdout: |-
DEVELOPMENT
time_delta: '0:00:00.001907'
user: root
- cmd_type: testcase_test
command: /bin/true
extra_info:
action: |-
1. Run /bin/true
dependencies: coreutils
description: System sanity check
expected_results: |-
1. /bin/true exits with status 0
returncode: 0
start_time: '2012-11-08 14:08:22.004614'
stderr: ''
stdout: ''
testcase: test_one
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.029548'
user: utah
errors: 0
failures: 0
fetch_errors: 0
install_type: desktop
media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
name: unnamed
passes: 1
ran_at: '2012-11-08 14:08:21.972824'
release: quantal
runlist: /tmp/utah_howto/master.run
uname:
- Linux
- xps8300
- 3.5.0-18-generic
- '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
- x86_64
- x86_64
|
The more important things to note for now are:
- lines 5-6: the test suite is fetched from its location.
- lines 22-23: the test case is executed
- line 45: the test case passed successfully
Including/excluding test cases¶
Let’s continue the example by adding a new test case to the test suite we’ve already created:
$ phoenix . test_two
Note
phoenix
will add test_two
to tslist.run
automatically
After that, let’s edit test_two/tc_control
and set the following contents:
description: Test FAIL protocol
dependencies: wget
action: |
1. Use fail protocol to retrieve example.com
expected_results: |
1. example.com retrieved
type: userland
timeout: 60
command: wget fail://example.com
run_as: utah
As it can be seen, the call to wget
will fail because the protocol in the
URL is invalid.
Warning
there’s a bug and utah that will cause problems when trying this example depending on the locale configuration.
When we’re done editing the test case metadata, the utah client can be executed again:
$ sudo utah -r master.run > report.yaml
$ vim report.yaml
Looking at the test execution report, the part about the new test case command is as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | ---
arch: amd64
build_number: '20121017.5'
commands:
- cmd_type: testsuite_fetch
command: cp -r /tmp/utah_howto utah_howto
returncode: 0
start_time: '2012-11-08 15:02:46.993684'
stderr: ''
stdout: ''
time_delta: '0:00:00.003440'
user: root
- cmd_type: testsuite_fetch
command: echo 'DEVELOPMENT'
returncode: 0
start_time: '2012-11-08 15:02:46.997347'
stderr: ''
stdout: |-
DEVELOPMENT
time_delta: '0:00:00.001918'
user: root
- cmd_type: testcase_test
command: /bin/true
extra_info:
action: |-
1. Run /bin/true
dependencies: coreutils
description: System sanity check
expected_results: |-
1. /bin/true exits with status 0
returncode: 0
start_time: '2012-11-08 15:02:47.024652'
stderr: ''
stdout: ''
testcase: test_one
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.010179'
user: utah
- cmd_type: testcase_test
command: wget fail://example.com
extra_info:
action: |-
1. Use fail protocol to retrieve example.com
dependencies: wget
description: Test FAIL protocol
expected_results: |-
1. example.com retrieved
returncode: 1
start_time: '2012-11-08 15:02:47.064155'
stderr: |-
fail://example.com: Unsupported scheme `fail'.
stdout: ''
testcase: test_two
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.049322'
user: utah
errors: 0
failures: 1
fetch_errors: 0
install_type: desktop
media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
name: unnamed
passes: 1
ran_at: '2012-11-08 15:02:46.993684'
release: quantal
runlist: /tmp/utah_howto/master.run
uname:
- Linux
- xps8300
- 3.5.0-18-generic
- '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
- x86_64
- x86_64
|
where it can be seen that:
- line 48: the test case command failed
- lines 50-51: the problem was indeed using an invalid protocol in the url
- line 58: the command failure was considered a test case failure
Let’s say that we know the test case has a problem, but we don’t have time to fix it now. Instead, what we want to do is skip it until it’s fixed in the future.
To do that, edit master.run
and specify that test_two
must be excluded:
---
testsuites:
- name: utah_howto
fetch_method: dev
fetch_location: /tmp/utah_howto
exclude_tests:
- test_two
After this change, if the utah client is executed again:
$ sudo utah -r master.run > report.yaml
$ vim report.yaml
The report only shows a test case executed and no errors:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | ---
arch: amd64
build_number: '20121017.5'
commands:
- cmd_type: testsuite_fetch
command: cp -r /tmp/utah_howto utah_howto
returncode: 0
start_time: '2012-11-08 15:34:58.501273'
stderr: ''
stdout: ''
time_delta: '0:00:00.003482'
user: root
- cmd_type: testsuite_fetch
command: echo 'DEVELOPMENT'
returncode: 0
start_time: '2012-11-08 15:34:58.504980'
stderr: ''
stdout: |-
DEVELOPMENT
time_delta: '0:00:00.001902'
user: root
- cmd_type: testcase_test
command: /bin/true
extra_info:
action: |-
1. Run /bin/true
dependencies: coreutils
description: System sanity check
expected_results: |-
1. /bin/true exits with status 0
returncode: 0
start_time: '2012-11-08 15:34:58.526534'
stderr: ''
stdout: ''
testcase: test_one
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.010364'
user: utah
errors: 0
failures: 0
fetch_errors: 0
install_type: desktop
media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
name: unnamed
passes: 1
ran_at: '2012-11-08 15:34:58.501273'
release: quantal
runlist: /tmp/utah_howto/master.run
uname:
- Linux
- xps8300
- 3.5.0-18-generic
- '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
- x86_64
- x86_64
|
Build/setup/cleanup¶
Sometimes, it might happen that a test case is written in a compiled language or that it requires a special configuration to be in place before it’s executed. To handle those test cases, there’s a special metadata that can be added to the test case.
Let’s create another test case in our test suite:
$ phoenix . test_three
To simulate a test case that requires a build step, let’s write a Makefile
under the test_three
directory that generates the a script we want to
execute later in the test case:
test_three.sh:
echo 'test -f /tmp/foo' > test_three.sh
chmod +x test_three.sh
After that, let’s edit test_three/tc_control
to make sure that the make
command is used in a build step before running the test case:
description: Test that /tmp/foo exists
dependencies: make
action: |
1. Test that /tmp/foo exists
expected_results: |
1. /tmp/foo indeed exists
type: userland
timeout: 60
command: ./test_three.sh
run_as: utah
build_cmd: make
At this point, we can run the utah client:
$ sudo utah -r master.run > report.yaml
$ vim report.yaml
and check the test execution report:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | ---
arch: amd64
build_number: '20121017.5'
commands:
- cmd_type: testsuite_fetch
command: cp -r /tmp/utah_howto utah_howto
returncode: 0
start_time: '2012-11-08 16:23:18.733182'
stderr: ''
stdout: ''
time_delta: '0:00:00.003528'
user: root
- cmd_type: testsuite_fetch
command: echo 'DEVELOPMENT'
returncode: 0
start_time: '2012-11-08 16:23:18.736933'
stderr: ''
stdout: |-
DEVELOPMENT
time_delta: '0:00:00.001879'
user: root
- cmd_type: testcase_test
command: /bin/true
extra_info:
action: |-
1. Run /bin/true
dependencies: coreutils
description: System sanity check
expected_results: |-
1. /bin/true exits with status 0
returncode: 0
start_time: '2012-11-08 16:23:18.765374'
stderr: ''
stdout: ''
testcase: test_one
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.010362'
user: utah
- cmd_type: testcase_build
command: make
returncode: 0
start_time: '2012-11-08 16:23:18.796690'
stderr: ''
stdout: |-
echo 'test -f /tmp/foo' > test_three.sh
chmod +x test_three.sh
testcase: test_three
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.005390'
user: root
- cmd_type: testcase_test
command: ./test_three.sh
extra_info:
action: |-
1. Test that /tmp/foo exists
dependencies: make
description: Test that /tmp/foo exists
expected_results: |-
1. /tmp/foo indeed exists
returncode: 1
start_time: '2012-11-08 16:23:18.817905'
stderr: ''
stdout: ''
testcase: test_three
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.010506'
user: utah
errors: 0
failures: 1
fetch_errors: 0
install_type: desktop
media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
name: unnamed
passes: 1
ran_at: '2012-11-08 16:23:18.733182'
release: quantal
runlist: /tmp/utah_howto/master.run
uname:
- Linux
- xps8300
- 3.5.0-18-generic
- '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
- x86_64
- x86_64
|
What we see here is that:
- lines 39-40: there’s a new build command that generates the files needed to run the test case.
- line 60: the test case failed because the
/tmp/foo
doesn’t exist.
Hence, we managed to generate the file needed to run the test case, but
failed to configure the environment properly, that is, have
the /tmp/foo
file in place.
To address that issue, let’s edit again
test_three/tc_control
as follows:
description: Test that /tmp/foo exists
dependencies: make
action: |
1. Test that /tmp/foo exists
expected_results: |
1. /tmp/foo indeed exists
type: userland
timeout: 60
command: ./test_three.sh
run_as: utah
build_cmd: make
tc_setup: touch /tmp/foo
tc_cleanup: rm /tmp/foo
where:
tc_setup
is a command that is executed to take care of all the configuration needed for the test case to work correctly.tc_cleanup
is a command that is executed to undo whatever the setup command did and set the environment as it was before executing the test case.
Now if we run agan the utah client,
$ sudo utah -r master.run > report.yaml
$ vim report.yaml
we see the following test execution report:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | ---
arch: amd64
build_number: '20121017.5'
commands:
- cmd_type: testsuite_fetch
command: cp -r /tmp/utah_howto utah_howto
returncode: 0
start_time: '2012-11-08 16:37:12.817425'
stderr: ''
stdout: ''
time_delta: '0:00:00.003487'
user: root
- cmd_type: testsuite_fetch
command: echo 'DEVELOPMENT'
returncode: 0
start_time: '2012-11-08 16:37:12.821134'
stderr: ''
stdout: |-
DEVELOPMENT
time_delta: '0:00:00.001893'
user: root
- cmd_type: testcase_test
command: /bin/true
extra_info:
action: |-
1. Run /bin/true
dependencies: coreutils
description: System sanity check
expected_results: |-
1. /bin/true exits with status 0
returncode: 0
start_time: '2012-11-08 16:37:12.849988'
stderr: ''
stdout: ''
testcase: test_one
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.010241'
user: utah
- cmd_type: testcase_build
command: make
returncode: 0
start_time: '2012-11-08 16:37:12.874301'
stderr: ''
stdout: |-
echo 'test -f /tmp/foo' > test_three.sh
chmod +x test_three.sh
testcase: test_three
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.005345'
user: root
- cmd_type: testcase_setup
command: touch /tmp/foo
returncode: 0
start_time: '2012-11-08 16:37:12.889391'
stderr: ''
stdout: ''
testcase: test_three
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.002992'
user: root
- cmd_type: testcase_test
command: ./test_three.sh
extra_info:
action: |-
1. Test that /tmp/foo exists
dependencies: make
description: Test that /tmp/foo exists
expected_results: |-
1. /tmp/foo indeed exists
returncode: 0
start_time: '2012-11-08 16:37:12.900769'
stderr: ''
stdout: ''
testcase: test_three
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.010171'
user: utah
- cmd_type: testcase_cleanup
command: rm /tmp/foo
returncode: 0
start_time: '2012-11-08 16:37:12.919748'
stderr: ''
stdout: ''
testcase: test_three
testsuite: /var/lib/utah/testsuites/utah_howto
time_delta: '0:00:00.002942'
user: root
errors: 0
failures: 0
fetch_errors: 0
install_type: desktop
media-info: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
name: unnamed
passes: 2
ran_at: '2012-11-08 16:37:12.817425'
release: quantal
runlist: /tmp/utah_howto/master.run
uname:
- Linux
- xps8300
- 3.5.0-18-generic
- '#29-Ubuntu SMP Fri Oct 19 10:26:51 UTC 2012'
- x86_64
- x86_64
|
where:
- lines 51-52: there’s a new setup step before executing the test case.
- lines 78-79: there’s a new cleanup step after execution the test case.
- line 94: all test cases now pass.
Todo
Move the setup/cleanup code to the test suite to give an example about how to do the same thing at the suite level (useful when multiple test cases need the same configuration).
Timeout¶
Todo
Fix the formatting and provide more information about the example.
$ phoenix . test_four
Edit tc_control file:
description: Sleep test
despendencies: sleep
action: |
1. Sleep for 10 seconds
expected_results: |
system waits and returns 0
command: sleep 10
timeout: 5
- Run again
There's one failure because of the timeout
Note
Timeout returncode is -9 (process is killed). This is documented, but the yaml output file might explain this better in the future.
Different architectures? Override the timeout value in the master.run, so that the timeout value adjust to the target hardware.
- Edit master.run:
timeout: 15
(top level setting, not per test suite)
- Run again
Now we have three passes