]>
Commit | Line | Data |
---|---|---|
8fe5ef5a SA |
1 | Writing TAP Tests |
2 | ||
3 | Introduction | |
4 | ||
5 | This is a guide for users of the C TAP Harness package or similar | |
6 | TAP-based test harnesses explaining how to write tests. If your | |
7 | package uses C TAP Harness as the test suite driver, you may want to | |
8 | copy this document to an appropriate file name in your test suite as | |
9 | documentation for contributors. | |
10 | ||
11 | About TAP | |
12 | ||
13 | TAP is the Test Anything Protocol, a protocol for communication | |
14 | between test cases and a test harness. This is the protocol used by | |
15 | Perl for its internal test suite and for nearly all Perl modules, | |
16 | since it's the format used by the build tools for Perl modules to run | |
17 | tests and report their results. | |
18 | ||
19 | A TAP-based test suite works with a somewhat different set of | |
20 | assumptions than an xUnit test suite. In TAP, each test case is a | |
21 | separate program. That program, when run, must produce output in the | |
22 | following format: | |
23 | ||
24 | 1..4 | |
25 | ok 1 - the first test | |
26 | ok 2 | |
27 | # a diagnostic, ignored by the harness | |
28 | not ok 3 - a failing test | |
29 | ok 4 # skip a skipped test | |
30 | ||
31 | The output should all go to standard output. The first line specifies | |
32 | the number of tests to be run, and then each test produces output that | |
33 | looks like either "ok <n>" or "not ok <n>" depending on whether the | |
34 | test succeeded or failed. Additional information about the test can | |
35 | be provided after the "ok <n>" or "not ok <n>", but is optional. | |
36 | Additional diagnostics and information can be provided in lines | |
37 | beginning with a "#". | |
38 | ||
39 | Processing directives are supported after the "ok <n>" or "not ok <n>" | |
40 | and start with a "#". The main one of interest is "# skip" which says | |
41 | that the test was skipped rather than successful and optionally gives | |
42 | the reason. Also supported is "# todo", which normally annotates a | |
43 | failing test and indicates that test is expected to fail, optionally | |
44 | providing a reason for why. | |
45 | ||
46 | There are three more special cases. First, the initial line stating | |
47 | the number of tests to run, called the plan, may appear at the end of | |
48 | the output instead of the beginning. This can be useful if the number | |
49 | of tests to run is not known in advance. Second, a plan in the form: | |
50 | ||
51 | 1..0 # skip entire test case skipped | |
52 | ||
53 | can be given instead, which indicates that this entire test case has | |
54 | been skipped (generally because it depends on facilities or optional | |
55 | configuration which is not present). Finally, if the test case | |
56 | encounters a fatal error, it should print the text: | |
57 | ||
58 | Bail out! | |
59 | ||
60 | on standard output, optionally followed by an error message, and then | |
61 | exit. This tells the harness that the test aborted unexpectedly. | |
62 | ||
63 | The exit status of a successful test case should always be 0. The | |
64 | harness will report the test as "dubious" if all the tests appeared to | |
65 | succeed but it exited with a non-zero status. | |
66 | ||
67 | Writing TAP Tests | |
68 | ||
69 | Environment | |
70 | ||
71 | One of the special features of C TAP Harness is the environment that | |
72 | it sets up for your test cases. If your test program is called under | |
73 | the runtests driver, the environment variables C_TAP_SOURCE and | |
74 | C_TAP_BUILD will be set to the top of the test directory in the source | |
75 | tree and the top of the build tree, respectively. You can use those | |
76 | environment variables to locate additional test data, programs and | |
77 | libraries built as part of your software build, and other supporting | |
78 | information needed by tests. | |
79 | ||
80 | The C and shell TAP libraries support a test_file_path() function, | |
81 | which looks for a file under the build tree and then under the source | |
82 | tree, using the C_TAP_BUILD and C_TAP_SOURCE environment variables, | |
83 | and return the full path to the file. This can be used to locate | |
84 | supporting data files. They also support a test_tmpdir() function | |
85 | that returns a directory that can be used for temporary files during | |
86 | tests. | |
87 | ||
88 | Perl | |
89 | ||
90 | Since TAP is the native test framework for Perl, writing TAP tests in | |
91 | Perl is very easy and extremely well-supported. If you've never | |
92 | written tests in Perl before, start by reading the documentation for | |
93 | Test::Tutorial and Test::Simple, which walks you through the basics, | |
94 | including the TAP output syntax. Then, the best Perl module to use | |
95 | for serious testing is Test::More, which provides a lot of additional | |
96 | functions over Test::Simple including support for skipping tests, | |
97 | bailing out, and not planning tests in advance. See the documentation | |
98 | of Test::More for all the details and lots of examples. | |
99 | ||
100 | C TAP Harness can run Perl test scripts directly and interpret the | |
101 | results correctly, and similarly the Perl Test::Harness module and | |
102 | prove command can run TAP tests written in other languages using, for | |
103 | example, the TAP library that comes with C TAP Harness. You can, if | |
104 | you wish, use the library that comes with C TAP Harness but use prove | |
105 | instead of runtests for running the test suite. | |
106 | ||
107 | C | |
108 | ||
109 | C TAP Harness provides a basic TAP library that takes away most of the | |
110 | pain of writing TAP test cases in C. A C test case should start with | |
111 | a call to plan(), passing in the number of tests to run. Then, each | |
112 | test should use is_int(), is_string(), is_double(), or is_hex() as | |
113 | appropriate to compare expected and seen values, or ok() to do a | |
114 | simpler boolean test. The is_*() functions take expected and seen | |
115 | values and then a printf-style format string explaining the test | |
116 | (which may be NULL). ok() takes a boolean and then the printf-style | |
117 | string. | |
118 | ||
119 | Here's a complete example test program that uses the C TAP library: | |
120 | ||
121 | #include <stddef.h> | |
122 | #include <tap/basic.h> | |
123 | ||
124 | int | |
125 | main(void) | |
126 | { | |
127 | plan(4); | |
128 | ||
129 | ok(1, "the first test"); | |
130 | is_int(42, 42, NULL); | |
131 | diag("a diagnostic, ignored by the harness"); | |
132 | ok(0, "a failing test"); | |
133 | skip("a skipped test"); | |
134 | ||
135 | return 0; | |
136 | } | |
137 | ||
138 | This test program produces the output shown above in the section on | |
139 | TAP and demonstrates most of the functions. The other functions of | |
140 | interest are sysdiag() (like diag() but adds strerror() results), | |
141 | bail() and sysbail() for fatal errors, skip_block() to skip a whole | |
142 | block of tests, and skip_all() which is called instead of plan() to | |
143 | skip an entire test case. | |
144 | ||
145 | The C TAP library also provides plan_lazy(), which can be called | |
146 | instead of plan(). If plan_lazy() is called, the library will keep | |
147 | track of how many test results are reported and will print out the | |
148 | plan at the end of execution of the program. This should normally be | |
149 | avoided since the test may appear to be successful even if it exits | |
150 | prematurely, but it can make writing tests easier in some | |
151 | circumstances. | |
152 | ||
153 | Complete API documentation for the basic C TAP library that comes with | |
154 | C TAP Harness is available at: | |
155 | ||
156 | <https://www.eyrie.org/~eagle/software/c-tap-harness/> | |
157 | ||
158 | It's common to need additional test functions and utility functions | |
159 | for your C tests, particularly if you have to set up and tear down a | |
160 | test environment for your test programs, and it's useful to have them | |
161 | all in the libtap library so that you only have to link your test | |
162 | programs with one library. Rather than editing tap/basic.c and | |
163 | tap/basic.h to add those additional functions, add additional *.c and | |
164 | *.h files into the tap directory with the function implementations and | |
165 | prototypes, and then add those additional objects to the library. | |
166 | That way, you can update tap/basic.c and tap/basic.h from subsequent | |
167 | releases of C TAP Harness without having to merge changes with your | |
168 | own code. | |
169 | ||
170 | Libraries of additional useful TAP test functions are available in | |
171 | rra-c-util at: | |
172 | ||
173 | <https://www.eyrie.org/~eagle/software/rra-c-util/> | |
174 | ||
175 | Some of the code there is particularly useful when testing programs | |
176 | that require Kerberos keys. | |
177 | ||
178 | If you implement new test functions that compare an expected and seen | |
179 | value, it's best to name them is_<something> and take the expected | |
180 | value, the seen value, and then a printf-style format string and | |
181 | possible arguments to match the calling convention of the functions | |
182 | provided by C TAP Harness. | |
183 | ||
184 | Shell | |
185 | ||
186 | C TAP Harness provides a library of shell functions to make it easier | |
187 | to write TAP tests in shell. That library includes much of the same | |
188 | functionality as the C TAP library, but takes its parameters in a | |
189 | somewhat different order to make better use of shell features. | |
190 | ||
191 | The libtap.sh file should be installed in a directory named tap in | |
192 | your test suite area. It can then be loaded by tests written in shell | |
193 | using the environment set up by runtests with: | |
194 | ||
195 | . "$C_TAP_SOURCE"/tap/libtap.sh | |
196 | ||
197 | Here is a complete test case written in shell which produces the same | |
198 | output as the TAP sample above: | |
199 | ||
200 | #!/bin/sh | |
201 | ||
202 | . "$C_TAP_SOURCE"/tap/libtap.sh | |
203 | cd "$C_TAP_BUILD" | |
204 | ||
205 | plan 4 | |
206 | ok 'the first test' true | |
207 | ok '' [ 42 -eq 42 ] | |
208 | diag a diagnostic, ignored by the harness | |
209 | ok '' false | |
210 | skip 'a skipped test' | |
211 | ||
212 | The shell framework doesn't provide the is_* functions, so you'll use | |
213 | the ok function more. It takes a string describing the text and then | |
214 | treats all of its remaining arguments as a condition, evaluated the | |
215 | same way as the arguments to the "if" statement. If that condition | |
216 | evaluates to true, the test passes; otherwise, the test fails. | |
217 | ||
218 | The plan, plan_lazy, diag, and bail functions work the same as with | |
219 | the C library. skip takes a string and skips the next test with that | |
220 | explanation. skip_block takes a count and a string and skips that | |
221 | many tests with that explanation. skip_all takes an optional reason | |
222 | and skips the entire test case. | |
223 | ||
224 | Since it's common for shell programs to want to test the output of | |
225 | commands, there's an additional function ok_program provided by the | |
226 | shell test library. It takes the test description string, the | |
227 | expected exit status, the expected program output, and then treats the | |
228 | rest of its arguments as the program to run. That program is run with | |
229 | standard error and standard output combined, and then its exit status | |
230 | and output are tested against the provided values. | |
231 | ||
232 | A utility function, strip_colon_error, is provided that runs the | |
233 | command given as its arguments and strips text following a colon and a | |
234 | space from the output (unless there is no whitespace on the line | |
235 | before the colon and the space, normally indicating a prefix of the | |
236 | program name). This function can be used to wrap commands that are | |
237 | expected to fail with output that has a system- or locale-specific | |
238 | error message appended, such as the output of strerror(). | |
239 | ||
240 | License | |
241 | ||
242 | This file is part of the documentation of C TAP Harness, which can be | |
243 | found at <https://www.eyrie.org/~eagle/software/c-tap-harness/>. | |
244 | ||
245 | Copyright 2010, 2016 Russ Allbery <eagle@eyrie.org> | |
246 | ||
247 | Copying and distribution of this file, with or without modification, | |
248 | are permitted in any medium without royalty provided the copyright | |
249 | notice and this notice are preserved. This file is offered as-is, | |
250 | without any warranty. |