正如我们上个章节里面看到的， `Test::Nginx` 提供了一个简单规范的格式来表达测试用例。

== 测试前的准备

正如我们上个章节里面看到的， Test::Nginx 提供了一个简单规范的格式来表达测试用例。每个测试用例都用一个测试块来表示。一个测试块由一个标题、一个可选的描述和几个数据节footnote:[data section 本书统一翻译为『数据节』]，用于指定输入和期望的输出。在这一节我们将仔细看看对于不同的测试要求，如何准备这样的测试案例。

设计测试用例在很多方面都是一门艺术。有时候可能设计测试案例花费的时间和精力，多于实现要测试的功能，这个和我们自己的经验有关。Test::Nginx 努力尝试让编写测试用例尽可能的简单，但仍然做不到让整个测试用例设计的过程自动化。只有你才确切的知道测试什么，以及如何去测试。本节讲侧重于 Test::Nginx 提供的基本原语，基于此你可以设计出巧妙和有效的测试用例。

=== 准备 NGINX 的配置

在一个测试块里面，我们可以用不同的数据节来指定我们自定义的代码片段，放在由 Test::Nginx 生成的最终的 nginx.conf 配置文件的不同位置中。

最常见的是 config 节，它用来在默认测试服务器的 server {} 配置块中插入自定义代码片段。我们也可以在 nginx.conf 的 http {} 配置块中，用 http_config 节来插入自定义的内容。 main_config 节可以在 NGINX 配置的最上层范围中插入自定义内容。一起看看下面这个例子。

[source,test-base]

=== TEST 1: — main_config env MY_ENVIRONMENT;

— http_config init_worker_by_lua_block { print(“init”) }

— config location = /t { echo ok; }

— request GET /t — response_body ok —-

这个测试块会生成一个有如下基本结构的 nginx.conf 文件：

[source,nginx]

… env MY_ENVIRONMENT;

http { …

init_worker_by_lua_block {
    print("init")
}

server {
    ...

    location = /t {
        echo ok;
    }
} } ----

请注意 main_config, http_config, 和 config 这几个数据节的值，是如何映射到 NGINX 配置文件的不同位置的。

有疑问的时候，我们总是可以来检查这个测试框架生成的实际 nginx.conf 文件，它位于当前工作目录（通常是当前项目的根目录）的 t/servroot/conf/nginx.conf 位置中。

Test::Nginx 会为每一个测试块生成一个新的 nginx.conf 文件，这会让每个测试块独立存在成为可能。测试框架默认会在运行每一个测试块之前，自动启动一个新的 NGINX 服务，然后在这个测试块运行结束后立即关闭服务。好在 NGINX 是一个轻量的服务器，通常启动和关闭都非常快。所以，测试块运行起来并没有看上去那么慢。

=== 准备发起请求

准备一个请求，最简单的是用 request 数据节, 比如

[source,test-base]

— request GET /t?a=1&b=2 —-

默认使用的是 HTTP/1.1 协议。如果你愿意，可以指明来使用 HTTP/1.0 协议:

[source,test-base]

— request GET /t?a=1&b=2 HTTP/1.0 —-

request 小节的值中，前导空格或者空白行都会被自动丢弃。你甚至可以在最前面加一个 # 字符来增加注释，比如

[source,test-base]

— request

# this is a simple test:
GET /t ----

你可以通过如下 more_headers 小节来同时增加一些额外的请求头部信息。

[source,test-base]

— request GET /t — more_headers Foo: bar Bar: baz —-

==== Pipelined Requests

Preparing pipelined HTTP requests are also possible. But you need to use the pipelined_requests section instead of request. For instance,

[source,test-base]

=== TEST 1: pipelined requests — config location = /t { echo ok; }

— pipelined_requests eval [“GET /t”, “GET /t”]

— request_body eval [“ok\n”, “ok\n”] —-

It is worth noting that we use the eval filter with the pipelined_requests section to treat the literal value of that section as Perl code. This way we can construct a Perl array of the request strings, which is the expected data format for the pipelined_requests section. Similarly we need a similar trick for the response_body section when checking outputs. With an array of expected response body data, we can expect and check different values for different individual request in the pipeline. Note, however, not every data section supports the same array-typed value semantics as response_body.

=== Checking Responses

We have already visited the response_body and error_code data sections for checking the response body data and response status code, respectively.

The response_body data section always performs an exact whole-string comparison between the section value and the actual response body. It tries to be clever when long string value comparison fails. Consider the following sample output from prove.

…. t/foo.t .. 1/?

Failed test ‘TEST 1: long string test - response_body - response is expected (req 0)’

at …/test-nginx/lib/Test/Nginx/Socket.pm line 1282.

got: …“IT 2.x is enabled.\x{0a}\x{0a}”…

length: 409

expected: …“IT 2.x is not enabled.\x{0a}”…

length: 412

strings begin to differ at char 400 (line 1 column 400)

Looks like you failed 1 test of 2.

/tmp/foo.t .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/2 subtests

Test Summary Report

/tmp/foo.t (Wstat: 256 Tests: 2 Failed: 1) Failed test: 2 Non-zero exit status: 1 Files=1, Tests=2, 0 wallclock secs (0.01 usr 0.00 sys + 0.09 cusr 0.03 csys = 0.13 CPU) Result: FAIL ….

From this test report, we can clearly see that

. it is the test block with the title TEST 1: long string test that is failing, . it is the response_body data section check that fails, . the actual response body data is 409 bytes long while the expected value is 412 bytes, and . the expected value has an additional not word in the string fragment IT 2.x is enabled and the difference starts at the offset 400 in the long string.

Behind the scene, Test::Nginx uses the Perl module link:https://metacpan.org/pod/Test::LongString[Test::LongString] to do the long string comparisons. It is also particularly useful while checking response body data in binary formats.

If your response body data is in a multi-line textual format, then you may also want to use a diff-style output when the data does not match. To achieve this, we can call the no_long_string() Perl function before the run_tests() function call in the prologue part of the test file. Below is such an example.

[source,test-base]

use Test::Nginx::Socket ‘no_plan’;

no_long_string();

run_tests();

DATA

=== TEST 1: — config location = /t { echo “Life is short.”; echo “Moon is bright.”; echo “Sun is shining.”; } — request GET /t — response_body Life is short. Moon is deem. Sun is shining. —-

Note the no_long_string() call in the prologue part. It is important to place it before the run_tests() call otherwise it would be too late for it to take effect, obviously.

Invoking the prove utility (or any shell wrappers for it) to run this test file gives the following details about the test failure:

….

Failed test ‘TEST 1: - response_body - response is expected (req 0)’

at …/test-nginx/lib/Test/Nginx/Socket.pm line 1277.

@@ -1,3 +1,3 @@

Life is short.

-Moon is deem.

+Moon is bright.

Sun is shining.

Looks like you failed 1 test of 2.

….

It is obvious that the second line of the response body output is different.

You can even further disable the diff-style comparison mode by adding a no_diff() Perl function call in the prologue part. Then the failure report will look like this:

….

Failed test ‘TEST 1: - response_body - response is expected (req 0)’

at …/test-nginx/lib/Test/Nginx/Socket.pm line 1277.

got: ‘Life is short.

Moon is bright.

Sun is shining.

’

expected: ‘Life is short.

Moon is deem.

Sun is shining.

’

Looks like you failed 1 test of 2.

….

That is, Test::Nginx just gives full listing of the actual response body data and the expected one without any abbreviations or hand-holding.

==== Pattern Matching on Response Bodies

When the request body may change in some ways or you just care about certain key words in a long data string, you can specify a Perl regular expression to do a pattern match against the actual request body data. This is achieved by the response_body_like data section. For example,

[source,test-base]

— response_body_like: age: \d+ —-

Be careful when you are using the multi-line data section value form. A trailing newline character appended to your section value may make your pattern never match. In this case the chomp filter we introduced in an early section can be very helpful here. For example,

[source,test-base]

— response_body_like chomp age: \d+ —-

You can also use the eval filter to construct a Perl regular expression object with a Perl expression, as in

[source,test-base]

— response_body_like eval qr/age: \d+/ —-

This is the most flexible form to specify a pattern.

NOTE: Perl uses the qr quoting structure to explicitly construct regular expression objects. You can use various different quoting forms like qr/.../, qr!...!, qr#...#, and qr{...}.

==== Checking Response Headers

The response_headers data section can be used to validate response header entries. For example,

[source,test-base]

— response_headers Foo: bar Bar: baz !Blah —-

This section dictates 3 tests actually:

. The response header Foo must appear and must take the value bar; . The response header Bar must appear and must take the value baz; and . The response header Blah must not appear or take an empty value.

=== Checking NGINX Error Logs

In addition to responses, the NGINX error log file is also an important output channel for an NGINX server setup.

==== True-False Tests

One immediate testing requirement is to check whether or not a piece of text appears in any error log messages. Such checks can be done via the data sections error_log and no_error_log, respectively. The former ensures that some lines in the error log file contain the string specified as the section value while the latter tests the opposite: ensuring that no line contains the pattern.

For example,

[source,test-base]

— error_log Hello world from my server —-

Then the string Hello world from my server (without the trailing new-line) must appear in at least one line of the NGINX error log. You can specify multiple strings in separate lines of the section value to perform different checks, for instance,

[source,test-base]

— error_log This is a dog! Is it a cat? —-

Then it performs two error log checks, one is to ensure that the string This is a dog! appears in some error log lines. The order of these two string patterns do not matter at all.

If one of the string pattern failed to match any lines in the error log file, then we would get a test failure report from prove like below.

….

Failed test ‘TEST 1: simple test - pattern “This is a dog!” matches a line in error.log (req 0)’

….

If you want to specify a Perl regular expression (regex) as one of the patterns, then you should use the eval section filter to construct a Perl-array as the section value, as in

[source,test-base]

— error_log eval [ “This is a dog!”, qr/\w+ is a cat\?/, ] —-

As we have seen earlier, Perl regexes can be constructed via the qr/.../ quoting syntax. Perl string patterns in the Perl array specified by double quotes or single quotes are still treated as plain string patterns, as usual. If the array contains only one regex pattern, then you can omit the array itself, as in

[source,test-base]

— error_log eval qr/\w+ is a cat\?/ —-

Test::Nginx puts the error log file of the test NGINX server in the file path t/servroot/logs/error.log. As a test writer, we frequently check out this file directly when things go wrong. For example, it is common to make mistakes or typos in the patterns we specify for the error_log section. Also, scanning the raw log file can give us insight about the details of the NGINX internal working when the NGINX debugging logs are enabled in the NGINX build.

The no_error_log section is very similar to error_log but it checks the nonexistence of the string patterns in the NGINX error log file. One of the most frequent uses of the no_error_log section is to ensure that there is no error level messages in the log file.

[source,test-base]

— no_error_log [error] —-

If, however, there is a line in the nginx error log file that contains the string [error], then the test fails. Below is such an example.

….

Failed test ‘TEST 1: simple test - pattern “[error]” should not match any line in error.log but matches line “2016/02/01 11:59:50 [error] 1788#0: *1 lua entry thread aborted: runtime error: content_by_lua(nginx.conf:42):2: bad”’

….

This is a great way to find the details of the error quickly by just looking at the test report.

Like error_log, this section also supports Perl array values and Perl regex values though the eval filter.

==== Grep Tests

The error_log and no_error_log sections are very handy in quickly checking the appearance of contain patterns in the NGINX error log file. But they have serious limitations in that it is impossible to impose stronger constraints on the relative order of the messages containing the patterns nor on the number of their occurrences.

To address such limitations, Test::Nginx::Socket provides an alternative way to check NGINX error logs in a way similar to the famous UNIX tool, grep. The sections grep_error_log and grep_error_log_out are used for this purpose. The test writer uses the grep_error_log section to specify a pattern, with which the test framework scans through the NGINX error log file and collect all the matched parts of the log file lines along the way, forming a final result. This aggregated log data result is then matched against the expected value specified as the value of the grep_error_log_out section, in a similar way as with the response_body section discussed above.

It is easiest to explain with a simple example.

[source,test-base]

=== TEST 1: simple grep test for error logs — config location = /t { content_by_lua_block { print(“it is matched!”) print(“it is matched!”) print(“it is matched!”) } } — request GET /t — grep_error_log: it is matched! — grep_error_log_out it is matched! it is matched! it is matched! —-

Here we use the Lua function print() provided by the link:https://github.com/openresty/lua-nginx-module#readme[ngx_http_lua] module to generate NGINX error log messages at the notice level. This test case tests the number of the log messages containing the string it is matched!. It is important to note that only the matched part of the log file lines are collected in the final result instead of the whole log lines. This simplifies the comparison a lot since NGINX error log messages can contain varying details like timestamps and connection numbers.

A more useful form of this test is to specify a Perl regex pattern in the grep_error_log section. Consider the following example.

[source,test-base]

=== TEST 1: simple grep test for error logs — config location = /t { content_by_lua_block { print(“test: before sleeping…”) ngx.sleep(0.001) – sleeping for 1ms print(“test: after sleeping…”) } } — request GET /t — grep_error_log eval: qr/test: .*?.../ — grep_error_log_out test: before sleeping… test: after sleeping… —-

We specify a Perl regex pattern, test: .*?\.\.\., here to filter out all the error log messages starting with test: and ending with .... And naturally in this test we also require the relative order of these two messages, that is, before sleeping must appear before after sleeping. Otherwise, we shall see failure reports like below:

….

Failed test ‘TEST 1: simple grep test for error logs - grep_error_log_out (req 0)’

at …./lib/Test/Nginx/Socket.pm line 1048.

got: “test: after sleeping…\x{0a}test: before sleeping…\x{0a}”

length: 49

expected: “test: before sleeping…\x{0a}test: after sleeping…\x{0a}”

length: 49

strings begin to differ at char 7 (line 1 column 7)

….

As with the response_body section, we can also call the no_long_string() Perl function before run_tests() in the test file prologue, so as to disable the long string output mode and enable the diff mode. Then the test failure would look like this:

….

Failed test ‘TEST 1: simple grep test for error logs - grep_error_log_out (req 0)’

at …/lib/Test/Nginx/Socket.pm line 1044.

@@ -1,2 +1,2 @@

-test: before sleeping…

test: after sleeping…

+test: before sleeping…

….

Obviously, for this test case, the diff format looks better.

==== Extra Delay Before Log Checks

By default, Test::Nginx::Socket performs the NGINX error log checks not long after it receives the complete HTTP response for the test request. Sometimes, when the log messages are generated by the server after sending out the response, the error log checks may be carried out too early that the messages are not yet written into the log file. In this case, we can specify an extra delay via the wait data section for the test scaffold to wait for the error log messages. Here is an example:

[source,test-base]

=== TEST 1: wait for the timer — config location = /t { content_by_lua_block { local function f(premature) print(“HERE!”) end assert(ngx.timer.at(0.1, f)) } } — request GET /t — error_log HERE! — no_error_log [error] — wait: 0.12 —-

Here we create a timer via the ngx.timer.at Lua function, which expires after 0.1 seconds. Due to the asynchronous nature of timers, the request handler does not wait for the timer to expire and immediately finishes processing the current request and sends out a response with an empty body. To check for the log message HERE! generated by the timer handler f, we have to specify an extra delay for the test scaffold to wait. The 0.12 seconds time is specified in this example but any values larger than 0.1 would suffice. Without the wait section, this test case would fail with the following output:

….

Failed test ‘TEST 1: wait for the timer - pattern “HERE!” matches a line in error.log (req 0)’

….

Obviously the test scaffold checks the error log too soon, even before the timer handler runs.

=== Section Review

Test::Nginx::Socket offers a rich set of data sections for specifying various different input data and expected output data, ranging from NGINX configuration file snippets, test requests, to expected responses and error log messages. We have already demonstrated the power of data driven testing and declarative test case crafting. We want to achieve multiple goals at the same time, that is, not only to make the tests self-contained and highly readable, but also to make the test report easy to interpret and analyze when some of the tests fail. Raw files automatically generated by the test scaffold, like t/servroot/conf/nginx.conf and t/servroot/logs/error.log, should be checked frequently when manually debugging the test cases. The next section extends the discussion of this section with a focus on testing erroneous cases.

Tags: