(Release 1.7. The RegTest script is in the public domain.)
RegTest is a very simple script for writing IF unit tests. You write down a list of commands and the output you want from each one. RegTest will run the list through your game, and check for that output.
RegTest can work with any interpreter which uses stdin/stdout, such as DumbFrotz or Glulxe/CheapGlk. (But see "Limitations", below.)
All of what RegTest does is defined by a test file. The easiest way to explain it is to paste one in. (With a soupçon of syntax coloring, for documentation's sake.)
# advent-regtest: test script for regtest.py
# For a full description, see <http://eblong.com/zarf/plotex/regtest.html>
** game: /Users/zarf/Documents/IF/Advent.ulx
** interpreter: /Users/zarf/bin/glulxec -q
* south-from-start
# A simple, one-command test.
> south
You are in a valley in the forest beside a stream tumbling along a rocky bed.
* in-well-house
# Test the opening text, followed by two commands. Lines starting
# with "!" are negated; lines starting with "/" are regular expressions.
Welcome to Adventure!
Crowther
Woods
> go east
There is tasty food here.
some keys
!grue
> get all
/b[aeiou]ttle.*water
!/^Taken
* test-that-fails
# All three of the tests in this run will fail.
> go east
There is a bucket of cheese here.
/[xqz]
! Inside Building
* remglk-test
# This test will only work if your interpreter uses the RemGlk library,
# and regtest is invoked with the --rem option. Otherwise, you'll see
# three test failures.
> go east
{status} Inside Building
spring
> help
{status} About Adventure
>{char} N
>{char} N
>{char}
The probabilities are as in the original game.
>{char} 32
>{char} Q
{status} Score
You are inside a building
> get food
Taken.
The first two lines are comments. Lines beginning with "#", and blank lines, are ignored.
The next two lines (beginning with "**") define test parameters -- the location of the game file and interpreter. The game will be the Glulx version of Adventure (compiled with Inform 6). The interpreter will be Glulxe/CheapGlk. I've defined pathnames in my computer's filesystem; you'd want to change those, of course. (You can also supply these values from the command line.)
A line beginning with "** precommand:" is an extra command that will be stuck onto the beginning of every test defined in the file.
A line beginning with "** checkclass:" specifies a (Python) file containing extra check classes. I won't get into the details here, but see this sample file.
The rest of the test file is a set of tests. Each test is a separate run through the game. A test contains a sequence of commands. A command can contain various checks, validating the output of that command.
(All the "**" lines should appear before the tests begin.) (Okay, you could customize the game file or interpreter for a specific test if you really wanted. But why?) (This is a rhetorical question.)
The line "* south-from-start" defines the beginning of the first test. south-from-start is the test name. (You can name tests anything you want; it's just a convenient label.)
This test contains just one command -- south. The next line is a check: RegTest will search the command's output for this line. It's the room description for the room to the south, obviously.
The second test is called in-well-house. Here we start by performing some checks on the banner text of the game. (Note that this test is a fresh start; the previous "south" command was in a different run.) RegTest verifies that "Welcome to Adventure!" occurs somewhere in the game's initial output. Then it looks for "Crowther" and "Woods", which also occur. (These aren't complete lines, but that's fine -- the check line just has to occur somewhere in one of the paragraphs that begin the game. The two name tests happen to occur in the same line; that's fine too.)
After the initial text, we go east. We're applying three different checks to the output of "go east". RegTest verifies that "There is tasty food here." and "some keys" both occur. (Remember, we're looking only at the output of the latest command, not the entire transcript.)
A check line starting with "!" is negated: RegTest verifies that none of the output contains the word grue. Which is good, because there are no grues in Colossal Cave. You can also use "{invert}" as the line prefix.
These are independent checks; order doesn't matter. (The line about the keys actually occurs before the one about the food.)
The idea is that you don't usually want to verify every single character of your game output. During development, you're going to be changing descriptions, adding objects, and so on. But you still might want to write a test sequence for particular actions. By checking only for the important bits of each response, you don't have to fix the test every time a room description or timer event changes.
The next command demonstrates regular expressions. A check line that begins with "/" is matched as a regular expression. (See the Python documentation for the syntax of regular expressions.) Here we have a (contrived) regex which matches the output line "stream: The bottle is now full of water."
A line starting with "!/" is, unsurprisingly, a negated regex check. The line "!/^Taken" verifies that no line of the output begins with the word Taken. (The word occurs within several lines, but not at the beginning of any.)
The last test, remglk-test, is its own crazy thing. We will discuss it momentarily.
To run all tests, paste the test script into a file, and then type:
python regtest.py TESTFILE
When you do this, you will see the output:
* south-from-start * in-well-house * test-that-fails <LiteralCheck "There is a bucket of cheese here...">: not found <RegExpCheck "[xqz]">: not found <LiteralCheck !"Inside Building">: inverse test should fail * remglk-test <LiteralCheck "Inside Building">: not found <LiteralCheck "About Adventure">: not found Exception: Cheap mode only supports line input FAILED: 6 errors
The first two tests show no problems. The other two fail three checks each. When you see failures, you'll probably want to re-run a single test:
python regtest.py TESTFILE -v test-that-fails
This runs only the specified test. The -v (or --verbose) argument displays the complete transcript of the test run, with the failures marked, so you can see exactly what went wrong.
You can run several tests by naming them all, or by using a glob-style wildcard. (You'll probably have to quote the wildcard to keep your shell from mangling it.)
python regtest.py TESTFILE south-from-start in-well-house python regtest.py TESTFILE 'test-*'
These options are available:
**game: line in the test script.)
**interpreter: line in the test script.)
**precommand: lines in the test script.)
**checkclass: lines in the test script.)
Sometimes you want to wrap up a sequence of commands as a "macro", to be invoked in several different tests.
To do this, add a command line like this:
>{include} TESTNAME
You can name any other test in the file. Its commands (and checks) will be executed at this point in your test.
(No space between the ">" and the "{". Checks after an >{include} line are meaningless; they are ignored.)
You typically won't want a subtest to be invoked by itself. (The player won't start in the right place, so the subtest's checks will fail.) To make this convenient, give the subtest a name beginning with "-" or "_". Such tests will not be run when you invoke RegTest in all-tests mode (or with "*").
Normally, RegTest handles IF output in a very simplistic way. Because the stdin/stdout model has no facility for a status line, there's no way to test the status line's contents. Also, RegTest will only work with a game that abides by these rules:
The prompt must always be ">" at the beginning of a line.
In particular, Inform's "if the player consents" (yes/no) questions will confuse RegTest -- it won't recognize them as input requests. The same goes for menu-based input.
">" at the beginning of a line must always be a prompt.
If your game prints ">" at the beginning of a line, even if text follows, RegTest will think it is an input request and fire the next command.
This is not very flexible. Can we do better? We can -- but we'll require a special interpreter.
If your interpreter is compiled with the RemGlk library, it will output the full display state of the game, structured as a JSON file. This means that RegTest can see the contents of the status line, and handle more complex I/O requests.
(The JSON format for the game's output, and its input, is described in this document. But you don't need to understand the details to use RegTest.)
The last test in the test file, remglk-test, makes use of this feature. To make it work, compile Glulxe and RemGlk, and then change the **interpreter line to refer to the new interpreter binary. You can then run RegTest with the --rem option. (This tells RegTest to expect JSON-formatted output, rather than plain text.)
python regtest.py --rem TESTFILE
The remglk-test will now succeed. (test-that-fails will still throw its three errors.)
The test demonstrates two special features: character input and status line output. We enter the game menus by typing "help"; we then navigate to one of the menu options and trigger it. We test the option's output -- this is the "How authentic is this edition?" text. Then we hit space (ASCII 32) to return to the menu, then "Q" to return to the game. We can then proceed with game commands as before.
> help
{status} About Adventure
>{char} N
These features are signified by lines with {curly brace tags}, as shown above.
When writing these input forms in a test, do not put any whitespace between the ">" and "{" characters. An input line like "> {foo}" is treated as regular line input, entering the string "{foo}".
left, escape, etc) are accepted, as are ASCII or Unicode code as decimal or hexadecimal. If you do not provide a value, RegTest assumes a Return keystroke.
-support timer option to RemGlk.)
-support hyperlinks option to RemGlk.)
debuginput event. Note that RegTest cannot currently check debug output.
You may put any of these prefixes before a check. They may be combined freely (except for {status} and {graphics}).
RegTest currently assumes that there is no more than one status (textgrid) window and no more than one graphics window.
Any of these checks may be combined with the modifiers listed above.
{graphics} {json special:'fill' color:'#FF0000' width:10 height:10}
{graphics} modifier (see above) on a {json} check.
Last updated January 23, 2017.
Other IF scripts: PlotEx