t3st

Lightweight, flexible shell TAP testing library

TLDR

• clone this repo and run prove -v t/t3st-hello.t :: --help (...helo.t is a lightweight demo that also prints further instructions)
• read the self-tests in t3st.t (which uses the t3st-ttt0.sh framework)
• run /src/t3st/git-t3st-setup inside an existing git project and look in t/

t3st is a shell library that produces TAP testing output. It can test any shell function, or commands or scripts. It requires only a POSIX shell (but works under bash / zsh as well as several sh variants — busybox / mksh / BSD sh / others). A TAP framework (prove comes with any system perl) is also assumed. The defaults make tests easy to write without sacrificing correctness or flexibility. The test API completely avoids namespace pollution, which guarantees complete interoperability with any source. Features:

• easy setup: adding to existing projects (git-t3st-setup), and running multiple tests under multiple shells (git t3st-prove / make -C t/) are one-liners. In addition to the basic library, there is an "opinionated" framework reducing boilerplate / friction to a minimum.
• output + exit code testing, with precise final newline handling
• full shell integration: tests are real shell scripts with no special formatting or syntax magic; you can request tests conditionally / from loops / inside subshells and pipes / from wrapper functions
• usable for POSIX sh code, as well as bash / zsh
• errexit (set -e) and set -u tests
• repeated tests (for stress-testing / benchmarking)
• TAP directives (TODO)

Notes

• development takes place at GitLab t3st but you can also report issues at the GitHub t3st mirror.
• watch the changelog if using the git version — there are no API guarantees at this stage

Highlights

• Installing
• Usage
  • t3st-ttt0 framework
  • Hand-rolled
  • errexit & nounset
• Test function
  • Subshells, pipes
  • Redirects
• Extras
• File-based (_me) tests
• Using prove
• Supported shells
• Copyright & license

Installing

Add t3st to a project directly:

cd myproject
URL=https://gitlab.com/kstr0k/t3st/-/raw/master/git-t3st-setup
curl -s "$URL" | sh  # or ... | sh -s -- --tdir=./mytests

Use prove, or the git aliases and Makefile targets, to run / add tests :

prove -v t/t3st-hello.t :: --help
git t3st-new         # or make -C t/ new
git t3st-prove [-v]  # from anywhere in repo, multiple shells
git t3st-setup       # update / repair
git config t3st.prove-shells 'sh,bash#,etc'  # saved in repo's .git/config

The script adds to repo/.git/config file (displayed at startup), but won't overwrite unrelated (or subsequently modified) settings. Specifically, it

• sets up a no-tags, no-push kstr0k-t3st git remote
• creates a test directory (--tdir=t/ by default) and copies the library to it directly from git. It also adds a simple t3st-hello.t example with instructions, and a more complete framework (t/t3st-lib/t3st-ttt0.sh) for heavier development.
• adds git aliases (which conveniently work from anywhere in the repo) to
  • run the tests in multiple shells: git t3st-prove (controlled by git config t3st.prove-shells)
  • create a new test: git t3st-new
  • re-run itself: git t3st-setup
• creates a t3st.mk makefile (symlinked to Makefile) with t3st-prove (default) and new targets. Use with make -C t/
• accepts special setup parameters: --reset (removes all t3st-related git settings as a first step); --no-setup: don't re-add t3st settings (combine with --reset)

For manual installation, clone this repo and run git-t3st-setup [--help] from another project. Or just copy k9s0ke_t3st_lib.sh in a testsuite.

Usage

Once you have a t/ folder with "*.t" tests, run them using prove (or any TAP test harness):

prove -v  # or change the shell:
prove -e 'busybox sh'
prove -vr tests  # rename t/, recursive

With the t3st-ttt0 framework

ttt0 is a framework that builds on t3st, organizes boilerplate / tricky code into overridable methods, and provides sensible defaults. It can be helpful, but is not required (see Hand-rolled).

To get started, run git t3st-new (or make -C t/ new) after importing t3st into your project (git-t3st-setup). Rename and edit the generated new.t and new-e.t (or delete the latter). new.t sources t/t3st-lib/t3st-ttt0.sh and defines a set of tests in TTT__tfile_tests; if needed, extend (see below) or replace methods inside TTT__tfile_entry. See the test function, now aliased to TTT, TTT_de / TTT_ee and TTT_xe (the _?e ones for specific errexit disable / enable modes; _xe calls TTT_de, then TTT_ee — it's not a single test).

The *.t methods (TTT__tfile_*, — a file-wide namespace prefix) receive whatever arguments the test harness supplies (e.g. prove .. :: ARG..). You can override them; once ...METHOD is overridden, you can still access the initial default with the corresponding ..._METHOD_0 (i.e. internally, ...METHOD simply calls ...METHOD_0). After _early, they can all use ..._my{path|name|dirn} path-related globals. The methods are (users are normally concerned with the first three, and possibly ..._parse_args):

• ..._tests (in new.t): add tests here
• ..._setup (in t3st-ttt0): by default, sources k9s0ke_t3st_lib.sh and defines TTT*. Extend this to source other libraries.
• ..._thelp (in t3st-ttt0): by default, prints some debug info.
• ..._entry (in new.t): called by *.t itself if run as a script, or by any script using *.t as a library. Sets up ..._mypath based on the absolute path to *.t and sources the ttt0 framework (getting this right is rather tricky — some naive attempts to use $0 fail under different scenarios / shells)
• ..._early (in t3st-ttt0): called right after sourcing t3st-ttt0 with an additional $1 argument bound to the original script's $0. Sets up some globals ($...myname = basename of *.t, $..._mydirn). If you want to override it, do so after ttt0 is sourced, for obvious reasons.
• ..._runme (in t3st-ttt0): sequences the other functions. Checks for --help, calls ..._setup, k9s0ke_t3st_enter's, then calls ..._tests, and finally k9s0ke_t3st_leave's. Internally, it uses …
• ..._parse_args: processes command-line switches (optional parameters) one at a time and calls itself recursively. When switches are exhausted (or a -- is encountered), calls ..._parse_args_end with the remaining positional parameters. Override / extend ..._parse_args to define new switches.

Hand-rolled

A sample.t file (using defaults aggressively, and peculiar formatting to highlight tested code) might look like

#!/bin/sh
# real shell script -- no syntax / formatting magic

. "$(dirname -- "$0")/k9s0ke_t3st_lib.sh  # or wherever
TTT() { k9s0ke_t3st_one "$@"; }           # or whatever

k9s0ke_t3st_enter

TTT spec='as bare as it gets' \
  -- echo
TTT nl=false rc='-ne 0' spec='standard command "false" -> non-0 exit status' \
  -- false
TTT out=// hook_test_pre='cd /' spec='use eval for multiple commands' \
  -- eval 'printf $PWD; pwd'

if (type __str_subst >/dev/null 2>&1); then
  TTT out=abcbcbc nl=false \
  -- __str_subst abbb b bc
else k9s0ke_t3st_skip 1 \
  'no str_subst (http://gitlab.com/kstr0k/mru-files.kak/-/tree/master/k9s0ke-shlib)'
fi

k9s0ke_t3st_leave

That is: source k9s0ke_t3st_lib.sh (along with any tested code you need to reference) and call

• k9s0ke_t3st_enter [test_plan] to start TAP output. Omit the plan to have the library count tests automatically and print the plan at the end.
• k9s0ke_t3st_one [param=value]... [-- cmd args...] (see test function for defaults) for each test; it executes everything after "--" (a single command or function call — use eval '...' otherwise) in a subshell and checks the output and exit status. Usually aliased to TTT.
• k9s0ke_t3st_me — alternative file-based tests (see _me tests)
• k9s0ke_t3st_leave [test_plan]: ends TAP output. If no plan was given to ..._enter, it prints the supplied test plan, or generates one that matches the total number of test calls ("1..k9s0ke_t3st_one-call-count"). The simplest setup is to not pass a test plan to either _enter or leave.

errexit

Don't "set -e" globally (i.e. outside a _one or _me call): the library code will refuse to run (it can't properly record exit statuses with set -e). Instead, either

• define a shortcut function (TTT_ee() { k9s0ke_t3st_one errexit=true $@; }), OR
• use errexit=true in individual ..._one / ..._me calls, OR
• set a global errexit default (k9s0ke_t3st_g_errexit=true) in the .t file or in the environment (k9s0ke_t3st_g_errexit=true prove...), OR
• set -e inside the tested code

To run tests with both set -e and set +e, create a ...-e.t file which adds a global errexit default, then sources the base .t file. The -e.t file can also define additional tests. t/t3st-e.t implements this; so do the new.t / new-e.t generated files (see t3st-ttt0);

set -u does not affect operation — set it either way globally and/or use set_pre=[-/+]u parameters (or the $k9s0ke_t3st_g_set_pre global default).

Test function

Call k9s0ke_t3st_one once per test in *.t (you may want to alias it, e.g. to TTT, possibly with some pre-set parameters). Minimal, though contrived, succeeding tests are the first tests in t/t3st.t

TTT() { k9s0ke_t3st_one "$@"; }

TTT -- echo   # expect out=('' + default \n)
TTT in=       # stdin = '' + \n; implied command = cat; expect '' + \n
TTT nl=false  # stdin = /dev/null; expect out=''

These illustrate the defaults: call cat if no command is supplied, expect exit status rc=0, expect output out='', add a final newline to the expected output (nl=true), stdin from /dev/null. The available arguments (before "--"; all optional, in any order; some defaults can be overridden by setting a corresponding k9s0ke_t3st_g_... global) are:

• nl={ true | false }: adds a newline to the expected out=, as well as any in= parameters described below (but not to infile= / outfile=). This is the default (most commands work with full lines); override with nl=false.
• out='expected...' (default: empty) compare the command's output (including any final newline) to the specified string (plus a newline if nl=true). For more complex conditions, use pp= (extras).
• outfile='...': load out= from a file (won't clobber host files, despite the name); nl= does not apply.
• infile={ 'path' | [-] }: redirect the command's input, or leave stdin alone (use caller's environment); without any infile= (or in=), all tests run with input from /dev/null. nl= has no influence.
• in='...': input to pass on stdin to the command; an additional newline is added with (the default) nl=true (even for an empty in). For a completely empty input, don't specify either in= or infile=; or use in= nl=false. For a single newline, use in= or in="$k9s0ke_t3st_nl" nl=false. As noted above, nl= also affects expected output.
• rc={ $rc | '-$cmp $rc'}: compare the command's exit status ($?) to the supplied value / uses the value as a shell test condition (e.g. rc='-lt 2' checks that $? is 0 or 1). If omitted, the expected exit is 0; if set to '', the exit status is ignored.
• spec='...': print this after the ok / not ok result (in particular, "# TODO" directives mark sub-tests as possibly failing, without causing the entire test file to fail). prove displays test names using this bit of output. Defaults to the first word of the command. You can also use todo='comment..' (syntactic sugar) and specfmt= below.
• pp='shell code...': post-process the output ($1) and exit status ($2). This code runs within a temporary function; whatever it outputs replaces the original output, and its exit status (from its last statement, or an explicit return) replaces the original $?. The rc= and out= parameters then match against these post-processed values. Enables arbitrarily complex tests.
• errexit={ true | false }: run the test under set -e conditions. Defaults to false or the global *_g_* override. Do not 'set -e' globally in *.t.
• set_pre={ -? | +? }* (e.g. set_pre=-f turns off globbing). Defaults to nothing or the global *_g_* override.
• repeat=N: repeat this test N times, or until it first fails. Defaults to 1, or $k9s0ke_t3st_g_repeat.
• cnt={ true | false }: the test counter $k9s0ke_t3st_cnt normally auto-increments; this can be disabled for pipes and subshells — see below

Note that shell variables (including in=, out=, and the internal variable that stores actual output) cannot hold NUL (\0) characters. The infile=, however, as well as pipes / redirects, can. Preprocess any NULs' before they reach the library (e.g. eval '... | tr \\0 \\n'; pp= won't help).

Redirects

infile= can be used with any local files (permanent or created on the fly, e.g. in $k9s0ke_t3st_tmp_dir). Here-doc (<<'EOF' / <<EOF with expansions) redirects work with infile=-, but can only create newline-terminated inputs.

You can specify redirects (or anything that changes the environment) in the pre-test hook, which runs in the same subshell as the tested command:

TTT hook_test_pre='cd /tmp || exit; exec 2>&1' ...

The standard error log of each test is normally pasted as TAP '# ' comments below the test (prove -v displays them); exec 2>/dev/null in the hook gets rid of it.

Subshells and pipes

k9s0ke_t3st_one can run in a pipeline, but it might execute in a different (forked) process than the main script. Pass infile=- (avoids the default </dev/null), cnt=false (in case the test part of the pipeline runs in the script process after all), and increment the counter manually after each such test:

echo 'XX YY' | k9s0ke_t3st_one out=XX infile=- cnt=false \
  -- eval 'read -r x rest; echo "$x"'
k9s0ke_t3st_cnt=$(( k9s0ke_t3st_cnt + 1 ))

This is also necessary if you call k9s0ke_t3st_one from a subshell. If the forked process might run an undetermined number of tests, use

• k9s0ke_t3st_cnt_save at the end of a subshell / pipe
• k9s0ke_t3st_cnt_load back in the top-level shell

Extras

• k9s0ke_t3st_bailout [message]: stop testing, output a TAP bailout marker, exit. Undefined behavior if you call this from a subshell / pipe.
• k9s0ke_t3st_skip skip_count reason: mark a few tests as skipped. This keeps the total number of tests constant with conditional tests. By default the plan (including the final test counter) is printed at the end, so this is not required (but helps with debugging).
• k9s0ke_t3st_g_on_fail={bailout | skip-rest | ignore-rest } (experimental): bailout or skip / ignore all tests after first (non-TODO) failure
• k9s0ke_t3st_g_specfmt (- $1 by default): a format string applied to both the implicit and explicit spec; set it to - $* (and possibly include other variables) to make implicit test names show all ..._one arguments instead of just $1. Double quotes must be escaped within specfmt. Also available as a specfmt= parameter in ..._one.
• ..._one hook_test_pre=...: code to be eval'd before the test command (defaults to k9s0ke_t3st_g_hook_test_pre, or empty). The framework adds additional code to this hook (errexit / set_pre setup, redirects).
• ..._one diff_on={ ok, | notok, }*: print actual vs expected results (as TAP "# ..." comments) for some tests. The default is notok, or $k9s0ke_t3st_g_diff_on if defined. Use '=ok,notok' to print all diffs or '=,' to print none.
• ..._one supports key+=value arguments (which append to previous values, or the default). For example, you can have a TTT_myfun wrapper which calls ..._one including a spec= argument, then call TTT_myfun spec+='...'
• you can inject arbitrary variables into ..._one via v:VARNAME=value (no "+=" support yet)

Utilities

• out_rc=$( k9s0ke_t3st_slurp_exec 'prelude' [cmd args]... ): load a command's output plus exit code into a shell variable. Before executing the command, eval() the prelude (e.g. set -e). Use this, along with k9s0ke_t3st_slurp_split, to avoid truncating final newlines, as the $() construct does in all shells. If no command is supplied, it runs cat; to slurp a file, use ...slurp_exec <...
• k9s0ke_t3st_slurp_split "$out_rc" outvar [rcvar]: split an out_rc string (as obtained above); sets outvar to the output and rcvar (if supplied and not empty) to the exit status. Both *var parameters are variable names (don't prepend a "$")
• $k9s0ke_t3st_tmp_dir is a temporary workdir (removed at the end, unless k9s0ke_t3st_g_keep_tmp = true). You can use it, but paths starting with .t3st* are reserved for the library.
• k9s0ke_t3st_mktemp outvar creates a temporary file and sets outvar to its path. It is automatically removed when testing ends (..._leave or ..._bailout).
• k9s0ke_t3st_dump_str str outputs a compact one-line representation of a string (using perl if available and not explicitly disabled by setting k9s0ke_t3st__perl to '').

For convenience, the library defines a few character constants, most notably k9s0ke_t3st_nl (\n), but also a tab, ['"<>|&;] etc (named k9s0ke_t3st_ch_ + the HTML entity name mostly — see the source)

File-based (_me) tests

TLDR: t/t3st.t contains two _me tests, complete with .out, .rc and .exec.

Call k9s0ke_t3st_me FILE [PARAM=..].. (like _one but without -- cmd..); instead of rc=, in= and out=, create FILE.{rc,in,out} files. nl=false is assumed. The actual command can be supplied as an initial exec=... argument (after FILE). If no exec= is supplied, _me() (unlike _one() which defaults to cat) looks for an .exec file and uses that.

_me() calls _one() internally, so the in, rc, and out defaults still apply, and other parameters can be supplied.

The expected output .out is read in a shell variable, so it still can't contain NUL (\0) characters. The .in file (if any), however, is used as an infile= parameter (a real redirect) and thus can contain anything.

..._me() can be called multiple times with different arguments, and doesn't preclude invoking regular _one() in the same .t file. You may wish to stick to one style per test file for clarity, though.

Supported shells

While the library itself only uses POSIX shell code, it can test scripts that require bash (or others) — the library code works in several shells. Use an appropriate shebang in *.t, or pass prove a -e SHELL argument. The following is being used to test t3st itself (with no errors):

for shell in dash bash bash44 bash32 'busybox sh' mksh yash zsh 'zsh --emulate sh' posh
  do printf '\n%s\n' "$shell"; prove -e "$shell"
done
# or `git t3st-prove`

Individual shell notes

• posh doesn't honor set -e inside eval; explicit errexit=true tests are auto-skipped.
• FreeBSD sh: has exhibited nested parameter expansion bugs (t3st does not currently use this)

zsh

t3st itself does not depend on the following behaviors, but zsh has not been fully tested, so there may be other problems. The major differences from POSIX seem to be that by default:

• sh_option_letters = off; some set -? options have a different meaning (in particular, 'set -F', rather than '-f', is noglob). Avoid this by using set {+o|-o} OPTNAME instead of option letters (+o = disabled); this makes code portable to both POSIX shells and zsh (for POSIX options, that is).
• shwordsplit = off
• nomatch = on (causes failures with set -e when globs fail to match)
• posixcd = off (directories starting with +/- are reinterpreted as dirstack entries)
• posixargzero = off ($0 switches to the function name inside a function)
• glob_subst = off: zsh does not honor globs (*, ?) in ${var##$pat}, ${var%$pat} etc parameter expansions (unless in ksh / sh emulation). This only applies to patterns supplied via a variable. For example, pat='/*'; var=/etc; echo ${var##$pat} yields '' in sh, but not in zsh.

Use set {-|+}o OPTNAME individually to turn these on/off (don't combine, for maximum portability, and definitely don't omit {+|-}o before each option). Or invoke zsh with --emulate [k]sh to turn on POSIX mode / ksh mode (the latter is close to bash). emulate also works as a command, or as a wrapper (emulate sh -c '...'). Check "[ "${ZSH_VERSION:-}" ]" to see if running under zsh (possibly in emulation).

Using prove

prove (distributed with perl) acts as both a test harness and a TAP producer. t3st relies on prove for shell scripts (not perl modules), so some command-line options are more relevant than others (for the full details, review the prove perldoc):

• prove [options] [FILE-or-DIR].. [ :: SCRIPT_ARG..]: prove accepts multiple directories and/or files; with no positional parameters, it looks for a t/ directory. Inside directories (but not their subdirs), it looks for .t files.
• everything after :: — passed to every *.t (e.g. --help, --db=...).
• -r: recurse into subdirs too
• -v: verbose mode; without it, you won't see individual test names, actual-vs-expected lines, SKIP / TODO's etc — only a summary. It doesn't combine nicely with other flags (-j), however.
• -e SHELL: controls what shell prove uses for all *.t's. Otherwise, executable .t's use their own shebang (#!) line, i.e. probably /bin/sh. Non-executable .t's get run with Perl, which won't do much good for shell scripts. SHELL can be a command, such as 'busybox sh' or 'zsh --emulate sh'.
• -j9: enables parallel execution (don't combine with -v)
• -Q: really quiet — no progress
• --timer
• -a tap.tgz: produces an archive of TAP results, which you can pass to other utilities. The *.t in the archive are, somewhat confusingly, TAP logs named identically to the tests that produced them.
• --formatter=TAP::Formatter::HTML

Splitting up tests

prove -j<INT> runs test files (but not individual tests) in parallel. Put slow tests in separate .t's (and use e.g. prove -j9) if testing takes too long.

See also

• Projects using t3st:
  • t3st includes self-tests based on ttt0 (t/t3st.t)
  • The mru-files.kak test branch has a self-contained t3st + git setup, with separate worktrees / branches. That project includes a POSIX shell library, the test file for which can also serve as inspiration.
  • bashaaparse's min-template.sh (a sh / bash / zsh argument parser) has tests that use temporary files, pp= post-processing and hook_test_pre to enforce complex conditions (grep in stderr, check globals assigned by code)
• TAP consumers: if you want to go beyond the widely available prove command. The language they're written in doesn't matter as long as they can parse TAP output. For example, ESR's tapview. You'll still need to generate the TAP output in the first place; see "Using prove" (-a tap.tgz).
• other frameworks: shellspec, sharness, bats-core, shspec, assert.sh

Copyright

Alin Mr. <almr.oss@outlook.com> / MIT license