undef/git-commit-vandalism
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

test-lib: Adjust output to be valid TAP format

Browse Source 
TAP, the Test Anything Protocol, is a simple text-based interface
between testing modules in a test harness. test-lib.sh's output was
already very close to being valid TAP. This change brings it all the
way there. Before:

    $ ./t0005-signals.sh
    *   ok 1: sigchain works
    * passed all 1 test(s)

And after:

    $ ./t0005-signals.sh
    ok 1 - sigchain works
    # passed all 1 test(s)
    1..1

The advantage of using TAP is that any program that reads the format
(a "test harness") can run the tests. The most popular of these is the
prove(1) utility that comes with Perl. It can run tests in parallel,
display colored output, format the output to console, file, HTML etc.,
and much more. An example:

    $ prove ./t0005-signals.sh
    ./t0005-signals.sh .. ok
    All tests successful.
    Files=1, Tests=1,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.01 cusr  0.02 csys =  0.06 CPU)
    Result: PASS

prove(1) gives you human readable output without being too
verbose. Running the test suite in parallel with `make test -j15`
produces a flood of text. Running them with `prove -j 15 ./t[0-9]*.sh`
makes it easy to follow what's going on.

All this patch does is re-arrange the output a bit so that it conforms
with the TAP spec, everything that the test suite did before continues
to work. That includes aggregating results in t/test-results/, the
--verbose, --debug and other options for tests, and the test color
output.

TAP harnesses ignore everything that they don't know about, so running
the tests with --verbose works:

    $ prove ./t0005-signals.sh :: --verbose --debug
    ./t0005-signals.sh .. Terminated
    ./t0005-signals.sh .. ok
    All tests successful.
    Files=1, Tests=1,  0 wallclock secs ( 0.02 usr  0.01 sys +  0.01 cusr  0.01 csys =  0.05 CPU)
    Result: PASS

Just supply the -v option to prove itself to get all the verbose
output that it suppresses:

    $ prove -v ./t0005-signals.sh :: --verbose --debug
    ./t0005-signals.sh ..
    Initialized empty Git repository in /home/avar/g/git/t/trash directory.t0005-signals/.git/
    expecting success:
            test-sigchain >actual
            case "$?" in
            143) true ;; # POSIX w/ SIGTERM=15
              3) true ;; # Windows
              *) false ;;
            esac &&
            test_cmp expect actual
    Terminated
    ok 1 - sigchain works
    # passed all 1 test(s)
    1..1
    ok
    All tests successful.
    Files=1, Tests=1,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.01 cusr  0.01 csys =  0.04 CPU)
    Result: PASS

As a further example, consider this test script that uses a lot of
test-lib.sh features by Jakub Narebski:

    #!/bin/sh

    test_description='this is a sample test.

    This test is here to see various test outputs.'

    . ./test-lib.sh

    say 'diagnostic message'

    test_expect_success 'true  test' 'true'
    test_expect_success 'false test' 'false'

    test_expect_failure 'true  test (todo)' 'true'
    test_expect_failure 'false test (todo)' 'false'

    test_debug 'echo "debug message"'

    test_done

The output of that was previously:

    * diagnostic message                      # yellow
    *   ok 1: true  test
    * FAIL 2: false test                      # bold red
            false
    *   FIXED 3: true  test (todo)
    *   still broken 4: false test (todo)     # bold green
    * fixed 1 known breakage(s)               # green
    * still have 1 known breakage(s)          # bold red
    * failed 1 among remaining 3 test(s)      # bold red

But is now:

    diagnostic message                                    # yellow
    ok 1 - true  test
    not ok - 2 false test                                 # bold red
    #       false
    ok 3 - true  test (todo) # TODO known breakage
    not ok 4 - false test (todo) # TODO known breakage    # bold green
    # fixed 1 known breakage(s)                           # green
    # still have 1 known breakage(s)                      # bold red
    # failed 1 among remaining 3 test(s)                  # bold red
    1..4

All the coloring is preserved when the test is run manually. Under
prove(1) the test performs as expected, even with --debug and
--verbose options:

    $ prove ./example.sh :: --debug --verbose
    ./example.sh .. Dubious, test returned 1 (wstat 256, 0x100)
    Failed 1/4 subtests
            (1 TODO test unexpectedly succeeded)

    Test Summary Report
    -------------------
    ./example.sh (Wstat: 256 Tests: 4 Failed: 1)
      Failed test:  2
      TODO passed:   3
      Non-zero exit status: 1
    Files=1, Tests=4,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.00 cusr  0.01 csys =  0.03 CPU)
    Result: FAIL

The TAP harness itself doesn't get confused by the color output, they
aren't used by test-lib.sh stdout isn't open to a terminal (test -t 1).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This commit is contained in:
Ævar Arnfjörð Bjarmason 2010-06-24 21:52:12 +00:00 committed by Junio C Hamano
parent ba4d01bd74
commit 5099b99d25
2 changed files with 57 additions and 28 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

55
t/README
View File

@ -18,25 +18,48 @@ The easiest way to run tests is to say "make". This runs all
the tests.
*** t0000-basic.sh ***
* ok 1: .git/objects should be empty after git-init in an empty repo.
* ok 2: .git/objects should have 256 subdirectories.
* ok 3: git-update-index without --add should fail adding.
...
* ok 23: no diff after checkout and git-update-index --refresh.
* passed all 23 test(s)
*** t0100-environment-names.sh ***
* ok 1: using old names should issue warnings.
* ok 2: using old names but having new names should not issue warnings.
ok 1 - .git/objects should be empty after git init in an empty repo.
ok 2 - .git/objects should have 3 subdirectories.
ok 3 - success is reported like this
...
ok 43 - very long name in the index handled sanely
# fixed 1 known breakage(s)
# still have 1 known breakage(s)
# passed all remaining 42 test(s)
1..43
*** t0001-init.sh ***
ok 1 - plain
ok 2 - plain with GIT_WORK_TREE
ok 3 - plain bare
Or you can run each test individually from command line, like
this:
Since the tests all output TAP (see http://testanything.org) they can
be run with any TAP harness. Here's an example of paralell testing
powered by a recent version of prove(1):
$ sh ./t3001-ls-files-killed.sh
* ok 1: git-update-index --add to add various paths.
* ok 2: git-ls-files -k to show killed files.
* ok 3: validate git-ls-files -k output.
* passed all 3 test(s)
$ prove --timer --jobs 15 ./t[0-9]*.sh
[19:17:33] ./t0005-signals.sh ................................... ok 36 ms
[19:17:33] ./t0022-crlf-rename.sh ............................... ok 69 ms
[19:17:33] ./t0024-crlf-archive.sh .............................. ok 154 ms
[19:17:33] ./t0004-unwritable.sh ................................ ok 289 ms
[19:17:33] ./t0002-gitfile.sh ................................... ok 480 ms
===( 102;0 25/? 6/? 5/? 16/? 1/? 4/? 2/? 1/? 3/? 1... )===
prove and other harnesses come with a lot of useful options. The
--state option in particular is very useful:
# Repeat until no more failures
$ prove -j 15 --state=failed,save ./t[0-9]*.sh
You can also run each test individually from command line, like this:
$ sh ./t3010-ls-files-killed-modified.sh
ok 1 - git update-index --add to add various paths.
ok 2 - git ls-files -k to show killed files.
ok 3 - validate git ls-files -k output.
ok 4 - git ls-files -m to show modified files.
ok 5 - validate git ls-files -m output.
# passed all 5 test(s)
1..5
You can pass --verbose (or -v), --debug (or -d), and --immediate
(or -i) command line argument to the test, or by setting GIT_TEST_OPTS

30
t/test-lib.sh
View File

@ -160,7 +160,7 @@ if test -n "$color"; then
*) test -n "$quiet" && return;;
esac
shift
printf "* %s" "$*"
printf "%s" "$*"
tput sgr0
echo
)
@ -169,7 +169,7 @@ else
say_color() {
test -z "$1" && test -n "$quiet" && return
shift
echo "* $*"
echo "$*"
}
fi
@ -339,25 +339,25 @@ test_have_prereq () {
test_ok_ () {
test_success=$(($test_success + 1))
say_color "" " ok $test_count: $@"
say_color "" "ok $test_count - $@"
}
test_failure_ () {
test_failure=$(($test_failure + 1))
say_color error "FAIL $test_count: $1"
say_color error "not ok - $test_count $1"
shift
echo "$@" | sed -e 's/^/ /'
echo "$@" | sed -e 's/^/# /'
test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
}
test_known_broken_ok_ () {
test_fixed=$(($test_fixed+1))
say_color "" " FIXED $test_count: $@"
say_color "" "ok $test_count - $@ # TODO known breakage"
}
test_known_broken_failure_ () {
test_broken=$(($test_broken+1))
say_color skip " still broken $test_count: $@"
say_color skip "not ok $test_count - $@ # TODO known breakage"
}
test_debug () {
@ -390,7 +390,7 @@ test_skip () {
case "$to_skip" in
t)
say_color skip >&3 "skipping test: $@"
say_color skip "skip $test_count: $1"
say_color skip "ok $test_count: # skip $1"
: true
;;
*)
@ -620,18 +620,22 @@ test_done () {
if test "$test_fixed" != 0
then
say_color pass "fixed $test_fixed known breakage(s)"
say_color pass "# fixed $test_fixed known breakage(s)"
fi
if test "$test_broken" != 0
then
say_color error "still have $test_broken known breakage(s)"
say_color error "# still have $test_broken known breakage(s)"
msg="remaining $(($test_count-$test_broken)) test(s)"
else
msg="$test_count test(s)"
fi
case "$test_failure" in
0)
say_color pass "passed all $msg"
# Maybe print SKIP message
[ -z "$skip_all" ] || skip_all=" # SKIP $skip_all"
say_color pass "# passed all $msg"
say "1..$test_count$skip_all"
test -d "$remove_trash" &&
cd "$(dirname "$remove_trash")" &&
@ -640,7 +644,9 @@ test_done () {
exit 0 ;;
*)
say_color error "failed $test_failure among $msg"
say_color error "# failed $test_failure among $msg"
say "1..$test_count"
exit 1 ;;
esac