2020-06-07 11:48:26 +02:00
|
|
|
Core Git Tests
|
2005-05-14 09:25:06 +02:00
|
|
|
==============
|
|
|
|
|
2020-06-07 11:48:26 +02:00
|
|
|
This directory holds many test scripts for core Git tools. The
|
2005-05-14 09:25:06 +02:00
|
|
|
first part of this short document describes how to run the tests
|
|
|
|
and read their output.
|
|
|
|
|
|
|
|
When fixing the tools or adding enhancements, you are strongly
|
|
|
|
encouraged to add tests in this directory to cover what you are
|
|
|
|
trying to fix or enhance. The later part of this short document
|
|
|
|
describes how your test scripts should be organized.
|
|
|
|
|
|
|
|
|
|
|
|
Running Tests
|
|
|
|
-------------
|
|
|
|
|
|
|
|
The easiest way to run tests is to say "make". This runs all
|
|
|
|
the tests.
|
|
|
|
|
|
|
|
*** t0000-basic.sh ***
|
test-lib: Adjust output to be valid TAP format
TAP, the Test Anything Protocol, is a simple text-based interface
between testing modules in a test harness. test-lib.sh's output was
already very close to being valid TAP. This change brings it all the
way there. Before:
$ ./t0005-signals.sh
* ok 1: sigchain works
* passed all 1 test(s)
And after:
$ ./t0005-signals.sh
ok 1 - sigchain works
# passed all 1 test(s)
1..1
The advantage of using TAP is that any program that reads the format
(a "test harness") can run the tests. The most popular of these is the
prove(1) utility that comes with Perl. It can run tests in parallel,
display colored output, format the output to console, file, HTML etc.,
and much more. An example:
$ prove ./t0005-signals.sh
./t0005-signals.sh .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.01 cusr 0.02 csys = 0.06 CPU)
Result: PASS
prove(1) gives you human readable output without being too
verbose. Running the test suite in parallel with `make test -j15`
produces a flood of text. Running them with `prove -j 15 ./t[0-9]*.sh`
makes it easy to follow what's going on.
All this patch does is re-arrange the output a bit so that it conforms
with the TAP spec, everything that the test suite did before continues
to work. That includes aggregating results in t/test-results/, the
--verbose, --debug and other options for tests, and the test color
output.
TAP harnesses ignore everything that they don't know about, so running
the tests with --verbose works:
$ prove ./t0005-signals.sh :: --verbose --debug
./t0005-signals.sh .. Terminated
./t0005-signals.sh .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.01 cusr 0.01 csys = 0.05 CPU)
Result: PASS
Just supply the -v option to prove itself to get all the verbose
output that it suppresses:
$ prove -v ./t0005-signals.sh :: --verbose --debug
./t0005-signals.sh ..
Initialized empty Git repository in /home/avar/g/git/t/trash directory.t0005-signals/.git/
expecting success:
test-sigchain >actual
case "$?" in
143) true ;; # POSIX w/ SIGTERM=15
3) true ;; # Windows
*) false ;;
esac &&
test_cmp expect actual
Terminated
ok 1 - sigchain works
# passed all 1 test(s)
1..1
ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.01 cusr 0.01 csys = 0.04 CPU)
Result: PASS
As a further example, consider this test script that uses a lot of
test-lib.sh features by Jakub Narebski:
#!/bin/sh
test_description='this is a sample test.
This test is here to see various test outputs.'
. ./test-lib.sh
say 'diagnostic message'
test_expect_success 'true test' 'true'
test_expect_success 'false test' 'false'
test_expect_failure 'true test (todo)' 'true'
test_expect_failure 'false test (todo)' 'false'
test_debug 'echo "debug message"'
test_done
The output of that was previously:
* diagnostic message # yellow
* ok 1: true test
* FAIL 2: false test # bold red
false
* FIXED 3: true test (todo)
* still broken 4: false test (todo) # bold green
* fixed 1 known breakage(s) # green
* still have 1 known breakage(s) # bold red
* failed 1 among remaining 3 test(s) # bold red
But is now:
diagnostic message # yellow
ok 1 - true test
not ok - 2 false test # bold red
# false
ok 3 - true test (todo) # TODO known breakage
not ok 4 - false test (todo) # TODO known breakage # bold green
# fixed 1 known breakage(s) # green
# still have 1 known breakage(s) # bold red
# failed 1 among remaining 3 test(s) # bold red
1..4
All the coloring is preserved when the test is run manually. Under
prove(1) the test performs as expected, even with --debug and
--verbose options:
$ prove ./example.sh :: --debug --verbose
./example.sh .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/4 subtests
(1 TODO test unexpectedly succeeded)
Test Summary Report
-------------------
./example.sh (Wstat: 256 Tests: 4 Failed: 1)
Failed test: 2
TODO passed: 3
Non-zero exit status: 1
Files=1, Tests=4, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.00 cusr 0.01 csys = 0.03 CPU)
Result: FAIL
The TAP harness itself doesn't get confused by the color output, they
aren't used by test-lib.sh stdout isn't open to a terminal (test -t 1).
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-06-24 23:52:12 +02:00
|
|
|
ok 1 - .git/objects should be empty after git init in an empty repo.
|
|
|
|
ok 2 - .git/objects should have 3 subdirectories.
|
|
|
|
ok 3 - success is reported like this
|
2005-05-14 09:25:06 +02:00
|
|
|
...
|
test-lib: Adjust output to be valid TAP format
TAP, the Test Anything Protocol, is a simple text-based interface
between testing modules in a test harness. test-lib.sh's output was
already very close to being valid TAP. This change brings it all the
way there. Before:
$ ./t0005-signals.sh
* ok 1: sigchain works
* passed all 1 test(s)
And after:
$ ./t0005-signals.sh
ok 1 - sigchain works
# passed all 1 test(s)
1..1
The advantage of using TAP is that any program that reads the format
(a "test harness") can run the tests. The most popular of these is the
prove(1) utility that comes with Perl. It can run tests in parallel,
display colored output, format the output to console, file, HTML etc.,
and much more. An example:
$ prove ./t0005-signals.sh
./t0005-signals.sh .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.01 cusr 0.02 csys = 0.06 CPU)
Result: PASS
prove(1) gives you human readable output without being too
verbose. Running the test suite in parallel with `make test -j15`
produces a flood of text. Running them with `prove -j 15 ./t[0-9]*.sh`
makes it easy to follow what's going on.
All this patch does is re-arrange the output a bit so that it conforms
with the TAP spec, everything that the test suite did before continues
to work. That includes aggregating results in t/test-results/, the
--verbose, --debug and other options for tests, and the test color
output.
TAP harnesses ignore everything that they don't know about, so running
the tests with --verbose works:
$ prove ./t0005-signals.sh :: --verbose --debug
./t0005-signals.sh .. Terminated
./t0005-signals.sh .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.01 cusr 0.01 csys = 0.05 CPU)
Result: PASS
Just supply the -v option to prove itself to get all the verbose
output that it suppresses:
$ prove -v ./t0005-signals.sh :: --verbose --debug
./t0005-signals.sh ..
Initialized empty Git repository in /home/avar/g/git/t/trash directory.t0005-signals/.git/
expecting success:
test-sigchain >actual
case "$?" in
143) true ;; # POSIX w/ SIGTERM=15
3) true ;; # Windows
*) false ;;
esac &&
test_cmp expect actual
Terminated
ok 1 - sigchain works
# passed all 1 test(s)
1..1
ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.01 cusr 0.01 csys = 0.04 CPU)
Result: PASS
As a further example, consider this test script that uses a lot of
test-lib.sh features by Jakub Narebski:
#!/bin/sh
test_description='this is a sample test.
This test is here to see various test outputs.'
. ./test-lib.sh
say 'diagnostic message'
test_expect_success 'true test' 'true'
test_expect_success 'false test' 'false'
test_expect_failure 'true test (todo)' 'true'
test_expect_failure 'false test (todo)' 'false'
test_debug 'echo "debug message"'
test_done
The output of that was previously:
* diagnostic message # yellow
* ok 1: true test
* FAIL 2: false test # bold red
false
* FIXED 3: true test (todo)
* still broken 4: false test (todo) # bold green
* fixed 1 known breakage(s) # green
* still have 1 known breakage(s) # bold red
* failed 1 among remaining 3 test(s) # bold red
But is now:
diagnostic message # yellow
ok 1 - true test
not ok - 2 false test # bold red
# false
ok 3 - true test (todo) # TODO known breakage
not ok 4 - false test (todo) # TODO known breakage # bold green
# fixed 1 known breakage(s) # green
# still have 1 known breakage(s) # bold red
# failed 1 among remaining 3 test(s) # bold red
1..4
All the coloring is preserved when the test is run manually. Under
prove(1) the test performs as expected, even with --debug and
--verbose options:
$ prove ./example.sh :: --debug --verbose
./example.sh .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/4 subtests
(1 TODO test unexpectedly succeeded)
Test Summary Report
-------------------
./example.sh (Wstat: 256 Tests: 4 Failed: 1)
Failed test: 2
TODO passed: 3
Non-zero exit status: 1
Files=1, Tests=4, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.00 cusr 0.01 csys = 0.03 CPU)
Result: FAIL
The TAP harness itself doesn't get confused by the color output, they
aren't used by test-lib.sh stdout isn't open to a terminal (test -t 1).
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-06-24 23:52:12 +02:00
|
|
|
ok 43 - very long name in the index handled sanely
|
|
|
|
# fixed 1 known breakage(s)
|
|
|
|
# still have 1 known breakage(s)
|
|
|
|
# passed all remaining 42 test(s)
|
|
|
|
1..43
|
|
|
|
*** t0001-init.sh ***
|
|
|
|
ok 1 - plain
|
|
|
|
ok 2 - plain with GIT_WORK_TREE
|
|
|
|
ok 3 - plain bare
|
|
|
|
|
|
|
|
Since the tests all output TAP (see http://testanything.org) they can
|
2010-07-02 16:59:44 +02:00
|
|
|
be run with any TAP harness. Here's an example of parallel testing
|
test-lib: Adjust output to be valid TAP format
TAP, the Test Anything Protocol, is a simple text-based interface
between testing modules in a test harness. test-lib.sh's output was
already very close to being valid TAP. This change brings it all the
way there. Before:
$ ./t0005-signals.sh
* ok 1: sigchain works
* passed all 1 test(s)
And after:
$ ./t0005-signals.sh
ok 1 - sigchain works
# passed all 1 test(s)
1..1
The advantage of using TAP is that any program that reads the format
(a "test harness") can run the tests. The most popular of these is the
prove(1) utility that comes with Perl. It can run tests in parallel,
display colored output, format the output to console, file, HTML etc.,
and much more. An example:
$ prove ./t0005-signals.sh
./t0005-signals.sh .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.01 cusr 0.02 csys = 0.06 CPU)
Result: PASS
prove(1) gives you human readable output without being too
verbose. Running the test suite in parallel with `make test -j15`
produces a flood of text. Running them with `prove -j 15 ./t[0-9]*.sh`
makes it easy to follow what's going on.
All this patch does is re-arrange the output a bit so that it conforms
with the TAP spec, everything that the test suite did before continues
to work. That includes aggregating results in t/test-results/, the
--verbose, --debug and other options for tests, and the test color
output.
TAP harnesses ignore everything that they don't know about, so running
the tests with --verbose works:
$ prove ./t0005-signals.sh :: --verbose --debug
./t0005-signals.sh .. Terminated
./t0005-signals.sh .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.01 cusr 0.01 csys = 0.05 CPU)
Result: PASS
Just supply the -v option to prove itself to get all the verbose
output that it suppresses:
$ prove -v ./t0005-signals.sh :: --verbose --debug
./t0005-signals.sh ..
Initialized empty Git repository in /home/avar/g/git/t/trash directory.t0005-signals/.git/
expecting success:
test-sigchain >actual
case "$?" in
143) true ;; # POSIX w/ SIGTERM=15
3) true ;; # Windows
*) false ;;
esac &&
test_cmp expect actual
Terminated
ok 1 - sigchain works
# passed all 1 test(s)
1..1
ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.01 cusr 0.01 csys = 0.04 CPU)
Result: PASS
As a further example, consider this test script that uses a lot of
test-lib.sh features by Jakub Narebski:
#!/bin/sh
test_description='this is a sample test.
This test is here to see various test outputs.'
. ./test-lib.sh
say 'diagnostic message'
test_expect_success 'true test' 'true'
test_expect_success 'false test' 'false'
test_expect_failure 'true test (todo)' 'true'
test_expect_failure 'false test (todo)' 'false'
test_debug 'echo "debug message"'
test_done
The output of that was previously:
* diagnostic message # yellow
* ok 1: true test
* FAIL 2: false test # bold red
false
* FIXED 3: true test (todo)
* still broken 4: false test (todo) # bold green
* fixed 1 known breakage(s) # green
* still have 1 known breakage(s) # bold red
* failed 1 among remaining 3 test(s) # bold red
But is now:
diagnostic message # yellow
ok 1 - true test
not ok - 2 false test # bold red
# false
ok 3 - true test (todo) # TODO known breakage
not ok 4 - false test (todo) # TODO known breakage # bold green
# fixed 1 known breakage(s) # green
# still have 1 known breakage(s) # bold red
# failed 1 among remaining 3 test(s) # bold red
1..4
All the coloring is preserved when the test is run manually. Under
prove(1) the test performs as expected, even with --debug and
--verbose options:
$ prove ./example.sh :: --debug --verbose
./example.sh .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/4 subtests
(1 TODO test unexpectedly succeeded)
Test Summary Report
-------------------
./example.sh (Wstat: 256 Tests: 4 Failed: 1)
Failed test: 2
TODO passed: 3
Non-zero exit status: 1
Files=1, Tests=4, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.00 cusr 0.01 csys = 0.03 CPU)
Result: FAIL
The TAP harness itself doesn't get confused by the color output, they
aren't used by test-lib.sh stdout isn't open to a terminal (test -t 1).
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-06-24 23:52:12 +02:00
|
|
|
powered by a recent version of prove(1):
|
|
|
|
|
|
|
|
$ prove --timer --jobs 15 ./t[0-9]*.sh
|
|
|
|
[19:17:33] ./t0005-signals.sh ................................... ok 36 ms
|
|
|
|
[19:17:33] ./t0022-crlf-rename.sh ............................... ok 69 ms
|
|
|
|
[19:17:33] ./t0024-crlf-archive.sh .............................. ok 154 ms
|
|
|
|
[19:17:33] ./t0004-unwritable.sh ................................ ok 289 ms
|
|
|
|
[19:17:33] ./t0002-gitfile.sh ................................... ok 480 ms
|
|
|
|
===( 102;0 25/? 6/? 5/? 16/? 1/? 4/? 2/? 1/? 3/? 1... )===
|
|
|
|
|
|
|
|
prove and other harnesses come with a lot of useful options. The
|
|
|
|
--state option in particular is very useful:
|
|
|
|
|
|
|
|
# Repeat until no more failures
|
|
|
|
$ prove -j 15 --state=failed,save ./t[0-9]*.sh
|
2005-05-14 09:25:06 +02:00
|
|
|
|
2010-10-14 10:53:36 +02:00
|
|
|
You can give DEFAULT_TEST_TARGET=prove on the make command (or define it
|
|
|
|
in config.mak) to cause "make test" to run tests under prove.
|
|
|
|
GIT_PROVE_OPTS can be used to pass additional options, e.g.
|
|
|
|
|
|
|
|
$ make DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS='--timer --jobs 16' test
|
|
|
|
|
test-lib: Adjust output to be valid TAP format
TAP, the Test Anything Protocol, is a simple text-based interface
between testing modules in a test harness. test-lib.sh's output was
already very close to being valid TAP. This change brings it all the
way there. Before:
$ ./t0005-signals.sh
* ok 1: sigchain works
* passed all 1 test(s)
And after:
$ ./t0005-signals.sh
ok 1 - sigchain works
# passed all 1 test(s)
1..1
The advantage of using TAP is that any program that reads the format
(a "test harness") can run the tests. The most popular of these is the
prove(1) utility that comes with Perl. It can run tests in parallel,
display colored output, format the output to console, file, HTML etc.,
and much more. An example:
$ prove ./t0005-signals.sh
./t0005-signals.sh .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.01 cusr 0.02 csys = 0.06 CPU)
Result: PASS
prove(1) gives you human readable output without being too
verbose. Running the test suite in parallel with `make test -j15`
produces a flood of text. Running them with `prove -j 15 ./t[0-9]*.sh`
makes it easy to follow what's going on.
All this patch does is re-arrange the output a bit so that it conforms
with the TAP spec, everything that the test suite did before continues
to work. That includes aggregating results in t/test-results/, the
--verbose, --debug and other options for tests, and the test color
output.
TAP harnesses ignore everything that they don't know about, so running
the tests with --verbose works:
$ prove ./t0005-signals.sh :: --verbose --debug
./t0005-signals.sh .. Terminated
./t0005-signals.sh .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.01 cusr 0.01 csys = 0.05 CPU)
Result: PASS
Just supply the -v option to prove itself to get all the verbose
output that it suppresses:
$ prove -v ./t0005-signals.sh :: --verbose --debug
./t0005-signals.sh ..
Initialized empty Git repository in /home/avar/g/git/t/trash directory.t0005-signals/.git/
expecting success:
test-sigchain >actual
case "$?" in
143) true ;; # POSIX w/ SIGTERM=15
3) true ;; # Windows
*) false ;;
esac &&
test_cmp expect actual
Terminated
ok 1 - sigchain works
# passed all 1 test(s)
1..1
ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.01 cusr 0.01 csys = 0.04 CPU)
Result: PASS
As a further example, consider this test script that uses a lot of
test-lib.sh features by Jakub Narebski:
#!/bin/sh
test_description='this is a sample test.
This test is here to see various test outputs.'
. ./test-lib.sh
say 'diagnostic message'
test_expect_success 'true test' 'true'
test_expect_success 'false test' 'false'
test_expect_failure 'true test (todo)' 'true'
test_expect_failure 'false test (todo)' 'false'
test_debug 'echo "debug message"'
test_done
The output of that was previously:
* diagnostic message # yellow
* ok 1: true test
* FAIL 2: false test # bold red
false
* FIXED 3: true test (todo)
* still broken 4: false test (todo) # bold green
* fixed 1 known breakage(s) # green
* still have 1 known breakage(s) # bold red
* failed 1 among remaining 3 test(s) # bold red
But is now:
diagnostic message # yellow
ok 1 - true test
not ok - 2 false test # bold red
# false
ok 3 - true test (todo) # TODO known breakage
not ok 4 - false test (todo) # TODO known breakage # bold green
# fixed 1 known breakage(s) # green
# still have 1 known breakage(s) # bold red
# failed 1 among remaining 3 test(s) # bold red
1..4
All the coloring is preserved when the test is run manually. Under
prove(1) the test performs as expected, even with --debug and
--verbose options:
$ prove ./example.sh :: --debug --verbose
./example.sh .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/4 subtests
(1 TODO test unexpectedly succeeded)
Test Summary Report
-------------------
./example.sh (Wstat: 256 Tests: 4 Failed: 1)
Failed test: 2
TODO passed: 3
Non-zero exit status: 1
Files=1, Tests=4, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.00 cusr 0.01 csys = 0.03 CPU)
Result: FAIL
The TAP harness itself doesn't get confused by the color output, they
aren't used by test-lib.sh stdout isn't open to a terminal (test -t 1).
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-06-24 23:52:12 +02:00
|
|
|
You can also run each test individually from command line, like this:
|
2005-05-14 09:25:06 +02:00
|
|
|
|
test-lib: Adjust output to be valid TAP format
TAP, the Test Anything Protocol, is a simple text-based interface
between testing modules in a test harness. test-lib.sh's output was
already very close to being valid TAP. This change brings it all the
way there. Before:
$ ./t0005-signals.sh
* ok 1: sigchain works
* passed all 1 test(s)
And after:
$ ./t0005-signals.sh
ok 1 - sigchain works
# passed all 1 test(s)
1..1
The advantage of using TAP is that any program that reads the format
(a "test harness") can run the tests. The most popular of these is the
prove(1) utility that comes with Perl. It can run tests in parallel,
display colored output, format the output to console, file, HTML etc.,
and much more. An example:
$ prove ./t0005-signals.sh
./t0005-signals.sh .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.01 cusr 0.02 csys = 0.06 CPU)
Result: PASS
prove(1) gives you human readable output without being too
verbose. Running the test suite in parallel with `make test -j15`
produces a flood of text. Running them with `prove -j 15 ./t[0-9]*.sh`
makes it easy to follow what's going on.
All this patch does is re-arrange the output a bit so that it conforms
with the TAP spec, everything that the test suite did before continues
to work. That includes aggregating results in t/test-results/, the
--verbose, --debug and other options for tests, and the test color
output.
TAP harnesses ignore everything that they don't know about, so running
the tests with --verbose works:
$ prove ./t0005-signals.sh :: --verbose --debug
./t0005-signals.sh .. Terminated
./t0005-signals.sh .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.01 cusr 0.01 csys = 0.05 CPU)
Result: PASS
Just supply the -v option to prove itself to get all the verbose
output that it suppresses:
$ prove -v ./t0005-signals.sh :: --verbose --debug
./t0005-signals.sh ..
Initialized empty Git repository in /home/avar/g/git/t/trash directory.t0005-signals/.git/
expecting success:
test-sigchain >actual
case "$?" in
143) true ;; # POSIX w/ SIGTERM=15
3) true ;; # Windows
*) false ;;
esac &&
test_cmp expect actual
Terminated
ok 1 - sigchain works
# passed all 1 test(s)
1..1
ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.01 cusr 0.01 csys = 0.04 CPU)
Result: PASS
As a further example, consider this test script that uses a lot of
test-lib.sh features by Jakub Narebski:
#!/bin/sh
test_description='this is a sample test.
This test is here to see various test outputs.'
. ./test-lib.sh
say 'diagnostic message'
test_expect_success 'true test' 'true'
test_expect_success 'false test' 'false'
test_expect_failure 'true test (todo)' 'true'
test_expect_failure 'false test (todo)' 'false'
test_debug 'echo "debug message"'
test_done
The output of that was previously:
* diagnostic message # yellow
* ok 1: true test
* FAIL 2: false test # bold red
false
* FIXED 3: true test (todo)
* still broken 4: false test (todo) # bold green
* fixed 1 known breakage(s) # green
* still have 1 known breakage(s) # bold red
* failed 1 among remaining 3 test(s) # bold red
But is now:
diagnostic message # yellow
ok 1 - true test
not ok - 2 false test # bold red
# false
ok 3 - true test (todo) # TODO known breakage
not ok 4 - false test (todo) # TODO known breakage # bold green
# fixed 1 known breakage(s) # green
# still have 1 known breakage(s) # bold red
# failed 1 among remaining 3 test(s) # bold red
1..4
All the coloring is preserved when the test is run manually. Under
prove(1) the test performs as expected, even with --debug and
--verbose options:
$ prove ./example.sh :: --debug --verbose
./example.sh .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/4 subtests
(1 TODO test unexpectedly succeeded)
Test Summary Report
-------------------
./example.sh (Wstat: 256 Tests: 4 Failed: 1)
Failed test: 2
TODO passed: 3
Non-zero exit status: 1
Files=1, Tests=4, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.00 cusr 0.01 csys = 0.03 CPU)
Result: FAIL
The TAP harness itself doesn't get confused by the color output, they
aren't used by test-lib.sh stdout isn't open to a terminal (test -t 1).
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-06-24 23:52:12 +02:00
|
|
|
$ sh ./t3010-ls-files-killed-modified.sh
|
|
|
|
ok 1 - git update-index --add to add various paths.
|
|
|
|
ok 2 - git ls-files -k to show killed files.
|
|
|
|
ok 3 - validate git ls-files -k output.
|
|
|
|
ok 4 - git ls-files -m to show modified files.
|
|
|
|
ok 5 - validate git ls-files -m output.
|
|
|
|
# passed all 5 test(s)
|
|
|
|
1..5
|
2005-05-14 09:25:06 +02:00
|
|
|
|
|
|
|
You can pass --verbose (or -v), --debug (or -d), and --immediate
|
2009-02-04 00:25:59 +01:00
|
|
|
(or -i) command line argument to the test, or by setting GIT_TEST_OPTS
|
test-lib: allow short options to be bundled
When debugging a test (or a set of tests), it's common to execute it
with some combination of short options, such as:
$ ./txxx-testname.sh -d -x -i
In cases like this, CLIs usually allow the short options to be bundled
in a single argument, for convenience and agility. Let's add this
feature to test-lib, allowing the above command to be run as:
$ ./txxx-testname.sh -dxi
(or any other permutation, e.g. '-ixd')
Note: Short options that require an argument can also be used in a
bundle, in any position. So, for example, '-r 5 -x', '-xr 5' and '-rx 5'
are all valid and equivalent. A special case would be having a bundle
with more than one of such options. To keep things simple, this case is
not allowed for now. This shouldn't be a major limitation, though, as
the only short option that requires an argument today is '-r'. And
concatenating '-r's as in '-rr 5 6' would probably not be very
practical: its unbundled format would be '-r 5 -r 6', for which test-lib
currently considers only the last argument. Therefore, if '-rr 5 6' were
to be allowed, it would have the same effect as just typing '-r 6'.
Note: the test-lib currently doesn't support '-r5', as an alternative
for '-r 5', so the former is not supported in bundles as well.
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-03-22 16:58:57 +01:00
|
|
|
appropriately before running "make". Short options can be bundled, i.e.
|
|
|
|
'-d -v' is the same as '-dv'.
|
2005-05-14 09:25:06 +02:00
|
|
|
|
2014-04-30 11:50:42 +02:00
|
|
|
-v::
|
2005-05-14 09:25:06 +02:00
|
|
|
--verbose::
|
|
|
|
This makes the test more verbose. Specifically, the
|
|
|
|
command being run and their output if any are also
|
|
|
|
output.
|
|
|
|
|
2013-06-23 20:12:56 +02:00
|
|
|
--verbose-only=<pattern>::
|
|
|
|
Like --verbose, but the effect is limited to tests with
|
|
|
|
numbers matching <pattern>. The number matched against is
|
|
|
|
simply the running count of the test within the file.
|
|
|
|
|
test-lib.sh: support -x option for shell-tracing
Usually running a test under "-v" makes it clear which
command is failing. However, sometimes it can be useful to
also see a complete trace of the shell commands being run in
the test. You can do so without any support from the test
suite by running "sh -x tXXXX-foo.sh". However, this
produces quite a large bit of output, as we see a trace of
the entire test suite.
This patch instead introduces a "-x" option to the test
scripts (i.e., "./tXXXX-foo.sh -x"). When enabled, this
turns on "set -x" only for the tests themselves. This can
still be a bit verbose, but should keep things to a more
manageable level. You can even use "--verbose-only" to see
the trace only for a specific test.
The implementation is a little invasive. We turn on the "set
-x" inside the "eval" of the test code. This lets the eval
itself avoid being reported in the trace (which would be
long, and redundant with the verbose listing we already
showed). And then after the eval runs, we do some trickery
with stderr to avoid showing the "set +x" to the user.
We also show traces for test_cleanup functions (since they
can impact the test outcome, too). However, we do avoid
running the noop ":" cleanup (the default if the test does
not use test_cleanup at all), as it creates unnecessary
noise in the "set -x" output.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-10-10 08:47:27 +02:00
|
|
|
-x::
|
|
|
|
Turn on shell tracing (i.e., `set -x`) during the tests
|
2018-02-24 00:39:50 +01:00
|
|
|
themselves. Implies `--verbose`.
|
2018-02-24 00:39:42 +01:00
|
|
|
Ignored in test scripts that set the variable 'test_untraceable'
|
|
|
|
to a non-empty value, unless it's run with a Bash version
|
|
|
|
supporting BASH_XTRACEFD, i.e. v4.1 or later.
|
test-lib.sh: support -x option for shell-tracing
Usually running a test under "-v" makes it clear which
command is failing. However, sometimes it can be useful to
also see a complete trace of the shell commands being run in
the test. You can do so without any support from the test
suite by running "sh -x tXXXX-foo.sh". However, this
produces quite a large bit of output, as we see a trace of
the entire test suite.
This patch instead introduces a "-x" option to the test
scripts (i.e., "./tXXXX-foo.sh -x"). When enabled, this
turns on "set -x" only for the tests themselves. This can
still be a bit verbose, but should keep things to a more
manageable level. You can even use "--verbose-only" to see
the trace only for a specific test.
The implementation is a little invasive. We turn on the "set
-x" inside the "eval" of the test code. This lets the eval
itself avoid being reported in the trace (which would be
long, and redundant with the verbose listing we already
showed). And then after the eval runs, we do some trickery
with stderr to avoid showing the "set +x" to the user.
We also show traces for test_cleanup functions (since they
can impact the test outcome, too). However, we do avoid
running the noop ":" cleanup (the default if the test does
not use test_cleanup at all), as it creates unnecessary
noise in the "set -x" output.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-10-10 08:47:27 +02:00
|
|
|
|
2014-04-30 11:50:42 +02:00
|
|
|
-d::
|
2005-05-14 09:25:06 +02:00
|
|
|
--debug::
|
|
|
|
This may help the person who is developing a new test.
|
|
|
|
It causes the command defined with test_debug to run.
|
2011-03-15 20:58:14 +01:00
|
|
|
The "trash" directory (used to store all temporary data
|
|
|
|
during testing) is not deleted even if there are no
|
|
|
|
failed tests so that you can inspect its contents after
|
|
|
|
the test finished.
|
2005-05-14 09:25:06 +02:00
|
|
|
|
2014-04-30 11:50:42 +02:00
|
|
|
-i::
|
2005-05-14 09:25:06 +02:00
|
|
|
--immediate::
|
|
|
|
This causes the test to immediately exit upon the first
|
2013-04-09 23:48:36 +02:00
|
|
|
failed test. Cleanup commands requested with
|
|
|
|
test_when_finished are not executed if the test failed,
|
|
|
|
in order to keep the state for inspection by the tester
|
|
|
|
to diagnose the bug.
|
2005-05-14 09:25:06 +02:00
|
|
|
|
2014-04-30 11:50:42 +02:00
|
|
|
-l::
|
2008-06-17 03:29:02 +02:00
|
|
|
--long-tests::
|
|
|
|
This causes additional long-running tests to be run (where
|
|
|
|
available), for more exhaustive testing.
|
|
|
|
|
2014-04-30 11:50:44 +02:00
|
|
|
-r::
|
|
|
|
--run=<test-selector>::
|
|
|
|
Run only the subset of tests indicated by
|
|
|
|
<test-selector>. See section "Skipping Tests" below for
|
|
|
|
<test-selector> syntax.
|
|
|
|
|
2013-03-31 10:00:16 +02:00
|
|
|
--valgrind=<tool>::
|
|
|
|
Execute all Git binaries under valgrind tool <tool> and exit
|
|
|
|
with status 126 on errors (just like regular tests, this will
|
|
|
|
only stop the test script when running under -i).
|
2005-05-14 09:25:06 +02:00
|
|
|
|
2009-02-04 00:26:26 +01:00
|
|
|
Since it makes no sense to run the tests with --valgrind and
|
|
|
|
not see any output, this option implies --verbose. For
|
|
|
|
convenience, it also implies --tee.
|
|
|
|
|
2013-03-31 10:00:16 +02:00
|
|
|
<tool> defaults to 'memcheck', just like valgrind itself.
|
|
|
|
Other particularly useful choices include 'helgrind' and
|
|
|
|
'drd', but you may use any tool recognized by your valgrind
|
|
|
|
installation.
|
|
|
|
|
2013-03-31 10:00:17 +02:00
|
|
|
As a special case, <tool> can be 'memcheck-fast', which uses
|
|
|
|
memcheck but disables --track-origins. Use this if you are
|
|
|
|
running tests in bulk, to see if there are _any_ memory
|
|
|
|
issues.
|
|
|
|
|
2013-03-31 10:00:16 +02:00
|
|
|
Note that memcheck is run with the option --leak-check=no,
|
2011-03-15 10:32:11 +01:00
|
|
|
as the git process is short-lived and some errors are not
|
|
|
|
interesting. In order to run a single command under the same
|
|
|
|
conditions manually, you should set GIT_VALGRIND to point to
|
|
|
|
the 't/valgrind/' directory and use the commands under
|
|
|
|
't/valgrind/bin/'.
|
|
|
|
|
2013-06-23 20:12:57 +02:00
|
|
|
--valgrind-only=<pattern>::
|
|
|
|
Like --valgrind, but the effect is limited to tests with
|
|
|
|
numbers matching <pattern>. The number matched against is
|
|
|
|
simply the running count of the test within the file.
|
|
|
|
|
test-lib.sh: optionally output to test-results/$TEST.out, too
When tests are run in parallel and a few tests fail, it does not help
that the output of the terminal is totally confusing, as you rarely know
which test which line came from.
So introduce the option '--tee' which triggers that the output of the
tests will be written to t/test-results/$TEST.out in addition to the
terminal, where $TEST is the basename of the script.
Unfortunately, there seems to be no way to redirect a given file
descriptor to a specified subprocess in POSIX shell, only redirection
to a file is supported via 'exec > $FILE'.
At least with bash, one might think that 'exec >($COMMAND)' would work
as intended, but it does not.
The common way to work around the lack of proper tools support is to
work with named pipes, alas, one of our most beloved platforms does not
really support named pipes. Besides, we would need a pipe for every
script, as the whole point of this patch is to allow parallel execution.
Therefore, we handle the redirection in the following way: when '--tee'
was passed to the test script, the variable GIT_TEST_TEE_STARTED is set
(to avoid triggering that code path again) and the script is started
_again_, in a subshell, redirected to the command "tee".
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-02-04 00:26:12 +01:00
|
|
|
--tee::
|
|
|
|
In addition to printing the test output to the terminal,
|
|
|
|
write it to files named 't/test-results/$TEST_NAME.out'.
|
|
|
|
As the names depend on the tests' file names, it is safe to
|
|
|
|
run the tests with this option in parallel.
|
|
|
|
|
2018-10-29 13:13:59 +01:00
|
|
|
-V::
|
test-lib: add --verbose-log option
The "--verbose" option redirects output from arbitrary
test commands to stdout. This is useful for examining the
output manually, like:
./t5547-push-quarantine.sh -v | less
But it also means that the output is intermingled with the
TAP directives, which can confuse a TAP parser like "prove".
This has always been a potential problem, but became an
issue recently when one test happened to output the word
"ok" on a line by itself, which prove interprets as a test
success:
$ prove t5547-push-quarantine.sh :: -v
t5547-push-quarantine.sh .. 1/? To dest.git
* [new branch] HEAD -> master
To dest.git
! [remote rejected] reject -> reject (pre-receive hook declined)
error: failed to push some refs to 'dest.git'
fatal: git cat-file d08c8eba97f4e683ece08654c7c8d2ba0c03b129: bad file
t5547-push-quarantine.sh .. Failed -1/4 subtests
Test Summary Report
-------------------
t5547-push-quarantine.sh (Wstat: 0 Tests: 5 Failed: 0)
Parse errors: Tests out of sequence. Found (2) but expected (3)
Tests out of sequence. Found (3) but expected (4)
Tests out of sequence. Found (4) but expected (5)
Bad plan. You planned 4 tests but ran 5.
Files=1, Tests=5, 0 wallclock secs ( 0.01 usr + 0.01 sys = 0.02 CPU)
Result: FAIL
One answer is "if it hurts, don't do it", but that's not
quite the whole story. The Travis tests use "--verbose
--tee" so that they can get the benefit of prove's parallel
options, along with a verbose log in case there is a
failure. We just need the verbose output to go to the log,
but keep stdout clean.
Getting this right turns out to be surprisingly difficult.
Here's the progression of alternatives I considered:
1. Add an option to write verbose output to stderr. This is
hard to capture, though, because we want each test to
have its own log (because they're all run in parallel
and the jumbled output would be useless).
2. Add an option to write verbose output to a file in
test-results. This works, but the log is missing all of
the non-verbose output, which gives context.
3. Like (2), but teach say_color() to additionally output
to the log. This mostly works, but misses any output
that happens outside of the say() functions (which isn't
a lot, but is a potential maintenance headache).
4. Like (2), but make the log file the same as the "--tee"
file. That almost works, but now we have two processes
opening the same file. That gives us two separate
descriptors, each with their own idea of the current
position. They'll each start writing at offset 0, and
overwrite each other's data.
5. Like (4), but in each case open the file for appending.
That atomically positions each write at the end of the
file.
It's possible we may still get sheared writes between
the two processes, but this is already the case when
writing to stdout. It's not a problem in practice
because the test harness generally waits for snippets to
finish before writing the TAP output.
We can ignore buffering issues with tee, because POSIX
mandates that it does not buffer. Likewise, POSIX
specifies "tee -a", so it should be available
everywhere.
This patch implements option (5), which seems to work well
in practice.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-21 12:48:00 +02:00
|
|
|
--verbose-log::
|
|
|
|
Write verbose output to the same logfile as `--tee`, but do
|
|
|
|
_not_ write it to stdout. Unlike `--tee --verbose`, this option
|
|
|
|
is safe to use when stdout is being consumed by a TAP parser
|
|
|
|
like `prove`. Implies `--tee` and `--verbose`.
|
|
|
|
|
2009-12-03 06:14:06 +01:00
|
|
|
--with-dashes::
|
|
|
|
By default tests are run without dashed forms of
|
|
|
|
commands (like git-commit) in the PATH (it only uses
|
|
|
|
wrappers from ../bin-wrappers). Use this option to include
|
|
|
|
the build directory (..) in the PATH, which contains all
|
|
|
|
the dashed forms of commands. This option is currently
|
|
|
|
implied by other options like --valgrind and
|
|
|
|
GIT_TEST_INSTALLED.
|
|
|
|
|
2019-01-29 15:19:37 +01:00
|
|
|
--no-bin-wrappers::
|
|
|
|
By default, the test suite uses the wrappers in
|
|
|
|
`../bin-wrappers/` to execute `git` and friends. With this option,
|
|
|
|
`../git` and friends are run directly. This is not recommended
|
|
|
|
in general, as the wrappers contain safeguards to ensure that no
|
|
|
|
files from an installed Git are used, but can speed up test runs
|
|
|
|
especially on platforms where running shell scripts is expensive
|
|
|
|
(most notably, Windows).
|
|
|
|
|
2010-06-10 20:24:46 +02:00
|
|
|
--root=<directory>::
|
|
|
|
Create "trash" directories used to store all temporary data during
|
|
|
|
testing under <directory>, instead of the t/ directory.
|
|
|
|
Using this option with a RAM-based filesystem (such as tmpfs)
|
|
|
|
can massively speed up the test suite.
|
|
|
|
|
t/test-lib: introduce --chain-lint option
It's easy to miss an "&&"-chain in a test script, like:
test_expect_success 'check something important' '
cmd1 &&
cmd2
cmd3
'
The test harness will notice if cmd3 fails, but a failure of
cmd1 or cmd2 will go unnoticed, as their exit status is lost
after cmd3 runs.
The toy example above is easy to spot because the "cmds" are
all the same length, but real code is much more complicated.
It's also difficult to detect these situations by statically
analyzing the shell code with regexps (like the
check-non-portable-shell script does); there's too much
context required to know whether a &&-chain is appropriate
on a given line or not.
This patch instead lets the shell check each test by
sticking a command with a specific and unusual return code
at the top of each test, like:
(exit 117) &&
cmd1 &&
cmd2
cmd3
In a well-formed test, the non-zero exit from the first
command prevents any of the rest from being run, and the
test's exit code is 117. In a bad test (like the one above),
the 117 is lost, and cmd3 is run.
When we encounter a failure of this check, we abort the test
script entirely. For one thing, we have no clue which subset
of the commands in the test snippet were actually run.
Running further tests would be pointless, because we're now
in an unknown state. And two, this is not a "test failure"
in the traditional sense. The test script is buggy, not the
code it is testing. We should be able to fix these problems
in the script once, and not have them come back later as a
regression in git's code.
After checking a test snippet for --chain-lint, we do still
run the test itself. We could actually have a pure-lint
mode which just checks each test, but there are a few
reasons not to. One, because the tests are executing
arbitrary code, which could impact the later environment
(e.g., that could impact which set of tests we run at all).
And two, because a pure-lint mode would still be expensive
to run, because a significant amount of code runs outside of
the test_expect_* blocks. Instead, this option is designed
to be used as part of a normal test suite run, where it adds
very little overhead.
Turning on this option detects quite a few problems in
existing tests, which will be fixed in subsequent patches.
However, there are a number of places it cannot reach:
- it cannot find a failure to break out of loops on error,
like:
cmd1 &&
for i in a b c; do
cmd2 $i
done &&
cmd3
which will not notice failures of "cmd2 a" or "cmd b"
- it cannot find a missing &&-chain inside a block or
subfunction, like:
foo () {
cmd1
cmd2
}
foo &&
bar
which will not notice a failure of cmd1.
- it only checks tests that you run; every platform will
have some tests skipped due to missing prequisites,
so it's impossible to say from one run that the test
suite is free of broken &&-chains. However, all tests get
run by _somebody_, so eventually we will notice problems.
- it does not operate on test_when_finished or prerequisite
blocks. It could, but these tends to be much shorter and
less of a problem, so I punted on them in this patch.
This patch was inspired by an earlier patch by Jonathan
Nieder:
http://article.gmane.org/gmane.comp.version-control.git/235913
This implementation and all bugs are mine.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-20 11:05:48 +01:00
|
|
|
--chain-lint::
|
|
|
|
--no-chain-lint::
|
|
|
|
If --chain-lint is enabled, the test harness will check each
|
|
|
|
test to make sure that it properly "&&-chains" all commands (so
|
|
|
|
that a failure in the middle does not go unnoticed by the final
|
|
|
|
exit code of the test). This check is performed in addition to
|
|
|
|
running the tests themselves. You may also enable or disable
|
|
|
|
this feature by setting the GIT_TEST_CHAIN_LINT environment
|
|
|
|
variable to "1" or "0", respectively.
|
|
|
|
|
test-lib: add the '--stress' option to run a test repeatedly under load
Unfortunately, we have a few flaky tests, whose failures tend to be
hard to reproduce. We've found that the best we can do to reproduce
such a failure is to run the test script repeatedly while the machine
is under load, and wait in the hope that the load creates enough
variance in the timing of the test's commands that a failure is
evenually triggered. I have a command to do that, and I noticed that
two other contributors have rolled their own scripts to do the same,
all choosing slightly different approaches.
To help reproduce failures in flaky tests, introduce the '--stress'
option to run a test script repeatedly in multiple parallel jobs until
one of them fails, thereby using the test script itself to increase
the load on the machine.
The number of parallel jobs is determined by, in order of precedence:
the number specified as '--stress=<N>', or the value of the
GIT_TEST_STRESS_LOAD environment variable, or twice the number of
available processors (as reported by the 'getconf' utility), or 8.
Make '--stress' imply '--verbose -x --immediate' to get the most
information about rare failures; there is really no point in spending
all the extra effort to reproduce such a failure, and then not know
which command failed and why.
To prevent the several parallel invocations of the same test from
interfering with each other:
- Include the parallel job's number in the name of the trash
directory and the various output files under 't/test-results/' as
a '.stress-<Nr>' suffix.
- Add the parallel job's number to the port number specified by the
user or to the test number, so even tests involving daemons
listening on a TCP socket can be stressed.
- Redirect each parallel test run's verbose output to
't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the
output of several parallel running tests to the terminal would
create a big ugly mess.
For convenience, print the output of the failed test job at the end,
and rename its trash directory to end with the '.stress-failed'
suffix, so it's easy to find in a predictable path (OTOH, all absolute
paths recorded in the trash directory become invalid; we'll see
whether this causes any issues in practice). If, in an unlikely case,
more than one jobs were to fail nearly at the same time, then print
the output of all failed jobs, and rename the trash directory of only
the last one (i.e. with the highest job number), as it is the trash
directory of the test whose output will be at the bottom of the user's
terminal.
Based on Jeff King's 'stress' script.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
|
|
|
--stress::
|
|
|
|
Run the test script repeatedly in multiple parallel jobs until
|
|
|
|
one of them fails. Useful for reproducing rare failures in
|
|
|
|
flaky tests. The number of parallel jobs is, in order of
|
2019-03-03 15:44:55 +01:00
|
|
|
precedence: the value of the GIT_TEST_STRESS_LOAD
|
test-lib: add the '--stress' option to run a test repeatedly under load
Unfortunately, we have a few flaky tests, whose failures tend to be
hard to reproduce. We've found that the best we can do to reproduce
such a failure is to run the test script repeatedly while the machine
is under load, and wait in the hope that the load creates enough
variance in the timing of the test's commands that a failure is
evenually triggered. I have a command to do that, and I noticed that
two other contributors have rolled their own scripts to do the same,
all choosing slightly different approaches.
To help reproduce failures in flaky tests, introduce the '--stress'
option to run a test script repeatedly in multiple parallel jobs until
one of them fails, thereby using the test script itself to increase
the load on the machine.
The number of parallel jobs is determined by, in order of precedence:
the number specified as '--stress=<N>', or the value of the
GIT_TEST_STRESS_LOAD environment variable, or twice the number of
available processors (as reported by the 'getconf' utility), or 8.
Make '--stress' imply '--verbose -x --immediate' to get the most
information about rare failures; there is really no point in spending
all the extra effort to reproduce such a failure, and then not know
which command failed and why.
To prevent the several parallel invocations of the same test from
interfering with each other:
- Include the parallel job's number in the name of the trash
directory and the various output files under 't/test-results/' as
a '.stress-<Nr>' suffix.
- Add the parallel job's number to the port number specified by the
user or to the test number, so even tests involving daemons
listening on a TCP socket can be stressed.
- Redirect each parallel test run's verbose output to
't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the
output of several parallel running tests to the terminal would
create a big ugly mess.
For convenience, print the output of the failed test job at the end,
and rename its trash directory to end with the '.stress-failed'
suffix, so it's easy to find in a predictable path (OTOH, all absolute
paths recorded in the trash directory become invalid; we'll see
whether this causes any issues in practice). If, in an unlikely case,
more than one jobs were to fail nearly at the same time, then print
the output of all failed jobs, and rename the trash directory of only
the last one (i.e. with the highest job number), as it is the trash
directory of the test whose output will be at the bottom of the user's
terminal.
Based on Jeff King's 'stress' script.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
|
|
|
environment variable, or twice the number of available
|
|
|
|
processors (as shown by the 'getconf' utility), or 8.
|
|
|
|
Implies `--verbose -x --immediate` to get the most information
|
|
|
|
about the failure. Note that the verbose output of each test
|
|
|
|
job is saved to 't/test-results/$TEST_NAME.stress-<nr>.out',
|
|
|
|
and only the output of the failed test job is shown on the
|
|
|
|
terminal. The names of the trash directories get a
|
|
|
|
'.stress-<nr>' suffix, and the trash directory of the failed
|
|
|
|
test job is renamed to end with a '.stress-failed' suffix.
|
|
|
|
|
2019-03-03 15:44:55 +01:00
|
|
|
--stress-jobs=<N>::
|
|
|
|
Override the number of parallel jobs. Implies `--stress`.
|
|
|
|
|
test-lib: make '--stress' more bisect-friendly
Let's suppose that a test somehow becomes flaky between 'master' and
'pu', and tends to fail within the first 50 repetitions when run with
'--stress'. In such a case we could use 'git bisect' to find the
culprit: if the test script fails with '--stress', then the commit is
definitely bad, but if it survives, say, 300 repetitions, then we could
consider it good with reasonable confidence.
Unfortunately, all this could only be done manually, because
'--stress' would run the test script repeatedly for all eternity on a
good commit, and it would exit with success even when it found a
failure on a bad commit.
So let's make '--stress' usable with 'git bisect run':
- Make it exit with failure if a failure is found.
- Add the '--stress-limit=<N>' option to repeat the test script
at most N times in each of the parallel jobs, and exit with
success when the limit is reached.
And then we could simply run something like:
$ git bisect start origin/pu master
$ git bisect run sh -c 'make && cd t &&
./t1234-foo.sh --stress --stress-limit=300'
Sure, as a brand new feature it won't be any useful right now, but in
a release or three most cooking topics will already contain this, so
we could automatically bisect at least newly introduced flakiness.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-08 12:50:45 +01:00
|
|
|
--stress-limit=<N>::
|
|
|
|
When combined with --stress run the test script repeatedly
|
|
|
|
this many times in each of the parallel jobs or until one of
|
2019-03-03 15:44:54 +01:00
|
|
|
them fails, whichever comes first. Implies `--stress`.
|
test-lib: make '--stress' more bisect-friendly
Let's suppose that a test somehow becomes flaky between 'master' and
'pu', and tends to fail within the first 50 repetitions when run with
'--stress'. In such a case we could use 'git bisect' to find the
culprit: if the test script fails with '--stress', then the commit is
definitely bad, but if it survives, say, 300 repetitions, then we could
consider it good with reasonable confidence.
Unfortunately, all this could only be done manually, because
'--stress' would run the test script repeatedly for all eternity on a
good commit, and it would exit with success even when it found a
failure on a bad commit.
So let's make '--stress' usable with 'git bisect run':
- Make it exit with failure if a failure is found.
- Add the '--stress-limit=<N>' option to repeat the test script
at most N times in each of the parallel jobs, and exit with
success when the limit is reached.
And then we could simply run something like:
$ git bisect start origin/pu master
$ git bisect run sh -c 'make && cd t &&
./t1234-foo.sh --stress --stress-limit=300'
Sure, as a brand new feature it won't be any useful right now, but in
a release or three most cooking topics will already contain this, so
we could automatically bisect at least newly introduced flakiness.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-08 12:50:45 +01:00
|
|
|
|
2009-11-30 07:19:28 +01:00
|
|
|
You can also set the GIT_TEST_INSTALLED environment variable to
|
|
|
|
the bindir of an existing git installation to test that installation.
|
|
|
|
You still need to have built this git sandbox, from which various
|
|
|
|
test-* support programs, templates, and perl libraries are used.
|
|
|
|
If your installed git is incomplete, it will silently test parts of
|
|
|
|
your built version instead.
|
|
|
|
|
|
|
|
When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to
|
|
|
|
override the location of the dashed-form subcommands (what
|
|
|
|
GIT_EXEC_PATH would be used for during normal operation).
|
|
|
|
GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`.
|
|
|
|
|
|
|
|
|
2008-06-20 23:10:50 +02:00
|
|
|
Skipping Tests
|
|
|
|
--------------
|
|
|
|
|
|
|
|
In some environments, certain tests have no way of succeeding
|
|
|
|
due to platform limitation, such as lack of 'unzip' program, or
|
|
|
|
filesystem that do not allow arbitrary sequence of non-NUL bytes
|
|
|
|
as pathnames.
|
|
|
|
|
|
|
|
You should be able to say something like
|
|
|
|
|
|
|
|
$ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh
|
|
|
|
|
|
|
|
and even:
|
|
|
|
|
|
|
|
$ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make
|
|
|
|
|
|
|
|
to omit such tests. The value of the environment variable is a
|
|
|
|
SP separated list of patterns that tells which tests to skip,
|
|
|
|
and either can match the "t[0-9]{4}" part to skip the whole
|
|
|
|
test, or t[0-9]{4} followed by ".$number" to say which
|
|
|
|
particular test to skip.
|
|
|
|
|
2014-04-30 11:50:44 +02:00
|
|
|
For an individual test suite --run could be used to specify that
|
|
|
|
only some tests should be run or that some tests should be
|
|
|
|
excluded from a run.
|
|
|
|
|
|
|
|
The argument for --run is a list of individual test numbers or
|
|
|
|
ranges with an optional negation prefix that define what tests in
|
|
|
|
a test suite to include in the run. A range is two numbers
|
|
|
|
separated with a dash and matches a range of tests with both ends
|
|
|
|
been included. You may omit the first or the second number to
|
|
|
|
mean "from the first test" or "up to the very last test"
|
|
|
|
respectively.
|
|
|
|
|
|
|
|
Optional prefix of '!' means that the test or a range of tests
|
|
|
|
should be excluded from the run.
|
|
|
|
|
|
|
|
If --run starts with an unprefixed number or range the initial
|
|
|
|
set of tests to run is empty. If the first item starts with '!'
|
|
|
|
all the tests are added to the initial set. After initial set is
|
|
|
|
determined every test number or range is added or excluded from
|
|
|
|
the set one by one, from left to right.
|
|
|
|
|
|
|
|
Individual numbers or ranges could be separated either by a space
|
|
|
|
or a comma.
|
|
|
|
|
|
|
|
For example, to run only tests up to a specific test (21), one
|
|
|
|
could do this:
|
|
|
|
|
|
|
|
$ sh ./t9200-git-cvsexport-commit.sh --run='1-21'
|
|
|
|
|
|
|
|
or this:
|
|
|
|
|
|
|
|
$ sh ./t9200-git-cvsexport-commit.sh --run='-21'
|
|
|
|
|
|
|
|
Common case is to run several setup tests (1, 2, 3) and then a
|
|
|
|
specific test (21) that relies on that setup:
|
|
|
|
|
|
|
|
$ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'
|
|
|
|
|
|
|
|
or:
|
|
|
|
|
|
|
|
$ sh ./t9200-git-cvsexport-commit.sh --run=1,2,3,21
|
|
|
|
|
|
|
|
or:
|
|
|
|
|
|
|
|
$ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'
|
|
|
|
|
2017-09-17 12:18:15 +02:00
|
|
|
As noted above, the test set is built by going through the items
|
|
|
|
from left to right, so this:
|
2014-04-30 11:50:44 +02:00
|
|
|
|
|
|
|
$ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'
|
|
|
|
|
2017-09-17 12:18:15 +02:00
|
|
|
will run tests 1, 2, and 4. Items that come later have higher
|
2016-08-09 10:53:38 +02:00
|
|
|
precedence. It means that this:
|
2014-04-30 11:50:44 +02:00
|
|
|
|
|
|
|
$ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'
|
|
|
|
|
|
|
|
would just run tests from 1 to 4, including 3.
|
|
|
|
|
|
|
|
You may use negation with ranges. The following will run all
|
|
|
|
test in the test suite except from 7 up to 11:
|
|
|
|
|
|
|
|
$ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'
|
|
|
|
|
|
|
|
Some tests in a test suite rely on the previous tests performing
|
|
|
|
certain actions, specifically some tests are designated as
|
|
|
|
"setup" test, so you cannot _arbitrarily_ disable one test and
|
|
|
|
expect the rest to function correctly.
|
|
|
|
|
|
|
|
--run is mostly useful when you want to focus on a specific test
|
|
|
|
and know what setup is needed for it. Or when you want to run
|
|
|
|
everything up to a certain test.
|
2008-06-20 23:10:50 +02:00
|
|
|
|
|
|
|
|
2018-04-14 17:34:59 +02:00
|
|
|
Running tests with special setups
|
|
|
|
---------------------------------
|
|
|
|
|
|
|
|
The whole test suite could be run to test some special features
|
|
|
|
that cannot be easily covered by a few specific test cases. These
|
|
|
|
could be enabled by running the test suite with correct GIT_TEST_
|
|
|
|
environment set.
|
|
|
|
|
2019-06-21 12:18:12 +02:00
|
|
|
GIT_TEST_FAIL_PREREQS=<boolean> fails all prerequisites. This is
|
2019-05-13 20:32:42 +02:00
|
|
|
useful for discovering issues with the tests where say a later test
|
|
|
|
implicitly depends on an optional earlier test.
|
|
|
|
|
|
|
|
There's a "FAIL_PREREQS" prerequisite that can be used to test for
|
|
|
|
whether this mode is active, and e.g. skip some tests that are hard to
|
|
|
|
refactor to deal with it. The "SYMLINKS" prerequisite is currently
|
|
|
|
excluded as so much relies on it, but this might change in the future.
|
|
|
|
|
2019-06-21 12:18:09 +02:00
|
|
|
GIT_TEST_GETTEXT_POISON=<boolean> turns all strings marked for
|
2019-06-21 12:18:10 +02:00
|
|
|
translation into gibberish if true. Used for spotting those tests that
|
|
|
|
need to be marked with a C_LOCALE_OUTPUT prerequisite when adding more
|
|
|
|
strings for translation. See "Testing marked strings" in po/README for
|
|
|
|
details.
|
i18n: make GETTEXT_POISON a runtime option
Change the GETTEXT_POISON compile-time + runtime GIT_GETTEXT_POISON
test parameter to only be a GIT_TEST_GETTEXT_POISON=<non-empty?>
runtime parameter, to be consistent with other parameters documented
in "Running tests with special setups" in t/README.
When I added GETTEXT_POISON in bb946bba76 ("i18n: add GETTEXT_POISON
to simulate unfriendly translator", 2011-02-22) I was concerned with
ensuring that the _() function would get constant folded if NO_GETTEXT
was defined, and likewise that GETTEXT_POISON would be compiled out
unless it was defined.
But as the benchmark in my [1] shows doing a one-off runtime
getenv("GIT_TEST_[...]") is trivial, and since GETTEXT_POISON was
originally added the GIT_TEST_* env variables have become the common
idiom for turning on special test setups.
So change GETTEXT_POISON to work the same way. Now the
GETTEXT_POISON=YesPlease compile-time option is gone, and running the
tests with GIT_TEST_GETTEXT_POISON=[YesPlease|] can be toggled on/off
without recompiling.
This allows for conditionally amending tests to test with/without
poison, similar to what 859fdc0c3c ("commit-graph: define
GIT_TEST_COMMIT_GRAPH", 2018-08-29) did for GIT_TEST_COMMIT_GRAPH. Do
some of that, now we e.g. always run the t0205-gettext-poison.sh test.
I did enough there to remove the GETTEXT_POISON prerequisite, but its
inverse C_LOCALE_OUTPUT is still around, and surely some tests using
it can be converted to e.g. always set GIT_TEST_GETTEXT_POISON=.
Notes on the implementation:
* We still compile a dedicated GETTEXT_POISON build in Travis
CI. Perhaps this should be revisited and integrated into the
"linux-gcc" build, see ae59a4e44f ("travis: run tests with
GIT_TEST_SPLIT_INDEX", 2018-01-07) for prior art in that area. Then
again maybe not, see [2].
* We now skip a test in t0000-basic.sh under
GIT_TEST_GETTEXT_POISON=YesPlease that wasn't skipped before. This
test relies on C locale output, but due to an edge case in how the
previous implementation of GETTEXT_POISON worked (reading it from
GIT-BUILD-OPTIONS) wasn't enabling poison correctly. Now it does,
and needs to be skipped.
* The getenv() function is not reentrant, so out of paranoia about
code of the form:
printf(_("%s"), getenv("some-env"));
call use_gettext_poison() in our early setup in git_setup_gettext()
so we populate the "poison_requested" variable in a codepath that's
won't suffer from that race condition.
* We error out in the Makefile if you're still saying
GETTEXT_POISON=YesPlease to prompt users to change their
invocation.
* We should not print out poisoned messages during the test
initialization itself to keep it more readable, so the test library
hides the variable if set in $GIT_TEST_GETTEXT_POISON_ORIG during
setup. See [3].
See also [4] for more on the motivation behind this patch, and the
history of the GETTEXT_POISON facility.
1. https://public-inbox.org/git/871s8gd32p.fsf@evledraar.gmail.com/
2. https://public-inbox.org/git/20181102163725.GY30222@szeder.dev/
3. https://public-inbox.org/git/20181022202241.18629-2-szeder.dev@gmail.com/
4. https://public-inbox.org/git/878t2pd6yu.fsf@evledraar.gmail.com/
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-08 22:15:29 +01:00
|
|
|
|
2018-04-14 17:34:59 +02:00
|
|
|
GIT_TEST_SPLIT_INDEX=<boolean> forces split-index mode on the whole
|
|
|
|
test suite. Accept any boolean values that are accepted by git-config.
|
|
|
|
|
2019-12-24 02:02:28 +01:00
|
|
|
GIT_TEST_PROTOCOL_VERSION=<n>, when set, makes 'protocol.version'
|
|
|
|
default to n.
|
2019-02-25 22:54:06 +01:00
|
|
|
|
2018-04-14 17:35:05 +02:00
|
|
|
GIT_TEST_FULL_IN_PACK_ARRAY=<boolean> exercises the uncommon
|
|
|
|
pack-objects code path where there are more than 1024 packs even if
|
|
|
|
the actual number of packs in repository is below this limit. Accept
|
|
|
|
any boolean values that are accepted by git-config.
|
|
|
|
|
2018-04-14 17:35:10 +02:00
|
|
|
GIT_TEST_OE_SIZE=<n> exercises the uncommon pack-objects code path
|
|
|
|
where we do not cache object size in memory and read it from existing
|
|
|
|
packs on demand. This normally only happens when the object size is
|
|
|
|
over 2GB. This variable forces the code path on any object larger than
|
|
|
|
<n> bytes.
|
|
|
|
|
2018-09-19 01:29:34 +02:00
|
|
|
GIT_TEST_OE_DELTA_SIZE=<n> exercises the uncommon pack-objects code
|
pack-objects: fix performance issues on packing large deltas
Let's start with some background about oe_delta_size() and
oe_set_delta_size(). If you already know, skip the next paragraph.
These two are added in 0aca34e826 (pack-objects: shrink delta_size
field in struct object_entry - 2018-04-14) to help reduce 'struct
object_entry' size. The delta size field in this struct is reduced to
only contain max 1MB. So if any new delta is produced and larger than
1MB, it's dropped because we can't really save such a large size
anywhere. Fallback is provided in case existing packfiles already have
large deltas, then we can retrieve it from the pack.
While this should help small machines repacking large repos without
large deltas (i.e. less memory pressure), dropping large deltas during
the delta selection process could end up with worse pack files. And if
existing packfiles already have >1MB delta and pack-objects is
instructed to not reuse deltas, all of them will be dropped on the
floor, and the resulting pack would be definitely bigger.
There is also a regression in terms of CPU/IO if we have large on-disk
deltas because fallback code needs to parse the pack every time the
delta size is needed and just access to the mmap'd pack data is enough
for extra page faults when memory is under pressure.
Both of these issues were reported on the mailing list. Here's some
numbers for comparison.
Version Pack (MB) MaxRSS(kB) Time (s)
------- --------- ---------- --------
2.17.0 5498 43513628 2494.85
2.18.0 10531 40449596 4168.94
This patch provides a better fallback that is
- cheaper in terms of cpu and io because we won't have to read
existing pack files as much
- better in terms of pack size because the pack heuristics is back to
2.17.0 time, we do not drop large deltas at all
If we encounter any delta (on-disk or created during try_delta phase)
that is larger than the 1MB limit, we stop using delta_size_ field for
this because it can't contain such size anyway. A new array of delta
size is dynamically allocated and can hold all the deltas that 2.17.0
can. This array only contains delta sizes that delta_size_ can't
contain.
With this, we do not have to drop deltas in try_delta() anymore. Of
course the downside is we use slightly more memory, even compared to
2.17.0. But since this is considered an uncommon case, a bit more
memory consumption should not be a problem.
Delta size limit is also raised from 1MB to 16MB to better cover
common case and avoid that extra memory consumption (99.999% deltas in
this reported repo are under 12MB; Jeff noted binary artifacts topped
out at about 3MB in some other private repos). Other fields are
shuffled around to keep this struct packed tight. We don't use more
memory in common case even with this limit update.
A note about thread synchronization. Since this code can be run in
parallel during delta searching phase, we need a mutex. The realloc
part in packlist_alloc() is not protected because it only happens
during the object counting phase, which is always single-threaded.
Access to e->delta_size_ (and by extension
pack->delta_size[e - pack->objects]) is unprotected as before, the
thread scheduler in pack-objects must make sure "e" is never updated
by two different threads.
The area under the new lock is as small as possible, avoiding locking
at all in common case, since lock contention with high thread count
could be expensive (most blobs are small enough that delta compute
time is short and we end up taking the lock very often). The previous
attempt to always hold a lock in oe_delta_size() and
oe_set_delta_size() increases execution time by 33% when repacking
linux.git with with 40 threads.
Reported-by: Elijah Newren <newren@gmail.com>
Helped-by: Elijah Newren <newren@gmail.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-22 10:04:21 +02:00
|
|
|
path where deltas larger than this limit require extra memory
|
|
|
|
allocation for bookkeeping.
|
|
|
|
|
2018-08-25 15:02:09 +02:00
|
|
|
GIT_TEST_VALIDATE_INDEX_CACHE_ENTRIES=<boolean> checks that cache-tree
|
|
|
|
records are valid when the index is written out or after a merge. This
|
|
|
|
is mostly to catch missing invalidation. Default is true.
|
|
|
|
|
2018-08-29 14:49:04 +02:00
|
|
|
GIT_TEST_COMMIT_GRAPH=<boolean>, when true, forces the commit-graph to
|
|
|
|
be written after every 'git commit' command, and overrides the
|
|
|
|
'core.commitGraph' setting to true.
|
|
|
|
|
2020-04-06 18:59:55 +02:00
|
|
|
GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=<boolean>, when true, forces
|
|
|
|
commit-graph write to compute and write changed path Bloom filters for
|
|
|
|
every 'git commit-graph write', as if the `--changed-paths` option was
|
|
|
|
passed in.
|
|
|
|
|
2018-09-19 01:29:35 +02:00
|
|
|
GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor
|
|
|
|
code path for utilizing a file system monitor to speed up detecting
|
|
|
|
new or changed files.
|
|
|
|
|
2018-09-19 01:29:36 +02:00
|
|
|
GIT_TEST_INDEX_VERSION=<n> exercises the index read/write code path
|
|
|
|
for the index version specified. Can be set to any valid version
|
|
|
|
(currently 2, 3, or 4).
|
|
|
|
|
2020-03-20 13:38:10 +01:00
|
|
|
GIT_TEST_PACK_SPARSE=<boolean> if disabled will default the pack-objects
|
|
|
|
builtin to use the non-sparse object walk. This can still be overridden by
|
|
|
|
the --sparse command-line argument.
|
2019-01-16 19:26:01 +01:00
|
|
|
|
2018-09-19 01:29:37 +02:00
|
|
|
GIT_TEST_PRELOAD_INDEX=<boolean> exercises the preload-index code path
|
|
|
|
by overriding the minimum number of cache entries required per thread.
|
|
|
|
|
2019-11-13 13:40:57 +01:00
|
|
|
GIT_TEST_ADD_I_USE_BUILTIN=<boolean>, when true, enables the
|
|
|
|
built-in version of git add -i. See 'add.interactive.useBuiltin' in
|
|
|
|
git-config(1).
|
|
|
|
|
2018-10-10 17:59:35 +02:00
|
|
|
GIT_TEST_INDEX_THREADS=<n> enables exercising the multi-threaded loading
|
|
|
|
of the index for the whole test suite by bypassing the default number of
|
|
|
|
cache entries and thread minimums. Setting this to 1 will make the
|
|
|
|
index loading single threaded.
|
|
|
|
|
2018-10-12 19:34:20 +02:00
|
|
|
GIT_TEST_MULTI_PACK_INDEX=<boolean>, when true, forces the multi-pack-
|
|
|
|
index to be written after every 'git repack' command, and overrides the
|
|
|
|
'core.multiPackIndex' setting to true.
|
|
|
|
|
2019-01-16 20:28:15 +01:00
|
|
|
GIT_TEST_SIDEBAND_ALL=<boolean>, when true, overrides the
|
|
|
|
'uploadpack.allowSidebandAll' setting to true, and when false, forces
|
|
|
|
fetch-pack to not request sideband-all (even if the server advertises
|
|
|
|
sideband-all).
|
|
|
|
|
tests: disallow the use of abbreviated options (by default)
Git's command-line parsers support uniquely abbreviated options, e.g.
`git init --ba` would automatically expand `--ba` to `--bare`.
This is a very convenient feature in every day life for Git users, in
particular when tab completion is not available.
However, it is not a good idea to rely on that in Git's test suite, as
something that is a unique abbreviation of a command line option today
might no longer be a unique abbreviation tomorrow.
For example, if a future contribution added a new mode
`git init --babyproofing` and a previously-introduced test case used the
fact that `git init --ba` expanded to `git init --bare`, that future
contribution would now have to touch seemingly unrelated tests just to
keep the test suite from failing.
So let's disallow abbreviated options in the test suite by default.
Note: for ease of implementation, this patch really only touches the
`parse-options` machinery: more and more hand-rolled option parsers are
converted to use that internal API, and more and more scripts are
converted to built-ins (naturally using the parse-options API, too), so
in practice this catches most issues, and is definitely the biggest bang
for the buck.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-12 11:37:24 +02:00
|
|
|
GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS=<boolean>, when true (which is
|
|
|
|
the default when running tests), errors out when an abbreviated option
|
|
|
|
is used.
|
|
|
|
|
Rename some test scripts and describe the naming convention
First digit: "family", e.g. the absolute basics and global stuff (0),
the basic db-side commands (read-tree, write-tree, commit-tree), the
basic working-tree-side commands (checkout-cache, update-cache), the
other basic commands (ls-files), the diff commands, the pull commands,
exporting commands, revision tree commands...
Second digit: the particular command we are testing
Third digit: (optionally) the particular switch or group of switches
we are testing
Freeform part: commandname-details
Described in the README.
mv t1000-checkout-cache.sh t2000-checkout-cache-clash.sh
mv t1001-checkout-cache.sh t2001-checkout-cache-clash.sh
mv t0200-update-cache.sh t2010-update-cache-badpath.sh
mv t0400-ls-files.sh t3000-ls-files-others.sh
mv t0500-ls-files.sh t3010-ls-files-killed.sh
2005-05-15 01:34:22 +02:00
|
|
|
Naming Tests
|
|
|
|
------------
|
|
|
|
|
|
|
|
The test files are named as:
|
|
|
|
|
|
|
|
tNNNN-commandname-details.sh
|
|
|
|
|
|
|
|
where N is a decimal digit.
|
|
|
|
|
|
|
|
First digit tells the family:
|
|
|
|
|
|
|
|
0 - the absolute basics and global stuff
|
|
|
|
1 - the basic commands concerning database
|
|
|
|
2 - the basic commands concerning the working tree
|
|
|
|
3 - the other basic commands (e.g. ls-files)
|
|
|
|
4 - the diff commands
|
|
|
|
5 - the pull and exporting commands
|
|
|
|
6 - the revision tree commands (even e.g. merge-base)
|
2006-06-28 20:45:52 +02:00
|
|
|
7 - the porcelainish commands concerning the working tree
|
2006-12-29 14:39:09 +01:00
|
|
|
8 - the porcelainish commands concerning forensics
|
|
|
|
9 - the git tools
|
Rename some test scripts and describe the naming convention
First digit: "family", e.g. the absolute basics and global stuff (0),
the basic db-side commands (read-tree, write-tree, commit-tree), the
basic working-tree-side commands (checkout-cache, update-cache), the
other basic commands (ls-files), the diff commands, the pull commands,
exporting commands, revision tree commands...
Second digit: the particular command we are testing
Third digit: (optionally) the particular switch or group of switches
we are testing
Freeform part: commandname-details
Described in the README.
mv t1000-checkout-cache.sh t2000-checkout-cache-clash.sh
mv t1001-checkout-cache.sh t2001-checkout-cache-clash.sh
mv t0200-update-cache.sh t2010-update-cache-badpath.sh
mv t0400-ls-files.sh t3000-ls-files-others.sh
mv t0500-ls-files.sh t3010-ls-files-killed.sh
2005-05-15 01:34:22 +02:00
|
|
|
|
|
|
|
Second digit tells the particular command we are testing.
|
|
|
|
|
|
|
|
Third digit (optionally) tells the particular switch or group of switches
|
|
|
|
we are testing.
|
|
|
|
|
2005-07-07 20:39:10 +02:00
|
|
|
If you create files under t/ directory (i.e. here) that is not
|
|
|
|
the top-level test script, never name the file to match the above
|
|
|
|
pattern. The Makefile here considers all such files as the
|
2011-02-22 18:15:00 +01:00
|
|
|
top-level test script and tries to run all of them. Care is
|
2005-07-07 20:39:10 +02:00
|
|
|
especially needed if you are creating a common test library
|
|
|
|
file, similar to test-lib.sh, because such a library file may
|
|
|
|
not be suitable for standalone execution.
|
|
|
|
|
Rename some test scripts and describe the naming convention
First digit: "family", e.g. the absolute basics and global stuff (0),
the basic db-side commands (read-tree, write-tree, commit-tree), the
basic working-tree-side commands (checkout-cache, update-cache), the
other basic commands (ls-files), the diff commands, the pull commands,
exporting commands, revision tree commands...
Second digit: the particular command we are testing
Third digit: (optionally) the particular switch or group of switches
we are testing
Freeform part: commandname-details
Described in the README.
mv t1000-checkout-cache.sh t2000-checkout-cache-clash.sh
mv t1001-checkout-cache.sh t2001-checkout-cache-clash.sh
mv t0200-update-cache.sh t2010-update-cache-badpath.sh
mv t0400-ls-files.sh t3000-ls-files-others.sh
mv t0500-ls-files.sh t3010-ls-files-killed.sh
2005-05-15 01:34:22 +02:00
|
|
|
|
2005-05-14 09:25:06 +02:00
|
|
|
Writing Tests
|
|
|
|
-------------
|
|
|
|
|
|
|
|
The test script is written as a shell script. It should start
|
2017-11-26 21:20:59 +01:00
|
|
|
with the standard "#!/bin/sh", and an
|
2005-05-14 09:25:06 +02:00
|
|
|
assignment to variable 'test_description', like this:
|
|
|
|
|
|
|
|
#!/bin/sh
|
|
|
|
|
2005-05-15 23:21:13 +02:00
|
|
|
test_description='xxx test (option --frotz)
|
2005-05-14 09:25:06 +02:00
|
|
|
|
|
|
|
This test registers the following structure in the cache
|
|
|
|
and tries to run git-ls-files with option --frotz.'
|
|
|
|
|
Rename some test scripts and describe the naming convention
First digit: "family", e.g. the absolute basics and global stuff (0),
the basic db-side commands (read-tree, write-tree, commit-tree), the
basic working-tree-side commands (checkout-cache, update-cache), the
other basic commands (ls-files), the diff commands, the pull commands,
exporting commands, revision tree commands...
Second digit: the particular command we are testing
Third digit: (optionally) the particular switch or group of switches
we are testing
Freeform part: commandname-details
Described in the README.
mv t1000-checkout-cache.sh t2000-checkout-cache-clash.sh
mv t1001-checkout-cache.sh t2001-checkout-cache-clash.sh
mv t0200-update-cache.sh t2010-update-cache-badpath.sh
mv t0400-ls-files.sh t3000-ls-files-others.sh
mv t0500-ls-files.sh t3010-ls-files-killed.sh
2005-05-15 01:34:22 +02:00
|
|
|
|
2005-05-14 09:25:06 +02:00
|
|
|
Source 'test-lib.sh'
|
|
|
|
--------------------
|
|
|
|
|
|
|
|
After assigning test_description, the test script should source
|
|
|
|
test-lib.sh like this:
|
|
|
|
|
|
|
|
. ./test-lib.sh
|
|
|
|
|
|
|
|
This test harness library does the following things:
|
|
|
|
|
|
|
|
- If the script is invoked with command line argument --help
|
|
|
|
(or -h), it shows the test_description and exits.
|
|
|
|
|
2010-07-02 16:59:43 +02:00
|
|
|
- Creates an empty test directory with an empty .git/objects database
|
|
|
|
and chdir(2) into it. This directory is 't/trash
|
|
|
|
directory.$test_name_without_dotsh', with t/ subject to change by
|
test-lib: add the '--stress' option to run a test repeatedly under load
Unfortunately, we have a few flaky tests, whose failures tend to be
hard to reproduce. We've found that the best we can do to reproduce
such a failure is to run the test script repeatedly while the machine
is under load, and wait in the hope that the load creates enough
variance in the timing of the test's commands that a failure is
evenually triggered. I have a command to do that, and I noticed that
two other contributors have rolled their own scripts to do the same,
all choosing slightly different approaches.
To help reproduce failures in flaky tests, introduce the '--stress'
option to run a test script repeatedly in multiple parallel jobs until
one of them fails, thereby using the test script itself to increase
the load on the machine.
The number of parallel jobs is determined by, in order of precedence:
the number specified as '--stress=<N>', or the value of the
GIT_TEST_STRESS_LOAD environment variable, or twice the number of
available processors (as reported by the 'getconf' utility), or 8.
Make '--stress' imply '--verbose -x --immediate' to get the most
information about rare failures; there is really no point in spending
all the extra effort to reproduce such a failure, and then not know
which command failed and why.
To prevent the several parallel invocations of the same test from
interfering with each other:
- Include the parallel job's number in the name of the trash
directory and the various output files under 't/test-results/' as
a '.stress-<Nr>' suffix.
- Add the parallel job's number to the port number specified by the
user or to the test number, so even tests involving daemons
listening on a TCP socket can be stressed.
- Redirect each parallel test run's verbose output to
't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the
output of several parallel running tests to the terminal would
create a big ugly mess.
For convenience, print the output of the failed test job at the end,
and rename its trash directory to end with the '.stress-failed'
suffix, so it's easy to find in a predictable path (OTOH, all absolute
paths recorded in the trash directory become invalid; we'll see
whether this causes any issues in practice). If, in an unlikely case,
more than one jobs were to fail nearly at the same time, then print
the output of all failed jobs, and rename the trash directory of only
the last one (i.e. with the highest job number), as it is the trash
directory of the test whose output will be at the bottom of the user's
terminal.
Based on Jeff King's 'stress' script.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-05 02:08:59 +01:00
|
|
|
the --root option documented above, and a '.stress-<N>' suffix
|
|
|
|
appended by the --stress option.
|
2005-05-14 09:25:06 +02:00
|
|
|
|
|
|
|
- Defines standard test helper functions for your scripts to
|
|
|
|
use. These functions are designed to make all scripts behave
|
|
|
|
consistently when command line arguments --verbose (or -v),
|
|
|
|
--debug (or -d), and --immediate (or -i) is given.
|
|
|
|
|
2018-10-05 23:54:01 +02:00
|
|
|
Do's & don'ts
|
|
|
|
-------------
|
2010-07-02 16:59:49 +02:00
|
|
|
|
2010-07-05 20:37:30 +02:00
|
|
|
Here are a few examples of things you probably should and shouldn't do
|
2010-07-02 16:59:49 +02:00
|
|
|
when writing tests.
|
|
|
|
|
2018-10-05 23:54:01 +02:00
|
|
|
Here are the "do's:"
|
2010-07-02 16:59:49 +02:00
|
|
|
|
2010-07-05 20:37:30 +02:00
|
|
|
- Put all code inside test_expect_success and other assertions.
|
2010-07-02 16:59:49 +02:00
|
|
|
|
|
|
|
Even code that isn't a test per se, but merely some setup code
|
2010-07-05 20:37:30 +02:00
|
|
|
should be inside a test assertion.
|
2010-07-02 16:59:49 +02:00
|
|
|
|
|
|
|
- Chain your test assertions
|
|
|
|
|
|
|
|
Write test code like this:
|
|
|
|
|
|
|
|
git merge foo &&
|
|
|
|
git push bar &&
|
|
|
|
test ...
|
|
|
|
|
|
|
|
Instead of:
|
|
|
|
|
|
|
|
git merge hla
|
|
|
|
git push gh
|
|
|
|
test ...
|
|
|
|
|
|
|
|
That way all of the commands in your tests will succeed or fail. If
|
2010-10-03 22:00:14 +02:00
|
|
|
you must ignore the return value of something, consider using a
|
|
|
|
helper function (e.g. use sane_unset instead of unset, in order
|
|
|
|
to avoid unportable return value for unsetting a variable that was
|
|
|
|
already unset), or prepending the command with test_might_fail or
|
|
|
|
test_must_fail.
|
2010-07-02 16:59:49 +02:00
|
|
|
|
2010-07-25 21:52:44 +02:00
|
|
|
- Check the test coverage for your tests. See the "Test coverage"
|
|
|
|
below.
|
|
|
|
|
2011-02-22 18:15:00 +01:00
|
|
|
Don't blindly follow test coverage metrics; if a new function you added
|
|
|
|
doesn't have any coverage, then you're probably doing something wrong,
|
2010-07-25 21:52:45 +02:00
|
|
|
but having 100% coverage doesn't necessarily mean that you tested
|
|
|
|
everything.
|
|
|
|
|
|
|
|
Tests that are likely to smoke out future regressions are better
|
|
|
|
than tests that just inflate the coverage metrics.
|
|
|
|
|
2011-01-11 08:44:30 +01:00
|
|
|
- When a test checks for an absolute path that a git command generated,
|
|
|
|
construct the expected value using $(pwd) rather than $PWD,
|
|
|
|
$TEST_DIRECTORY, or $TRASH_DIRECTORY. It makes a difference on
|
|
|
|
Windows, where the shell (MSYS bash) mangles absolute path names.
|
|
|
|
For details, see the commit message of 4114156ae9.
|
|
|
|
|
2018-10-05 23:54:01 +02:00
|
|
|
- Remember that inside the <script> part, the standard output and
|
|
|
|
standard error streams are discarded, and the test harness only
|
|
|
|
reports "ok" or "not ok" to the end user running the tests. Under
|
|
|
|
--verbose, they are shown to help debug the tests.
|
|
|
|
|
2020-03-27 18:55:09 +01:00
|
|
|
- Be careful when you loop
|
|
|
|
|
|
|
|
You may need to verify multiple things in a loop, but the
|
|
|
|
following does not work correctly:
|
|
|
|
|
|
|
|
test_expect_success 'test three things' '
|
|
|
|
for i in one two three
|
|
|
|
do
|
|
|
|
test_something "$i"
|
|
|
|
done &&
|
|
|
|
test_something_else
|
|
|
|
'
|
|
|
|
|
|
|
|
Because the status of the loop itself is the exit status of the
|
|
|
|
test_something in the last round, the loop does not fail when
|
|
|
|
"test_something" for "one" or "two" fails. This is not what you
|
|
|
|
want.
|
|
|
|
|
|
|
|
Instead, you can break out of the loop immediately when you see a
|
|
|
|
failure. Because all test_expect_* snippets are executed inside
|
|
|
|
a function, "return 1" can be used to fail the test immediately
|
|
|
|
upon a failure:
|
|
|
|
|
|
|
|
test_expect_success 'test three things' '
|
|
|
|
for i in one two three
|
|
|
|
do
|
|
|
|
test_something "$i" || return 1
|
|
|
|
done &&
|
|
|
|
test_something_else
|
|
|
|
'
|
|
|
|
|
|
|
|
Note that we still &&-chain the loop to propagate failures from
|
|
|
|
earlier commands.
|
|
|
|
|
|
|
|
|
2018-10-05 23:54:01 +02:00
|
|
|
And here are the "don'ts:"
|
2010-07-02 16:59:49 +02:00
|
|
|
|
2018-10-05 23:54:01 +02:00
|
|
|
- Don't exit() within a <script> part.
|
2010-07-02 16:59:49 +02:00
|
|
|
|
|
|
|
The harness will catch this as a programming error of the test.
|
|
|
|
Use test_done instead if you need to stop the tests early (see
|
|
|
|
"Skipping tests" below).
|
|
|
|
|
2018-10-05 23:54:01 +02:00
|
|
|
- Don't use '! git cmd' when you want to make sure the git command
|
|
|
|
exits with failure in a controlled way by calling "die()". Instead,
|
2012-06-12 18:44:56 +02:00
|
|
|
use 'test_must_fail git cmd'. This will signal a failure if git
|
|
|
|
dies in an unexpected way (e.g. segfault).
|
|
|
|
|
2013-06-04 18:50:12 +02:00
|
|
|
On the other hand, don't use test_must_fail for running regular
|
2014-11-24 18:47:07 +01:00
|
|
|
platform commands; just use '! cmd'. We are not in the business
|
|
|
|
of verifying that the world given to us sanely works.
|
2013-06-04 18:50:12 +02:00
|
|
|
|
2018-10-05 23:54:02 +02:00
|
|
|
- Don't feed the output of a git command to a pipe, as in:
|
|
|
|
|
|
|
|
git -C repo ls-files |
|
|
|
|
xargs -n 1 basename |
|
|
|
|
grep foo
|
|
|
|
|
|
|
|
which will discard git's exit code and may mask a crash. In the
|
|
|
|
above example, all exit codes are ignored except grep's.
|
|
|
|
|
|
|
|
Instead, write the output of that command to a temporary
|
|
|
|
file with ">" or assign it to a variable with "x=$(git ...)" rather
|
|
|
|
than pipe it.
|
|
|
|
|
|
|
|
- Don't use command substitution in a way that discards git's exit
|
|
|
|
code. When assigning to a variable, the exit code is not discarded,
|
|
|
|
e.g.:
|
|
|
|
|
|
|
|
x=$(git cat-file -p $sha) &&
|
|
|
|
...
|
|
|
|
|
|
|
|
is OK because a crash in "git cat-file" will cause the "&&" chain
|
|
|
|
to fail, but:
|
|
|
|
|
|
|
|
test "refs/heads/foo" = "$(git symbolic-ref HEAD)"
|
|
|
|
|
|
|
|
is not OK and a crash in git could go undetected.
|
|
|
|
|
2018-10-05 23:54:01 +02:00
|
|
|
- Don't use perl without spelling it as "$PERL_PATH". This is to help
|
|
|
|
our friends on Windows where the platform Perl often adds CR before
|
2012-06-12 18:44:56 +02:00
|
|
|
the end of line, and they bundle Git with a version of Perl that
|
t: provide a perl() function which uses $PERL_PATH
Once upon a time, we assumed that calling a bare "perl" in
the test scripts was OK, because we would find the perl from
the user's PATH, and we were only asking that perl to do
basic operations that work even on old versions of perl.
Later, we found that some systems really prefer to use
$PERL_PATH even for these basic cases, because the system
perl misbehaves in some way (e.g., by handling line endings
differently). We then switched "perl" invocations to
"$PERL_PATH" to respect the user's choice.
Having to use "$PERL_PATH" is ugly and cumbersome, though.
Instead, let's provide a perl() shell function that tests
can use, which will transparently do the right thing.
Unfortunately, test writers still have to use $PERL_PATH in
certain situations, so we still need to keep the advice in
the README.
Note that this may fix test failures in t5004, t5503, t6002,
t6003, t6300, t8001, and t8002, depending on your system's
perl setup. All of these can be detected by running:
ln -s /bin/false bin-wrappers/perl
make test
which fails before this patch, and passes after.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-29 02:22:07 +01:00
|
|
|
does not do so, whose path is specified with $PERL_PATH. Note that we
|
|
|
|
provide a "perl" function which uses $PERL_PATH under the hood, so
|
|
|
|
you do not need to worry when simply running perl in the test scripts
|
|
|
|
(but you do, for example, on a shebang line or in a sub script
|
|
|
|
created via "write_script").
|
2012-06-12 18:44:56 +02:00
|
|
|
|
2018-10-05 23:54:01 +02:00
|
|
|
- Don't use sh without spelling it as "$SHELL_PATH", when the script
|
|
|
|
can be misinterpreted by broken platform shell (e.g. Solaris).
|
2012-06-12 18:44:56 +02:00
|
|
|
|
2018-10-05 23:54:01 +02:00
|
|
|
- Don't chdir around in tests. It is not sufficient to chdir to
|
2012-06-12 18:44:56 +02:00
|
|
|
somewhere and then chdir back to the original location later in
|
|
|
|
the test, as any intermediate step can fail and abort the test,
|
|
|
|
causing the next test to start in an unexpected directory. Do so
|
|
|
|
inside a subshell if necessary.
|
|
|
|
|
2018-10-05 23:54:01 +02:00
|
|
|
- Don't save and verify the standard error of compound commands, i.e.
|
|
|
|
group commands, subshells, and shell functions (except test helper
|
2018-02-24 00:39:50 +01:00
|
|
|
functions like 'test_must_fail') like this:
|
|
|
|
|
|
|
|
( cd dir && git cmd ) 2>error &&
|
|
|
|
test_cmp expect error
|
|
|
|
|
|
|
|
When running the test with '-x' tracing, then the trace of commands
|
|
|
|
executed in the compound command will be included in standard error
|
|
|
|
as well, quite possibly throwing off the subsequent checks examining
|
|
|
|
the output. Instead, save only the relevant git command's standard
|
|
|
|
error:
|
|
|
|
|
|
|
|
( cd dir && git cmd 2>../error ) &&
|
|
|
|
test_cmp expect error
|
|
|
|
|
2018-10-05 23:54:01 +02:00
|
|
|
- Don't break the TAP output
|
2010-07-02 16:59:49 +02:00
|
|
|
|
2010-07-05 20:37:30 +02:00
|
|
|
The raw output from your test may be interpreted by a TAP harness. TAP
|
|
|
|
harnesses will ignore everything they don't know about, but don't step
|
|
|
|
on their toes in these areas:
|
2010-07-02 16:59:49 +02:00
|
|
|
|
|
|
|
- Don't print lines like "$x..$y" where $x and $y are integers.
|
|
|
|
|
|
|
|
- Don't print lines that begin with "ok" or "not ok".
|
|
|
|
|
2010-07-05 20:37:30 +02:00
|
|
|
TAP harnesses expect a line that begins with either "ok" and "not
|
2010-07-02 16:59:49 +02:00
|
|
|
ok" to signal a test passed or failed (and our harness already
|
|
|
|
produces such lines), so your script shouldn't emit such lines to
|
|
|
|
their output.
|
|
|
|
|
|
|
|
You can glean some further possible issues from the TAP grammar
|
2017-03-22 23:18:52 +01:00
|
|
|
(see https://metacpan.org/pod/TAP::Parser::Grammar#TAP-GRAMMAR)
|
2010-07-02 16:59:49 +02:00
|
|
|
but the best indication is to just run the tests with prove(1),
|
|
|
|
it'll complain if anything is amiss.
|
|
|
|
|
|
|
|
|
2010-07-02 16:59:48 +02:00
|
|
|
Skipping tests
|
|
|
|
--------------
|
|
|
|
|
2011-03-09 21:25:09 +01:00
|
|
|
If you need to skip tests you should do so by using the three-arg form
|
2010-07-28 12:34:59 +02:00
|
|
|
of the test_* functions (see the "Test harness library" section
|
|
|
|
below), e.g.:
|
|
|
|
|
2012-06-12 18:44:56 +02:00
|
|
|
test_expect_success PERL 'I need Perl' '
|
t: provide a perl() function which uses $PERL_PATH
Once upon a time, we assumed that calling a bare "perl" in
the test scripts was OK, because we would find the perl from
the user's PATH, and we were only asking that perl to do
basic operations that work even on old versions of perl.
Later, we found that some systems really prefer to use
$PERL_PATH even for these basic cases, because the system
perl misbehaves in some way (e.g., by handling line endings
differently). We then switched "perl" invocations to
"$PERL_PATH" to respect the user's choice.
Having to use "$PERL_PATH" is ugly and cumbersome, though.
Instead, let's provide a perl() shell function that tests
can use, which will transparently do the right thing.
Unfortunately, test writers still have to use $PERL_PATH in
certain situations, so we still need to keep the advice in
the README.
Note that this may fix test failures in t5004, t5503, t6002,
t6003, t6300, t8001, and t8002, depending on your system's
perl setup. All of these can be detected by running:
ln -s /bin/false bin-wrappers/perl
make test
which fails before this patch, and passes after.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-29 02:22:07 +01:00
|
|
|
perl -e "hlagh() if unf_unf()"
|
2012-06-12 18:44:56 +02:00
|
|
|
'
|
2010-07-28 12:34:59 +02:00
|
|
|
|
|
|
|
The advantage of skipping tests like this is that platforms that don't
|
|
|
|
have the PERL and other optional dependencies get an indication of how
|
|
|
|
many tests they're missing.
|
|
|
|
|
|
|
|
If the test code is too hairy for that (i.e. does a lot of setup work
|
|
|
|
outside test assertions) you can also skip all remaining tests by
|
|
|
|
setting skip_all and immediately call test_done:
|
2010-07-02 16:59:48 +02:00
|
|
|
|
|
|
|
if ! test_have_prereq PERL
|
|
|
|
then
|
|
|
|
skip_all='skipping perl interface tests, perl not available'
|
|
|
|
test_done
|
|
|
|
fi
|
2005-05-15 23:21:13 +02:00
|
|
|
|
2010-07-28 12:34:59 +02:00
|
|
|
The string you give to skip_all will be used as an explanation for why
|
|
|
|
the test was skipped.
|
|
|
|
|
2005-05-14 09:25:06 +02:00
|
|
|
End with test_done
|
|
|
|
------------------
|
|
|
|
|
|
|
|
Your script will be a sequence of tests, using helper functions
|
|
|
|
from the test harness library. At the end of the script, call
|
|
|
|
'test_done'.
|
|
|
|
|
|
|
|
|
|
|
|
Test harness library
|
|
|
|
--------------------
|
|
|
|
|
|
|
|
There are a handful helper functions defined in the test harness
|
|
|
|
library for your script to use.
|
|
|
|
|
2010-07-02 16:59:45 +02:00
|
|
|
- test_expect_success [<prereq>] <message> <script>
|
2005-05-14 09:25:06 +02:00
|
|
|
|
2011-04-26 12:33:26 +02:00
|
|
|
Usually takes two strings as parameters, and evaluates the
|
2005-05-14 09:25:06 +02:00
|
|
|
<script>. If it yields success, test is considered
|
|
|
|
successful. <message> should state what it is testing.
|
|
|
|
|
|
|
|
Example:
|
|
|
|
|
|
|
|
test_expect_success \
|
|
|
|
'git-write-tree should be able to write an empty tree.' \
|
|
|
|
'tree=$(git-write-tree)'
|
|
|
|
|
2010-07-02 16:59:45 +02:00
|
|
|
If you supply three parameters the first will be taken to be a
|
2011-04-26 12:33:26 +02:00
|
|
|
prerequisite; see the test_set_prereq and test_have_prereq
|
2010-07-02 16:59:45 +02:00
|
|
|
documentation below:
|
|
|
|
|
|
|
|
test_expect_success TTY 'git --paginate rev-list uses a pager' \
|
|
|
|
' ... '
|
|
|
|
|
2010-08-06 23:19:23 +02:00
|
|
|
You can also supply a comma-separated list of prerequisites, in the
|
|
|
|
rare case where your test depends on more than one:
|
|
|
|
|
|
|
|
test_expect_success PERL,PYTHON 'yo dawg' \
|
|
|
|
' test $(perl -E 'print eval "1 +" . qx[python -c "print 2"]') == "4" '
|
|
|
|
|
2010-07-02 16:59:45 +02:00
|
|
|
- test_expect_failure [<prereq>] <message> <script>
|
2005-05-14 09:25:06 +02:00
|
|
|
|
2008-02-01 10:50:53 +01:00
|
|
|
This is NOT the opposite of test_expect_success, but is used
|
|
|
|
to mark a test that demonstrates a known breakage. Unlike
|
|
|
|
the usual test_expect_success tests, which say "ok" on
|
|
|
|
success and "FAIL" on failure, this will say "FIXED" on
|
|
|
|
success and "still broken" on failure. Failures from these
|
|
|
|
tests won't cause -i (immediate) to stop.
|
2005-05-14 09:25:06 +02:00
|
|
|
|
2010-07-02 16:59:45 +02:00
|
|
|
Like test_expect_success this function can optionally use a three
|
|
|
|
argument invocation with a prerequisite as the first argument.
|
|
|
|
|
2005-05-14 09:25:06 +02:00
|
|
|
- test_debug <script>
|
|
|
|
|
|
|
|
This takes a single argument, <script>, and evaluates it only
|
|
|
|
when the test script is started with --debug command line
|
|
|
|
argument. This is primarily meant for use during the
|
|
|
|
development of a new test script.
|
|
|
|
|
2015-10-30 20:02:56 +01:00
|
|
|
- debug <git-command>
|
|
|
|
|
|
|
|
Run a git command inside a debugger. This is primarily meant for
|
|
|
|
use when debugging a failing test script.
|
|
|
|
|
2005-05-14 09:25:06 +02:00
|
|
|
- test_done
|
|
|
|
|
|
|
|
Your test script must have test_done at the end. Its purpose
|
|
|
|
is to summarize successes and failures in the test script and
|
|
|
|
exit with an appropriate error code.
|
|
|
|
|
2009-01-27 23:34:48 +01:00
|
|
|
- test_tick
|
|
|
|
|
|
|
|
Make commit and tag names consistent by setting the author and
|
2011-02-22 18:15:00 +01:00
|
|
|
committer times to defined state. Subsequent calls will
|
2009-01-27 23:34:48 +01:00
|
|
|
advance the times by a fixed amount.
|
|
|
|
|
|
|
|
- test_commit <message> [<filename> [<contents>]]
|
|
|
|
|
|
|
|
Creates a commit with the given message, committing the given
|
|
|
|
file with the given contents (default for both is to reuse the
|
|
|
|
message string), and adds a tag (again reusing the message
|
|
|
|
string as name). Calls test_tick to make the SHA-1s
|
|
|
|
reproducible.
|
|
|
|
|
|
|
|
- test_merge <message> <commit-or-tag>
|
|
|
|
|
|
|
|
Merges the given rev using the given message. Like test_commit,
|
|
|
|
creates a tag and calls test_tick before committing.
|
2005-05-14 09:25:06 +02:00
|
|
|
|
2011-04-26 12:33:26 +02:00
|
|
|
- test_set_prereq <prereq>
|
2010-07-02 16:59:45 +02:00
|
|
|
|
|
|
|
Set a test prerequisite to be used later with test_have_prereq. The
|
2010-08-06 23:19:25 +02:00
|
|
|
test-lib will set some prerequisites for you, see the
|
|
|
|
"Prerequisites" section below for a full list of these.
|
|
|
|
|
|
|
|
Others you can set yourself and use later with either
|
|
|
|
test_have_prereq directly, or the three argument invocation of
|
|
|
|
test_expect_success and test_expect_failure.
|
2010-07-02 16:59:45 +02:00
|
|
|
|
2011-04-26 12:33:26 +02:00
|
|
|
- test_have_prereq <prereq>
|
2010-07-02 16:59:45 +02:00
|
|
|
|
2017-03-22 23:18:54 +01:00
|
|
|
Check if we have a prerequisite previously set with test_set_prereq.
|
|
|
|
The most common way to use this explicitly (as opposed to the
|
|
|
|
implicit use when an argument is passed to test_expect_*) is to skip
|
|
|
|
all the tests at the start of the test script if we don't have some
|
|
|
|
essential prerequisite:
|
2010-07-02 16:59:45 +02:00
|
|
|
|
|
|
|
if ! test_have_prereq PERL
|
|
|
|
then
|
|
|
|
skip_all='skipping perl interface tests, perl not available'
|
|
|
|
test_done
|
|
|
|
fi
|
|
|
|
|
2010-07-02 16:59:46 +02:00
|
|
|
- test_external [<prereq>] <message> <external> <script>
|
|
|
|
|
|
|
|
Execute a <script> with an <external> interpreter (like perl). This
|
|
|
|
was added for tests like t9700-perl-git.sh which do most of their
|
|
|
|
work in an external test script.
|
|
|
|
|
|
|
|
test_external \
|
|
|
|
'GitwebCache::*FileCache*' \
|
t: provide a perl() function which uses $PERL_PATH
Once upon a time, we assumed that calling a bare "perl" in
the test scripts was OK, because we would find the perl from
the user's PATH, and we were only asking that perl to do
basic operations that work even on old versions of perl.
Later, we found that some systems really prefer to use
$PERL_PATH even for these basic cases, because the system
perl misbehaves in some way (e.g., by handling line endings
differently). We then switched "perl" invocations to
"$PERL_PATH" to respect the user's choice.
Having to use "$PERL_PATH" is ugly and cumbersome, though.
Instead, let's provide a perl() shell function that tests
can use, which will transparently do the right thing.
Unfortunately, test writers still have to use $PERL_PATH in
certain situations, so we still need to keep the advice in
the README.
Note that this may fix test failures in t5004, t5503, t6002,
t6003, t6300, t8001, and t8002, depending on your system's
perl setup. All of these can be detected by running:
ln -s /bin/false bin-wrappers/perl
make test
which fails before this patch, and passes after.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-29 02:22:07 +01:00
|
|
|
perl "$TEST_DIRECTORY"/t9503/test_cache_interface.pl
|
2010-07-02 16:59:46 +02:00
|
|
|
|
|
|
|
If the test is outputting its own TAP you should set the
|
|
|
|
test_external_has_tap variable somewhere before calling the first
|
|
|
|
test_external* function. See t9700-perl-git.sh for an example.
|
|
|
|
|
|
|
|
# The external test will outputs its own plan
|
|
|
|
test_external_has_tap=1
|
|
|
|
|
|
|
|
- test_external_without_stderr [<prereq>] <message> <external> <script>
|
|
|
|
|
|
|
|
Like test_external but fail if there's any output on stderr,
|
|
|
|
instead of checking the exit code.
|
|
|
|
|
|
|
|
test_external_without_stderr \
|
|
|
|
'Perl API' \
|
t: provide a perl() function which uses $PERL_PATH
Once upon a time, we assumed that calling a bare "perl" in
the test scripts was OK, because we would find the perl from
the user's PATH, and we were only asking that perl to do
basic operations that work even on old versions of perl.
Later, we found that some systems really prefer to use
$PERL_PATH even for these basic cases, because the system
perl misbehaves in some way (e.g., by handling line endings
differently). We then switched "perl" invocations to
"$PERL_PATH" to respect the user's choice.
Having to use "$PERL_PATH" is ugly and cumbersome, though.
Instead, let's provide a perl() shell function that tests
can use, which will transparently do the right thing.
Unfortunately, test writers still have to use $PERL_PATH in
certain situations, so we still need to keep the advice in
the README.
Note that this may fix test failures in t5004, t5503, t6002,
t6003, t6300, t8001, and t8002, depending on your system's
perl setup. All of these can be detected by running:
ln -s /bin/false bin-wrappers/perl
make test
which fails before this patch, and passes after.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-29 02:22:07 +01:00
|
|
|
perl "$TEST_DIRECTORY"/t9700/test.pl
|
2010-07-02 16:59:46 +02:00
|
|
|
|
2010-10-03 21:59:59 +02:00
|
|
|
- test_expect_code <exit-code> <command>
|
|
|
|
|
|
|
|
Run a command and ensure that it exits with the given exit code.
|
|
|
|
For example:
|
|
|
|
|
|
|
|
test_expect_success 'Merge with d/f conflicts' '
|
|
|
|
test_expect_code 1 git merge "merge msg" B master
|
|
|
|
'
|
|
|
|
|
2018-02-09 03:42:33 +01:00
|
|
|
- test_must_fail [<options>] <git-command>
|
2010-07-06 22:04:10 +02:00
|
|
|
|
|
|
|
Run a git command and ensure it fails in a controlled way. Use
|
2010-07-20 19:17:12 +02:00
|
|
|
this instead of "! <git-command>". When git-command dies due to a
|
|
|
|
segfault, test_must_fail diagnoses it as an error; "! <git-command>"
|
|
|
|
treats it as just another expected failure, which would let such a
|
|
|
|
bug go unnoticed.
|
2010-07-06 22:04:10 +02:00
|
|
|
|
2018-02-09 03:42:33 +01:00
|
|
|
Accepts the following options:
|
|
|
|
|
|
|
|
ok=<signal-name>[,<...>]:
|
|
|
|
Don't treat an exit caused by the given signal as error.
|
|
|
|
Multiple signals can be specified as a comma separated list.
|
|
|
|
Currently recognized signal names are: sigpipe, success.
|
|
|
|
(Don't use 'success', use 'test_might_fail' instead.)
|
|
|
|
|
|
|
|
- test_might_fail [<options>] <git-command>
|
2010-07-06 22:04:10 +02:00
|
|
|
|
|
|
|
Similar to test_must_fail, but tolerate success, too. Use this
|
|
|
|
instead of "<git-command> || :" to catch failures due to segv.
|
|
|
|
|
2018-02-09 03:42:33 +01:00
|
|
|
Accepts the same options as test_must_fail.
|
|
|
|
|
2010-07-06 22:04:10 +02:00
|
|
|
- test_cmp <expected> <actual>
|
|
|
|
|
|
|
|
Check whether the content of the <actual> file matches the
|
|
|
|
<expected> file. This behaves like "cmp" but produces more
|
|
|
|
helpful output when the test is run with "-v" option.
|
|
|
|
|
2017-11-26 21:21:00 +01:00
|
|
|
- test_cmp_rev <expected> <actual>
|
|
|
|
|
|
|
|
Check whether the <expected> rev points to the same commit as the
|
|
|
|
<actual> rev.
|
|
|
|
|
2010-10-31 08:33:50 +01:00
|
|
|
- test_line_count (= | -lt | -ge | ...) <length> <file>
|
|
|
|
|
|
|
|
Check whether a file has the length it is expected to.
|
|
|
|
|
2011-04-26 12:33:26 +02:00
|
|
|
- test_path_is_file <path> [<diagnosis>]
|
|
|
|
test_path_is_dir <path> [<diagnosis>]
|
2010-08-10 17:17:52 +02:00
|
|
|
test_path_is_missing <path> [<diagnosis>]
|
|
|
|
|
2011-04-26 12:33:26 +02:00
|
|
|
Check if the named path is a file, if the named path is a
|
|
|
|
directory, or if the named path does not exist, respectively,
|
|
|
|
and fail otherwise, showing the <diagnosis> text.
|
2010-08-10 17:17:52 +02:00
|
|
|
|
2010-07-06 22:04:10 +02:00
|
|
|
- test_when_finished <script>
|
|
|
|
|
|
|
|
Prepend <script> to a list of commands to run to clean up
|
|
|
|
at the end of the current test. If some clean-up command
|
|
|
|
fails, the test will not pass.
|
|
|
|
|
|
|
|
Example:
|
|
|
|
|
|
|
|
test_expect_success 'branch pointing to non-commit' '
|
|
|
|
git rev-parse HEAD^{tree} >.git/refs/heads/invalid &&
|
|
|
|
test_when_finished "git update-ref -d refs/heads/invalid" &&
|
|
|
|
...
|
|
|
|
'
|
|
|
|
|
test-lib: introduce 'test_atexit'
When running Apache, 'git daemon', or p4d, we want to kill them at the
end of the test script, otherwise a leftover daemon process will keep
its port open indefinitely, and thus will interfere with subsequent
executions of the same test script.
So far, we stop these daemon processes "manually", i.e.:
- by registering functions or commands in the trap on EXIT to stop
the daemon while preserving the last seen exit code before the
trap (to deal with a failure when run with '--immediate' or with
interrupts by ctrl-C),
- and by invoking these functions/commands last thing before
'test_done' (and sometimes restoring the test framework's default
trap on EXIT, to prevent the daemons from being killed twice).
On one hand, we do this inconsistently, e.g. 'git p4' tests invoke
different functions in the trap on EXIT and in the last test before
'test_done', and they neither restore the test framework's default trap
on EXIT nor preserve the last seen exit code. On the other hand, this
is error prone, because, as shown in a previous patch in this series,
any output from the cleanup commands in the trap on EXIT can prevent a
proper cleanup when a test script run with '--verbose-log' and certain
shells, notably 'dash', is interrupted.
Let's introduce 'test_atexit', which is loosely modeled after
'test_when_finished', but has a broader scope: rather than running the
commands after the current test case, run them when the test script
finishes, and also run them when the test is interrupted, or exits
early in case of a failure while the '--immediate' option is in
effect.
When running the cleanup commands at the end of a successful test,
then they will be run in 'test_done' before it removes the trash
directory, i.e. the cleanup commands will still be able to access any
pidfiles or socket files in there. When running the cleanup commands
after an interrupt or failure with '--immediate', then they will be
run in the trap on EXIT. In both cases they will be run in
'test_eval_', i.e. both standard error and output of all cleanup
commands will go where they should according to the '-v' or
'--verbose-log' options, and thus won't cause any troubles when
interrupting a test script run with '--verbose-log'.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-13 13:24:11 +01:00
|
|
|
- test_atexit <script>
|
|
|
|
|
|
|
|
Prepend <script> to a list of commands to run unconditionally to
|
|
|
|
clean up before the test script exits, e.g. to stop a daemon:
|
|
|
|
|
|
|
|
test_expect_success 'test git daemon' '
|
|
|
|
git daemon &
|
|
|
|
daemon_pid=$! &&
|
|
|
|
test_atexit 'kill $daemon_pid' &&
|
|
|
|
hello world
|
|
|
|
'
|
|
|
|
|
|
|
|
The commands will be executed before the trash directory is removed,
|
|
|
|
i.e. the atexit commands will still be able to access any pidfiles or
|
|
|
|
socket files.
|
|
|
|
|
|
|
|
Note that these commands will be run even when a test script run
|
|
|
|
with '--immediate' fails. Be careful with your atexit commands to
|
|
|
|
minimize any changes to the failed state.
|
|
|
|
|
2014-05-06 01:51:43 +02:00
|
|
|
- test_write_lines <lines>
|
2014-04-27 20:15:47 +02:00
|
|
|
|
2014-05-06 01:51:43 +02:00
|
|
|
Write <lines> on standard output, one line per argument.
|
2014-04-27 20:15:47 +02:00
|
|
|
Useful to prepare multi-line files in a compact form.
|
|
|
|
|
|
|
|
Example:
|
|
|
|
|
2014-05-06 01:51:43 +02:00
|
|
|
test_write_lines a b c d e f g >foo
|
2014-04-27 20:15:47 +02:00
|
|
|
|
|
|
|
Is a more compact equivalent of:
|
|
|
|
cat >foo <<-EOF
|
|
|
|
a
|
|
|
|
b
|
|
|
|
c
|
|
|
|
d
|
|
|
|
e
|
|
|
|
f
|
|
|
|
g
|
|
|
|
EOF
|
|
|
|
|
|
|
|
|
2012-01-17 22:04:31 +01:00
|
|
|
- test_pause
|
|
|
|
|
|
|
|
This command is useful for writing and debugging tests and must be
|
|
|
|
removed before submitting. It halts the execution of the test and
|
|
|
|
spawns a shell in the trash directory. Exit the shell to continue
|
|
|
|
the test. Example:
|
|
|
|
|
|
|
|
test_expect_success 'test' '
|
|
|
|
git do-something >actual &&
|
|
|
|
test_pause &&
|
|
|
|
test_cmp expected actual
|
|
|
|
'
|
|
|
|
|
2013-06-07 22:53:27 +02:00
|
|
|
- test_ln_s_add <path1> <path2>
|
|
|
|
|
|
|
|
This function helps systems whose filesystem does not support symbolic
|
|
|
|
links. Use it to add a symbolic link entry to the index when it is not
|
|
|
|
important that the file system entry is a symbolic link, i.e., instead
|
|
|
|
of the sequence
|
|
|
|
|
|
|
|
ln -s foo bar &&
|
|
|
|
git add bar
|
|
|
|
|
|
|
|
Sometimes it is possible to split a test in a part that does not need
|
|
|
|
the symbolic link in the file system and a part that does; then only
|
|
|
|
the latter part need be protected by a SYMLINKS prerequisite (see below).
|
|
|
|
|
t: add test functions to translate hash-related values
Add several test functions to make working with various hash-related
values easier.
Add test_oid_init, which loads common hash-related constants and
placeholder object IDs from the newly added files in t/oid-info.
Provide values for these constants for both SHA-1 and SHA-256.
Add test_oid_cache, which accepts data on standard input in the form of
hash-specific key-value pairs that can be looked up later, using the
same format as the files in t/oid-info. Document this format in a
t/oid-info/README directory so that it's easier to use in the future.
Add test_oid, which is used to specify look up a per-hash value
(produced on standard output) based on the key specified as its
argument. Usually the data to be looked up will be a hash-related
constant (such as the size of the hash in binary or hexadecimal), a
well-known or placeholder object ID (such as the all-zeros object ID or
one consisting of "deadbeef" repeated), or something similar. For these
reasons, test_oid will usually be used within a command substitution.
Consequently, redirect the error output to standard error, since
otherwise it will not be displayed.
Add test_detect_hash, which currently only detects SHA-1, and
test_set_hash, which can be used to set a different hash algorithm for
test purposes. In the future, test_detect_hash will learn to actually
detect the hash depending on how the testsuite is to be run.
Use the local keyword within these functions to avoid overwriting other
shell variables. We have had a test balloon in place for a couple of
releases to catch shells that don't have this keyword and have not
received any reports of failure. Note that the varying usages of local
used here are supported by all common open-source shells supporting the
local keyword.
Test these new functions as part of t0000, which also serves to
demonstrate basic usage of them. In addition, add documentation on how
to format the lookup data and how to use the test functions.
Implement two basic lookup charts, one for common invalid or synthesized
object IDs, and one for various facts about the hash function in use.
Provide versions of the data for both SHA-1 and SHA-256.
Since we use shell variables for storage, names used for lookup can
currently consist only of shell identifier characters. If this is a
problem in the future, we can hash the names before use.
Improved-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-13 07:17:31 +02:00
|
|
|
- test_oid_init
|
|
|
|
|
|
|
|
This function loads facts and useful object IDs related to the hash
|
|
|
|
algorithm(s) in use from the files in t/oid-info.
|
|
|
|
|
|
|
|
- test_oid_cache
|
|
|
|
|
|
|
|
This function reads per-hash algorithm information from standard
|
|
|
|
input (usually a heredoc) in the format described in
|
|
|
|
t/oid-info/README. This is useful for test-specific values, such as
|
|
|
|
object IDs, which must vary based on the hash algorithm.
|
|
|
|
|
|
|
|
Certain fixed values, such as hash sizes and common placeholder
|
|
|
|
object IDs, can be loaded with test_oid_init (described above).
|
|
|
|
|
|
|
|
- test_oid <key>
|
|
|
|
|
|
|
|
This function looks up a value for the hash algorithm in use, based
|
|
|
|
on the key given. The value must have been loaded using
|
|
|
|
test_oid_init or test_oid_cache. Providing an unknown key is an
|
|
|
|
error.
|
|
|
|
|
2019-02-09 19:25:26 +01:00
|
|
|
- yes [<string>]
|
|
|
|
|
|
|
|
This is often seen in modern UNIX but some platforms lack it, so
|
|
|
|
the test harness overrides the platform implementation with a
|
|
|
|
more limited one. Use this only when feeding a handful lines of
|
|
|
|
output to the downstream---unlike the real version, it generates
|
|
|
|
only up to 99 lines.
|
|
|
|
|
tests: add 'test_bool_env' to catch non-bool GIT_TEST_* values
Since 3b072c577b (tests: replace test_tristate with "git env--helper",
2019-06-21) we get the normalized bool values of various GIT_TEST_*
environment variables via 'git env--helper'. Now, while the 'git
env--helper' command itself does catch invalid values in the
environment variable or in the given --default and exits with error
(exit code 128 or 129, respectively), it's invoked in conditions like
'if ! git env--helper ...', which means that all invalid bool values
are interpreted the same as the ordinary 'false' (exit code 1). This
has led to inadvertently skipped httpd tests in our CI builds for a
couple of weeks, see 3960290675 (ci: restore running httpd tests,
2019-09-06).
Let's be more careful about what the test suite accepts as bool values
in GIT_TEST_* environment variables, and error out loud and clear on
invalid values instead of simply skipping tests. Add the
'test_bool_env' helper function to encapsulate the invocation of 'git
env--helper' and the verification of its exit code, and replace all
invocations of that command in our test framework and test suite with
a call to this new helper (except in 't0017-env-helper.sh', of
course).
$ GIT_TEST_GIT_DAEMON=YesPlease ./t5570-git-daemon.sh
fatal: bad numeric config value 'YesPlease' for 'GIT_TEST_GIT_DAEMON': invalid unit
error: test_bool_env requires bool values both for $GIT_TEST_GIT_DAEMON and for the default fallback
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-22 14:14:36 +01:00
|
|
|
- test_bool_env <env-variable-name> <default-value>
|
|
|
|
|
|
|
|
Given the name of an environment variable with a bool value,
|
|
|
|
normalize its value to a 0 (true) or 1 (false or empty string)
|
|
|
|
return code. Return with code corresponding to the given default
|
|
|
|
value if the variable is unset.
|
|
|
|
Abort the test script if either the value of the variable or the
|
|
|
|
default are not valid bool values.
|
|
|
|
|
2019-02-09 19:25:26 +01:00
|
|
|
|
2010-08-06 23:19:25 +02:00
|
|
|
Prerequisites
|
|
|
|
-------------
|
|
|
|
|
|
|
|
These are the prerequisites that the test library predefines with
|
|
|
|
test_have_prereq.
|
|
|
|
|
|
|
|
See the prereq argument to the test_* functions in the "Test harness
|
|
|
|
library" section above and the "test_have_prereq" function for how to
|
|
|
|
use these, and "test_set_prereq" for how to define your own.
|
|
|
|
|
2013-10-28 20:22:16 +01:00
|
|
|
- PYTHON
|
2010-08-06 23:19:25 +02:00
|
|
|
|
2013-10-28 20:22:16 +01:00
|
|
|
Git wasn't compiled with NO_PYTHON=YesPlease. Wrap any tests that
|
|
|
|
need Python with this.
|
|
|
|
|
|
|
|
- PERL
|
|
|
|
|
|
|
|
Git wasn't compiled with NO_PERL=YesPlease.
|
|
|
|
|
|
|
|
Even without the PERL prerequisite, tests can assume there is a
|
|
|
|
usable perl interpreter at $PERL_PATH, though it need not be
|
|
|
|
particularly modern.
|
2010-08-06 23:19:25 +02:00
|
|
|
|
|
|
|
- POSIXPERM
|
|
|
|
|
|
|
|
The filesystem supports POSIX style permission bits.
|
|
|
|
|
|
|
|
- BSLASHPSPEC
|
|
|
|
|
|
|
|
Backslashes in pathspec are not directory separators. This is not
|
|
|
|
set on Windows. See 6fd1106a for details.
|
|
|
|
|
|
|
|
- EXECKEEPSPID
|
|
|
|
|
|
|
|
The process retains the same pid across exec(2). See fb9a2bea for
|
|
|
|
details.
|
|
|
|
|
2013-04-11 04:07:04 +02:00
|
|
|
- PIPE
|
|
|
|
|
|
|
|
The filesystem we're on supports creation of FIFOs (named pipes)
|
|
|
|
via mkfifo(1).
|
|
|
|
|
2010-08-06 23:19:25 +02:00
|
|
|
- SYMLINKS
|
|
|
|
|
|
|
|
The filesystem we're on supports symbolic links. E.g. a FAT
|
|
|
|
filesystem doesn't support these. See 704a3143 for details.
|
2010-07-02 16:59:46 +02:00
|
|
|
|
2010-08-07 00:09:09 +02:00
|
|
|
- SANITY
|
|
|
|
|
|
|
|
Test is not run by root user, and an attempt to write to an
|
|
|
|
unwritable file is expected to fail correctly.
|
2010-07-02 16:59:46 +02:00
|
|
|
|
2017-05-20 23:42:06 +02:00
|
|
|
- PCRE
|
2011-05-09 23:52:07 +02:00
|
|
|
|
2017-05-20 23:42:06 +02:00
|
|
|
Git was compiled with support for PCRE. Wrap any tests
|
2011-05-09 23:52:07 +02:00
|
|
|
that use git-grep --perl-regexp or git-grep -P in these.
|
|
|
|
|
2017-11-23 15:16:57 +01:00
|
|
|
- LIBPCRE1
|
|
|
|
|
|
|
|
Git was compiled with PCRE v1 support via
|
|
|
|
USE_LIBPCRE1=YesPlease. Wrap any PCRE using tests that for some
|
|
|
|
reason need v1 of the PCRE library instead of v2 in these.
|
|
|
|
|
|
|
|
- LIBPCRE2
|
|
|
|
|
|
|
|
Git was compiled with PCRE v2 support via
|
|
|
|
USE_LIBPCRE2=YesPlease. Wrap any PCRE using tests that for some
|
|
|
|
reason need v2 of the PCRE library instead of v1 in these.
|
|
|
|
|
2012-07-26 15:39:53 +02:00
|
|
|
- CASE_INSENSITIVE_FS
|
|
|
|
|
|
|
|
Test is run on a case insensitive file system.
|
|
|
|
|
2012-07-26 15:39:56 +02:00
|
|
|
- UTF8_NFD_TO_NFC
|
|
|
|
|
|
|
|
Test is run on a filesystem which converts decomposed utf-8 (nfd)
|
|
|
|
to precomposed utf-8 (nfc).
|
|
|
|
|
2017-05-25 21:45:31 +02:00
|
|
|
- PTHREADS
|
|
|
|
|
|
|
|
Git wasn't compiled with NO_PTHREADS=YesPlease.
|
|
|
|
|
2005-05-14 09:25:06 +02:00
|
|
|
Tips for Writing Tests
|
|
|
|
----------------------
|
|
|
|
|
|
|
|
As with any programming projects, existing programs are the best
|
|
|
|
source of the information. However, do _not_ emulate
|
|
|
|
t0000-basic.sh when writing your tests. The test is special in
|
2020-06-07 11:48:26 +02:00
|
|
|
that it tries to validate the very core of Git. For example, it
|
2005-05-14 09:25:06 +02:00
|
|
|
knows that there will be 256 subdirectories under .git/objects/,
|
|
|
|
and it knows that the object ID of an empty tree is a certain
|
|
|
|
40-byte string. This is deliberately done so in t0000-basic.sh
|
|
|
|
because the things the very basic core test tries to achieve is
|
2020-06-07 11:48:26 +02:00
|
|
|
to serve as a basis for people who are changing the Git internals
|
2005-05-14 09:25:06 +02:00
|
|
|
drastically. For these people, after making certain changes,
|
|
|
|
not seeing failures from the basic test _is_ a failure. And
|
2020-06-07 11:48:26 +02:00
|
|
|
such drastic changes to the core Git that even changes these
|
2005-05-14 09:25:06 +02:00
|
|
|
otherwise supposedly stable object IDs should be accompanied by
|
|
|
|
an update to t0000-basic.sh.
|
|
|
|
|
|
|
|
However, other tests that simply rely on basic parts of the core
|
2020-06-07 11:48:26 +02:00
|
|
|
Git working properly should not have that level of intimate
|
|
|
|
knowledge of the core Git internals. If all the test scripts
|
2005-05-14 09:25:06 +02:00
|
|
|
hardcoded the object IDs like t0000-basic.sh does, that defeats
|
|
|
|
the purpose of t0000-basic.sh, which is to isolate that level of
|
|
|
|
validation in one place. Your test also ends up needing
|
|
|
|
updating when such a change to the internal happens, so do _not_
|
|
|
|
do it and leave the low level of validation to t0000-basic.sh.
|
2010-08-08 16:49:25 +02:00
|
|
|
|
2010-07-25 21:52:44 +02:00
|
|
|
Test coverage
|
|
|
|
-------------
|
|
|
|
|
|
|
|
You can use the coverage tests to find code paths that are not being
|
|
|
|
used or properly exercised yet.
|
|
|
|
|
|
|
|
To do that, run the coverage target at the top-level (not in the t/
|
|
|
|
directory):
|
|
|
|
|
|
|
|
make coverage
|
|
|
|
|
|
|
|
That'll compile Git with GCC's coverage arguments, and generate a test
|
|
|
|
report with gcov after the tests finish. Running the coverage tests
|
|
|
|
can take a while, since running the tests in parallel is incompatible
|
|
|
|
with GCC's coverage mode.
|
|
|
|
|
|
|
|
After the tests have run you can generate a list of untested
|
|
|
|
functions:
|
|
|
|
|
|
|
|
make coverage-untested-functions
|
|
|
|
|
|
|
|
You can also generate a detailed per-file HTML report using the
|
|
|
|
Devel::Cover module. To install it do:
|
|
|
|
|
|
|
|
# On Debian or Ubuntu:
|
|
|
|
sudo aptitude install libdevel-cover-perl
|
|
|
|
|
|
|
|
# From the CPAN with cpanminus
|
|
|
|
curl -L http://cpanmin.us | perl - --sudo --self-upgrade
|
|
|
|
cpanm --sudo Devel::Cover
|
|
|
|
|
|
|
|
Then, at the top-level:
|
|
|
|
|
|
|
|
make cover_db_html
|
|
|
|
|
|
|
|
That'll generate a detailed cover report in the "cover_db_html"
|
|
|
|
directory, which you can then copy to a webserver, or inspect locally
|
|
|
|
in a browser.
|