Commit Graph

12036 Commits

Author SHA1 Message Date
Ævar Arnfjörð Bjarmason
d1edee4ada grep: given --threads with NO_PTHREADS=YesPlease, warn
Add a warning about missing thread support when grep.threads or
--threads is set to a non 0 (default) or 1 (no parallelism) value
under NO_PTHREADS=YesPlease.

This is for consistency with the index-pack & pack-objects commands,
which also take a --threads option & are configurable via
pack.threads, and have long warned about the same under
NO_PTHREADS=YesPlease.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-26 12:52:37 +09:00
Ævar Arnfjörð Bjarmason
2e96d8154f pack-objects: fix buggy warning about threads
Fix a buggy warning about threads under NO_PTHREADS=YesPlease. Due to
re-using the delta_search_threads variable for both the state of the
"pack.threads" config & the --threads option, setting "pack.threads"
but not supplying --threads would trigger the warning for both
"pack.threads" & --threads.

Solve this bug by resetting the delta_search_threads variable in
git_pack_config(), it might then be set by --threads again and be
subsequently warned about, as the test I'm changing here asserts.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-26 12:52:37 +09:00
Ævar Arnfjörð Bjarmason
967a3eaf43 pack-objects & index-pack: add test for --threads warning
Add a test for the warning that's emitted when --threads or
pack.threads is provided under NO_PTHREADS=YesPlease. This uses the
new PTHREADS prerequisite.

The assertion for C_LOCALE_OUTPUT in the latter test is currently
redundant, since unlike index-pack the pack-objects warnings aren't
i18n'd. However they might be changed to be i18n'd in the future, and
there's no harm in future-proofing the test.

There's an existing bug in the implementation of pack-objects which
this test currently tests for as-is. Details about the bug & the fix
are included in a follow-up change.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-26 12:52:37 +09:00
Ævar Arnfjörð Bjarmason
68c7d2761d test-lib: add a PTHREADS prerequisite
Add a PTHREADS prerequisite which is false when git is compiled with
NO_PTHREADS=YesPlease.

There's lots of custom code that runs when threading isn't available,
but before this prerequisite there was no way to test it.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-26 12:52:37 +09:00
Ævar Arnfjörð Bjarmason
3485bea157 grep: change the internal PCRE macro names to be PCRE1
Change the internal USE_LIBPCRE define, & build options flag to use a
naming convention ending in PCRE1, without changing the long-standing
USE_LIBPCRE Makefile flag which enables this code.

This is for preparation for libpcre2 support where having things like
USE_LIBPCRE and USE_LIBPCRE2 in any more places than we absolutely
need to for backwards compatibility with old Makefile arguments would
be confusing.

In some ways it would be better to change everything that now uses
USE_LIBPCRE to use USE_LIBPCRE1, and to make specifying
USE_LIBPCRE (or --with-pcre) an error. This would impose a one-time
burden on packagers of git to s/USE_LIBPCRE/USE_LIBPCRE1/ in their
build scripts.

However I'd like to leave the door open to making
USE_LIBPCRE=YesPlease eventually mean USE_LIBPCRE2=YesPlease,
i.e. once PCRE v2 is ubiquitous enough that it makes sense to make it
the default.

This code and the USE_LIBPCRE Makefile argument was added in commit
63e7e9d8b6 ("git-grep: Learn PCRE", 2011-05-09). At the time there was
no indication that the PCRE project would release an entirely new &
incompatible API around 3 years later.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-26 12:52:37 +09:00
Ævar Arnfjörð Bjarmason
723fc5a6e1 perf: add a comparison test of log --grep regex engines with -F
Add a performance comparison test of log --grepgrep regex engines
given fixed strings.

See the preceding fixed-string t/perf change ("perf: add a comparison
test of grep regex engines with -F", 2017-04-21) for notes about this,
in particular this mostly tests exactly the same codepath now, but
might not in the future:

    $ GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux ./run p4221-log-grep-engines-fixed.sh
    [...]
    Test                                     this tree
    --------------------------------------------------------
    4221.1: fixed log --grep='int'           5.99(5.55+0.40)
    4221.2: basic log --grep='int'           5.92(5.56+0.31)
    4221.3: extended log --grep='int'        6.01(5.51+0.45)
    4221.4: perl log --grep='int'            5.99(5.56+0.38)
    4221.6: fixed log --grep='uncommon'      5.06(4.76+0.27)
    4221.7: basic log --grep='uncommon'      5.02(4.78+0.21)
    4221.8: extended log --grep='uncommon'   4.99(4.78+0.20)
    4221.9: perl log --grep='uncommon'       5.00(4.72+0.26)
    4221.11: fixed log --grep='æ'            5.35(5.12+0.20)
    4221.12: basic log --grep='æ'            5.34(5.11+0.20)
    4221.13: extended log --grep='æ'         5.39(5.10+0.22)
    4221.14: perl log --grep='æ'             5.44(5.16+0.23)

Only the non-ASCII -i case is different:

    $ GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_4221_LOG_OPTS=' -i' ./run p4221-log-grep-engines-fixed.sh
    [...]
    Test                                        this tree
    -----------------------------------------------------------
    4221.1: fixed log -i --grep='int'           6.17(5.77+0.35)
    4221.2: basic log -i --grep='int'           6.16(5.59+0.39)
    4221.3: extended log -i --grep='int'        6.15(5.70+0.39)
    4221.4: perl log -i --grep='int'            6.15(5.69+0.38)
    4221.6: fixed log -i --grep='uncommon'      5.10(4.88+0.21)
    4221.7: basic log -i --grep='uncommon'      5.04(4.76+0.25)
    4221.8: extended log -i --grep='uncommon'   5.07(4.82+0.23)
    4221.9: perl log -i --grep='uncommon'       5.03(4.78+0.22)
    4221.11: fixed log -i --grep='æ'            5.93(5.65+0.25)
    4221.12: basic log -i --grep='æ'            5.88(5.62+0.25)
    4221.13: extended log -i --grep='æ'         6.02(5.69+0.29)
    4221.14: perl log -i --grep='æ'             5.36(5.06+0.29)

See commit ("perf: add a comparison test of grep regex engines",
2017-04-19) for details on the machine the above test run was executed
on.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-26 12:52:37 +09:00
Ævar Arnfjörð Bjarmason
c8f39be67e perf: add a comparison test of log --grep regex engines
Add a very basic performance comparison test comparing the POSIX
basic, extended and perl engines with patterns matching log messages
via --grep=<pattern>.

    $ GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux ./run p4220-log-grep-engines.sh
    [...]
    Test                                                  this tree
    ---------------------------------------------------------------------
    4220.1: basic log --grep='how.to'                     6.22(6.00+0.21)
    4220.2: extended log --grep='how.to'                  6.23(5.98+0.23)
    4220.3: perl log --grep='how.to'                      6.07(5.79+0.25)
    4220.5: basic log --grep='^how to'                    6.19(5.93+0.22)
    4220.6: extended log --grep='^how to'                 6.19(5.93+0.23)
    4220.7: perl log --grep='^how to'                     6.14(5.88+0.24)
    4220.9: basic log --grep='[how] to'                   6.96(6.65+0.28)
    4220.10: extended log --grep='[how] to'               6.96(6.69+0.24)
    4220.11: perl log --grep='[how] to'                   6.95(6.58+0.33)
    4220.13: basic log --grep='\(e.t[^ ]*\|v.ry\) rare'   7.10(6.80+0.27)
    4220.14: extended log --grep='(e.t[^ ]*|v.ry) rare'   7.07(6.80+0.26)
    4220.15: perl log --grep='(e.t[^ ]*|v.ry) rare'       7.70(7.46+0.22)
    4220.17: basic log --grep='m\(ú\|u\)lt.b\(æ\|y\)te'   6.12(5.87+0.24)
    4220.18: extended log --grep='m(ú|u)lt.b(æ|y)te'      6.14(5.84+0.26)
    4220.19: perl log --grep='m(ú|u)lt.b(æ|y)te'          6.16(5.93+0.20)

With -i:

    $ GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_4220_LOG_OPTS=' -i' ./run p4220-log-grep-engines.sh
    [...]
    Test                                                     this tree
    ------------------------------------------------------------------------
    4220.1: basic log -i --grep='how.to'                     6.74(6.41+0.32)
    4220.2: extended log -i --grep='how.to'                  6.78(6.55+0.22)
    4220.3: perl log -i --grep='how.to'                      6.06(5.77+0.28)
    4220.5: basic log -i --grep='^how to'                    6.80(6.57+0.22)
    4220.6: extended log -i --grep='^how to'                 6.83(6.52+0.29)
    4220.7: perl log -i --grep='^how to'                     6.16(5.94+0.20)
    4220.9: basic log -i --grep='[how] to'                   7.87(7.61+0.24)
    4220.10: extended log -i --grep='[how] to'               7.85(7.57+0.27)
    4220.11: perl log -i --grep='[how] to'                   7.03(6.75+0.25)
    4220.13: basic log -i --grep='\(e.t[^ ]*\|v.ry\) rare'   8.68(8.41+0.25)
    4220.14: extended log -i --grep='(e.t[^ ]*|v.ry) rare'   8.80(8.44+0.28)
    4220.15: perl log -i --grep='(e.t[^ ]*|v.ry) rare'       7.85(7.56+0.26)
    4220.17: basic log -i --grep='m\(ú\|u\)lt.b\(æ\|y\)te'   6.94(6.68+0.24)
    4220.18: extended log -i --grep='m(ú|u)lt.b(æ|y)te'      7.04(6.76+0.24)
    4220.19: perl log -i --grep='m(ú|u)lt.b(æ|y)te'          6.26(5.92+0.29)

See commit ("perf: add a comparison test of grep regex engines",
2017-04-19) for details on the machine the above test run was executed
on.

Before commit ("log: make --regexp-ignore-case work with
--perl-regexp", 2017-05-20) this test will almost definitely
fail (depending on the repo) if passed the -i option, since it wasn't
properly supported under PCRE.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-26 12:52:37 +09:00
Ævar Arnfjörð Bjarmason
bc22d81370 perf: add a comparison test of grep regex engines with -F
Add a performance comparison test of grep regex engines given fixed
strings.

The current logic in compile_regexp() ignores the engine parameter and
uses kwset() to search for these, so this test shows no difference
between engines right now:

    $ GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux ./run p7821-grep-engines-fixed.sh
    [...]
    Test                             this tree
    ------------------------------------------------
    7821.1: fixed grep int           0.56(1.67+0.68)
    7821.2: basic grep int           0.57(1.70+0.57)
    7821.3: extended grep int        0.59(1.76+0.51)
    7821.4: perl grep int            1.08(1.71+0.55)
    7821.6: fixed grep uncommon      0.23(0.55+0.50)
    7821.7: basic grep uncommon      0.24(0.55+0.50)
    7821.8: extended grep uncommon   0.26(0.55+0.52)
    7821.9: perl grep uncommon       0.24(0.58+0.47)
    7821.11: fixed grep æ            0.36(1.30+0.42)
    7821.12: basic grep æ            0.36(1.32+0.40)
    7821.13: extended grep æ         0.38(1.30+0.42)
    7821.14: perl grep æ             0.35(1.24+0.48)

Only when run with -i via GIT_PERF_7821_GREP_OPTS=' -i' do we avoid
avoid going through the same kwset.[ch] codepath, see the "Even when
-F..."  comment in grep.c. This only kicks for the non-ASCII case:

    $ GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_7821_GREP_OPTS=' -i' ./run p7821-grep-engines-fixed.sh
    [...]
    Test                                this tree
    ---------------------------------------------------
    7821.1: fixed grep -i int           0.62(2.10+0.57)
    7821.2: basic grep -i int           0.68(1.90+0.61)
    7821.3: extended grep -i int        0.78(1.94+0.57)
    7821.4: perl grep -i int            0.98(1.78+0.74)
    7821.6: fixed grep -i uncommon      0.24(0.44+0.64)
    7821.7: basic grep -i uncommon      0.25(0.56+0.54)
    7821.8: extended grep -i uncommon   0.27(0.62+0.45)
    7821.9: perl grep -i uncommon       0.24(0.59+0.49)
    7821.11: fixed grep -i æ            0.30(0.96+0.39)
    7821.12: basic grep -i æ            0.27(0.92+0.44)
    7821.13: extended grep -i æ         0.28(0.90+0.46)
    7821.14: perl grep -i æ             0.28(0.74+0.49)

I'm planning to change how fixed-string searching happens. This test
gives a baseline for comparing performance before & after any such
change.

See commit ("perf: add a comparison test of grep regex engines",
2017-04-19) for details on the machine the above test run was executed
on.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-26 12:52:36 +09:00
Ævar Arnfjörð Bjarmason
3878c7a540 perf: add a comparison test of grep regex engines
Add a very basic performance comparison test comparing the POSIX
basic, extended and perl engines.

In theory the "basic" and "extended" engines should be implemented
using the same underlying code with a slightly different pattern
parser, but some implementations may not do this. Jump through some
slight hoops to test both, which is worthwhile since "basic" is the
default.

Running this on an i7 3.4GHz Linux 4.9.0-2 Debian testing against a
checkout of linux.git & latest upstream PCRE, both PCRE and git
compiled with -O3 using gcc 7.1.1:

    $ GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux ./run p7820-grep-engines.sh
    [...]
    Test                                            this tree
    ---------------------------------------------------------------
    7820.1: basic grep 'how.to'                     0.34(1.24+0.53)
    7820.2: extended grep 'how.to'                  0.33(1.23+0.45)
    7820.3: perl grep 'how.to'                      0.31(1.05+0.56)
    7820.5: basic grep '^how to'                    0.32(1.24+0.42)
    7820.6: extended grep '^how to'                 0.33(1.20+0.44)
    7820.7: perl grep '^how to'                     0.57(2.67+0.42)
    7820.9: basic grep '[how] to'                   0.51(2.16+0.45)
    7820.10: extended grep '[how] to'               0.49(2.20+0.43)
    7820.11: perl grep '[how] to'                   0.56(2.60+0.43)
    7820.13: basic grep '\(e.t[^ ]*\|v.ry\) rare'   0.66(3.25+0.40)
    7820.14: extended grep '(e.t[^ ]*|v.ry) rare'   0.65(3.19+0.46)
    7820.15: perl grep '(e.t[^ ]*|v.ry) rare'       1.05(5.74+0.34)
    7820.17: basic grep 'm\(ú\|u\)lt.b\(æ\|y\)te'   0.34(1.28+0.47)
    7820.18: extended grep 'm(ú|u)lt.b(æ|y)te'      0.34(1.38+0.38)
    7820.19: perl grep 'm(ú|u)lt.b(æ|y)te'          0.39(1.56+0.44)

Options can also be passed to git-grep via the GIT_PERF_7820_GREP_OPTS
environment variable. There are various modes such as "-v" that have
very different performance profiles, but handling the combinatorial
explosion of testing all those options would make this script much
more complex and harder to maintain. Instead just add the ability to
do one-shot runs with arbitrary options, e.g.:

    $ GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_7820_GREP_OPTS=" -i" ./run p7820-grep-engines.sh
    [...]
    Test                                               this tree
    ------------------------------------------------------------------
    7820.1: basic grep -i 'how.to'                     0.49(1.72+0.38)
    7820.2: extended grep -i 'how.to'                  0.46(1.64+0.42)
    7820.3: perl grep -i 'how.to'                      0.44(1.45+0.45)
    7820.5: basic grep -i '^how to'                    0.47(1.76+0.38)
    7820.6: extended grep -i '^how to'                 0.47(1.70+0.42)
    7820.7: perl grep -i '^how to'                     0.65(2.72+0.37)
    7820.9: basic grep -i '[how] to'                   0.86(3.64+0.42)
    7820.10: extended grep -i '[how] to'               0.84(3.62+0.46)
    7820.11: perl grep -i '[how] to'                   0.73(3.06+0.39)
    7820.13: basic grep -i '\(e.t[^ ]*\|v.ry\) rare'   1.63(8.13+0.36)
    7820.14: extended grep -i '(e.t[^ ]*|v.ry) rare'   1.64(8.01+0.44)
    7820.15: perl grep -i '(e.t[^ ]*|v.ry) rare'       1.44(6.88+0.44)
    7820.17: basic grep -i 'm\(ú\|u\)lt.b\(æ\|y\)te'   0.66(2.67+0.44)
    7820.18: extended grep -i 'm(ú|u)lt.b(æ|y)te'      0.66(2.67+0.43)
    7820.19: perl grep -i 'm(ú|u)lt.b(æ|y)te'          0.59(2.31+0.37)

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-26 12:52:36 +09:00
Ævar Arnfjörð Bjarmason
b11ad029cb perf: emit progress output when unpacking & building
Amend the t/perf/run output so that in addition to the "Running N
tests" heading currently being emitted, it also emits "Unpacking $rev"
and "Building $rev" when setting up the build/$rev directory & when
building it, respectively.

This makes it easier to see what's going on and what revision is being
tested as the output scrolls by.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:38 +09:00
Ævar Arnfjörð Bjarmason
88b6197d0b perf: add a GIT_PERF_MAKE_COMMAND for when *_MAKE_OPTS won't do
Add a git GIT_PERF_MAKE_COMMAND variable to compliment the existing
GIT_PERF_MAKE_OPTS facility. This allows specifying an arbitrary shell
command to execute instead of 'make'.

This is useful e.g. in cases where the name, semantics or defaults of
a Makefile flag have changed over time. It can even be used to change
the contents of the tree, useful for monkeypatching ancient versions
of git to get them to build.

This opens Pandora's box in some ways, it's now possible to
"jailbreak" the perf environment and e.g. modify the source tree via
this arbitrary instead of just issuing a custom "make" command, such a
command has to be re-entrant in the sense that subsequent perf runs
will re-use the possibly modified tree.

It would be pointless to try to mitigate or work around that caveat in
a tool purely aimed at Git developers, so this change makes no attempt
to do so.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:38 +09:00
Ævar Arnfjörð Bjarmason
966be95549 grep: add tests to fix blind spots with \0 patterns
Address a big blind spot in the tests for patterns containing \0. The
is_fixed() function considers any string that contains \0 fixed, even
if it contains regular expression metacharacters, those patterns are
currently matched with kwset.

Before this change removing that memchr(s, 0, len) check from
is_fixed() wouldn't change the result of any of the tests, since
regcomp() will happily match the part before the \0.

The kwset path is dependent on whether the the -i flag is on, and
whether the pattern has any non-ASCII characters, but none of this was
tested for.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:38 +09:00
Ævar Arnfjörð Bjarmason
12fc32faa8 grep: prepare for testing binary regexes containing rx metacharacters
Add setup code needed for testing regexes that contain both binary
data and regex metacharacters.

The POSIX regcomp() function inherently can't support that, because it
takes a \0-delimited char *, but other regex engines APIs like PCRE v2
take a pattern/length pair, and are thus able to handle \0s in
patterns as well as any other character.

When kwset was imported in commit 9eceddeec6 ("Use kwset in grep",
2011-08-21) this limitation was fixed, but at the expense of
introducing the undocumented limitation that any pattern containing \0
implicitly becomes a fixed match (equivalent to -F having been
provided).

That's not something we'd like to keep in the future. The inability to
match patterns containing \0 is a leaky implementation detail.

So add tests as a first step towards changing that. In order to test
that \0-patterns can properly match as regexes the test string needs
to have some regex metacharacters in it.

There were other blind spots in the tests. The code around kwset
specially handles case-insensitive & non-ASCII data, but there were no
tests for this.

Fix all of that by amending the text being matched to contain both
regex metacharacters & non-ASCII data.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:37 +09:00
Ævar Arnfjörð Bjarmason
77f6f4406f grep: add a test helper function for less verbose -f \0 tests
Add a helper function to make the tests which check for patterns with
\0 in them more succinct. Right now this isn't a big win, but
subsequent commits will add a lot more of these tests.

The helper is based on the match() function in t3070-wildmatch.sh.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:37 +09:00
Ævar Arnfjörð Bjarmason
5ee6f1a21b grep: add tests for grep pattern types being passed to submodules
Add testing for grep pattern types being correctly passed to
submodules. The pattern "(.|.)[\d]" matches differently under
fixed (not at all), and then matches different lines under
basic/extended & perl regular expressions, so this change asserts that
the pattern type is passed along correctly.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:37 +09:00
Ævar Arnfjörð Bjarmason
5d52a30eda grep: amend submodule recursion test for regex engine testing
Amend the submodule recursion test to prepare it for subsequent tests
of whether it passes along the grep.patternType to the submodule
greps.

This is the result of searching & replacing:

    foobar -> (1|2)d(3|4)
    foo    -> (1|2)
    bar    -> (3|4)

Currently there's no tests for whether e.g. -P or -E is correctly
passed along, tests for that will be added in a follow-up change, but
first add content to the tests which will match differently under
different regex engines.

Reuse the pattern established in an earlier commit of mine in this
series ("log: add exhaustive tests for pattern style options &
config", 2017-04-07). The pattern "(.|.)[\d]" will match this content
differently under fixed/basic/extended & perl.

This test code was originally added in commit 0281e487fd ("grep:
optionally recurse into submodules", 2016-12-16).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:37 +09:00
Ævar Arnfjörð Bjarmason
c5813658f7 grep: add tests for --threads=N and grep.threads
Add tests for --threads=N being supplied on the command-line, or when
grep.threads=N being supplied in the configuration.

When the threading support was made run-time configurable in commit
89f09dd34e ("grep: add --threads=<num> option and grep.threads
configuration", 2015-12-15) no tests were added for it.

In developing a change to the grep code I was able to make
'--threads=1 <pat>` segfault, while the test suite still passed. This
change fixes that blind spot in the tests.

In addition to asserting that asking for N threads shouldn't segfault,
test that the grep output given any N is the same.

The choice to test only 1..10 as opposed to 1..8 or 1..16 or whatever
is arbitrary. Testing 1..1024 works locally for me (but gets
noticeably slower as more threads are spawned). Given the structure of
the code there's no reason to test an arbitrary number of threads,
only 0, 1 and >=2 are special modes of operation.

A later patch introduces a PTHREADS test prerequisite which is true
under NO_PTHREADS=UnfortunatelyYes, but even under NO_PTHREADS it's
fine to test --threads=N, we'll just ignore it and not use
threading. So these tests also make sense under that mode to assert
that --threads=N without pthreads still returns expected results.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:37 +09:00
Ævar Arnfjörð Bjarmason
e01b4dab01 grep: change non-ASCII -i test to stop using --debug
Change a non-ASCII case-insensitive test case to stop using --debug,
and instead simply test for the expected results.

The test coverage remains the same with this change, but the test
won't break due to internal refactoring.

This test was added in commit 793dc676e0 ("grep/icase: avoid kwsset
when -F is specified", 2016-06-25). It was asserting that the regex
must be compiled with compile_fixed_regexp(), instead test for the
expected results, allowing the underlying implementation to change.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:37 +09:00
Ævar Arnfjörð Bjarmason
4aeb720d3f grep: add a test for backreferences in PCRE patterns
Add a test for backreferences such as (.)\1 in PCRE patterns. This
test ensures that the PCRE_NO_AUTO_CAPTURE option isn't turned
on. Before this change turning it on would break these sort of
patterns, but wouldn't break any tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:37 +09:00
Ævar Arnfjörð Bjarmason
9001c1920c grep: add a test asserting that --perl-regexp dies when !PCRE
Add a test asserting that when --perl-regexp (and -P for grep) is
given to git-grep & git-log that we die with an error.

In developing the PCRE v2 series I introduced a regression where -P
would (through control-flow fall-through) become synonymous with basic
POSIX matching. I.e. 'git grep -P '[\d]' would match "d" instead of
digits.

The entire test suite would still pass with this serious regression,
since everything that tested for --perl-regexp would be guarded by the
PCRE prerequisite, fix that blind-spot by adding tests under !PCRE
asserting that git must die when given --perl-regexp or -P.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:37 +09:00
Ævar Arnfjörð Bjarmason
9e3cbc59d5 log: make --regexp-ignore-case work with --perl-regexp
Make the --regexp-ignore-case option work with --perl-regexp. This
never worked, and there was no test for this. Fix the bug and add a
test.

When PCRE support was added in commit 63e7e9d8b6 ("git-grep: Learn
PCRE", 2011-05-09) compile_pcre_regexp() would only check
opt->ignore_case, but when the --perl-regexp option was added in
commit 727b6fc3ed ("log --grep: accept --basic-regexp and
--perl-regexp", 2012-10-03) the code didn't set the opt->ignore_case.

Change the test suite to test for -i and --invert-regexp with
basic/extended/perl patterns in addition to fixed, which was the only
patternType that was tested for before in combination with those
options.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:37 +09:00
Ævar Arnfjörð Bjarmason
9df46763ef log: add exhaustive tests for pattern style options & config
Add exhaustive tests for how the different grep.patternType options &
the corresponding command-line options affect git-log.

Before this change it was possible to patch revision.c so that the
--basic-regexp option was synonymous with --extended-regexp, and
--perl-regexp wasn't recognized at all, and still have 100% of the
test suite pass.

This was because the first test being modified here, added in commit
34a4ae55b2 ("log --grep: use the same helper to set -E/-F options as
"git grep"", 2012-10-03), didn't actually check whether we'd enabled
extended regular expressions as distinct from re-toggling non-fixed
string support.

Fix that by changing the pattern to a pattern that'll only match if
--extended-regexp option is provided, but won't match under the
default --basic-regexp option.

Other potential regressions were possible since there were no tests
for the rest of the combinations of grep.patternType configuration
toggles & corresponding git-log command-line options. Add exhaustive
tests for those.

The patterns being passed to fixed/basic/extended/PCRE are carefully
crafted to return the wrong thing if the grep engine were to pick any
other matching method than the one it's told to use.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:37 +09:00
Ævar Arnfjörð Bjarmason
3eb585c112 test-lib: rename the LIBPCRE prerequisite to PCRE
Rename the LIBPCRE prerequisite to PCRE. This is for preparation for
libpcre2 support, where having just "LIBPCRE" would be confusing as it
implies v1 of the library.

None of these tests are incompatible between versions 1 & 2 of
libpcre, it's less confusing to give them a more general name to make
it clear that they work on both library versions.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-21 08:25:37 +09:00
Junio C Hamano
9b669787fc Merge branch 'jh/verify-index-checksum-only-in-fsck'
* jh/verify-index-checksum-only-in-fsck:
  t1450: avoid use of "sed" on the index, which is a binary file
2017-05-09 12:17:42 +09:00
Junio C Hamano
0998b648d6 Merge branch 'sh/rebase-i-reread-todo-after-exec'
"git rebase -i" failed to re-read the todo list file when the
command specified with the `exec` instruction updated it.

* sh/rebase-i-reread-todo-after-exec:
  rebase -i: reread the todo list if `exec` touched it
2017-05-01 14:14:44 +09:00
Junio C Hamano
d92dbf1934 Merge branch 'jk/submodule-init-segv-fix'
Fix a segv in 'submodule init' when url is not given for a submodule.

* jk/submodule-init-segv-fix:
  submodule_init: die cleanly on submodules without url defined
2017-05-01 14:14:44 +09:00
Junio C Hamano
d9dfed9e47 Merge branch 'ab/align-perf-descriptions'
Output from perf tests have been updated to align their titles.

* ab/align-perf-descriptions:
  t/perf: correctly align non-ASCII descriptions in output
2017-05-01 14:14:42 +09:00
Stephen Hicks
54fd3243da rebase -i: reread the todo list if exec touched it
In the scripted version of the interactive rebase, there was no internal
representation of the todo list; it was re-read before every command.
That allowed the hack that an `exec` command could append (or even
completely rewrite) the todo list.

This hack was broken by the partial conversion of the interactive rebase
to C, and this patch reinstates it.

We also add a small test to verify that this fix does not regress in the
future.

Signed-off-by: Stephen Hicks <sdh@google.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-27 10:56:26 +09:00
Jeff Hostetler
4d9bc37fbe t1450: avoid use of "sed" on the index, which is a binary file
The previous step added a path zzzzzzzz to the index, and then used
"sed" to replace this string to yyyyyyyy to create a test case where
the checksum at the end of the file does not match the contents.

Unfortunately, use of "sed" on a non-text file is not portable.
Instead, use a Perl script that seeks to the end and modifies the
last byte of the file (where we _know_ stores the trailing
checksum).

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-27 09:41:19 +09:00
Junio C Hamano
e31159746e Merge branch 'nd/worktree-add-lock'
Allow to lock a worktree immediately after it's created. This helps
prevent a race between "git worktree add; git worktree lock" and
"git worktree prune".

* nd/worktree-add-lock:
  worktree add: add --lock option
2017-04-26 15:39:12 +09:00
Junio C Hamano
f70b541188 Merge branch 'ab/completion-push-delete-ref'
The completion script (in contrib/) learned to complete "git push
--delete b<TAB>" to complete branch name to be deleted.

* ab/completion-push-delete-ref:
  completion: expand "push --delete <remote> <ref>" for refs on that <remote>
2017-04-26 15:39:09 +09:00
Junio C Hamano
6cbc478d83 Merge branch 'jh/add-index-entry-optim'
"git checkout" that handles a lot of paths has been optimized by
reducing the number of unnecessary checks of paths in the
has_dir_name() function.

* jh/add-index-entry-optim:
  read-cache: speed up has_dir_name (part 2)
  read-cache: speed up has_dir_name (part 1)
  read-cache: speed up add_index_entry during checkout
  p0006-read-tree-checkout: perf test to time read-tree
  read-cache: add strcmp_offset function
2017-04-26 15:39:07 +09:00
Junio C Hamano
c9672ba4c8 Merge branch 'nd/conditional-config-in-early-config'
The recently introduced conditional inclusion of configuration did
not work well when early-config mechanism was involved.

* nd/conditional-config-in-early-config:
  config: correct file reading order in read_early_config()
  config: handle conditional include when $GIT_DIR is not set up
  config: prepare to pass more info in git_config_with_options()
2017-04-26 15:39:05 +09:00
Junio C Hamano
46bdfa3975 Merge branch 'ab/push-cas-doc-n-test'
Doc update.

* ab/push-cas-doc-n-test:
  push: document & test --force-with-lease with multiple remotes
2017-04-26 15:39:05 +09:00
Junio C Hamano
44427a0c18 Merge branch 'ps/pathspec-empty-prefix-origin'
A recent update broke "git add -p ../foo" from a subdirectory.

* ps/pathspec-empty-prefix-origin:
  pathspec: honor `PATHSPEC_PREFIX_ORIGIN` with empty prefix
2017-04-26 15:39:03 +09:00
Junio C Hamano
da4c600730 Merge branch 'pc/t2027-git-to-pipe-cleanup'
Having a git command on the upstream side of a pipe in a test
script will hide the exit status from the command, which may cause
us to fail to notice a breakage; rewrite tests in a script to avoid
this issue.

* pc/t2027-git-to-pipe-cleanup:
  t2027: avoid using pipes
2017-04-26 15:39:02 +09:00
Junio C Hamano
768c7cb710 Merge branch 'gb/rebase-signoff'
"git rebase" learns "--signoff" option.

* gb/rebase-signoff:
  rebase: pass --[no-]signoff option to git am
  builtin/am: fold am_signoff() into am_append_signoff()
  builtin/am: honor --signoff also when --rebasing
2017-04-26 15:39:02 +09:00
Jeff King
627fde1025 submodule_init: die cleanly on submodules without url defined
When we init a submodule, we try to die when it has no URL
defined:

  url = xstrdup(sub->url);
  if (!url)
	  die(...);

But that's clearly nonsense. xstrdup() will never return
NULL, and if sub->url is NULL, we'll segfault.

These two bits of code need to be flipped, so we check
sub->url before looking at it.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-24 19:00:15 -07:00
Junio C Hamano
2d646e3e1f Merge branch 'jk/ls-files-recurse-submodules-fix'
"ls-files --recurse-submodules" did not quite work well in a
project with nested submodules.

* jk/ls-files-recurse-submodules-fix:
  ls-files: fix path used when recursing into submodules
  ls-files: fix recurse-submodules with nested submodules
2017-04-23 22:07:57 -07:00
Junio C Hamano
b30cf6761a Merge branch 'km/t1400-modernization'
Code cleanup.

* km/t1400-modernization:
  t1400: use consistent style for test_expect_success calls
2017-04-23 22:07:52 -07:00
Junio C Hamano
9f1384f711 Merge branch 'jk/quarantine-received-objects'
Add finishing touches to a recent topic.

* jk/quarantine-received-objects:
  refs: reject ref updates while GIT_QUARANTINE_PATH is set
  receive-pack: document user-visible quarantine effects
  receive-pack: drop tmp_objdir_env from run_update_hook
2017-04-23 22:07:52 -07:00
Junio C Hamano
5bceab4113 Merge branch 'bw/submodule-with-bs-path'
"git submodule" script does not work well with strange pathnames.
Protect it from a path with slashes in them, at least.

* bw/submodule-with-bs-path:
  submodule: prevent backslash expantion in submodule names
2017-04-23 22:07:50 -07:00
Junio C Hamano
cdfe138b36 Merge branch 'jh/verify-index-checksum-only-in-fsck'
The index file has a trailing SHA-1 checksum to detect file
corruption, and historically we checked it every time the index
file is used.  Omit the validation during normal use, and instead
verify only in "git fsck".

* jh/verify-index-checksum-only-in-fsck:
  read-cache: force_verify_index_checksum
2017-04-23 22:07:49 -07:00
Junio C Hamano
8b6bba6663 Merge branch 'jh/string-list-micro-optim'
The string-list API used a custom reallocation strategy that was
very inefficient, instead of using the usual ALLOC_GROW() macro,
which has been fixed.

* jh/string-list-micro-optim:
  string-list: use ALLOC_GROW macro when reallocing string_list
2017-04-23 22:07:47 -07:00
Junio C Hamano
a2e2c04683 Merge branch 'nd/conditional-config-include'
$GIT_DIR may in some cases be normalized with all symlinks resolved
while "gitdir" path expansion in the pattern does not receive the
same treatment, leading to incorrect mismatch.  This has been fixed.

* nd/conditional-config-include:
  config: resolve symlinks in conditional include's patterns
  path.c: and an option to call real_path() in expand_user_path()
2017-04-23 22:07:46 -07:00
Junio C Hamano
848d9a9bb7 Merge branch 'tb/doc-eol-normalization'
Doc update.

* tb/doc-eol-normalization:
  gitattributes.txt: document how to normalize the line endings
2017-04-23 22:07:45 -07:00
Ævar Arnfjörð Bjarmason
db7ed0f20c t/perf: correctly align non-ASCII descriptions in output
Change the test descriptions from being treated as binary blobs by
perl to being treated as UTF-8. This ensures that e.g. a test
description like "æ" is counted as 1 character, not 2.

I have WIP performance tests for non-ASCII grep patterns on another
topic that are affected by this.

Now instead of:

    $ ./run p0000-perf-lib-sanity.sh
    [...]
    0000.4: export a weird var                                    0.00(0.00+0.00)
    0000.5: éḿíẗ ńöń-ÁŚĆÍÍ ćḧáŕáćẗéŕś   0.00(0.00+0.00)
    0000.7: important variables available in subshells            0.00(0.00+0.00)
    [...]

We emit:

    [...]
    0000.4: export a weird var                                 0.00(0.00+0.00)
    0000.5: éḿíẗ ńöń-ÁŚĆÍÍ ćḧáŕáćẗéŕś                          0.00(0.00+0.00)
    0000.7: important variables available in subshells         0.00(0.00+0.00)
    [...]

Fixes code originally added in 342e9ef2d9 ("Introduce a performance
testing framework", 2012-02-17).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-23 21:33:15 -07:00
Ævar Arnfjörð Bjarmason
723c1d526f completion: expand "push --delete <remote> <ref>" for refs on that <remote>
Change the completion of "push --delete <remote> <ref>" to complete
refs on that <remote>, not all refs.

Before this cloning git.git and doing "git push --delete origin
p<TAB>" will complete nothing, since a fresh clone of git.git will
have no "pu" branch, whereas origin/p<TAB> will uselessly complete
origin/pu, but fully qualified references aren't accepted by
"--delete".

Now p<TAB> will complete as "pu". The completion of giving --delete
later, e.g. "git push origin --delete p<TAB>" remains unchanged, this
is a bug, but is a general existing limitation of the bash completion,
and not how git-push is documented, so I'm not fixing that case, but
adding a failing TODO test for it.

The testing code was supplied by SZEDER Gábor in
<20170421122832.24617-1-szeder.dev@gmail.com> with minor setup
modifications on my part.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Reviewed-by: SZEDER Gábor <szeder.dev@gmail.com>
Test-code-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-23 17:30:59 -07:00
Nguyễn Thái Ngọc Duy
507e6e9eec worktree add: add --lock option
As explained in the document. This option has an advantage over the
command sequence "git worktree add && git worktree lock": there will be
no gap that somebody can accidentally "prune" the new worktree (or soon,
explicitly "worktree remove" it).

"worktree add" does keep a lock on while it's preparing the worktree.
If --lock is specified, this lock remains after the worktree is created.

Suggested-by: David Taylor <David.Taylor@dell.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-20 17:59:02 -07:00
Junio C Hamano
8377f34540 Merge branch 'jh/memihash-opt'
Hotfix for a topic that is already in 'master'.

* jh/memihash-opt:
  p0004: make perf test executable
  t3008: skip lazy-init test on a single-core box
  test-online-cpus: helper to return cpu count
  name-hash: fix buffer overrun
2017-04-19 21:37:25 -07:00