Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there
are two codepaths in pack-objects: with & without using bitmap
reachability index.
However add_object_entry_from_bitmap(), despite its non-bitmapped
counterpart add_object_entry(), in no way does check for whether --local
or --honor-pack-keep or --incremental should be respected. In
non-bitmapped codepath this is handled in want_object_in_pack(), but
bitmapped codepath has simply no such checking at all.
The bitmapped codepath however was allowing to pass in all those options
and with bitmap indices still being used under such conditions -
potentially giving wrong output (e.g. including objects from non-local or
.keep'ed pack).
We can easily fix this by noting the following: when an object comes to
add_object_entry_from_bitmap() it can come for two reasons:
1. entries coming from main pack covered by bitmap index, and
2. object coming from, possibly alternate, loose or other packs.
"2" can be already handled by want_object_in_pack() and to cover
"1" we can teach want_object_in_pack() to expect that *found_pack can be
non-NULL, meaning calling client already found object's pack entry.
In want_object_in_pack() we care to start the checks from already found
pack, if we have one, this way determining the answer right away
in case neither --local nor --honour-pack-keep are active. In
particular, as p5310-pack-bitmaps.sh shows (3 consecutive runs), we do
not do harm to served-with-bitmap clones performance-wise:
Test 56dfeb62 this tree
-----------------------------------------------------------------
5310.2: repack to disk 9.08(8.20+0.25) 9.09(8.14+0.32) +0.1%
5310.3: simulated clone 1.92(2.12+0.08) 1.93(2.12+0.09) +0.5%
5310.4: simulated fetch 0.82(1.07+0.04) 0.82(1.06+0.04) +0.0%
5310.6: partial bitmap 1.96(2.42+0.13) 1.95(2.40+0.15) -0.5%
Test 56dfeb62 this tree
-----------------------------------------------------------------
5310.2: repack to disk 9.11(8.16+0.32) 9.11(8.19+0.28) +0.0%
5310.3: simulated clone 1.93(2.14+0.07) 1.92(2.11+0.10) -0.5%
5310.4: simulated fetch 0.82(1.06+0.04) 0.82(1.04+0.05) +0.0%
5310.6: partial bitmap 1.95(2.38+0.16) 1.94(2.39+0.14) -0.5%
Test 56dfeb62 this tree
-----------------------------------------------------------------
5310.2: repack to disk 9.13(8.17+0.31) 9.07(8.13+0.28) -0.7%
5310.3: simulated clone 1.92(2.13+0.07) 1.91(2.12+0.06) -0.5%
5310.4: simulated fetch 0.82(1.08+0.03) 0.82(1.08+0.03) +0.0%
5310.6: partial bitmap 1.96(2.43+0.14) 1.96(2.42+0.14) +0.0%
with delta timings showing they are all within noise from run to run.
In the general case we do not want to call find_pack_entry_one() more than
once, because it is expensive. This patch splits the loop in
want_object_in_pack() into two parts: finding the object and seeing if it
impacts our choice to include it in the pack. We may call the inexpensive
want_found_object() twice, but we will never call find_pack_entry_one() if we
do not need to.
I appreciate help and discussing this change with Junio C Hamano and
Jeff King.
Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Git's pack storage does efficient (log n) lookups in a
single packfile's index, but if we have multiple packfiles,
we have to linearly search each for a given object. This
patch introduces some timing tests for cases where we have a
large number of packs, so that we can measure any
improvements we make in the following patches.
The main thing we want to time is object lookup. To do this,
we measure "git rev-list --objects --all", which does a
fairly large number of object lookups (essentially one per
object in the repository).
However, we also measure the time to do a full repack, which
is interesting for two reasons. One is that in addition to
the usual pack lookup, it has its own linear iteration over
the list of packs. And two is that because it it is the tool
one uses to go from an inefficient many-pack situation back
to a single pack, we care about its performance not only at
marginal numbers of packs, but at the extreme cases (e.g.,
if you somehow end up with 5,000 packs, it is the only way
to get back to 1 pack, so we need to make sure it performs
well).
We measure the performance of each command in three
scenarios: 1 pack, 50 packs, and 1,000 packs.
The 1-pack case is a baseline; any optimizations we do to
handle multiple packs cannot possibly perform better than
this.
The 50-pack case is as far as Git should generally allow
your repository to go, if you have auto-gc enabled with the
default settings. So this represents the maximum performance
improvement we would expect under normal circumstances.
The 1,000-pack case is hopefully rare, though I have seen it
in the wild where automatic maintenance was broken for some
time (and the repository continued to receive pushes). This
represents cases where we care less about general
performance, but want to make sure that a full repack
command does not take excessively long.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git p4" used a location outside $GIT_DIR/refs/ to place its
temporary branches, which has been moved to refs/git-p4-tmp/.
* ls/p4-tmp-refs:
git-p4: place temporary refs used for branch import under refs/git-p4-tmp
Improve the look of the way "git fetch" reports what happened to
each ref that was fetched.
* nd/fetch-ref-summary:
fetch: reduce duplicate in ref update status lines with placeholder
fetch: align all "remote -> local" output
fetch: change flag code for displaying tag update and deleted ref
fetch: refactor ref update status formatting code
git-fetch.txt: document fetch output
The test framework learned a new helper test_match_signal to
check an exit code from getting killed by an expected signal.
* jk/test-match-signal:
t/lib-git-daemon: use test_match_signal
test_must_fail: use test_match_signal
t0005: use test_match_signal as appropriate
tests: factor portable signal check out of t0005
There are certain house-keeping tasks that need to be performed at
the very beginning of any Git program, and programs that are not
built-in commands had to do them exactly the same way as "git"
potty does. It was easy to make mistakes in one-off standalone
programs (like test helpers). A common "main()" function that
calls cmd_main() of individual program has been introduced to
make it harder to make mistakes.
* jk/common-main:
mingw: declare main()'s argv as const
common-main: call git_setup_gettext()
common-main: call restore_sigpipe_to_default()
common-main: call sanitize_stdfds()
common-main: call git_extract_argv0_path()
add an extra level of indirection to main()
A test that unconditionally used "mktemp" learned that the command
is not necessarily available everywhere.
* ak/lazy-prereq-mktemp:
t7610: test for mktemp before test execution
"git grep -i" has been taught to fold case in non-ascii locales
correctly.
* nd/icase:
grep.c: reuse "icase" variable
diffcore-pickaxe: support case insensitive match on non-ascii
diffcore-pickaxe: Add regcomp_or_die()
grep/pcre: support utf-8
gettext: add is_utf8_locale()
grep/pcre: prepare locale-dependent tables for icase matching
grep: rewrite an if/else condition to avoid duplicate expression
grep/icase: avoid kwsset when -F is specified
grep/icase: avoid kwsset on literal non-ascii strings
test-regex: expose full regcomp() to the command line
test-regex: isolate the bug test code
grep: break down an "if" stmt in preparation for next changes
The commands in the "log/diff" family have had an FILE* pointer in the
data structure they pass around for a long time, but some codepaths
used to always write to the standard output. As a preparatory step
to make "git format-patch" available to the internal callers, these
codepaths have been updated to consistently write into that FILE*
instead.
* js/log-to-diffopt-file:
mingw: fix the shortlog --output=<file> test
diff: do not color output when --color=auto and --output=<file> is given
t4211: ensure that log respects --output=<file>
shortlog: respect the --output=<file> setting
format-patch: use stdout directly
format-patch: avoid freopen()
format-patch: explicitly switch off color when writing to files
shortlog: support outputting to streams other than stdout
graph: respect the diffopt.file setting
line-log: respect diffopt's configured output file stream
log-tree: respect diffopt's configured output file stream
log: prepare log/log-tree to reuse the diffopt.close_file attribute
As we are not yet moving everything to size_t but still using ulong
internally when talking about the size of object, platforms with
32-bit long will not be able to produce tar archive with 4GB+ file,
and cannot grok 077777777777UL as a constant. Disable the extended
header feature and do not test it on them.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Skip tests that are unrunnable on platforms without 64-bit long
to avoid unnecessary test failures.
* jk/tzoffset-fix:
t0006: skip "far in the future" test when unsigned long is not long enough
Git's source code refers to timestamps as unsigned longs. On 32-bit
platforms, as well as on Windows, unsigned long is not large enough
to capture dates that are "absurdly far in the future".
While we can fix this issue properly by replacing unsigned long with
a larger type, we want to be a bit more conservative and just skip
those tests on the maint track.
Signed-off-by: Jeff King <peff@peff.net>
Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we tried to fix in 58461bd (t1308: do not get fooled by symbolic
links to the source tree, 2016-06-02) an obscure case where the user
cd's into Git's source code via a symbolic link, a regression was
introduced that affects all test runs on Windows.
The original patch introducing the test case in question was careful to
use `$(pwd)` instead of `$PWD`.
This was done to account for the fact that Git's test suite uses shell
scripting even on Windows, where the shell's Unix-y paths are
incompatible with the main Git executable's idea of paths: it only
accepts Windows paths.
It is an awkward but necessary thing, then, to use `$(pwd)` (which gives
us a Windows path) when interacting with the Git executable and `$PWD`
(which gives the shell's idea of the current working directory in Unix-y
form) for shell scripts, including the test suite itself.
Obviously this broke the use case of the Git maintainer when changing
the working directory into Git's source code directory via a symlink,
i.e. when `$(pwd)` does not agree with `$PWD`.
However, we must not fix that use case at the expense of regressing
another use case.
Let's special-case Windows here, even if it is ugly, for lack of a more
elegant solution.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git archive" learned to handle files that are larger than 8GB and
commits far in the future than expressible by the traditional US-TAR
format.
* jk/big-and-future-archive-tar:
archive-tar: drop return value
archive-tar: write extended headers for far-future mtime
archive-tar: write extended headers for file sizes >= 8GB
t5000: test tar files that overflow ustar headers
t9300: factor out portable "head -c" replacement
Git does not know what the contents in the index should be for a
path added with "git add -N" yet, so "git grep --cached" should not
show hits (or show lack of hits, with -L) in such a path, but that
logic does not apply to "git grep", i.e. searching in the working
tree files. But we did so by mistake, which has been corrected.
* nd/ita-cleanup:
grep: fix grepping for "intent to add" files
t7810-grep.sh: fix a whitespace inconsistency
t7810-grep.sh: fix duplicated test name
"git rebase -i --autostash" did not restore the auto-stashed change
when the operation was aborted.
* ps/rebase-i-auto-unstash-upon-abort:
rebase -i: restore autostash on abort
Add a test to specify the desired behaviour that currently is not
available in "git rebase -Xsubtree=...".
* dg/subtree-rebase-test:
contrib/subtree: Add a test for subtree rebase that loses commits
More markings of messages for i18n, with updates to various tests
to pass GETTEXT_POISON tests.
One patch from the original submission dropped due to conflicts
with jk/upload-pack-hook, which is still in flux.
* va/i18n-even-more: (38 commits)
t5541: become resilient to GETTEXT_POISON
i18n: branch: mark comment when editing branch description for translation
i18n: unmark die messages for translation
i18n: submodule: escape shell variables inside eval_gettext
i18n: submodule: join strings marked for translation
i18n: init-db: join message pieces
i18n: remote: allow translations to reorder message
i18n: remote: mark URL fallback text for translation
i18n: standardise messages
i18n: sequencer: add period to error message
i18n: merge: change command option help to lowercase
i18n: merge: mark messages for translation
i18n: notes: mark options for translation
i18n: notes: mark strings for translation
i18n: transport-helper.c: change N_() call to _()
i18n: bisect: mark strings for translation
t5523: use test_i18ngrep for negation
t4153: fix negated test_i18ngrep call
t9003: become resilient to GETTEXT_POISON
tests: unpack-trees: update to use test_i18n* functions
...
Adjust t4201 to pass on Windows; a couple of test cases need to be
skipped on Windows which leads to a different shortlog than on Linux.
Let's just fix that by limiting the shortlog's commit range to traverse
only one commit: that guarantees that it does not matter how many test
cases were skipped.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
One among four invocations of readlink(1) in our test suite has
been rewritten so that the test can run on systems without the
command (others are in valgrind test framework and t9802).
* ak/t7800-wo-readlink:
t7800: readlink may not be available
The internal code used to show local timezone offset is not
prepared to handle timestamps beyond year 2100, and gave a
bogus offset value to the caller. Use a more benign looking
+0000 instead and let "git log" going in such a case, instead
of aborting.
* jk/tzoffset-fix:
local_tzoffset: detect errors from tm_to_time_t
t0006: test various date formats
t0006: rename test-date's "show" to "relative"
Fix an unintended regression in v2.9 that breaks "clone --depth"
that recurses down to submodules by forcing the submodules to also
be cloned shallowly, which many server instances that host upstream
of the submodules are not prepared for.
* sb/clone-shallow-passthru:
clone: do not let --depth imply --shallow-submodules
"log --graph --format=" learned that "%>|(N)" specifies the width
relative to the terminal's left edge, not relative to the area to
draw text that is to the right of the ancestry-graph section. It
also now accepts negative N that means the column limit is relative
to the right border.
* nd/graph-width-padded:
pretty.c: support <direction>|(<negative number>) forms
pretty: pass graph width to pretty formatting for use in '%>|(N)'
"git log" learns log.showSignature configuration variable, and a
command line option "--no-show-signature" to countermand it.
* mj/log-show-signature-conf:
log: add log.showSignature configuration variable
log: add "--no-show-signature" command line option
t4202: refactor test
A helper function that takes the contents of a commit object and
finds its subject line did not ignore leading blank lines, as is
commonly done by other codepaths. Make it ignore leading blank
lines to match.
* js/find-commit-subject-ignore-leading-blanks:
reset --hard: skip blank lines when reporting the commit subject
sequencer: use skip_blank_lines() to find the commit subject
commit -C: skip blank lines at the beginning of the message
commit.c: make find_commit_subject() more robust
pretty: make the skip_blank_lines() function public
Allow t/perf framework to use the features from the most recent
version of Git even when testing an older installed version.
* jk/perf-any-version:
p4211: explicitly disable renames in no-rename test
t/perf: fix regression in testing older versions of git
The output coloring scheme learned two new attributes, italic and
strike, in addition to existing bold, reverse, etc.
* jk/ansi-color:
color: support strike-through attribute
color: support "italic" attribute
color: allow "no-" for negating attributes
color: refactor parse_attr
add skip_prefix_mem helper
doc: refactor description of color format
color: fix max-size comment
Git-P4 used to place temporary refs under "git-p4-tmp". Since 3da1f37
Git checks that all refs are placed under "refs". Instruct Git-P4 to
place temporary refs under "refs/git-p4-tmp". There are no backwards
compatibility considerations as these refs are transient.
Use "git show-ref --verify" to check the (non-)existience of the refs
instead of file checks assuming the file-based ref backend.
All refs under "refs" are shared across all worktrees. This is not
desired for temporary Git-P4 refs and will be adressed in a later patch.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Reviewed-by: Vitor Antunes <vitor.hda@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
One among four invocations of readlink(1) in our test suite has
been rewritten so that the test can run on systems without the
command (others are in valgrind test framework and t9802).
* ak/t7800-wo-readlink:
t7800: readlink may not be available
The internal code used to show local timezone offset is not
prepared to handle timestamps beyond year 2100, and gave a
bogus offset value to the caller. Use a more benign looking
+0000 instead and let "git log" going in such a case, instead
of aborting.
* jk/tzoffset-fix:
local_tzoffset: detect errors from tm_to_time_t
t0006: test various date formats
t0006: rename test-date's "show" to "relative"
Fix an unintended regression in v2.9 that breaks "clone --depth"
that recurses down to submodules by forcing the submodules to also
be cloned shallowly, which many server instances that host upstream
of the submodules are not prepared for.
* sb/clone-shallow-passthru:
clone: do not let --depth imply --shallow-submodules
A new run-command API function pipe_command() is introduced to
sanely feed data to the standard input while capturing data from
the standard output and the standard error of an external process,
which is cumbersome to hand-roll correctly without deadlocking.
The codepath to sign data in a prepared buffer with GPG has been
updated to use this API to read from the status-fd to check for
errors (instead of relying on GPG's exit status).
* jk/gpg-interface-cleanup:
gpg-interface: check gpg signature creation status
sign_buffer: use pipe_command
verify_signed_buffer: use pipe_command
run-command: add pipe_command helper
verify_signed_buffer: use tempfile object
verify_signed_buffer: drop pbuf variable
gpg-interface: use child_process.args
"log --graph --format=" learned that "%>|(N)" specifies the width
relative to the terminal's left edge, not relative to the area to
draw text that is to the right of the ancestry-graph section. It
also now accepts negative N that means the column limit is relative
to the right border.
* nd/graph-width-padded:
pretty.c: support <direction>|(<negative number>) forms
pretty: pass graph width to pretty formatting for use in '%>|(N)'
"git repack" learned the "--keep-unreachable" option, which sends
loose unreachable objects to a pack instead of leaving them loose.
This helps heuristics based on the number of loose objects
(e.g. "gc --auto").
* jk/repack-keep-unreachable:
repack: extend --keep-unreachable to loose objects
repack: add --keep-unreachable option
repack: document --unpack-unreachable option
Teach format-patch and mailsplit (hence "am") how a line that
happens to begin with "From " in the e-mail message is quoted with
">", so that these lines can be restored to their original shape.
* ew/mboxrd-format-am:
am: support --patch-format=mboxrd
mailsplit: support unescaping mboxrd messages
pretty: support "mboxrd" output format
"upload-pack" allows a custom "git pack-objects" replacement when
responding to "fetch/clone" via the uploadpack.packObjectsHook.
* jk/upload-pack-hook:
upload-pack: provide a hook for running pack-objects
t1308: do not get fooled by symbolic links to the source tree
config: add a notion of "scope"
config: return configset value for current_config_ functions
config: set up config_source for command-line config
git_config_parse_parameter: refactor cleanup code
git_config_with_options: drop "found" counting
HTTPd tests learned to show the server error log to help diagnosing
a failing tests.
* nd/test-lib-httpd-show-error-log-in-verbose:
lib-httpd.sh: print error.log on error
