Fix for a long-standing bug that leaves the index file corrupt when
it shrinks during a partial commit.
* jk/reopen-tempfile-truncate:
reopen_tempfile(): truncate opened file
The test 'switching trees does not invalidate shared index' in
't0090-cache-tree.sh' is about verifying the behaviour of the split
index feature, therefore it should be in full control of when index
splitting is performed, like all the tests in 't1700-split-index.sh'.
Unset GIT_TEST_SPLIT_INDEX for this test to avoid unintended random
index splitting.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We provide a reopen_tempfile() function, which is in turn
used by reopen_lockfile(). The idea is that a caller may
want to rewrite the tempfile without letting go of the lock.
And that's what our one caller does: after running
add--interactive, "commit -p" will update the cache-tree
extension of the index and write out the result, all while
holding the lock.
However, because we open the file with only the O_WRONLY
flag, the existing index content is left in place, and we
overwrite it starting at position 0. If the new index after
updating the cache-tree is smaller than the original, those
final bytes are not overwritten and remain in the file. This
results in a corrupt index, since those cruft bytes are
interpreted as part of the trailing hash (or even as an
extension, if there are enough bytes).
This bug actually pre-dates reopen_tempfile(); the original
code from 9c4d6c0297 (cache-tree: Write updated cache-tree
after commit, 2014-07-13) has the same bug, and those lines
were eventually refactored into the tempfile module. Nobody
noticed until now for two reasons:
- the bug can only be triggered in interactive mode
("commit -p" or "commit -i")
- the size of the index must shrink after updating the
cache-tree, which implies a non-trivial deletion. Notice
that the included test actually has to create a 2-deep
hierarchy. A single level is not enough to actually cause
shrinkage.
The fix is to truncate the file before writing out the
second index. We can do that at the caller by using
ftruncate(). But we shouldn't have to do that. There is no
other place in Git where we want to open a file and
overwrite bytes, making reopen_tempfile() a confusing and
error-prone interface. Let's pass O_TRUNC there, which gives
callers the same state they had after initially opening the
file or lock.
It's possible that we could later add a caller that wants
something else (e.g., to open with O_APPEND). But this is
the only caller we've had in the history of the codebase.
Let's punt on doing anything more clever until another one
comes along.
Reported-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Test fixes.
* sg/test-must-be-empty:
tests: use 'test_must_be_empty' instead of 'test_cmp <empty> <out>'
tests: use 'test_must_be_empty' instead of 'test_cmp /dev/null <out>'
tests: use 'test_must_be_empty' instead of 'test ! -s'
tests: use 'test_must_be_empty' instead of '! test -s'
Using 'test_must_be_empty' is preferable to '! test -s', because it
gives a helpful error message if the given file is unexpectedly not
empty, while the latter remains completely silent. Furthermore, it
also catches cases when the given file unexpectedly does not exist at
all.
This patch was basically created by:
sed -i -e 's/! test -s/test_must_be_empty/' t[0-9]*.sh
with the following notable exceptions:
- The '! test -s' check in '.gitmodules ignore=dirty suppresses
submodules with untracked content' in 't7508-status.sh' is left
as-is, because it's bogus and, therefore, it's subject of a
dedicated patch.
- The '! test -s' checks in 't9131-git-svn-empty-symlink.sh' and
't9135-git-svn-moved-branch-empty-file.sh' are immediately
preceeded by a 'test -f' to ensure that the files exist in the
first place. 'test_must_be_empty' ensures that as well, so those
'test -f' commands are removed as well.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
These tests employ a noisy subshell (with missing &&-chain) to feed
input into Git commands or files:
(echo a; echo b; echo c) | git some-command ...
Simplify by taking advantage of test_write_lines():
test_write_lines a b c | git some-command ...
Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Switch all uses of $_x40 to $OID_REGEX so that they work correctly with
larger hashes. This commit was created by using the following sed
command to modify all files in the t directory except t/test-lib.sh:
sed -i 's/\$_x40/$OID_REGEX/g'
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When re-priming the cache-tree opportunistically while committing
the in-core index as-is, we mistakenly invalidated the in-core
index too aggressively, causing the experimental split-index code
to unnecessarily rewrite the on-disk index file(s).
* dt/commit-preserve-base-index-upon-opportunistic-cache-tree-update:
commit: don't rewrite shared index unnecessarily
Remove a cache invalidation which would cause the shared index to be
rewritten on as-is commits.
When the cache-tree has changed, we need to update it. But we don't
necessarily need to update the shared index. So setting
active_cache_changed to SOMETHING_CHANGED is unnecessary. Instead, we
let update_main_cache_tree just update the CACHE_TREE_CHANGED bit.
In order to test this, make test-dump-split-index not segfault on
missing replace_bitmap/delete_bitmap. This new codepath is not called
now that the test passes, but is necessary to avoid a segfault when the
new test is run with the old builtin/commit.c code.
Signed-off-by: David Turner <dturner@twopensource.com>
Acked-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we unpack trees into an existing index, we discard the old
index and replace it with the new, merged index. Ensure that this
index has its cache-tree populated. This will make subsequent git
status and commit commands faster.
Signed-off-by: Brian Degenhardt <bmd@bmdhacks.com>
Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The awk statements previously used in this test weren't compatible
with the native versions of awk on Solaris:
echo "dir" | /bin/awk -v c=0 '$1 {++c} END {print c}'
awk: syntax error near line 1
awk: bailing out near line 1
echo "dir" | /usr/xpg4/bin/awk -v c=0 '$1 {++c} END {print c}'
0
Even though we do not cater to tools in /usr/bin on Solaris that
have and are overridden by corresponding ones in /usr/xpg?/bin,
in this case, even the XPG version does not work correctly.
With GNU awk for comparison:
echo "dir" | /opt/csw/gnu/awk -v c=0 '$1 {++c} END {print c}'
1
which is what this test expects (and is in line with POSIX; non-empty
string is true and an empty string is false).
Work this issue around by using $1 != "" to state more explicitly
that we are skipping empty lines.
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Ben Walton <bdwalton@gmail.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The add-interactive system is built in perl. If you build
with NO_PERL, running "git commit --interactive" will exit
with an error and the test will fail.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
FreeBSD's printf(1) doesn't accept empty strings for numerical format
specifiers:
$ printf "%d\n" "" >/dev/null; echo $?
printf: : expected numeric value
1
Initialize the AWK variable c to make sure the shell variable
subtree_count always contains a numerical value, in order to keep the
subsequently called printf happy.
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We punt from repairing the cache-tree during a branch switching if
it involves having to create a new tree object that does not yet
exist in the object store. "mkdir dir && >dir/file && git add dir"
followed by "git checkout" is one example, when a tree that records
the state of such "dir/" is not in the object store.
However, after discovering that we do not have a tree object that
records the state of "dir/", the caller failed to remember the fact
that it noticed the cache-tree entry it received for "dir/" is
invalidated, it already knows it should not be populating the level
that has "dir/" as its immediate subdirectory, and it is not an
error at all for the sublevel cache-tree entry gave it a bogus
object name it shouldn't even look at.
This led the caller to detect and report a non-existent error. The
end result was the same and we avoided stuffing a non-existent tree
to the cache-tree, but we shouldn't have issued an alarming error
message to the user.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
During the commit process, update the cache-tree. Write this updated
cache-tree so that it's ready for subsequent commands.
Add test code which demonstrates that git commit now writes the cache
tree. Make all tests test the entire cache-tree, not just the root
level.
Signed-off-by: David Turner <dturner@twitter.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add tests to confirm that invalidation of subdirectories neither over-
nor under-invalidates.
Signed-off-by: David Turner <dturner@twitter.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When git checkout checks out a branch, create or update the
cache-tree so that subsequent operations are faster.
update_main_cache_tree learned a new flag, WRITE_TREE_REPAIR. When
WRITE_TREE_REPAIR is set, portions of the cache-tree which do not
correspond to existing tree objects are invalidated (and portions which
do are marked as valid). No new tree objects are created.
Signed-off-by: David Turner <dturner@twitter.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Use 'printf %d $(whatever|wc -l)' so that the shell removes the blanks
for us.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Acked-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the case of --mixed and --hard, we throw away the old index and
rebuild everything from the tree argument (or HEAD). So we have an
opportunity here to fill in the cache-tree data, just as read-tree
did.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In prepare_index(), we refresh the index, and then write it to disk if
this changed the index data. After running hooks we re-read the index
and compute the root tree sha1 with the cache-tree machinery.
This gives us a mostly free opportunity to write up-to-date cache-tree
data: we can compute it in prepare_index() immediately before writing
the index to disk.
If we do this, we were going to write the index anyway, and the later
cache-tree update has no further work to do. If we don't do it, we
don't do any extra work, though we still don't have have cache-tree
data after the commit.
The only case that suffers badly is when the pre-commit hook changes
many trees in the index. I'm writing this off as highly unusual.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The cache-tree optimization originally helped speed up write-tree
operation. However, many commands no longer properly maintain -- or
use an opportunity to cheaply generate -- the cache-tree data. In
particular, this affects commit, checkout and reset. The notable
examples that *do* write cache-tree data are read-tree and write-tree.
This sadly means most people no longer benefit from the optimization,
as they would not normally use the plumbing commands.
Document the current state of affairs in a test file, in preparation
for improvements in the area.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
