git-commit-vandalism/t/t1060-object-corruption.sh

150 lines
3.7 KiB
Bash
Raw Normal View History

#!/bin/sh
test_description='see how we handle various forms of corruption'
revisions API: have release_revisions() release "cmdline" Extend the the release_revisions() function so that it frees the "cmdline" in the "struct rev_info". This in combination with a preceding change to free "commits" and "mailmap" means that we can whitelist another test under "TEST_PASSES_SANITIZE_LEAK=true". There was a proposal in [1] to do away with xstrdup()-ing this add_rev_cmdline(), perhaps that would be worthwhile, but for now let's just free() it. We could also make that a "char *" in "struct rev_cmdline_entry" itself, but since we own it let's expose it as a constant to outside callers. I proposed that in [2] but have since changed my mind. See 14d30cdfc04 (ref-filter: fix memory leak in `free_array_item()`, 2019-07-10), c514c62a4fd (checkout: fix leak of non-existent branch names, 2020-08-14) and other log history hits for "free((char *)" for prior art. This includes the tests we had false-positive passes on before my 6798b08e848 (perl Git.pm: don't ignore signalled failure in _cmd_close(), 2022-02-01), now they pass for real. Since there are 66 tests matching t/t[0-9]*git-svn*.sh it's easier to list those that don't pass than to touch most of those 66. So let's introduce a "TEST_FAILS_SANITIZE_LEAK=true", which if set in the tests won't cause lib-git-svn.sh to set "TEST_PASSES_SANITIZE_LEAK=true. This change also marks all the tests that we removed "TEST_FAILS_SANITIZE_LEAK=true" from in an earlier commit due to removing the UNLEAK() from cmd_format_patch(), we can now assert that its API use doesn't leak any "struct rev_info" memory. This change also made commit "t5503-tagfollow.sh" pass on current master, but that would regress when combined with ps/fetch-atomic-fixup's de004e848a9 (t5503: simplify setup of test which exercises failure of backfill, 2022-03-03) (through no fault of that topic, that change started using "git clone" in the test, which has an outstanding leak). Let's leave that test out for now to avoid in-flight semantic conflicts. 1. https://lore.kernel.org/git/YUj%2FgFRh6pwrZalY@carlos-mbp.lan/ 2. https://lore.kernel.org/git/87o88obkb1.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-04-13 22:01:47 +02:00
2022-04-13 22:01:48 +02:00
TEST_PASSES_SANITIZE_LEAK=true
. ./test-lib.sh
# convert "1234abcd" to ".git/objects/12/34abcd"
obj_to_file() {
echo "$(git rev-parse --git-dir)/objects/$(git rev-parse "$1" | sed 's,..,&/,')"
}
# Convert byte at offset "$2" of object "$1" into '\0'
corrupt_byte() {
obj_file=$(obj_to_file "$1") &&
chmod +w "$obj_file" &&
printf '\0' | dd of="$obj_file" bs=1 seek="$2" conv=notrunc
}
test_expect_success 'setup corrupt repo' '
git init bit-error &&
(
cd bit-error &&
test_commit content &&
corrupt_byte HEAD:content.t 10
index-pack: detect local corruption in collision check When we notice that we have a local copy of an incoming object, we compare the two objects to make sure we haven't found a collision. Before we get to the actual object bytes, though, we compare the type and size from sha1_object_info(). If our local object is corrupted, then the type will be OBJ_BAD, which obviously will not match the incoming type, and we'll report "SHA1 COLLISION FOUND" (with capital letters and everything). This is confusing, as the problem is not a collision but rather local corruption. We should report that instead (just like we do if reading the rest of the object content fails a few lines later). Note that we _could_ just ignore the error and mark it as a non-collision. That would let you "git fetch" to replace a corrupted object. But it's not a very reliable method for repairing a repository. The earlier want/have negotiation tries to get the other side to omit objects we already have, and it would not realize that we are "missing" this corrupted object. So we're better off complaining loudly when we see corruption, and letting the user take more drastic measures to repair (like making a full clone elsewhere and copying the pack into place). Note that the test sets transfer.unpackLimit in the receiving repository so that we use index-pack (which is what does the collision check). Normally for such a small push we'd use unpack-objects, which would simply try to write the loose object, and discard the new one when we see that there's already an old one. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-01 10:09:32 +02:00
) &&
git init no-bit-error &&
(
# distinct commit from bit-error, but containing a
# non-corrupted version of the same blob
cd no-bit-error &&
test_tick &&
test_commit content
)
'
test_expect_success 'setup repo with missing object' '
git init missing &&
(
cd missing &&
test_commit content &&
rm -f "$(obj_to_file HEAD:content.t)"
)
'
test_expect_success 'setup repo with misnamed object' '
git init misnamed &&
(
cd misnamed &&
test_commit content &&
good=$(obj_to_file HEAD:content.t) &&
blob=$(echo corrupt | git hash-object -w --stdin) &&
bad=$(obj_to_file $blob) &&
rm -f "$good" &&
mv "$bad" "$good"
)
'
test_expect_success 'streaming a corrupt blob fails' '
(
cd bit-error &&
test_must_fail git cat-file blob HEAD:content.t
)
'
test_expect_success 'getting type of a corrupt blob fails' '
(
cd bit-error &&
test_must_fail git cat-file -s HEAD:content.t
)
'
test_expect_success 'read-tree -u detects bit-errors in blobs' '
(
cd bit-error &&
rm -f content.t &&
test_must_fail git read-tree --reset -u HEAD
)
'
test_expect_success 'read-tree -u detects missing objects' '
(
cd missing &&
rm -f content.t &&
test_must_fail git read-tree --reset -u HEAD
)
'
# We use --bare to make sure that the transport detects it, not the checkout
# phase.
test_expect_success 'clone --no-local --bare detects corruption' '
test_must_fail git clone --no-local --bare bit-error corrupt-transport
'
test_expect_success 'clone --no-local --bare detects missing object' '
test_must_fail git clone --no-local --bare missing missing-transport
'
test_expect_success 'clone --no-local --bare detects misnamed object' '
test_must_fail git clone --no-local --bare misnamed misnamed-transport
'
# We do not expect --local to detect corruption at the transport layer,
# so we are really checking the checkout() code path.
test_expect_success 'clone --local detects corruption' '
test_must_fail git clone --local bit-error corrupt-checkout
'
test_expect_success 'error detected during checkout leaves repo intact' '
test_path_is_dir corrupt-checkout/.git
'
test_expect_success 'clone --local detects missing objects' '
test_must_fail git clone --local missing missing-checkout
'
test_expect_failure 'clone --local detects misnamed objects' '
test_must_fail git clone --local misnamed misnamed-checkout
'
index-pack: detect local corruption in collision check When we notice that we have a local copy of an incoming object, we compare the two objects to make sure we haven't found a collision. Before we get to the actual object bytes, though, we compare the type and size from sha1_object_info(). If our local object is corrupted, then the type will be OBJ_BAD, which obviously will not match the incoming type, and we'll report "SHA1 COLLISION FOUND" (with capital letters and everything). This is confusing, as the problem is not a collision but rather local corruption. We should report that instead (just like we do if reading the rest of the object content fails a few lines later). Note that we _could_ just ignore the error and mark it as a non-collision. That would let you "git fetch" to replace a corrupted object. But it's not a very reliable method for repairing a repository. The earlier want/have negotiation tries to get the other side to omit objects we already have, and it would not realize that we are "missing" this corrupted object. So we're better off complaining loudly when we see corruption, and letting the user take more drastic measures to repair (like making a full clone elsewhere and copying the pack into place). Note that the test sets transfer.unpackLimit in the receiving repository so that we use index-pack (which is what does the collision check). Normally for such a small push we'd use unpack-objects, which would simply try to write the loose object, and discard the new one when we see that there's already an old one. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-01 10:09:32 +02:00
test_expect_success 'fetch into corrupted repo with index-pack' '
cp -R bit-error bit-error-cp &&
test_when_finished "rm -rf bit-error-cp" &&
index-pack: detect local corruption in collision check When we notice that we have a local copy of an incoming object, we compare the two objects to make sure we haven't found a collision. Before we get to the actual object bytes, though, we compare the type and size from sha1_object_info(). If our local object is corrupted, then the type will be OBJ_BAD, which obviously will not match the incoming type, and we'll report "SHA1 COLLISION FOUND" (with capital letters and everything). This is confusing, as the problem is not a collision but rather local corruption. We should report that instead (just like we do if reading the rest of the object content fails a few lines later). Note that we _could_ just ignore the error and mark it as a non-collision. That would let you "git fetch" to replace a corrupted object. But it's not a very reliable method for repairing a repository. The earlier want/have negotiation tries to get the other side to omit objects we already have, and it would not realize that we are "missing" this corrupted object. So we're better off complaining loudly when we see corruption, and letting the user take more drastic measures to repair (like making a full clone elsewhere and copying the pack into place). Note that the test sets transfer.unpackLimit in the receiving repository so that we use index-pack (which is what does the collision check). Normally for such a small push we'd use unpack-objects, which would simply try to write the loose object, and discard the new one when we see that there's already an old one. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-01 10:09:32 +02:00
(
cd bit-error-cp &&
index-pack: detect local corruption in collision check When we notice that we have a local copy of an incoming object, we compare the two objects to make sure we haven't found a collision. Before we get to the actual object bytes, though, we compare the type and size from sha1_object_info(). If our local object is corrupted, then the type will be OBJ_BAD, which obviously will not match the incoming type, and we'll report "SHA1 COLLISION FOUND" (with capital letters and everything). This is confusing, as the problem is not a collision but rather local corruption. We should report that instead (just like we do if reading the rest of the object content fails a few lines later). Note that we _could_ just ignore the error and mark it as a non-collision. That would let you "git fetch" to replace a corrupted object. But it's not a very reliable method for repairing a repository. The earlier want/have negotiation tries to get the other side to omit objects we already have, and it would not realize that we are "missing" this corrupted object. So we're better off complaining loudly when we see corruption, and letting the user take more drastic measures to repair (like making a full clone elsewhere and copying the pack into place). Note that the test sets transfer.unpackLimit in the receiving repository so that we use index-pack (which is what does the collision check). Normally for such a small push we'd use unpack-objects, which would simply try to write the loose object, and discard the new one when we see that there's already an old one. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-01 10:09:32 +02:00
test_must_fail git -c transfer.unpackLimit=1 \
fetch ../no-bit-error 2>stderr &&
test_i18ngrep ! -i collision stderr
)
'
rev-list: allow cached objects in existence check This fixes a regression in 7c0fe330d5 (rev-list: handle missing tree objects properly, 2018-10-05) where rev-list will now complain about the empty tree when it doesn't physically exist on disk. Before that commit, we relied on the traversal code in list-objects.c to walk through the trees. Since it uses parse_tree(), we'd do a normal object lookup that includes looking in the set of "cached" objects (which is where our magic internal empty-tree kicks in). After that commit, we instead tell list-objects.c not to die on any missing trees, and we check them ourselves using has_object_file(). But that function uses OBJECT_INFO_SKIP_CACHED, which means we won't use our internal empty tree. This normally wouldn't come up. For most operations, Git will try to write out the empty tree object as it would any other object. And pack-objects in a push or fetch will send the empty tree (even if it's virtual on the sending side). However, there are cases where this can matter. One I found in the wild: 1. The root tree of a commit became empty by deleting all files, without using an index. In this case it was done using libgit2's tree builder API, but as the included test shows, it can easily be done with regular git using hash-object. The resulting repo works OK, as we'd avoid walking over our own reachable commits for a connectivity check. 2. Cloning with --reference pointing to the repository from (1) can trigger the problem, because we tell the other side we already have that commit (and hence the empty tree), but then walk over it during the connectivity check (where we complain about it missing). Arguably the workflow in step (1) should be more careful about writing the empty tree object if we're referencing it. But this workflow did work prior to 7c0fe330d5, so let's restore it. This patch makes the minimal fix, which is to swap out a direct call to oid_object_info_extended(), minus the SKIP_CACHED flag, instead of calling has_object_file(). This is all that has_object_file() is doing under the hood. And there's little danger of unrelated fallout from other unexpected "cached" objects, since there's only one call site that ends such a cached object, and it's in git-blame. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-04 18:40:54 +01:00
test_expect_success 'internal tree objects are not "missing"' '
git init missing-empty &&
(
cd missing-empty &&
empty_tree=$(git hash-object -t tree /dev/null) &&
commit=$(echo foo | git commit-tree $empty_tree) &&
git rev-list --objects $commit
)
'
t1060: check partial clone of misnamed blob A recent commit (upload-pack: skip parse-object re-hashing of "want" objects, 2022-09-06) loosened the behavior of upload-pack so that it does not verify the sha1 of objects it receives directly via "want" requests. The existing corruption tests in t1060 aren't affected by this: the corruptions are blobs reachable from commits, and the client requests the commits. The more interesting case here is a partial clone, where the client will directly ask for the corrupted blob when it does an on-demand fetch of the filtered object. And that is not covered at all, so let's add a test. It's important here that we use the "misnamed" corruption and not "bit-error". The latter is sufficiently corrupted that upload-pack cannot even figure out the type of the object, so it bails identically both before and after the recent change. But with "misnamed", with the hash-checks enabled it sees the problem (though the error messages are a bit confusing because of the inability to create a "struct object" to store the flags): error: hash mismatch d95f3ad14dee633a758d2e331151e950dd13e4ed fatal: git upload-pack: not our ref d95f3ad14dee633a758d2e331151e950dd13e4ed fatal: remote error: upload-pack: not our ref d95f3ad14dee633a758d2e331151e950dd13e4ed After the change to skip the hash check, the server side happily sends the bogus object, but the client correctly realizes that it did not get the necessary data: remote: Enumerating objects: 1, done. remote: Counting objects: 100% (1/1), done. remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (1/1), 49 bytes | 49.00 KiB/s, done. fatal: bad revision 'd95f3ad14dee633a758d2e331151e950dd13e4ed' error: [...]/misnamed did not send all necessary objects which is exactly what we expect to happen. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-09-07 23:02:20 +02:00
test_expect_success 'partial clone of corrupted repository' '
test_config -C misnamed uploadpack.allowFilter true &&
git clone --no-local --no-checkout --filter=blob:none \
misnamed corrupt-partial && \
test_must_fail git -C corrupt-partial checkout --force
'
test_done