3a251bac0d
Fix the overly verbose trace2 logging added in9a4987677d(trace2: add stats for fsync operations, 2022-03-30) (first released with v2.36.0). Since that change every single "git" command invocation has included these "data" events, even though we'll only make use of these with core.fsyncMethod=batch, and even then only have non-zero values if we're writing object data to disk. Seec0f4752ed2(core.fsyncmethod: batched disk flushes for loose-objects, 2022-04-04) for that feature. As we're needing to indent the trace2_data_intmax() lines let's introduce helper variables to ensure that our resulting lines (which were already too) don't exceed the recommendations of the CodingGuidelines. Doing that requires either wrapping them twice, or introducing short throwaway variable names, let's do the latter. The result was that e.g. "git version" would previously emit a total of 6 trace2 events with the GIT_TRACE2_EVENT target (version, start, cmd_ancestry, cmd_name, exit, atexit), but afterwards would emit 8. We'd emit 2 "data" events before the "exit" event. The reason we didn't catch this was that the trace2 unit tests added ina15860dca3(trace2: t/helper/test-trace2, t0210.sh, t0211.sh, t0212.sh, 2019-02-22) would omit any "data" events that weren't the ones it cared about. Before this change to the C code 6/7 of our "t/t0212-trace2-event.sh" tests would fail if this change was applied to "t/t0212/parse_events.perl". Let's make the trace2 testing more strict, and further append any new events types we don't know about in "t/t0212/parse_events.perl". Since we only invoke the "test-tool trace2" there's no guarantee that we'll catch other overly verbose events in the future, but we'll at least notice if we start emitting new events that are issues every time we log anything with trace2's JSON target. We exclude the "data_json" event type, we'd otherwise would fail on both "win test" and "win+VS test" CI due to the logging added in353d3d77f4(trace2: collect Windows-specific process information, 2019-02-22). It looks like that logging should really be using trace2_cmd_ancestry() instead, which was introduced later in2f732bf15e(tr2: log parent process name, 2021-07-21), but let's leave it for now. The fix-up toaaf81223f4(unpack-objects: use stream_loose_object() to unpack large objects, 2022-06-11) is needed because we're changing the behavior of these events as discussed above. Since we'd always emit a "hardware-flush" event the test added inaaf81223f4wasn't testing anything except that this trace2 data was unconditionally logged. Even if "core.fsyncMethod" wasn't set to "batch" we'd pass the test. Now we'll check the expected number of "writeout" v.s. "flush" calls under "core.fsyncMethod=batch", but note that this doesn't actually test if we carried out the sync using that method, on a platform where we'd have to fall back to fsync() each of those "writeout" would really be a "flush" (i.e. a full fsync()). But in this case what we're testing is that the logic in "unpack-objects" behaves as expected, not the OS-specific question of whether we actually were able to use the "bulk" method. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
97 lines
2.6 KiB
Bash
Executable File
97 lines
2.6 KiB
Bash
Executable File
#!/bin/sh
|
|
#
|
|
# Copyright (c) 2022 Han Xin
|
|
#
|
|
|
|
test_description='git unpack-objects with large objects'
|
|
|
|
. ./test-lib.sh
|
|
|
|
prepare_dest () {
|
|
test_when_finished "rm -rf dest.git" &&
|
|
git init --bare dest.git &&
|
|
git -C dest.git config core.bigFileThreshold "$1"
|
|
}
|
|
|
|
test_expect_success "create large objects (1.5 MB) and PACK" '
|
|
test-tool genrandom foo 1500000 >big-blob &&
|
|
test_commit --append foo big-blob &&
|
|
test-tool genrandom bar 1500000 >big-blob &&
|
|
test_commit --append bar big-blob &&
|
|
PACK=$(echo HEAD | git pack-objects --revs pack) &&
|
|
git verify-pack -v pack-$PACK.pack >out &&
|
|
sed -n -e "s/^\([0-9a-f][0-9a-f]*\).*\(commit\|tree\|blob\).*/\1/p" \
|
|
<out >obj-list
|
|
'
|
|
|
|
test_expect_success 'set memory limitation to 1MB' '
|
|
GIT_ALLOC_LIMIT=1m &&
|
|
export GIT_ALLOC_LIMIT
|
|
'
|
|
|
|
test_expect_success 'unpack-objects failed under memory limitation' '
|
|
prepare_dest 2m &&
|
|
test_must_fail git -C dest.git unpack-objects <pack-$PACK.pack 2>err &&
|
|
grep "fatal: attempting to allocate" err
|
|
'
|
|
|
|
test_expect_success 'unpack-objects works with memory limitation in dry-run mode' '
|
|
prepare_dest 2m &&
|
|
git -C dest.git unpack-objects -n <pack-$PACK.pack &&
|
|
test_stdout_line_count = 0 find dest.git/objects -type f &&
|
|
test_dir_is_empty dest.git/objects/pack
|
|
'
|
|
|
|
test_expect_success 'unpack big object in stream' '
|
|
prepare_dest 1m &&
|
|
git -C dest.git unpack-objects <pack-$PACK.pack &&
|
|
test_dir_is_empty dest.git/objects/pack
|
|
'
|
|
|
|
check_fsync_events () {
|
|
local trace="$1" &&
|
|
shift &&
|
|
|
|
cat >expect &&
|
|
sed -n \
|
|
-e '/^{"event":"data",.*"category":"fsync",/ {
|
|
s/.*"category":"fsync",//;
|
|
s/}$//;
|
|
p;
|
|
}' \
|
|
<"$trace" >actual &&
|
|
test_cmp expect actual
|
|
}
|
|
|
|
BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch'
|
|
|
|
test_expect_success 'unpack big object in stream (core.fsyncmethod=batch)' '
|
|
prepare_dest 1m &&
|
|
GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \
|
|
GIT_TEST_FSYNC=true \
|
|
git -C dest.git $BATCH_CONFIGURATION unpack-objects <pack-$PACK.pack &&
|
|
check_fsync_events trace2.txt <<-\EOF &&
|
|
"key":"fsync/writeout-only","value":"6"
|
|
"key":"fsync/hardware-flush","value":"1"
|
|
EOF
|
|
|
|
test_dir_is_empty dest.git/objects/pack &&
|
|
git -C dest.git cat-file --batch-check="%(objectname)" <obj-list >current &&
|
|
cmp obj-list current
|
|
'
|
|
|
|
test_expect_success 'do not unpack existing large objects' '
|
|
prepare_dest 1m &&
|
|
git -C dest.git index-pack --stdin <pack-$PACK.pack &&
|
|
git -C dest.git unpack-objects <pack-$PACK.pack &&
|
|
|
|
# The destination came up with the exact same pack...
|
|
DEST_PACK=$(echo dest.git/objects/pack/pack-*.pack) &&
|
|
test_cmp pack-$PACK.pack $DEST_PACK &&
|
|
|
|
# ...and wrote no loose objects
|
|
test_stdout_line_count = 0 find dest.git/objects -type f ! -name "pack-*"
|
|
'
|
|
|
|
test_done
|