builtin/repack.c: add '--geometric' option
Often it is useful to both:
- have relatively few packfiles in a repository, and
- avoid having so few packfiles in a repository that we repack its
entire contents regularly
This patch implements a '--geometric=<n>' option in 'git repack'. This
allows the caller to specify that they would like each pack to be at
least a factor times as large as the previous largest pack (by object
count).
Concretely, say that a repository has 'n' packfiles, labeled P1, P2,
..., up to Pn. Each packfile has an object count equal to 'objects(Pn)'.
With a geometric factor of 'r', it should be that:
objects(Pi) > r*objects(P(i-1))
for all i in [1, n], where the packs are sorted by
objects(P1) <= objects(P2) <= ... <= objects(Pn).
Since finding a true optimal repacking is NP-hard, we approximate it
along two directions:
1. We assume that there is a cutoff of packs _before starting the
repack_ where everything to the right of that cut-off already forms
a geometric progression (or no cutoff exists and everything must be
repacked).
2. We assume that everything smaller than the cutoff count must be
repacked. This forms our base assumption, but it can also cause
even the "heavy" packs to get repacked, for e.g., if we have 6
packs containing the following number of objects:
1, 1, 1, 2, 4, 32
then we would place the cutoff between '1, 1' and '1, 2, 4, 32',
rolling up the first two packs into a pack with 2 objects. That
breaks our progression and leaves us:
2, 1, 2, 4, 32
^
(where the '^' indicates the position of our split). To restore a
progression, we move the split forward (towards larger packs)
joining each pack into our new pack until a geometric progression
is restored. Here, that looks like:
2, 1, 2, 4, 32 ~> 3, 2, 4, 32 ~> 5, 4, 32 ~> ... ~> 9, 32
^ ^ ^ ^
This has the advantage of not repacking the heavy-side of packs too
often while also only creating one new pack at a time. Another wrinkle
is that we assume that loose, indexed, and reflog'd objects are
insignificant, and lump them into any new pack that we create. This can
lead to non-idempotent results.
Suggested-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Reviewed-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-02-23 03:25:27 +01:00
|
|
|
#!/bin/sh
|
|
|
|
|
|
|
|
|
|
test_description='git repack --geometric works correctly'
|
|
|
|
|
|
|
|
|
|
. ./test-lib.sh
|
|
|
|
|
|
|
|
|
|
GIT_TEST_MULTI_PACK_INDEX=0
|
|
|
|
|
|
|
|
|
|
objdir=.git/objects
|
|
|
|
|
midx=$objdir/pack/multi-pack-index
|
|
|
|
|
|
|
|
|
|
test_expect_success '--geometric with no packs' '
|
|
|
|
|
git init geometric &&
|
|
|
|
|
test_when_finished "rm -fr geometric" &&
|
|
|
|
|
(
|
|
|
|
|
cd geometric &&
|
|
|
|
|
|
builtin/repack.c: support writing a MIDX while repacking
Teach `git repack` a new `--write-midx` option for callers that wish to
persist a multi-pack index in their repository while repacking.
There are two existing alternatives to this new flag, but they don't
cover our particular use-case. These alternatives are:
- Call 'git multi-pack-index write' after running 'git repack', or
- Set 'GIT_TEST_MULTI_PACK_INDEX=1' in your environment when running
'git repack'.
The former works, but introduces a gap in bitmap coverage between
repacking and writing a new MIDX (since the repack may have deleted a
pack included in the existing MIDX, invalidating it altogether).
Setting the 'GIT_TEST_' environment variable is obviously unsupported.
In fact, even if it were supported officially, it still wouldn't work,
because it generates the MIDX *after* redundant packs have been dropped,
leading to the same issue as above.
Introduce a new option which eliminates this race by teaching `git
repack` to generate the MIDX at the critical point: after the new packs
have been written and moved into place, but before the redundant packs
have been removed.
This option is compatible with `git repack`'s '--bitmap' option (it
changes the interpretation to be: "write a bitmap corresponding to the
MIDX after one has been generated").
There is a little bit of additional noise in the patch below to avoid
repeating ourselves when selecting which packs to delete. Instead of a
single loop as before (where we iterate over 'existing_packs', decide if
a pack is worth deleting, and if so, delete it), we have two loops (the
first where we decide which ones are worth deleting, and the second
where we actually do the deleting). This makes it so we have a single
check we can make consistently when (1) telling the MIDX which packs we
want to exclude, and (2) actually unlinking the redundant packs.
There is also a tiny change to short-circuit the body of
write_midx_included_packs() when no packs remain in the case of an empty
repository. The MIDX code does not handle this, so avoid trying to
generate a MIDX covering zero packs in the first place.
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-29 03:55:18 +02:00
|
|
|
git repack --write-midx --geometric 2 >out &&
|
builtin/repack.c: add '--geometric' option
Often it is useful to both:
- have relatively few packfiles in a repository, and
- avoid having so few packfiles in a repository that we repack its
entire contents regularly
This patch implements a '--geometric=<n>' option in 'git repack'. This
allows the caller to specify that they would like each pack to be at
least a factor times as large as the previous largest pack (by object
count).
Concretely, say that a repository has 'n' packfiles, labeled P1, P2,
..., up to Pn. Each packfile has an object count equal to 'objects(Pn)'.
With a geometric factor of 'r', it should be that:
objects(Pi) > r*objects(P(i-1))
for all i in [1, n], where the packs are sorted by
objects(P1) <= objects(P2) <= ... <= objects(Pn).
Since finding a true optimal repacking is NP-hard, we approximate it
along two directions:
1. We assume that there is a cutoff of packs _before starting the
repack_ where everything to the right of that cut-off already forms
a geometric progression (or no cutoff exists and everything must be
repacked).
2. We assume that everything smaller than the cutoff count must be
repacked. This forms our base assumption, but it can also cause
even the "heavy" packs to get repacked, for e.g., if we have 6
packs containing the following number of objects:
1, 1, 1, 2, 4, 32
then we would place the cutoff between '1, 1' and '1, 2, 4, 32',
rolling up the first two packs into a pack with 2 objects. That
breaks our progression and leaves us:
2, 1, 2, 4, 32
^
(where the '^' indicates the position of our split). To restore a
progression, we move the split forward (towards larger packs)
joining each pack into our new pack until a geometric progression
is restored. Here, that looks like:
2, 1, 2, 4, 32 ~> 3, 2, 4, 32 ~> 5, 4, 32 ~> ... ~> 9, 32
^ ^ ^ ^
This has the advantage of not repacking the heavy-side of packs too
often while also only creating one new pack at a time. Another wrinkle
is that we assume that loose, indexed, and reflog'd objects are
insignificant, and lump them into any new pack that we create. This can
lead to non-idempotent results.
Suggested-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Reviewed-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-02-23 03:25:27 +01:00
|
|
|
test_i18ngrep "Nothing new to pack" out
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
2021-03-05 16:21:37 +01:00
|
|
|
test_expect_success '--geometric with one pack' '
|
|
|
|
|
git init geometric &&
|
|
|
|
|
test_when_finished "rm -fr geometric" &&
|
|
|
|
|
(
|
|
|
|
|
cd geometric &&
|
|
|
|
|
|
|
|
|
|
test_commit "base" &&
|
|
|
|
|
git repack -d &&
|
|
|
|
|
|
|
|
|
|
git repack --geometric 2 >out &&
|
|
|
|
|
|
|
|
|
|
test_i18ngrep "Nothing new to pack" out
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
builtin/repack.c: add '--geometric' option
Often it is useful to both:
- have relatively few packfiles in a repository, and
- avoid having so few packfiles in a repository that we repack its
entire contents regularly
This patch implements a '--geometric=<n>' option in 'git repack'. This
allows the caller to specify that they would like each pack to be at
least a factor times as large as the previous largest pack (by object
count).
Concretely, say that a repository has 'n' packfiles, labeled P1, P2,
..., up to Pn. Each packfile has an object count equal to 'objects(Pn)'.
With a geometric factor of 'r', it should be that:
objects(Pi) > r*objects(P(i-1))
for all i in [1, n], where the packs are sorted by
objects(P1) <= objects(P2) <= ... <= objects(Pn).
Since finding a true optimal repacking is NP-hard, we approximate it
along two directions:
1. We assume that there is a cutoff of packs _before starting the
repack_ where everything to the right of that cut-off already forms
a geometric progression (or no cutoff exists and everything must be
repacked).
2. We assume that everything smaller than the cutoff count must be
repacked. This forms our base assumption, but it can also cause
even the "heavy" packs to get repacked, for e.g., if we have 6
packs containing the following number of objects:
1, 1, 1, 2, 4, 32
then we would place the cutoff between '1, 1' and '1, 2, 4, 32',
rolling up the first two packs into a pack with 2 objects. That
breaks our progression and leaves us:
2, 1, 2, 4, 32
^
(where the '^' indicates the position of our split). To restore a
progression, we move the split forward (towards larger packs)
joining each pack into our new pack until a geometric progression
is restored. Here, that looks like:
2, 1, 2, 4, 32 ~> 3, 2, 4, 32 ~> 5, 4, 32 ~> ... ~> 9, 32
^ ^ ^ ^
This has the advantage of not repacking the heavy-side of packs too
often while also only creating one new pack at a time. Another wrinkle
is that we assume that loose, indexed, and reflog'd objects are
insignificant, and lump them into any new pack that we create. This can
lead to non-idempotent results.
Suggested-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Reviewed-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-02-23 03:25:27 +01:00
|
|
|
test_expect_success '--geometric with an intact progression' '
|
|
|
|
|
git init geometric &&
|
|
|
|
|
test_when_finished "rm -fr geometric" &&
|
|
|
|
|
(
|
|
|
|
|
cd geometric &&
|
|
|
|
|
|
|
|
|
|
# These packs already form a geometric progression.
|
|
|
|
|
test_commit_bulk --start=1 1 && # 3 objects
|
|
|
|
|
test_commit_bulk --start=2 2 && # 6 objects
|
|
|
|
|
test_commit_bulk --start=4 4 && # 12 objects
|
|
|
|
|
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >expect &&
|
|
|
|
|
git repack --geometric 2 -d &&
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >actual &&
|
|
|
|
|
|
|
|
|
|
test_cmp expect actual
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
2021-03-05 16:21:43 +01:00
|
|
|
test_expect_success '--geometric with loose objects' '
|
|
|
|
|
git init geometric &&
|
|
|
|
|
test_when_finished "rm -fr geometric" &&
|
|
|
|
|
(
|
|
|
|
|
cd geometric &&
|
|
|
|
|
|
|
|
|
|
# These packs already form a geometric progression.
|
|
|
|
|
test_commit_bulk --start=1 1 && # 3 objects
|
|
|
|
|
test_commit_bulk --start=2 2 && # 6 objects
|
|
|
|
|
# The loose objects are packed together, breaking the
|
|
|
|
|
# progression.
|
|
|
|
|
test_commit loose && # 3 objects
|
|
|
|
|
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >before &&
|
|
|
|
|
git repack --geometric 2 -d &&
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >after &&
|
|
|
|
|
|
|
|
|
|
comm -13 before after >new &&
|
|
|
|
|
comm -23 before after >removed &&
|
|
|
|
|
|
|
|
|
|
test_line_count = 1 new &&
|
|
|
|
|
test_must_be_empty removed &&
|
|
|
|
|
|
|
|
|
|
git repack --geometric 2 -d &&
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >after &&
|
|
|
|
|
|
|
|
|
|
# The progression (3, 3, 6) is combined into one new pack.
|
|
|
|
|
test_line_count = 1 after
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
builtin/repack.c: add '--geometric' option
Often it is useful to both:
- have relatively few packfiles in a repository, and
- avoid having so few packfiles in a repository that we repack its
entire contents regularly
This patch implements a '--geometric=<n>' option in 'git repack'. This
allows the caller to specify that they would like each pack to be at
least a factor times as large as the previous largest pack (by object
count).
Concretely, say that a repository has 'n' packfiles, labeled P1, P2,
..., up to Pn. Each packfile has an object count equal to 'objects(Pn)'.
With a geometric factor of 'r', it should be that:
objects(Pi) > r*objects(P(i-1))
for all i in [1, n], where the packs are sorted by
objects(P1) <= objects(P2) <= ... <= objects(Pn).
Since finding a true optimal repacking is NP-hard, we approximate it
along two directions:
1. We assume that there is a cutoff of packs _before starting the
repack_ where everything to the right of that cut-off already forms
a geometric progression (or no cutoff exists and everything must be
repacked).
2. We assume that everything smaller than the cutoff count must be
repacked. This forms our base assumption, but it can also cause
even the "heavy" packs to get repacked, for e.g., if we have 6
packs containing the following number of objects:
1, 1, 1, 2, 4, 32
then we would place the cutoff between '1, 1' and '1, 2, 4, 32',
rolling up the first two packs into a pack with 2 objects. That
breaks our progression and leaves us:
2, 1, 2, 4, 32
^
(where the '^' indicates the position of our split). To restore a
progression, we move the split forward (towards larger packs)
joining each pack into our new pack until a geometric progression
is restored. Here, that looks like:
2, 1, 2, 4, 32 ~> 3, 2, 4, 32 ~> 5, 4, 32 ~> ... ~> 9, 32
^ ^ ^ ^
This has the advantage of not repacking the heavy-side of packs too
often while also only creating one new pack at a time. Another wrinkle
is that we assume that loose, indexed, and reflog'd objects are
insignificant, and lump them into any new pack that we create. This can
lead to non-idempotent results.
Suggested-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Reviewed-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-02-23 03:25:27 +01:00
|
|
|
test_expect_success '--geometric with small-pack rollup' '
|
|
|
|
|
git init geometric &&
|
|
|
|
|
test_when_finished "rm -fr geometric" &&
|
|
|
|
|
(
|
|
|
|
|
cd geometric &&
|
|
|
|
|
|
|
|
|
|
test_commit_bulk --start=1 1 && # 3 objects
|
|
|
|
|
test_commit_bulk --start=2 1 && # 3 objects
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >small &&
|
|
|
|
|
test_commit_bulk --start=3 4 && # 12 objects
|
|
|
|
|
test_commit_bulk --start=7 8 && # 24 objects
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >before &&
|
|
|
|
|
|
|
|
|
|
git repack --geometric 2 -d &&
|
|
|
|
|
|
|
|
|
|
# Three packs in total; two of the existing large ones, and one
|
|
|
|
|
# new one.
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >after &&
|
|
|
|
|
test_line_count = 3 after &&
|
|
|
|
|
comm -3 small before | tr -d "\t" >large &&
|
|
|
|
|
grep -qFf large after
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
|
|
|
|
test_expect_success '--geometric with small- and large-pack rollup' '
|
|
|
|
|
git init geometric &&
|
|
|
|
|
test_when_finished "rm -fr geometric" &&
|
|
|
|
|
(
|
|
|
|
|
cd geometric &&
|
|
|
|
|
|
|
|
|
|
# size(small1) + size(small2) > size(medium) / 2
|
|
|
|
|
test_commit_bulk --start=1 1 && # 3 objects
|
|
|
|
|
test_commit_bulk --start=2 1 && # 3 objects
|
|
|
|
|
test_commit_bulk --start=2 3 && # 7 objects
|
|
|
|
|
test_commit_bulk --start=6 9 && # 27 objects &&
|
|
|
|
|
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >before &&
|
|
|
|
|
|
|
|
|
|
git repack --geometric 2 -d &&
|
|
|
|
|
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >after &&
|
|
|
|
|
comm -12 before after >untouched &&
|
|
|
|
|
|
|
|
|
|
# Two packs in total; the largest pack from before running "git
|
|
|
|
|
# repack", and one new one.
|
|
|
|
|
test_line_count = 1 untouched &&
|
|
|
|
|
test_line_count = 2 after
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
|
|
|
|
test_expect_success '--geometric ignores kept packs' '
|
|
|
|
|
git init geometric &&
|
|
|
|
|
test_when_finished "rm -fr geometric" &&
|
|
|
|
|
(
|
|
|
|
|
cd geometric &&
|
|
|
|
|
|
|
|
|
|
test_commit kept && # 3 objects
|
|
|
|
|
test_commit pack && # 3 objects
|
|
|
|
|
|
|
|
|
|
KEPT=$(git pack-objects --revs $objdir/pack/pack <<-EOF
|
|
|
|
|
refs/tags/kept
|
|
|
|
|
EOF
|
|
|
|
|
) &&
|
|
|
|
|
PACK=$(git pack-objects --revs $objdir/pack/pack <<-EOF
|
|
|
|
|
refs/tags/pack
|
|
|
|
|
^refs/tags/kept
|
|
|
|
|
EOF
|
|
|
|
|
) &&
|
|
|
|
|
|
|
|
|
|
# neither pack contains more than twice the number of objects in
|
|
|
|
|
# the other, so they should be combined. but, marking one as
|
|
|
|
|
# .kept on disk will "freeze" it, so the pack structure should
|
|
|
|
|
# remain unchanged.
|
|
|
|
|
touch $objdir/pack/pack-$KEPT.keep &&
|
|
|
|
|
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >before &&
|
|
|
|
|
git repack --geometric 2 -d &&
|
|
|
|
|
find $objdir/pack -name "*.pack" | sort >after &&
|
|
|
|
|
|
|
|
|
|
# both packs should still exist
|
|
|
|
|
test_path_is_file $objdir/pack/pack-$KEPT.pack &&
|
|
|
|
|
test_path_is_file $objdir/pack/pack-$PACK.pack &&
|
|
|
|
|
|
|
|
|
|
# and no new packs should be created
|
|
|
|
|
test_cmp before after &&
|
|
|
|
|
|
|
|
|
|
# Passing --pack-kept-objects causes packs with a .keep file to
|
|
|
|
|
# be repacked, too.
|
|
|
|
|
git repack --geometric 2 -d --pack-kept-objects &&
|
|
|
|
|
|
|
|
|
|
find $objdir/pack -name "*.pack" >after &&
|
|
|
|
|
test_line_count = 1 after
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
2021-09-29 03:55:20 +02:00
|
|
|
test_expect_success '--geometric chooses largest MIDX preferred pack' '
|
|
|
|
|
git init geometric &&
|
|
|
|
|
test_when_finished "rm -fr geometric" &&
|
|
|
|
|
(
|
|
|
|
|
cd geometric &&
|
|
|
|
|
|
|
|
|
|
# These packs already form a geometric progression.
|
|
|
|
|
test_commit_bulk --start=1 1 && # 3 objects
|
|
|
|
|
test_commit_bulk --start=2 2 && # 6 objects
|
|
|
|
|
ls $objdir/pack/pack-*.idx >before &&
|
|
|
|
|
test_commit_bulk --start=4 4 && # 12 objects
|
|
|
|
|
ls $objdir/pack/pack-*.idx >after &&
|
|
|
|
|
|
|
|
|
|
git repack --geometric 2 -dbm &&
|
|
|
|
|
|
|
|
|
|
comm -3 before after | xargs -n 1 basename >expect &&
|
|
|
|
|
test-tool read-midx --preferred-pack $objdir >actual &&
|
|
|
|
|
|
|
|
|
|
test_cmp expect actual
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
builtin/repack.c: add '--geometric' option
Often it is useful to both:
- have relatively few packfiles in a repository, and
- avoid having so few packfiles in a repository that we repack its
entire contents regularly
This patch implements a '--geometric=<n>' option in 'git repack'. This
allows the caller to specify that they would like each pack to be at
least a factor times as large as the previous largest pack (by object
count).
Concretely, say that a repository has 'n' packfiles, labeled P1, P2,
..., up to Pn. Each packfile has an object count equal to 'objects(Pn)'.
With a geometric factor of 'r', it should be that:
objects(Pi) > r*objects(P(i-1))
for all i in [1, n], where the packs are sorted by
objects(P1) <= objects(P2) <= ... <= objects(Pn).
Since finding a true optimal repacking is NP-hard, we approximate it
along two directions:
1. We assume that there is a cutoff of packs _before starting the
repack_ where everything to the right of that cut-off already forms
a geometric progression (or no cutoff exists and everything must be
repacked).
2. We assume that everything smaller than the cutoff count must be
repacked. This forms our base assumption, but it can also cause
even the "heavy" packs to get repacked, for e.g., if we have 6
packs containing the following number of objects:
1, 1, 1, 2, 4, 32
then we would place the cutoff between '1, 1' and '1, 2, 4, 32',
rolling up the first two packs into a pack with 2 objects. That
breaks our progression and leaves us:
2, 1, 2, 4, 32
^
(where the '^' indicates the position of our split). To restore a
progression, we move the split forward (towards larger packs)
joining each pack into our new pack until a geometric progression
is restored. Here, that looks like:
2, 1, 2, 4, 32 ~> 3, 2, 4, 32 ~> 5, 4, 32 ~> ... ~> 9, 32
^ ^ ^ ^
This has the advantage of not repacking the heavy-side of packs too
often while also only creating one new pack at a time. Another wrinkle
is that we assume that loose, indexed, and reflog'd objects are
insignificant, and lump them into any new pack that we create. This can
lead to non-idempotent results.
Suggested-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Reviewed-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-02-23 03:25:27 +01:00
|
|
|
test_done
|