git-commit-vandalism/builtin
Derrick Stolee 4f6d26b167 list-objects: consume sparse tree walk
When creating a pack-file using 'git pack-objects --revs' we provide
a list of interesting and uninteresting commits. For example, a push
operation would make the local topic branch be interesting and the
known remote refs as uninteresting. We want to discover the set of
new objects to send to the server as a thin pack.

We walk these commits until we discover a frontier of commits such
that every commit walk starting at interesting commits ends in a root
commit or unintersting commit. We then need to discover which
non-commit objects are reachable from  uninteresting commits. This
commit walk is not changing during this series.

The mark_edges_uninteresting() method in list-objects.c iterates on
the commit list and does the following:

* If the commit is UNINTERSTING, then mark its root tree and every
  object it can reach as UNINTERESTING.

* If the commit is interesting, then mark the root tree of every
  UNINTERSTING parent (and all objects that tree can reach) as
  UNINTERSTING.

At the very end, we repeat the process on every commit directly
given to the revision walk from stdin. This helps ensure we properly
cover shallow commits that otherwise were not included in the
frontier.

The logic to recursively follow trees is in the
mark_tree_uninteresting() method in revision.c. The algorithm avoids
duplicate work by not recursing into trees that are already marked
UNINTERSTING.

Add a new 'sparse' option to the mark_edges_uninteresting() method
that performs this logic in a slightly different way. As we iterate
over the commits, we add all of the root trees to an oidset. Then,
call mark_trees_uninteresting_sparse() on that oidset. Note that we
include interesting trees in this process. The current implementation
of mark_trees_unintersting_sparse() will walk the same trees as
the old logic, but this will be replaced in a later change.

Add a '--sparse' flag in 'git pack-objects' to call this new logic.
Add a new test script t/t5322-pack-objects-sparse.sh that tests this
option. The tests currently demonstrate that the resulting object
list is the same as the old algorithm. This includes a case where
both algorithms pack an object that is not needed by a remote due to
limits on the explored set of trees. When the sparse algorithm is
changed in a later commit, we will add a test that demonstrates a
change of behavior in some cases.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-17 13:44:39 -08:00
..
add.c Merge branch 'nd/attr-pathspec-fix' into maint 2018-11-21 22:57:45 +09:00
am.c Merge branch 'jk/unused-parameter-fixes' 2018-11-18 18:23:53 +09:00
annotate.c annotate: use argv_array 2014-07-16 11:10:11 -07:00
apply.c apply.c: make init_apply_state() take a struct repository 2018-08-13 14:14:44 -07:00
archive.c archive: initialize archivers earlier 2018-10-26 10:17:59 +09:00
bisect--helper.c bisect: mention "view" as an alternative to "visualize" 2017-11-13 10:51:14 +09:00
blame.c assert NOARG/NONEG behavior of parse-options callbacks 2018-11-06 12:56:29 +09:00
branch.c Merge branch 'tq/branch-style-fix' into maint 2018-11-21 22:58:06 +09:00
bundle.c
cat-file.c Merge branch 'tb/print-size-t-with-uintmax-format' 2018-11-19 16:24:41 +09:00
check-attr.c Make git_check_attr() a void function 2018-09-12 15:15:34 -07:00
check-ignore.c check-ignore: fix mix of directories and other file types 2018-02-12 13:09:35 -08:00
check-mailmap.c
check-ref-format.c Merge branch 'jc/check-ref-format-oor' into maint 2017-11-15 12:04:57 +09:00
checkout-index.c assert NOARG/NONEG behavior of parse-options callbacks 2018-11-06 12:56:29 +09:00
checkout.c Merge branch 'nd/the-index' 2018-10-19 13:34:02 +09:00
clean.c assert NOARG/NONEG behavior of parse-options callbacks 2018-11-06 12:56:29 +09:00
clone.c Merge branch 'nd/clone-case-smashing-warning' 2018-09-17 13:53:47 -07:00
column.c column: fix off-by-one default width 2018-05-13 10:45:05 +09:00
commit-graph.c Merge branch 'ds/commit-graph-leakfix' 2018-10-19 13:34:07 +09:00
commit-tree.c commit: add repository argument to lookup_commit 2018-06-29 10:43:39 -07:00
commit.c Merge branch 'jk/unused-parameter-fixes' 2018-11-18 18:23:53 +09:00
config.c worktree: add per-worktree config files 2018-10-22 13:17:04 +09:00
count-objects.c treewide: use get_all_packs 2018-08-20 15:31:40 -07:00
credential.c
describe.c refresh_index: remove unnecessary calls to preload_index() 2018-11-06 12:49:59 +09:00
diff-files.c revision.c: remove implicit dependency on the_index 2018-09-21 09:51:19 -07:00
diff-index.c revision.c: remove implicit dependency on the_index 2018-09-21 09:51:19 -07:00
diff-tree.c revision.c: remove implicit dependency on the_index 2018-09-21 09:51:19 -07:00
diff.c Merge branch 'nd/the-index' 2018-10-19 13:34:02 +09:00
difftool.c Merge branch 'nd/the-index' 2018-10-19 13:34:02 +09:00
fast-export.c Upcast size_t variables to uintmax_t when printing 2018-11-12 16:43:52 +09:00
fetch-pack.c builtin/fetch-pack: remove constants with parse_oid_hex 2018-10-15 12:53:15 +09:00
fetch.c Merge branch 'jk/unused-parameter-fixes' 2018-11-18 18:23:53 +09:00
fmt-merge-msg.c Merge branch 'nd/the-index' 2018-10-19 13:34:02 +09:00
for-each-ref.c
fsck.c Merge branch 'nd/per-worktree-ref-iteration' 2018-11-13 22:37:26 +09:00
gc.c Merge branch 'jn/gc-auto' 2018-10-16 16:16:02 +09:00
get-tar-commit-id.c
grep.c Merge branch 'jk/unused-parameter-fixes' 2018-11-18 18:23:53 +09:00
hash-object.c sha1-file.c: remove implicit dependency on the_index 2018-09-21 09:48:11 -07:00
help.c Merge branch 'rv/alias-help' 2018-10-26 14:22:13 +09:00
index-pack.c Merge branch 'tb/print-size-t-with-uintmax-format' 2018-11-19 16:24:41 +09:00
init-db.c assert NOARG/NONEG behavior of parse-options callbacks 2018-11-06 12:56:29 +09:00
interpret-trailers.c assert NOARG/NONEG behavior of parse-options callbacks 2018-11-06 12:56:29 +09:00
log.c range-diff: always pass at least minimal diff options 2018-12-04 10:36:14 +09:00
ls-files.c assert NOARG/NONEG behavior of parse-options callbacks 2018-11-06 12:56:29 +09:00
ls-remote.c Merge branch 'jk/proto-v2-ref-prefix-fix' 2018-11-13 22:37:17 +09:00
ls-tree.c Upcast size_t variables to uintmax_t when printing 2018-11-12 16:43:52 +09:00
mailinfo.c
mailsplit.c
merge-base.c Merge branch 'pk/rebase-in-c-4-opts' 2018-11-02 11:04:55 +09:00
merge-file.c assert NOARG/NONEG behavior of parse-options callbacks 2018-11-06 12:56:29 +09:00
merge-index.c
merge-ours.c Merge branch 'bw/diff-opt-impl-to-bitfields' 2017-11-09 14:31:27 +09:00
merge-recursive.c builtin/merge-recursive: make hash independent 2018-07-16 14:27:39 -07:00
merge-tree.c Merge branch 'jk/xdiff-interface' 2018-11-13 22:37:27 +09:00
merge.c Merge branch 'jk/verify-sig-merge-into-void' 2018-11-18 18:23:54 +09:00
mktag.c object-store: move object access functions to object-store.h 2018-05-16 11:42:03 +09:00
mktree.c builtin/mktree: remove hard-coded constant 2018-10-15 12:53:15 +09:00
multi-pack-index.c multi-pack-index: add 'verify' verb 2018-09-17 13:49:38 -07:00
mv.c Merge branch 'ma/lockfile-cleanup' 2018-05-30 14:04:05 +09:00
name-rev.c tag: add repository argument to deref_tag 2018-06-29 10:43:39 -07:00
notes.c Merge branch 'cb/notes-freeing-always-null-fix' 2018-11-18 18:23:57 +09:00
pack-objects.c list-objects: consume sparse tree walk 2019-01-17 13:44:39 -08:00
pack-redundant.c treewide: use get_all_packs 2018-08-20 15:31:40 -07:00
pack-refs.c refs: add repository argument to get_main_ref_store 2018-04-12 11:38:56 +09:00
patch-id.c
prune-packed.c for_each_*_object: move declarations to object-store.h 2018-08-14 12:29:57 -07:00
prune.c Merge branch 'md/exclude-promisor-objects-fix' into maint 2018-11-21 22:57:52 +09:00
pull.c Merge branch 'jk/verify-sig-merge-into-void' 2018-11-18 18:23:54 +09:00
push.c push: change needlessly ambiguous example in error 2018-11-14 17:17:09 +09:00
range-diff.c Merge branch 'nd/the-index' 2018-10-19 13:34:02 +09:00
read-tree.c assert NOARG/NONEG behavior of parse-options callbacks 2018-11-06 12:56:29 +09:00
rebase--interactive.c rebase -i: move rebase--helper modes to rebase--interactive 2018-10-09 10:44:10 +09:00
rebase.c Merge branch 'rt/rebase-in-c-message-fix' 2018-12-01 21:41:44 +09:00
receive-pack.c Merge branch 'jk/check-everything-connected-is-long-gone' into maint 2018-11-21 22:58:00 +09:00
reflog.c Merge branch 'nd/per-worktree-ref-iteration' 2018-11-13 22:37:26 +09:00
remote-ext.c
remote-fd.c
remote.c Merge branch 'ms/remote-error-message-update' 2018-09-24 10:30:52 -07:00
repack.c Merge branch 'ds/test-multi-pack-index' 2018-11-13 22:37:19 +09:00
replace.c Merge branch 'ab/replace-graft-with-replace-advice' 2018-12-01 21:41:42 +09:00
rerere.c Merge branch 'jk/xdiff-interface' 2018-11-13 22:37:27 +09:00
reset.c reset: warn when refresh_index() takes more than 2 seconds 2018-10-24 11:57:08 +09:00
rev-list.c list-objects: consume sparse tree walk 2019-01-17 13:44:39 -08:00
rev-parse.c Merge branch 'ag/rev-parse-all-exclude-fix' 2018-11-13 22:37:23 +09:00
revert.c revision.c: remove implicit dependency on the_index 2018-09-21 09:51:19 -07:00
rm.c convert "oidcmp() != 0" to "!oideq()" 2018-08-29 11:32:49 -07:00
send-pack.c Merge branch 'rs/opt-updates' 2018-08-27 14:33:43 -07:00
serve.c serve: introduce git-serve 2018-03-15 12:01:08 -07:00
shortlog.c revision.c: remove implicit dependency on the_index 2018-09-21 09:51:19 -07:00
show-branch.c assert NOARG/NONEG behavior of parse-options callbacks 2018-11-06 12:56:29 +09:00
show-index.c make show-index a builtin 2018-05-29 00:28:22 +09:00
show-ref.c assert NOARG/NONEG behavior of parse-options callbacks 2018-11-06 12:56:29 +09:00
stripspace.c
submodule--helper.c Merge branch 'ao/submodule-wo-gitmodules-checked-out' 2018-11-13 22:37:22 +09:00
symbolic-ref.c refs: rename constant REF_NODEREF to REF_NO_DEREF 2017-11-06 10:31:08 +09:00
tag.c assert NOARG/NONEG behavior of parse-options callbacks 2018-11-06 12:56:29 +09:00
unpack-file.c object-store: move object access functions to object-store.h 2018-05-16 11:42:03 +09:00
unpack-objects.c convert "hashcmp() != 0" to "!hasheq()" 2018-08-29 11:32:49 -07:00
update-index.c Merge branch 'jk/unused-parameter-fixes' 2018-11-18 18:23:53 +09:00
update-ref.c update-ref: allow --no-deref with --stdin 2018-09-12 15:17:17 -07:00
update-server-info.c parse-options: let OPT__FORCE take optional flags argument 2018-02-09 10:24:50 -08:00
upload-archive.c archive: initialize archivers earlier 2018-10-26 10:17:59 +09:00
upload-pack.c check_replace_refs: rename to read_replace_refs 2018-07-18 15:45:14 -07:00
var.c
verify-commit.c commit: add repository argument to lookup_commit 2018-06-29 10:43:39 -07:00
verify-pack.c
verify-tag.c ref-filter: use "struct object_id" consistently 2018-04-09 06:14:45 +09:00
worktree.c worktree: rename is_worktree_locked to worktree_lock_reason 2018-10-31 11:41:47 +09:00
write-tree.c Merge branch 'rs/parse-opt-lithelp' 2018-08-17 13:09:56 -07:00