undef/git-commit-vandalism
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
abd5a00268
git-commit-vandalism/Documentation
History
Jeff King abd5a00268 clear_delta_base_cache(): don't modify hashmap while iterating 
On Thu, Jan 19, 2017 at 03:03:46PM +0100, Ulrich Spörlein wrote:

> > I suspect the patch below may fix things for you. It works around it by
> > walking over the lru list (either is fine, as they both contain all
> > entries, and since we're clearing everything, we don't care about the
> > order).
>
> Confirmed. With the patch applied, I can import the whole 55G in one go
> without any crashes or aborts. Thanks much!

Thanks. Here it is rolled up with a commit message.

-- >8 --
Subject: clear_delta_base_cache(): don't modify hashmap while iterating

Removing entries while iterating causes fast-import to
access an already-freed `struct packed_git`, leading to
various confusing errors.

What happens is that clear_delta_base_cache() drops the
whole contents of the cache by iterating over the hashmap,
calling release_delta_base_cache() on each entry. That
function removes the item from the hashmap. The hashmap code
may then shrink the table, but the hashmap_iter struct
retains an offset from the old table.

As a result, the next call to hashmap_iter_next() may claim
that the iteration is done, even though some items haven't
been visited.

The only caller of clear_delta_base_cache() is fast-import,
which wants to clear the cache because it is discarding the
packed_git struct for its temporary pack. So by failing to
remove all of the entries, we still have references to the
freed packed_git.

To make things even more confusing, this doesn't seem to
trigger with the test suite, because it depends on
complexities like the size of the hash table, which entries
got cleared, whether we try to access them before they're
evicted from the cache, etc.

So I've been able to identify the problem with large
imports like freebsd's svn import, or a fast-export of
linux.git. But nothing that would be reasonable to run as
part of the normal test suite.

We can fix this easily by iterating over the lru linked list
instead of the hashmap. They both contain the same entries,
and we can use the "safe" variant of the list iterator,
which exists for exactly this case.

Let's also add a warning to the hashmap API documentation to
reduce the chances of getting bit by this again.

Reported-by: Ulrich Spörlein <uqs@freebsd.org>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-19 11:17:20 -08:00
..
howto Merge branch 'po/fix-doc-merge-base-illustration' into maint 2016-10-28 09:01:21 -07:00
RelNotes Almost ready for 2.11.1 2017-01-17 15:19:11 -08:00
technical clear_delta_base_cache(): don't modify hashmap while iterating 2017-01-19 11:17:20 -08:00
.gitattributes
.gitignore
asciidoc.conf
blame-options.txt blame: dwim "blame --reverse OLD" as "blame --reverse OLD.." 2016-06-14 12:13:07 -07:00
build-docdep.perl
cat-texi.perl
cmd-list.perl command-list: prepare machinery for upcoming "common groups" section 2015-05-21 13:03:37 -07:00
CodingGuidelines CodingGuidelines: formatting HEAD in documentation 2016-06-28 08:36:45 -07:00
config.txt Merge branch 'mm/gc-safety-doc' into maint 2017-01-17 15:19:11 -08:00
date-formats.txt Merge branch 'lr/doc-fix-cet' into maint 2017-01-17 15:19:08 -08:00
diff-config.txt Merge branch 'jc/ws-error-highlight' 2016-10-26 13:14:43 -07:00
diff-format.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
diff-generate-patch.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
diff-heuristic-options.txt blame: honor the diff heuristic options and config 2016-09-19 10:25:11 -07:00
diff-options.txt Merge branch 'nd/ita-empty-commit' 2016-10-27 14:58:50 -07:00
docbook-xsl.css
docbook.xsl
everyday.txto Documentation: fix linkgit references 2016-05-09 15:44:14 -07:00
fetch-options.txt Merge branch 'nd/shallow-deepen' 2016-10-10 14:03:50 -07:00
fix-texi.perl
fmt-merge-msg-config.txt Documentation: include 'merge.branchdesc' for merge and config as well 2015-05-28 12:38:46 -07:00
git-add.txt add: document the chmod option 2016-09-12 15:03:32 -07:00
git-am.txt Merge branch 'mm/doc-tt' 2016-07-13 11:24:14 -07:00
git-annotate.txt blame: honor the diff heuristic options and config 2016-09-19 10:25:11 -07:00
git-apply.txt git-apply.txt: mention the behavior inside a subdir 2016-03-24 10:16:52 -07:00
git-archimport.txt
git-archive.txt
git-bisect-lk2009.txt doc: more consistency in environment variables format 2016-06-08 12:04:37 -07:00
git-bisect.txt doc: typeset HEAD and variants as literal 2016-06-28 08:36:45 -07:00
git-blame.txt Merge branch 'jc/blame-reverse' 2016-10-10 14:03:51 -07:00
git-branch.txt doc: typeset HEAD and variants as literal 2016-06-28 08:36:45 -07:00
git-bundle.txt transport: drop support for git-over-rsync 2016-02-01 13:07:41 -08:00
git-cat-file.txt cat-file: support --textconv/--filters in batch mode 2016-09-11 14:48:15 -07:00
git-check-attr.txt usage: do not insist that standard input must come from a file 2015-10-16 15:27:52 -07:00
git-check-ignore.txt Documentation: fix linkgit references 2016-05-09 15:44:14 -07:00
git-check-mailmap.txt
git-check-ref-format.txt git-check-ref-format.txt: fixup documentation 2016-09-21 11:12:41 -07:00
git-checkout-index.txt
git-checkout.txt checkout.txt: document a common case that ignores ambiguation rules 2016-09-21 08:44:41 -07:00
git-cherry-pick.txt Merge branch 'mm/doc-tt' into maint 2016-07-28 11:25:54 -07:00
git-cherry.txt
git-citool.txt
git-clean.txt doc: typeset short command-line options as literal 2016-06-28 08:20:52 -07:00
git-clone.txt Merge branch 'nd/shallow-deepen' 2016-10-10 14:03:50 -07:00
git-column.txt doc: remote author/documentation sections from more pages 2014-01-27 08:34:34 -08:00
git-commit-tree.txt Merge branch 'mm/doc-tt' into maint 2016-07-28 11:25:54 -07:00
git-commit.txt Merge branch 'ak/commit-only-allow-empty' into maint 2017-01-17 15:11:03 -08:00
git-config.txt Merge branch 'dg/document-git-c-in-git-config-doc' into maint 2016-09-08 21:35:56 -07:00
git-count-objects.txt count-objects: report alternates via verbose mode 2016-10-10 13:52:37 -07:00
git-credential-cache--daemon.txt
git-credential-cache.txt credential-cache--daemon: disallow relative socket path 2016-02-23 12:56:27 -08:00
git-credential-store.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-credential.txt
git-cvsexportcommit.txt
git-cvsimport.txt Merge branch 'jk/doc-cvs-update' into maint 2016-10-03 13:22:25 -07:00
git-cvsserver.txt doc: typeset HEAD and variants as literal 2016-06-28 08:36:45 -07:00
git-daemon.txt doc: typeset HEAD and variants as literal 2016-06-28 08:36:45 -07:00
git-describe.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-diff-files.txt
git-diff-index.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-diff-tree.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-diff.txt
git-difftool.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-fast-export.txt doc: convert \--option to --option 2015-05-12 22:14:46 -07:00
git-fast-import.txt Merge branch 'mm/doc-tt' 2016-07-13 11:24:14 -07:00
git-fetch-pack.txt upload-pack: optionally allow fetching any sha1 2016-11-18 13:06:14 -08:00
git-fetch.txt Merge branch 'mm/push-social-engineering-attack-doc' into maint 2017-01-17 15:19:10 -08:00
git-filter-branch.txt doc: typeset '--' as literal 2016-06-28 08:36:45 -07:00
git-fmt-merge-msg.txt Documentation/fmt-merge-msg: fix markup in example 2016-10-28 05:51:51 -07:00
git-for-each-ref.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-format-patch.txt format-patch: add "--rfc" for the common case of [RFC PATCH] 2016-09-21 08:58:10 -07:00
git-fsck-objects.txt
git-fsck.txt fsck: optionally show more helpful info for broken links 2016-07-18 15:15:59 -07:00
git-gc.txt Merge branch 'mm/gc-safety-doc' into maint 2017-01-17 15:19:11 -08:00
git-get-tar-commit-id.txt usage: do not insist that standard input must come from a file 2015-10-16 15:27:52 -07:00
git-grep.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-gui.txt doc: typeset HEAD and variants as literal 2016-06-28 08:36:45 -07:00
git-hash-object.txt usage: do not insist that standard input must come from a file 2015-10-16 15:27:52 -07:00
git-help.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-http-backend.txt doc: more consistency in environment variables format 2016-06-08 12:04:37 -07:00
git-http-fetch.txt
git-http-push.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-imap-send.txt
git-index-pack.txt index-pack: add --max-input-size=<size> option 2016-08-24 12:31:05 -07:00
git-init-db.txt
git-init.txt Merge branch 'tr/doc-tt' into maint 2016-07-06 13:06:34 -07:00
git-instaweb.txt doc: change configuration variables format 2016-06-08 12:04:55 -07:00
git-interpret-trailers.txt doc: mention user-configured trailers 2016-11-21 12:49:57 -08:00
git-log.txt Merge branch 'mj/log-show-signature-conf' 2016-07-11 10:31:08 -07:00
git-ls-files.txt ls-files: add pathspec matching for submodules 2016-10-10 12:14:58 -07:00
git-ls-remote.txt ls-remote: add support for showing symrefs 2016-01-19 10:07:56 -08:00
git-ls-tree.txt doc: typeset HEAD and variants as literal 2016-06-28 08:36:45 -07:00
git-mailinfo.txt Merge branch 'va/mailinfo-doc-typofix' into maint 2016-05-26 13:17:14 -07:00
git-mailsplit.txt mailsplit: support unescaping mboxrd messages 2016-06-06 11:14:43 -07:00
git-merge-base.txt doc: fix merge-base ASCII art tab spacing 2016-10-21 09:46:48 -07:00
git-merge-file.txt merge-file: clamp exit code to maximum 127 2015-10-29 12:10:23 -07:00
git-merge-index.txt
git-merge-one-file.txt
git-merge-tree.txt
git-merge.txt pull: pass --allow-unrelated-histories to "git merge" 2016-04-21 11:58:51 -07:00
git-mergetool--lib.txt
git-mergetool.txt mergetool: honor -O<orderfile> 2016-10-11 10:04:31 -07:00
git-mktag.txt usage: do not insist that standard input must come from a file 2015-10-16 15:27:52 -07:00
git-mktree.txt doc: typeset short command-line options as literal 2016-06-28 08:20:52 -07:00
git-mv.txt doc: typeset short command-line options as literal 2016-06-28 08:20:52 -07:00
git-name-rev.txt
git-notes.txt doc: typeset short command-line options as literal 2016-06-28 08:20:52 -07:00
git-p4.txt Merge branch 'mm/doc-tt' into maint 2016-07-28 11:25:54 -07:00
git-pack-objects.txt document git-repack interaction of pack.threads and pack.windowMemory 2016-08-10 10:55:13 -07:00
git-pack-redundant.txt
git-pack-refs.txt
git-parse-remote.txt
git-patch-id.txt usage: do not insist that standard input must come from a file 2015-10-16 15:27:52 -07:00
git-prune-packed.txt
git-prune.txt worktree: new place for "git prune --worktrees" 2015-06-29 08:48:44 -07:00
git-pull.txt Merge branch 'mm/push-social-engineering-attack-doc' into maint 2017-01-17 15:19:10 -08:00
git-push.txt Merge branch 'mm/push-social-engineering-attack-doc' into maint 2017-01-17 15:19:10 -08:00
git-quiltimport.txt doc: more consistency in environment variables format 2016-06-08 12:04:37 -07:00
git-read-tree.txt
git-rebase.txt Merge branch 'mm/doc-tt' into maint 2016-07-28 11:25:54 -07:00
git-receive-pack.txt receive-pack: allow a maximum input size to be specified 2016-08-24 12:31:05 -07:00
git-reflog.txt git-reflog: add exists command 2015-07-21 14:08:14 -07:00
git-relink.txt
git-remote-ext.txt
git-remote-fd.txt Spelling fixes 2016-08-11 14:35:42 -07:00
git-remote-helpers.txto
git-remote-testgit.txt
git-remote.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-repack.txt Merge branch 'ms/document-pack-window-memory-is-per-thread' 2016-08-12 09:47:35 -07:00
git-replace.txt doc: change environment variables format 2016-06-08 12:04:37 -07:00
git-request-pull.txt doc: show usage of branch description 2015-09-14 12:50:33 -07:00
git-rerere.txt
git-reset.txt
git-rev-list.txt Merge branch 'jk/date-local' 2015-10-05 12:30:13 -07:00
git-rev-parse.txt rev-parse --parseopt: allow [*=?!] in argument hints 2015-07-15 10:30:54 -07:00
git-revert.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-rm.txt
git-send-email.txt Merge branch 'mm/doc-tt' into maint 2016-07-28 11:25:54 -07:00
git-send-pack.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-sh-i18n--envsubst.txt
git-sh-i18n.txt
git-sh-setup.txt doc: more consistency in environment variables format 2016-06-08 12:04:37 -07:00
git-shell.txt doc: typeset short command-line options as literal 2016-06-28 08:20:52 -07:00
git-shortlog.txt
git-show-branch.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-show-index.txt usage: do not insist that standard input must come from a file 2015-10-16 15:27:52 -07:00
git-show-ref.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-show.txt doc: convert \--option to --option 2015-05-12 22:14:46 -07:00
git-stage.txt
git-stash.txt stash: allow stashes to be referenced by index only 2016-10-26 09:58:10 -07:00
git-status.txt git-status.txt: describe --porcelain=v2 format 2016-08-11 11:15:56 -07:00
git-stripspace.txt usage: do not insist that standard input must come from a file 2015-10-16 15:27:52 -07:00
git-submodule.txt submodules doc: update documentation for "." used for submodule branches 2016-10-19 14:58:53 -07:00
git-svn.txt Merge branch 'mm/doc-tt' into maint 2016-07-28 11:25:54 -07:00
git-symbolic-ref.txt
git-tag.txt Merge branch 'yk/git-tag-remove-mention-of-old-layout-in-doc' into maint 2016-10-28 09:01:10 -07:00
git-tools.txt Documentation/git-tools: retire manually-maintained list 2015-07-28 13:21:59 -07:00
git-unpack-file.txt
git-unpack-objects.txt unpack-objects: add --max-input-size=<size> option 2016-08-24 12:31:05 -07:00
git-update-index.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-update-ref.txt update-ref and tag: add --create-reflog arg 2015-07-21 14:08:35 -07:00
git-update-server-info.txt
git-upload-archive.txt Documentation: match underline with the text 2015-10-22 10:16:12 -07:00
git-upload-pack.txt upload-pack.c: use parse-options API 2016-05-31 10:17:20 -07:00
git-var.txt
git-verify-commit.txt Merge branch 'dn/gpg-doc' into maint 2016-07-06 13:06:36 -07:00
git-verify-pack.txt git-verify-pack.txt: fix inconsistent spelling of "packfile" 2015-05-17 11:24:57 -07:00
git-verify-tag.txt verify-tag: add option to print raw gpg status information 2015-06-22 14:20:47 -07:00
git-web--browse.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
git-whatchanged.txt
git-worktree.txt Merge branch 'nd/worktree-lock' into maint 2016-11-29 13:28:02 -08:00
git-write-tree.txt
git.txt Almost ready for 2.11.1 2017-01-17 15:19:11 -08:00
gitattributes.txt docs: warn about possible '=' in clean/smudge filter process values 2016-12-06 11:29:52 -08:00
gitcli.txt
gitcore-tutorial.txt doc: omit needless "for" 2016-12-09 15:14:01 -08:00
gitcredentials.txt credential: let empty credential specs reset helper list 2016-02-26 10:58:14 -08:00
gitcvs-migration.txt docs/cvs-migration: mention cvsimport caveats 2016-09-22 11:23:45 -07:00
gitdiffcore.txt doc: typeset long command-line options as literal 2016-06-28 08:36:45 -07:00
giteveryday.txt Documentation/everyday: match undefline with the text 2015-10-22 10:14:44 -07:00
gitglossary.txt
githooks.txt push options: {pre,post}-receive hook learns about push options 2016-07-14 15:50:17 -07:00
gitignore.txt doc: change configuration variables format 2016-06-08 12:04:55 -07:00
gitk.txt Merge branch 'po/range-doc' into maint 2016-09-19 13:51:38 -07:00
gitmodules.txt Merge branch 'bw/submodule-branch-dot-doc' into maint 2016-10-28 09:01:19 -07:00
gitnamespaces.txt doc: mention transfer data leaks in more places 2016-11-14 11:23:07 -08:00
gitremote-helpers.txt Merge branch 'nd/shallow-deepen' 2016-10-10 14:03:50 -07:00
gitrepository-layout.txt doc: fix location of 'info/' with $GIT_COMMON_DIR 2016-11-11 09:37:33 -08:00
gitrevisions.txt doc: gitrevisions - clarify 'latter case' is revision walk 2016-08-13 19:36:44 -07:00
gittutorial-2.txt
gittutorial.txt transport: drop support for git-over-rsync 2016-02-01 13:07:41 -08:00
gitweb.conf.txt gitweb: use highlight's shebang detection 2016-09-25 16:39:11 -07:00
gitweb.txt doc: change environment variables format 2016-06-08 12:04:37 -07:00
gitworkflows.txt
glossary-content.txt Merge branch 'rn/glossary-typofix' into HEAD 2016-05-18 14:40:14 -07:00
howto-index.sh
i18n.txt Documentation/i18n.txt: clarify character encoding support 2015-07-01 14:55:53 -07:00
install-doc-quick.sh
install-webdoc.sh
line-range-format.txt Documentation: change -L:<regex> to -L:<funcname> 2015-04-20 11:05:50 -07:00
lint-gitlink.perl ci: validate "linkgit:" in documentation 2016-05-10 11:15:04 -07:00
mailmap.txt
Makefile Documentation/technical: describe signature formats 2016-06-17 11:39:05 -07:00
manpage-1.72.xsl
manpage-base-url.xsl.in
manpage-base.xsl
manpage-bold-literal.xsl
manpage-normal.xsl
manpage-quote-apos.xsl
manpage-suppress-sp.xsl
merge-config.txt doc: change environment variables format 2016-06-08 12:04:37 -07:00
merge-options.txt Merge branch 'kf/gpg-sig-verification-doc' 2016-05-17 14:38:39 -07:00
merge-strategies.txt merge-strategies.txt: fix typo 2016-02-22 10:42:52 -08:00
pretty-formats.txt Merge branch 'rs/pretty-format-color-doc-fix' into maint 2016-10-28 09:01:23 -07:00
pretty-options.txt Merge branch 'tr/doc-tt' into maint 2016-07-06 13:06:34 -07:00
pull-fetch-param.txt
rev-list-options.txt Merge branch 'pb/rev-list-reverse-with-count' into maint 2016-10-11 14:20:06 -07:00
revisions.txt revision: new rev^-n shorthand for rev^n..rev 2016-09-27 10:59:28 -07:00
sequencer.txt
SubmittingPatches SubmittingPatches: use gitk's "Copy commit summary" format 2016-08-26 15:58:10 -07:00
transfer-data-leaks.txt doc: mention transfer data leaks in more places 2016-11-14 11:23:07 -08:00
urls-remotes.txt Documentation: match underline with the text 2015-10-22 10:16:12 -07:00
urls.txt transport: drop support for git-over-rsync 2016-02-01 13:07:41 -08:00
user-manual.conf
user-manual.txt documentation: fix some typos 2016-03-03 13:43:36 -08:00