This commit integrates the fuzzy fingerprint heuristic into
guess_line_blames().
We actually make two passes. The first pass uses the fuzzy algorithm to
find a match within the current diff chunk. If that fails, the second
pass searches the entire parent file for the best match.
For an example of scanning the entire parent for a match, consider:
commit-a 30) #include <sys/header_a.h>
commit-b 31) #include <header_b.h>
commit-c 32) #include <header_c.h>
Then commit X alphabetizes them:
commit-X 30) #include <header_b.h>
commit-X 31) #include <header_c.h>
commit-X 32) #include <sys/header_a.h>
If we just check the parent's chunk (i.e. the first pass), we'd get:
commit-b 30) #include <header_b.h>
commit-c 31) #include <header_c.h>
commit-X 32) #include <sys/header_a.h>
That's because commit X actually consists of two chunks: one chunk is
removing sys/header_a.h, then some context, and the second chunk is
adding sys/header_a.h.
If we scan the entire parent file, we get:
commit-b 30) #include <header_b.h>
commit-c 31) #include <header_c.h>
commit-a 32) #include <sys/header_a.h>
Signed-off-by: Barret Rhoden <brho@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This algorithm will replace the heuristic used to identify lines from
ignored commits with one that finds likely candidate lines in the
parent's version of the file. The actual replacement occurs in an
upcoming commit.
The old heuristic simply assigned lines in the target to the same line
number (plus offset) in the parent. The new function uses a
fingerprinting algorithm to detect similarity between lines.
The new heuristic is designed to accurately match changes made
mechanically by formatting tools such as clang-format and clang-tidy.
These tools make changes such as breaking up lines to fit within a
character limit or changing identifiers to fit with a naming convention.
The heuristic is not intended to match more extensive refactoring
changes and may give misleading results in such cases.
In most cases formatting tools preserve line ordering, so the heuristic
is optimised for such cases. (Some types of changes do reorder lines
e.g. sorting keep the line content identical, the git blame -M option
can already be used to address this). The reason that it is advantageous
to rely on ordering is due to source code repeating the same character
sequences often e.g. declaring an identifier on one line and using that
identifier on several subsequent lines. This means that lines can look
very similar to each other which presents a problem when doing fuzzy
matching. Relying on ordering gives us extra clues to point towards the
true match.
The heuristic operates on a single diff chunk change at a time. It
creates a “fingerprint” for each line on each side of the change.
Fingerprints are described in detail in the comment for `struct
fingerprint`, but essentially are a multiset of the character pairs in a
line. The heuristic first identifies the line in the target entry whose
fingerprint is most clearly matched to a line fingerprint in the parent
entry. Where fingerprints match identically, the position of the lines
is used as a tie-break. The heuristic locks in the best match, and
subtracts the fingerprint of the line in the target entry from the
fingerprint of the line in the parent entry to prevent other lines being
matched on the same parts of that line. It then repeats the process
recursively on the section of the chunk before the match, and then the
section of the chunk after the match.
Here's an example of the difference the fingerprinting makes. Consider
a file with two commits:
commit-a 1) void func_1(void *x, void *y);
commit-b 2) void func_2(void *x, void *y);
After a commit 'X', we have:
commit-X 1) void func_1(void *x,
commit-X 2) void *y);
commit-X 3) void func_2(void *x,
commit-X 4) void *y);
When we blame-ignored with the old algorithm, we get:
commit-a 1) void func_1(void *x,
commit-b 2) void *y);
commit-X 3) void func_2(void *x,
commit-X 4) void *y);
Where commit-b is blamed for 2 instead of 3. With the fingerprint
algorithm, we get:
commit-a 1) void func_1(void *x,
commit-a 2) void *y);
commit-b 3) void func_2(void *x,
commit-b 4) void *y);
Note line 2 could be matched with either commit-a or commit-b as it is
equally similar to both lines, but is matched with commit-a because its
position as a fraction of the new line range is more similar to commit-a
as a fraction of the old line range. Line 4 is also equally similar to
both lines, but as it appears after line 3 which will be matched first
it cannot be matched with an earlier line.
For many more examples, see t/t8014-blame-ignore-fuzzy.sh which contains
example parent and target files and the line numbers in the parent that
must be matched.
Signed-off-by: Michael Platings <michael@platin.gs>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
fill_blame_origin() is a convenient place to store data that we will use
throughout the lifetime of a blame_origin. Some heuristics for
ignoring commits during a blame session can make use of this storage.
In particular, we will calculate a fingerprint for each line of a file
for blame_origins involved in an ignored commit.
In this commit, we only calculate the line_starts, reusing the existing
code from the scoreboard's line_starts. In an upcoming commit, we will
actually compute the fingerprints.
This feature will be used when we attempt to pass blame entries to
parents when we "ignore" a commit. Most uses of fill_blame_origin()
will not require this feature, hence the flag parameter. Multiple calls
to fill_blame_origin() are idempotent, and any of them can request the
creation of the fingerprints structure.
Suggested-by: Michael Platings <michael@platin.gs>
Signed-off-by: Barret Rhoden <brho@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When ignoring commits, the commit that is blamed might not be
responsible for the change, due to the inaccuracy of our heuristic.
Users might want to know when a particular line has a potentially
inaccurate blame.
Furthermore, guess_line_blames() may fail to find any parent commit for
a given line touched by an ignored commit. Those 'unblamable' lines
remain blamed on an ignored commit. Users might want to know if a line
is unblamable so that they do not spend time investigating a commit they
know is uninteresting.
This patch adds two config options to mark these two types of lines in
the output of blame.
The first option can identify ignored lines by specifying
blame.markIgnoredLines. When this option is set, each blame line that
was blamed on a commit other than the ignored commit is marked with a
'?'.
For example:
278b6158d6fdb (Barret Rhoden 2016-04-11 13:57:54 -0400 26)
appears as:
?278b6158d6fd (Barret Rhoden 2016-04-11 13:57:54 -0400 26)
where the '?' is placed before the commit, and the hash has one fewer
characters.
Sometimes we are unable to even guess at what ancestor commit touched a
line. These lines are 'unblamable.' The second option,
blame.markUnblamableLines, will mark the line with '*'.
For example, say we ignore e5e8d36d04cbe, yet we are unable to blame
this line on another commit:
e5e8d36d04cbe (Barret Rhoden 2016-04-11 13:57:54 -0400 26)
appears as:
*e5e8d36d04cb (Barret Rhoden 2016-04-11 13:57:54 -0400 26)
When these config options are used together, every line touched by an
ignored commit will be marked with either a '?' or a '*'.
Signed-off-by: Barret Rhoden <brho@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Commits that make formatting changes or function renames are often not
interesting when blaming a file. A user may deem such a commit as 'not
interesting' and want to ignore and its changes it when assigning blame.
For example, say a file has the following git history / rev-list:
---O---A---X---B---C---D---Y---E---F
Commits X and Y both touch a particular line, and the other commits do
not:
X: "Take a third parameter"
-MyFunc(1, 2);
+MyFunc(1, 2, 3);
Y: "Remove camelcase"
-MyFunc(1, 2, 3);
+my_func(1, 2, 3);
git-blame will blame Y for the change. I'd like to be able to ignore Y:
both the existence of the commit as well as any changes it made. This
differs from -S rev-list, which specifies the list of commits to
process for the blame. We would still process Y, but just don't let the
blame 'stick.'
This patch adds the ability for users to ignore a revision with
--ignore-rev=rev, which may be repeated. They can specify a set of
files of full object names of revs, e.g. SHA-1 hashes, one per line. A
single file may be specified with the blame.ignoreRevFile config option
or with --ignore-rev-file=file. Both the config option and the command
line option may be repeated multiple times. An empty file name "" will
clear the list of revs from previously processed files. Config options
are processed before command line options.
For a typical use case, projects will maintain the file containing
revisions for commits that perform mass reformatting, and their users
have the option to ignore all of the commits in that file.
Additionally, a user can use the --ignore-rev option for one-off
investigation. To go back to the example above, X was a substantive
change to the function, but not the change the user is interested in.
The user inspected X, but wanted to find the previous change to that
line - perhaps a commit that introduced that function call.
To make this work, we can't simply remove all ignored commits from the
rev-list. We need to diff the changes introduced by Y so that we can
ignore them. We let the blames get passed to Y, just like when
processing normally. When Y is the target, we make sure that Y does not
*keep* any blames. Any changes that Y is responsible for get passed to
its parent. Note we make one pass through all of the scapegoats
(parents) to attempt to pass blame normally; we don't know if we *need*
to ignore the commit until we've checked all of the parents.
The blame_entry will get passed up the tree until we find a commit that
has a diff chunk that affects those lines.
One issue is that the ignored commit *did* make some change, and there is
no general solution to finding the line in the parent commit that
corresponds to a given line in the ignored commit. That makes it hard
to attribute a particular line within an ignored commit's diff
correctly.
For example, the parent of an ignored commit has this, say at line 11:
commit-a 11) #include "a.h"
commit-b 12) #include "b.h"
Commit X, which we will ignore, swaps these lines:
commit-X 11) #include "b.h"
commit-X 12) #include "a.h"
We can pass that blame entry to the parent, but line 11 will be
attributed to commit A, even though "include b.h" came from commit B.
The blame mechanism will be looking at the parent's view of the file at
line number 11.
ignore_blame_entry() is set up to allow alternative algorithms for
guessing per-line blames. Any line that is not attributed to the parent
will continue to be blamed on the ignored commit as if that commit was
not ignored. Upcoming patches have the ability to detect these lines
and mark them in the blame output.
The existing algorithm is simple: blame each line on the corresponding
line in the parent's diff chunk. Any lines beyond that stay with the
target.
For example, the parent of an ignored commit has this, say at line 11:
commit-a 11) void new_func_1(void *x, void *y);
commit-b 12) void new_func_2(void *x, void *y);
commit-c 13) some_line_c
commit-d 14) some_line_d
After a commit 'X', we have:
commit-X 11) void new_func_1(void *x,
commit-X 12) void *y);
commit-X 13) void new_func_2(void *x,
commit-X 14) void *y);
commit-c 15) some_line_c
commit-d 16) some_line_d
Commit X nets two additionally lines: 13 and 14. The current
guess_line_blames() algorithm will not attribute these to the parent,
whose diff chunk is only two lines - not four.
When we ignore with the current algorithm, we get:
commit-a 11) void new_func_1(void *x,
commit-b 12) void *y);
commit-X 13) void new_func_2(void *x,
commit-X 14) void *y);
commit-c 15) some_line_c
commit-d 16) some_line_d
Note that line 12 was blamed on B, though B was the commit for
new_func_2(), not new_func_1(). Even when guess_line_blames() finds a
line in the parent, it may still be incorrect.
Signed-off-by: Barret Rhoden <brho@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The same code for splitting a blame_entry at a particular line was used
twice in blame_chunk(), and I'll use the helper again in an upcoming
patch.
Signed-off-by: Barret Rhoden <brho@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
init_skiplist() took a file consisting of SHA-1s and comments and added
the objects to an oidset. This functionality is useful for other
commands and will be moved to oidset.c in a future commit.
In preparation for that move, this commit renames it to
oidset_parse_file() to reflect its more generic usage and cleans up a
few of the names.
Signed-off-by: Barret Rhoden <brho@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git log -G<regex>" looked for a hunk in the "git log -p" patch
output that contained a string that matches the given pattern.
Optimize this code to ignore binary files, which by default will
not show any hunk that would match any pattern (unless textconv or
the --text option is in effect, that is).
* tb/log-G-binary:
log -G: ignore binary files
Lines that begin with a certain keyword that come over the wire, as
well as lines that consist only of one of these keywords, ought to
be painted in color for easier eyeballing, but the latter was
broken ever since the feature was introduced in 2.19, which has
been corrected.
* hn/highlight-sideband-keywords:
sideband: color lines with keyword only
"git checkout [<tree-ish>] path..." learned to report the number of
paths that have been checked out of the index or the tree-ish,
which gives it the same degree of noisy-ness as the case in which
the command checks out a branch.
* nd/checkout-noisy:
t0027: squelch checkout path run outside test_expect_* block
checkout: print something when checking out paths
The traversal over tree objects has learned to honor
":(attr:label)" pathspec match, which has been implemented only for
enumerating paths on the filesystem.
* nd/attr-pathspec-in-tree-walk:
tree-walk: support :(attr) matching
dir.c: move, rename and export match_attrs()
pathspec.h: clean up "extern" in function declarations
tree-walk.c: make tree_entry_interesting() take an index
tree.c: make read_tree*() take 'struct repository *'
"git rev-list --exclude-promisor-objects" had to take an object
that does not exist locally (and is lazily available) from the
command line without barfing, but the code dereferenced NULL.
* md/list-lazy-objects-fix:
list-objects.c: don't segfault for missing cmdline objects
The http-backend CGI process did not correctly clean up the child
processes it spawns to run upload-pack etc. when it dies itself,
which has been corrected.
* mk/http-backend-kill-children-before-exit:
http-backend: enable cleaning up forked upload/receive-pack on exit
A properly configured username/email is required under
user.useConfigOnly in order to create commits; now "git stash"
(even though it creates commit objects to represent stash entries)
command is exempt from the requirement.
* sd/stash-wo-user-name:
stash: tolerate missing user identity
Refspecs configured with "git -c var=val clone" did not propagate
to the resulting repository, which has been corrected.
* sg/clone-initial-fetch-configuration:
Documentation/clone: document ignored configuration variables
clone: respect additional configured fetch refspecs during initial fetch
clone: use a more appropriate variable name for the default refspec
"git checkout frotz" (without any double-dash) avoids ambiguity by
making sure 'frotz' cannot be interpreted as a revision and as a
path at the same time. This safety has been updated to check also
a unique remote-tracking branch 'frotz' in a remote, when dwimming
to create a local branch 'frotz' out of a remote-tracking branch
'frotz' from a remote.
* nd/checkout-dwim-fix:
checkout: disambiguate dwim tracking branches and local files
"git push $there $src:$dst" rejects when $dst is not a fully
qualified refname and not clear what the end user meant. The
codepath has been taught to give a clearer error message, and also
guess where the push should go by taking the type of the pushed
object into account (e.g. a tag object would want to go under
refs/tags/).
* ab/push-dwim-dst:
push doc: document the DWYM behavior pushing to unqualified <dst>
push: test that <src> doesn't DWYM if <dst> is unqualified
push: add an advice on unqualified <dst> push
push: move unqualified refname error into a function
push: improve the error shown on unqualified <dst> push
i18n: remote.c: mark error(...) messages for translation
remote.c: add braces in anticipation of a follow-up change
Small fixes and features for fast-export and fast-import, mostly on
the fast-export side.
* en/fast-export-import:
fast-export: add a --show-original-ids option to show original names
fast-import: remove unmaintained duplicate documentation
fast-export: add --reference-excluded-parents option
fast-export: ensure we export requested refs
fast-export: when using paths, avoid corrupt stream with non-existent mark
fast-export: move commit rewriting logic into a function for reuse
fast-export: avoid dying when filtering by paths and old tags exist
fast-export: use value from correct enum
git-fast-export.txt: clarify misleading documentation about rev-list args
git-fast-import.txt: fix documentation for --quiet option
fast-export: convert sha1 to oid
Code clean-up with optimization for the codepath that checks
(non-)existence of loose objects.
* jk/loose-object-cache:
odb_load_loose_cache: fix strbuf leak
fetch-pack: drop custom loose object cache
sha1-file: use loose object cache for quick existence check
object-store: provide helpers for loose_objects_cache
sha1-file: use an object_directory for the main object dir
handle alternates paths the same as the main object dir
sha1_file_name(): overwrite buffer instead of appending
rename "alternate_object_database" to "object_directory"
submodule--helper: prefer strip_suffix() to ends_with()
fsck: do not reuse child_process structs
The "http.version" configuration variable can be used with recent
enough cURL library to force the version of HTTP used to talk when
fetching and pushing.
* fc/http-version:
http: add support selecting http version
More _("i18n") markings.
* nd/i18n:
fsck: mark strings for translation
fsck: reduce word legos to help i18n
parse-options.c: mark more strings for translation
parse-options.c: turn some die() to BUG()
parse-options: replace opterror() with optname()
repack: mark more strings for translation
remote.c: mark messages for translation
remote.c: turn some error() or die() to BUG()
reflog: mark strings for translation
read-cache.c: add missing colon separators
read-cache.c: mark more strings for translation
read-cache.c: turn die("internal error") to BUG()
attr.c: mark more string for translation
archive.c: mark more strings for translation
alias.c: mark split_cmdline_strerror() strings for translation
git.c: mark more strings for translation
A regression for cygwin users was introduced with commit 05b458c,
"real_path: resolve symlinks by hand".
In the the commit message we read:
The current implementation of real_path uses chdir() in order to resolve
symlinks. Unfortunately this isn't thread-safe as chdir() affects a
process as a whole...
The old (and non-thread-save) OS calls chdir()/pwd() had been
replaced by a string operation.
The cygwin layer "knows" that "C:\cygwin" is an absolute path,
but the new string operation does not.
"git clone <url> C:\cygwin\home\USER\repo" fails like this:
fatal: Invalid path '/home/USER/repo/C:\cygwin\home\USER\repo'
The solution is to implement has_dos_drive_prefix(), skip_dos_drive_prefix()
is_dir_sep(), offset_1st_component() and convert_slashes() for cygwin
in the same way as it is done in 'Git for Windows' in compat/mingw.[ch]
Extract the needed code into compat/win32/path-utils.[ch] and use it
for cygwin as well.
Reported-by: Steven Penny <svnpenn@gmail.com>
Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Torsten Bögershausen <tboegi@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The -G<regex> option of log looks for the differences whose patch text
contains added/removed lines that match regex.
Currently -G looks also into patches of binary files (which
according to [1]) is binary as well.
This has a couple of issues:
- It makes the pickaxe search slow. In a proprietary repository of the
author with only ~5500 commits and a total .git size of ~300MB
searching takes ~13 seconds
$time git log -Gwave > /dev/null
real 0m13,241s
user 0m12,596s
sys 0m0,644s
whereas when we ignore binary files with this patch it takes ~4s
$time ~/devel/git/git log -Gwave > /dev/null
real 0m3,713s
user 0m3,608s
sys 0m0,105s
which is a speedup of more than fourfold.
- The internally used algorithm for generating patch text is based on
xdiff and its states in [1]
> The output format of the binary patch file is proprietary
> (and binary) and it is basically a collection of copy and insert
> commands [..]
which means that the current format could change once the internal
algorithm is changed as the format is not standardized. In addition
the git binary patch format used for preparing patches for git apply
is *different* from the xdiff format as can be seen by comparing
git log -p -a
commit 6e95bf4bafccf14650d02ab57f3affe669be10cf
Author: A U Thor <author@example.com>
Date: Thu Apr 7 15:14:13 2005 -0700
modify binary file
diff --git a/data.bin b/data.bin
index f414c84..edfeb6f 100644
--- a/data.bin
+++ b/data.bin
@@ -1,2 +1,4 @@
a
a^@a
+a
+a^@a
with git log --binary
commit 6e95bf4bafccf14650d02ab57f3affe669be10cf
Author: A U Thor <author@example.com>
Date: Thu Apr 7 15:14:13 2005 -0700
modify binary file
diff --git a/data.bin b/data.bin
index f414c84bd3aa25fa07836bb1fb73db784635e24b..edfeb6f501[..]
GIT binary patch
literal 12
QcmYe~N@Pgn0zx1O01)N^ZvX%Q
literal 6
NcmYe~N@Pgn0ssWg0XP5v
which seems unexpected.
To resolve these issues this patch makes -G<regex> ignore binary files
by default. Textconv filters are supported and also -a/--text for
getting the old and broken behaviour back.
The -S<block of text> option of log looks for differences that changes
the number of occurrences of the specified block of text (i.e.
addition/deletion) in a file. As we want to keep the current behaviour,
add a test to ensure it stays that way.
[1]: http://www.xmailserver.org/xdiff.html
Signed-off-by: Thomas Braun <thomas.braun@virtuell-zuhause.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* maint:
Git 2.20.1
.gitattributes: ensure t/oid-info/* has eol=lf
t9902: 'send-email' test case requires PERL
t4256: mark support files as LF-only
parse-options: fix SunCC compiler warning
help -a: handle aliases with long names gracefully
help.h: fix coding style
run-command: report exec failure
A recent update accidentally squelched an error message when the
run_command API failed to run a missing command, which has been
corrected.
* jc/run-command-report-exec-failure-fix:
run-command: report exec failure
"git help -a" did not work well when an overly long alias is
defined, which has been corrected.
* js/help-commands-verbose-by-default-fix:
help -a: handle aliases with long names gracefully
help.h: fix coding style
