The filter-process interface learned to allow a process with long
latency give a "delayed" response.
* ls/filter-process-delayed:
convert: add "status=delayed" to filter process protocol
convert: refactor capabilities negotiation
convert: move multiple file filter error handling to separate function
convert: put the flags field before the flag itself for consistent style
t0021: write "OUT <size>" only on success
t0021: make debug log file name configurable
t0021: keep filter log files on comparison
Some `clean` / `smudge` filters may require a significant amount of
time to process a single blob (e.g. the Git LFS smudge filter might
perform network requests). During this process the Git checkout
operation is blocked and Git needs to wait until the filter is done to
continue with the checkout.
Teach the filter process protocol, introduced in edcc8581 ("convert: add
filter.<driver>.process option", 2016-10-16), to accept the status
"delayed" as response to a filter request. Upon this response Git
continues with the checkout operation. After the checkout operation Git
calls "finish_delayed_checkout" which queries the filter for remaining
blobs. If the filter is still working on the completion, then the filter
is expected to block. If the filter has completed all remaining blobs
then an empty response is expected.
Git has a multiple code paths that checkout a blob. Support delayed
checkouts only in `clone` (in unpack-trees.c) and `checkout` operations
for now. The optimization is most effective in these code paths as all
files of the tree are processed.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When illustrating how to normalize the line endings, the
documentation in gitattributes tells the user to `rm .git/index`.
This is incorrect for two reasons:
- Users shouldn't be instructed to mess around with the internal
implementation of Git using raw file system tools like `rm`.
- Within a submodule or an additional working tree `.git` is just a
file containing a `gitdir: <path>` pointer into the real `.git`
directory. Therefore `rm .git/index` does not work.
The purpose of the `rm .git/index` instruction is to remove all entries
from the index without touching the working tree. The way to do this
with Git is to use `read-tree --empty`.
Signed-off-by: Andreas Heiduk <asheiduk@gmail.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Helped-by: Torsten Bögershausen <tboegi@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The instructions how to normalize the line endings should have been updated
as part of commit 6523728499 'convert: unify the "auto" handling of CRLF',
(but that part never made it into the commit).
Update the documentation in Documentation/gitattributes.txt and add
a test case in t0025.
Reported by Kristian Adrup
https://github.com/git-for-windows/git/issues/954
Signed-off-by: Torsten Bögershausen <tboegi@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Brandon Williams <bmwill@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Full pattern must be quoted. So 'pat"t"ern attr' will give exactly
'pat"t"ern', not 'pattern'. Also clarify that leading whitespaces are
not part of the pattern and document comment syntax.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Brandon Williams <bmwill@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A pathname value in a clean/smudge filter process "key=value" pair can
contain the '=' character (introduced in edcc858). Make the user aware
of this issue in the docs, add a corresponding test case, and fix the
issue in filter process value parser of the example implementation in
contrib.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add a simple pass-thru filter as example implementation for the Git
filter protocol version 2. See Documentation/gitattributes.txt, section
"Filter Protocol" for more info.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Git's clean/smudge mechanism invokes an external filter process for
every single blob that is affected by a filter. If Git filters a lot of
blobs then the startup time of the external filter processes can become
a significant part of the overall Git execution time.
In a preliminary performance test this developer used a clean/smudge
filter written in golang to filter 12,000 files. This process took 364s
with the existing filter mechanism and 5s with the new mechanism. See
details here: https://github.com/github/git-lfs/pull/1382
This patch adds the `filter.<driver>.process` string option which, if
used, keeps the external filter process running and processes all blobs
with the packet format (pkt-line) based protocol over standard input and
standard output. The full protocol is explained in detail in
`Documentation/gitattributes.txt`.
A few key decisions:
* The long running filter process is referred to as filter protocol
version 2 because the existing single shot filter invocation is
considered version 1.
* Git sends a welcome message and expects a response right after the
external filter process has started. This ensures that Git will not
hang if a version 1 filter is incorrectly used with the
filter.<driver>.process option for version 2 filters. In addition,
Git can detect this kind of error and warn the user.
* The status of a filter operation (e.g. "success" or "error) is set
before the actual response and (if necessary!) re-set after the
response. The advantage of this two step status response is that if
the filter detects an error early, then the filter can communicate
this and Git does not even need to create structures to read the
response.
* All status responses are pkt-line lists terminated with a flush
packet. This allows us to send other status fields with the same
protocol in the future.
Helped-by: Martin-Louis Bright <mlbright@gmail.com>
Reviewed-by: Jakub Narebski <jnareb@gmail.com>
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Update the documentation about text=auto:
text=auto now follows the core.autocrlf handling when files are not
normalized in the repository.
For a cross platform project recommend the usage of attributes for
line-ending conversions.
Signed-off-by: Torsten Bögershausen <tboegi@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
It's natural to expect %f to be an actual file on disk; help avoid that
mistake.
Signed-off-by: Joey Hess <joeyh@joeyh.name>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git merge" with renormalization did not work well with
merge-recursive, due to "safer crlf" conversion kicking in when it
shouldn't.
* jc/renormalize-merge-kill-safer-crlf:
merge: avoid "safer crlf" during recording of merge results
convert: unify the "auto" handling of CRLF
Before this change,
$ echo "* text=auto" >.gitattributes
$ echo "* eol=crlf" >>.gitattributes
would have the same effect as
$ echo "* text" >.gitattributes
$ git config core.eol crlf
Since the 'eol' attribute had higher priority than 'text=auto', this may
corrupt binary files and is not what most users expect to happen.
Make the 'eol' attribute to obey 'text=auto' and now
$ echo "* text=auto" >.gitattributes
$ echo "* eol=crlf" >>.gitattributes
behaves the same as
$ echo "* text=auto" >.gitattributes
$ git config core.eol crlf
In other words,
$ echo "* text=auto eol=crlf" >.gitattributes
has the same effect as
$ git config core.autocrlf true
and
$ echo "* text=auto eol=lf" >.gitattributes
has the same effect as
$ git config core.autocrlf input
Signed-off-by: Torsten Bögershausen <tboegi@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
CSS is widely used, motivating it being included as a built-in pattern.
It must be noted that the word_regex for CSS (i.e. the regex defining
what is a word in the language) does not consider '.' and '#' characters
(in CSS selectors) to be part of the word. This behavior is documented
by the test t/t4018/css-rule.
The logic behind this behavior is the following: identifiers in CSS
selectors are identifiers in a HTML/XML document. Therefore, the '.'/'#'
character are not part of the identifier, but an indicator of the nature
of the identifier in HTML/XML (class or id). Diffing ".class1" and
".class2" must show that the class name is changed, but we still are
selecting a class.
Logic behind the "pattern" regex is:
1. reject lines ending with a colon/semicolon (properties)
2. if a line begins with a name in column 1, pick the whole line
Credits to Johannes Sixt (j6t@kdbg.org) for the pattern regex and most
of the tests.
Signed-off-by: William Duclot <william.duclot@ensimag.grenoble-inp.fr>
Signed-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
Reviewed-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add support for Fountain, a plain text screenplay format. Git
facilitates not just programming specifically, but creative writing
in general, so it makes sense to also support other plain text
documents besides source code.
In the structure of a screenplay specifically, scenes are roughly
analogous to functions, in the sense that it makes your job easier
if you can see which ones were changed in a given range of patches.
More information about the Fountain format can be found on its
official website, at http://fountain.io .
Signed-off-by: Zoë Blade <zoe@bytenoise.co.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Traditionally, external low-level 3-way merge drivers are expected
to produce their results based solely on the contents of the three
variants given in temporary files named by %O, %A and %B on their
command line. Additionally allow them to look at the final path
(given by %P).
* jc/ll-merge-expose-path:
ll-merge: pass the original path to external drivers
The interface to custom low-level merge driver was modeled to be
capable of driving programs like "merge" (from the RCS suite) that
can produce result solely by looking at three files that hold
contents of common ancestor, ours and theirs. The information we
feed to the external drivers via the command line placeholders %O,
%A, and %B were designed to be purely about contents by giving
names of the temporary files that hold these variants without
exposing the original pathname. No matter where the result goes,
merging the same three variants should produce the same result,
contents is the king, that is the Git way.
The external driver interface, however, is meant to help people to
step outside the Git worldview, and sometimes people want to know
the final path that the resulting merged contents would be stored
in. Expose this to the external drivers via a new placeholder %P.
Requested-by: Andreas Gondek
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This should improve readability. Compare "thislongname" and
"thisLongName". The following keys are left in unchanged. We can
decide what to do with them later.
- am.keepcr
- core.autocrlf .safecrlf .trustctime
- diff.dirstat .noprefix
- gitcvs.usecrlfattr
- gui.blamehistoryctx .trustmtime
- pull.twohead
- receive.autogc
- sendemail.signedoffbycc .smtpsslcertpath .suppresscc
Helped-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In addition to fixing trivial and obvious typos, be careful about
the following points:
- Spell ASCII, URL and CRC in ALL CAPS;
- Spell Linux as Capitalized;
- Do not omit periods in "i.e." and "e.g.".
Signed-off-by: Thomas Ackermann <th.acker@arcor.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Too large files may lead to failure to allocate memory. If it happens
here, it could impact quite a few commands that involve
diff. Moreover, too large files are inefficient to compare anyway (and
most likely non-text), so mark them binary and skip looking at their
content.
Noticed-by: Dale R. Worley <worley@alum.mit.edu>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The old text made it sound like macros are only allowed in the
.gitattributes file at the top-level of the working tree. Make it
clear that they are also allowed in $GIT_DIR/info/attributes and in
the global and system-wide gitattributes files.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Start laying the foundation to build the "wildmatch" after we can
agree on its desired semantics.
* nd/attr-match-optim-more:
attr: more matching optimizations from .gitignore
gitignore: make pattern parsing code a separate function
exclude: split pathname matching code into a separate function
exclude: fix a bug in prefix compare optimization
exclude: split basename matching code into a separate function
exclude: stricten a length check in EXC_FLAG_ENDSWITH case
.gitattributes and .gitignore share the same pattern syntax but has
separate matching implementation. Over the years, ignore's
implementation accumulates more optimizations while attr's stays the
same.
This patch reuses the core matching functions that are also used by
excluded_from_list. excluded_from_list and path_matches can't be
merged due to differences in exclude and attr, for example:
* "!pattern" syntax is forbidden in .gitattributes. As an attribute
can be unset (i.e. set to a special value "false") or made back to
unspecified (i.e. not even set to "false"), "!pattern attr" is unclear
which one it means.
* we support attaching attributes to directories, but git-core
internally does not currently make use of attributes on
directories.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This is the documentation part of
1a9d7e9 (attr.c: read .gitattributes from index as well. - 2007-08-14)
06f33c1 (Read attributes from the index that is being checked out - 2009-03-13)
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add Ada xfuncname and wordRegex patterns to the list of builtin
patterns.
Signed-off-by: Adrian Johnson <ajohnson@redneon.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git merge -Xtheirs" did not help content-level merge of binary
files; it should just take their version. Also "*.jpg binary" in
the attributes did not imply they should use the binary ll-merge
driver.
* jc/ll-merge-binary-ours:
ll-merge: warn about inability to merge binary files only when we can't
attr: "binary" attribute should choose built-in "binary" merge driver
merge: teach -Xours/-Xtheirs to binary ll-merge driver
The built-in "binary" attribute macro expands to "-diff -text", so
that textual diff is not produced, and the contents will not go
through any CR/LF conversion ever. During a merge, it should also
choose the "binary" low-level merge driver, but it didn't.
Make it expand to "-diff -merge -text".
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This gives the default value for the core.attributesfile variable
following the exact same logic of the previous change for the
core.excludesfile setting.
Signed-off-by: Huynh Khoi Nguyen Nguyen <Huynh-Khoi-Nguyen.Nguyen@ensimag.imag.fr>
Signed-off-by: Valentin Duperray <Valentin.Duperray@ensimag.imag.fr>
Signed-off-by: Franck Jonas <Franck.Jonas@ensimag.imag.fr>
Signed-off-by: Lucien Kong <Lucien.Kong@ensimag.imag.fr>
Signed-off-by: Thomas Nguy <Thomas.Nguy@ensimag.imag.fr>
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
By default, a missing filter driver or a failure from the filter driver is
not an error, but merely makes the filter operation a no-op pass through.
This is useful to massage the content into a shape that is more convenient
for the platform, filesystem, and the user to use, and the content filter
mechanism is not used to turn something unusable into usable.
However, we could also use of the content filtering mechanism and store
the content that cannot be directly used in the repository (e.g. a UUID
that refers to the true content stored outside git, or an encrypted
content) and turn it into a usable form upon checkout (e.g. download the
external content, or decrypt the encrypted content). For such a use case,
the content cannot be used when filter driver fails, and we need a way to
tell Git to abort the whole operation for such a failing or missing filter
driver.
Add a new "filter.<driver>.required" configuration variable to mark the
second use case. When it is set, git will abort the operation when the
filter driver does not exist or exits with a non-zero status code.
Signed-off-by: Jehan Bing <jehan@orb.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
MATLAB is often used in industry and academia for scientific
computations motivating it being included as a built-in pattern.
Signed-off-by: Gustaf Hendeby <hendeby@isy.liu.se>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* mh/check-attr-relative: (29 commits)
test-path-utils: Add subcommand "prefix_path"
test-path-utils: Add subcommand "absolute_path"
git-check-attr: Normalize paths
git-check-attr: Demonstrate problems with relative paths
git-check-attr: Demonstrate problems with unnormalized paths
git-check-attr: test that no output is written to stderr
Rename git_checkattr() to git_check_attr()
git-check-attr: Fix command-line handling to match docs
git-check-attr: Drive two tests using the same raw data
git-check-attr: Add an --all option to show all attributes
git-check-attr: Error out if no pathnames are specified
git-check-attr: Process command-line args more systematically
git-check-attr: Handle each error separately
git-check-attr: Extract a function error_with_usage()
git-check-attr: Introduce a new variable
git-check-attr: Extract a function output_attr()
Allow querying all attributes on a file
Remove redundant check
Remove redundant call to bootstrap_attr_stack()
Extract a function collect_all_attrs()
...
The new wording makes it clearer that such a beast is an attribute in
addition to being a macro (as opposed to being only a macro that is
used for attributes).
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In particular, make it clear that attribute macros are themselves
recorded as attributes in addition to setting other attributes.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We already talk about how to use each one and how they work,
but it is a reasonable question to wonder why one might use
one over the other.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This was added long ago as part of the userdiff refactoring
for textconv, as internally it made the code simpler and
cleaner. However, there was never a concrete use case for
actually using the config variable.
Now that Matthieu Moy has provided such a use case, it's
easy to explain it using his example.
Signed-off-by: Jeff King <peff@peff.net>
Acked-by: Matthieu Moy <Matthieu.Moy@grenoble-inp.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The default function name discovery already works quite well for Perl
code... with the exception of here-documents (or rather their ending).
sub foo {
print <<END
here-document
END
return 1;
}
The default funcname pattern treats the unindented END line as a
function declaration and puts it in the @@ line of diff and "grep
--show-function" output.
With a little knowledge of perl syntax, we can do better. You can
try it out by adding "*.perl diff=perl" to the gitattributes file.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Filtering to support keyword expansion may need the name of
the file being filtered. In particular, to support p4 keywords
like
$File: //depot/product/dir/script.sh $
the smudge filter needs to know the name of the file it is
smudging.
Allow "%f" in the custom filter command line specified in the
configuration. This will be substituted by the filename
inside a single-quote pair to be passed to the shell.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A new whitespace "rule" is added that sets the tab width to use for
whitespace checks and fix-ups and replaces the hard-coded constant 8.
Since the setting is part of the rules, it can be set per file using
.gitattributes.
The new configuration is backwards compatible because older git versions
simply ignore unknown whitespace rules.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
