When creating a shallow clone of a repository with submodules, the depth
argument does not influence the submodules, i.e. the submodules are done
as non-shallow clones. It is unclear what the best default is for the
depth of submodules of a shallow clone, so we need to have the possibility
to do all kinds of combinations:
* shallow super project with shallow submodules
e.g. build bots starting always from scratch. They want to transmit
the least amount of network data as well as using the least amount
of space on their hard drive.
* shallow super project with unshallow submodules
e.g. The superproject is just there to track a collection of repositories
and it is not important to have the relationship between the repositories
intact. However the history of the individual submodules matter.
* unshallow super project with shallow submodules
e.g. The superproject is the actual project and the submodule is a
library which is rarely touched.
The new switch to select submodules to be shallow or unshallow supports
all of these three cases.
It is easy to transition from the first to the second case by just
unshallowing the submodules (`git submodule foreach git fetch
--unshallow`), but it is not possible to transition from the second to the
first case (as we would have already transmitted the non shallow over
the network). That is why we want to make the first case the default in
case of a shallow super project. This leads to the inconvenience in the
second case with the shallow super project and unshallow submodules,
as you need to pass `--no-shallow-submodules`.
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Just pass it along to "git submodule update", which may pick reasonable
defaults if you don't specify an explicit number.
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Expose possible parallelism either via the "--jobs" CLI parameter or
the "submodule.fetchJobs" setting.
By having the variable initialized to -1, we make sure 0 can be passed
into the parallel processing machine, which will then pick as many parallel
workers as there are CPUs.
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Reroute the error message for specified but initialized submodules
to stderr instead of stdout.
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This allows to configure fetching and updating in parallel
without having the command line option.
This moved the responsibility to determine how many parallel processes
to start from builtin/fetch to submodule.c as we need a way to communicate
"The user did not specify the number of parallel processes in the command
line options" in the builtin fetch. The submodule code takes care of
the precedence (CLI > config > default).
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This allows to run external commands in parallel with ordered output
on stderr.
If we run external commands in parallel we cannot pipe the output directly
to the our stdout/err as it would mix up. So each process's output will
flow through a pipe, which we buffer. One subprocess can be directly
piped to out stdout/err for a low latency feedback to the user.
Example:
Let's assume we have 5 submodules A,B,C,D,E and each fetch takes a
different amount of time as the different submodules vary in size, then
the output of fetches in sequential order might look like this:
time -->
output: |---A---| |-B-| |-------C-------| |-D-| |-E-|
When we schedule these submodules into maximal two parallel processes,
a schedule and sample output over time may look like this:
process 1: |---A---| |-D-| |-E-|
process 2: |-B-| |-------C-------|
output: |---A---|B|---C-------|DE
So A will be perceived as it would run normally in the single child
version. As B has finished by the time A is done, we can dump its whole
progress buffer on stderr, such that it looks like it finished in no
time. Once that is done, C is determined to be the visible child and
its progress will be reported in real time.
So this way of output is really good for human consumption, as it only
changes the timing, not the actual output.
For machine consumption the output needs to be prepared in the tasks,
by either having a prefix per line or per block to indicate whose tasks
output is displayed, because the output order may not follow the
original sequential ordering:
|----A----| |--B--| |-C-|
will be scheduled to be all parallel:
process 1: |----A----|
process 2: |--B--|
process 3: |-C-|
output: |----A----|CB
This happens because C finished before B did, so it will be queued for
output before B.
To detect when a child has finished executing, we check interleaved
with other actions (such as checking the liveliness of children or
starting new processes) whether the stderr pipe still exists. Once a
child closed its stderr stream, we assume it is terminating very soon,
and use `finish_command()` from the single external process execution
interface to collect the exit status.
By maintaining the strong assumption of stderr being open until the
very end of a child process, we can avoid other hassle such as an
implementation using `waitpid(-1)`, which is not implemented in Windows.
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The "Pushing submodule <foo>" progress output correctly goes to
stderr, but "Fetching submodule <foo>" is going to stdout by
mistake. Fix it to write to stderr.
Noticed while trying to implement a parallel submodule fetch. When
this particular output line went to a different file descriptor, it
was buffered separately, resulting in wrongly interleaved output if
we copied it to the terminal naively.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Fix some racy client/server tests by treating SIGPIPE the same as a
normal non-zero exit.
* ls/test-must-fail-sigpipe:
add "ok=sigpipe" to test_must_fail and use it to fix flaky tests
implement test_might_fail using a refactored test_must_fail
As CodingGuidelines says, some versions of bash errors out when
$variable substitution is used as the target for redirection without
being quoted (even though POSIX may not require such a quote).
Signed-off-by: Charles Bailey <cbailey32@bloomberg.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Produce correct "dirty" marker for shell prompts, even when we
are on an orphan or an unborn branch.
* sg/bash-prompt-dirty-orphan:
bash prompt: indicate dirty index even on orphan branches
bash prompt: remove a redundant 'git diff' option
bash prompt: test dirty index and worktree while on an orphan branch
Teach send-email to dump mail aliases, so that we can do tab completion
on the command line.
* jk/send-email-complete-aliases:
completion: add support for completing email aliases
sendemail: teach git-send-email to dump alias names
Speed up filter-branch for cases where we only care about rewriting
commits, not tree data.
* jk/filter-branch-no-index:
filter-branch: skip index read/write when possible
Fsck did not correctly detect a NUL-truncated header in a tag.
* rs/fsck-nul-header:
fsck: treat a NUL in a tag header as an error
t1450: add tests for NUL in headers of commits and tags
Work around some test flakiness with p4d.
* ls/p4-test-timeouts:
git-p4: add trap to kill p4d on test exit
git-p4: add p4d timeout in tests
git-p4: retry kill/cleanup operations in tests with timeout
Clean up style in an ancient test.
* js/test-modernize-t9300:
modernize t9300: move test preparations into test_expect_success
modernize t9300: mark here-doc words to ignore tab indentation
modernize t9300: use test_when_finished for clean-up
modernize t9300: wrap lines after &&
modernize t9300: use test_must_be_empty
modernize t9300: use test_must_fail
modernize t9300: single-quote placement and indentation
* maint:
http: treat config options sslCAPath and sslCAInfo as paths
Documentation/diff: give --word-diff-regex=. example
filter-branch: deal with object name vs. pathname ambiguity in tree-filter
check-ignore: correct documentation about output
git-p4: clean up after p4 submit failure
git-p4: work with a detached head
git-p4: add option to system() to return subshell status
git-p4: add failing test for submit from detached head
remote-http(s): support SOCKS proxies
t5813: avoid creating urls that break on cygwin
Escape Git's exec path in contrib/rerere-train.sh script
allow hooks to ignore their standard input stream
rebase-i-exec: Allow space in SHELL_PATH
Documentation: make environment variable formatting more consistent
Just like the working tree is cleaned up when the user cancelled
submission in P4Submit.applyCommit(), clean up the mess if "p4
submit" fails.
* eg/p4-submit-catch-failure:
git-p4: clean up after p4 submit failure
Make git-p4 work on a detached head.
* ld/p4-detached-head:
git-p4: work with a detached head
git-p4: add option to system() to return subshell status
git-p4: add failing test for submit from detached head
t5516 "75 - deny fetch unreachable SHA1, allowtipsha1inwant=true" is
flaky in the following case:
1. remote upload-pack finds out "not our ref"
2. remote sends a response and closes the pipe
3. fetch-pack still tries to write commands to the remote upload-pack
4. write call in wrapper.c dies with SIGPIPE
The test is flaky because the sending fetch-pack may or may
not have finished writing its output by step (3). If it did,
then we see a closed pipe on the next read() call. If it
didn't, then we get the SIGPIPE from step (4) above. Both
are fine, but the latter fools test_must_fail.
t5504 "9 - push with transfer.fsckobjects" is flaky, too, and returns
SIGPIPE once in a while. I had to remove the final "To dst..." output
check because there is no output if the process dies with SIGPIPE.
Accept such a death-with-sigpipe also as OK when we are expecting a
failure.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
Add an (optional) first parameter "ok=<special case>" to test_must_fail
and return success for "<special case>". Add "success" as
"<special case>" and use it to implement "test_might_fail". This removes
redundancies in test-lib-function.sh.
You can pass multiple <special case> arguments divided by comma (e.g.
"test_must_fail ok=success,something")
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Jeff King <peff@peff.net>
'git filter-branch' fails complaining about an ambiguous argument, if
a tree-filter renames a path and the new pathname happens to match an
existing object name.
After the tree-filter has been applied, 'git filter-branch' looks for
changed paths by running:
git diff-index -r --name-only --ignore-submodules $commit
which then, because of the lack of disambiguating double-dash, can't
decide whether to treat '$commit' as revision or path and errors out.
Add that disambiguating double-dash after 'git diff-index's revision
argument to make sure that '$commit' is interpreted as a revision.
Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
Signed-off-by: Jeff King <peff@peff.net>
Commit 1b0d400 refactored the prepare_final() function so
that it could be reused in multiple places. Originally, the
loop had two outputs: a commit to stuff into sb->final, and
the name of the commit from the rev->pending array.
After the refactor, that loop is put in its own function
with a single return value: the object_array_entry from the
rev->pending array. This contains both the name and the object,
but with one important difference: the object is the
_original_ object found by the revision parser, not the
dereferenced commit. If one feeds a tag to "git blame", we
end up casting the tag object to a "struct commit", which
causes a segfault.
Instead, let's return the commit (properly casted) directly
from the function, and take the "name" as an optional
out-parameter. This does the right thing, and actually
simplifies the callers, who no longer need to cast or
dereference the object_array_entry themselves.
[test case by Max Kirillov <max@max630.net>]
Signed-off-by: Jeff King <peff@peff.net>
When "p4 submit" command fails in P4Submit.applyCommit, the
workspace is left with the changes. We already have code to revert
the changes to the workspace when the user decides to cancel
submission by aborting the editor that edits the change description,
and we should treat the "p4 submit" failure the same way.
Clean the workspace if p4_write_pipe raised SystemExit, so that the
user don't have to do it themselves.
Signed-off-by: GIRARD Etienne <egirard@murex.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Jeff King <peff@peff.net>
__git_ps1() doesn't indicate dirty index while on an orphan branch.
To check the dirtiness of the index, __git_ps1() runs 'git diff-index
--cached ... HEAD', which doesn't work on an orphan branch,
because HEAD doesn't point to a valid commit.
Run 'git diff ... --cached' instead, as it does the right thing both
on valid and invalid HEAD, i.e. compares the index to the existing
HEAD in the former case and to the empty tree in the latter. This
fixes the two failing tests added in the first commit of this series.
The dirtiness of the worktree is already checked with 'git diff' and
is displayed correctly even on an orphan branch.
Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
Signed-off-by: Jeff King <peff@peff.net>
There is only a single test exercising the dirty state indicator on an
orphan branch, and in that test neither the index nor the worktree are
dirty.
Add two failing tests to check the dirty state indicator while either
the index is dirty or while both the index and the worktree are dirty
on an orphan branch, and to show that the dirtiness of the index is
not displayed in these cases (the fourth combination, i.e. clean index
and dirty worktree are impossible on an orphan branch). Update the
existing dirty state indicator on clean orphan branch test to match
the style of the two new tests, most importantly to use 'git checkout
--orphan' instead of cd-ing into a repository that just happens to be
empty and clean.
Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
Signed-off-by: Jeff King <peff@peff.net>
When submitting, git-p4 finds the current branch in
order to know if it is allowed to submit (configuration
"git-p4.allowSubmit").
On a detached head, detecting the branch would fail, and
git-p4 would report a cryptic error.
This change teaches git-p4 to recognise a detached head and
submit successfully.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Jeff King <peff@peff.net>
git-p4 can't submit from a detached head. This test case
demonstrates the problem.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Jeff King <peff@peff.net>
We check the return value of verify_header() for commits already, so do
the same for tags as well.
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Jeff King <peff@peff.net>
Sometimes the "prove" test runner hangs on test exit because p4d is
still running. Add a trap to always kill "p4d" on test exit.
You can reproduce the problem by commenting "P4D_TIMEOUT" in
"lib-git-p4.sh" and running "prove ./t9800-git-p4-basic.sh".
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
Our usual style these days is to execute everything inside
test_expect_success. Make it so.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Jeff King <peff@peff.net>
Add an option "--dump-aliases" which changes the default behavior of
git-send-email. This mode will simply read the alias files configured by
sendemail.aliasesfile and sendemail.aliasfiletype and dump a list of all
configured aliases, one per line. The intended use case for this option
is the bash-completion script which will use it to autocomplete aliases
on the options which take addresses.
Add some tests for the new option using various alias file formats.
A possible future extension to the alias dump format could be done by
extending the --dump-aliases to take an optional argument defining the
format to display. This has not been done in this patch as no user of
this information has been identified.
Signed-off-by: Jacob Keller <jacob.keller@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
In rare cases p4d seems to hang. This watchdog will kill the p4d
process after 300s in any case. That means each individual git p4 test
needs to finish before 300s or it will fail.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Jeff King <peff@peff.net>
In the next commit, we will indent test case preparations. This will
require that here-documents ignore the tab indentation. Prepare for
this change by marking the here-doc words accordingly. This does not
have an effect now, but will remove some noise from the git diff -b
output of the next commit.
The change here is entirely automated with this perl command:
perl -i -lpe 's/(cat.*<<) *((EOF|(EXPECT|INPUT)_END).*$)/$1-$2 &&/' t/t9300-fast-import.sh
i.e., inserts a dash between << and the EOF word (and removes blanks
that our style guide abhors) and appends the && that will become
necessary.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Jeff King <peff@peff.net>
In rare cases kill/cleanup operations in tests fail. Retry these
operations with a timeout to make the test less flaky.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
A number of clean-ups of test cases are performed outside of
test_expect_success. Replace these cases by using test_when_finished.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Jeff King <peff@peff.net>
It is customary to have each command in test snippets on its own line.
Fix those instances that do not follow this guideline.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Jeff King <peff@peff.net>
Instead of comparing actual output to an empty file, use
test_must_be_empty. In addition to the better error message provided by
the helper, allocation of an empty file during the setup sequence can be
avoided.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Jeff King <peff@peff.net>
One test case open-codes a test for an expected failure. Replace it by
test_must_fail.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Jeff King <peff@peff.net>
Many test cases do not follow our modern style that places the
single-quotes that surround the shell code snippets before and after
the shell code. Make it so.
Many of the lines changed in this way are indented other than by a
single tab. Change them (and some additional lines) to be indented
with a tab.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Jeff King <peff@peff.net>
Extend transfer.hideRefs to work better with use of namespaces.
* lf/ref-is-hidden-namespace:
t5509: add basic tests for hideRefs
hideRefs: add support for matching full refs
upload-pack: strip refs before calling ref_is_hidden()
config.txt: document the semantics of hideRefs with namespaces
Having a leftover .idx file without corresponding .pack file in
the repository hurts performance; "git gc" learned to prune them.
* dk/gc-idx-wo-pack:
gc: remove garbage .idx files from pack dir
t5304: test cleaning pack garbage
prepare_packed_git(): refactor garbage reporting in pack directory
When passed an ssh:// url, git strips ssh://host from the url but does
not remove leading slashes from the path. So when this test used
ssh://remote//path/to/pwd, the path accessed by our fake SSH is
//path/to/pwd, which cygwin interprets as a UNC path, causing the test
to fail.
We may want to actually fix this in git itself, making it remove extra
slashes from urls before feeding them to transports or helpers, but
that's for another topic as it could cause regressions.
Signed-off-by: Dennis Kaarsemaker <dennis@kaarsemaker.net>
Signed-off-by: Jeff King <peff@peff.net>
Since ec7dbd145 (receive-pack: allow hooks to ignore its
standard input stream) the pre-receive and post-receive
hooks ignore SIGPIPE. Do the same for the remaining hooks
pre-push and post-rewrite, which read from standard input.
The same arguments for ignoring SIGPIPE apply.
Include test by Jeff King which checks that SIGPIPE does not
cause pre-push hook failure. With the use of git update-ref
--stdin it is fast enough to be enabled by default.
Signed-off-by: Clemens Buchacher <clemens.buchacher@intel.com>
Signed-off-by: Jeff King <peff@peff.net>
If the user specifies an index filter but not a tree filter,
filter-branch cleverly avoids checking out the tree
entirely. But we don't do the next level of optimization: if
you have no index or tree filter, we do not need to read the
index at all.
This can greatly speed up cases where we are only changing
the commit objects (e.g., cementing a graft into place).
Here are numbers from the newly-added perf test:
Test HEAD^ HEAD
---------------------------------------------------------------
7000.2: noop filter 13.81(4.95+0.83) 5.43(0.42+0.43) -60.7%
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
