This allows to run external commands in parallel with ordered output
on stderr.
If we run external commands in parallel we cannot pipe the output directly
to the our stdout/err as it would mix up. So each process's output will
flow through a pipe, which we buffer. One subprocess can be directly
piped to out stdout/err for a low latency feedback to the user.
Example:
Let's assume we have 5 submodules A,B,C,D,E and each fetch takes a
different amount of time as the different submodules vary in size, then
the output of fetches in sequential order might look like this:
time -->
output: |---A---| |-B-| |-------C-------| |-D-| |-E-|
When we schedule these submodules into maximal two parallel processes,
a schedule and sample output over time may look like this:
process 1: |---A---| |-D-| |-E-|
process 2: |-B-| |-------C-------|
output: |---A---|B|---C-------|DE
So A will be perceived as it would run normally in the single child
version. As B has finished by the time A is done, we can dump its whole
progress buffer on stderr, such that it looks like it finished in no
time. Once that is done, C is determined to be the visible child and
its progress will be reported in real time.
So this way of output is really good for human consumption, as it only
changes the timing, not the actual output.
For machine consumption the output needs to be prepared in the tasks,
by either having a prefix per line or per block to indicate whose tasks
output is displayed, because the output order may not follow the
original sequential ordering:
|----A----| |--B--| |-C-|
will be scheduled to be all parallel:
process 1: |----A----|
process 2: |--B--|
process 3: |-C-|
output: |----A----|CB
This happens because C finished before B did, so it will be queued for
output before B.
To detect when a child has finished executing, we check interleaved
with other actions (such as checking the liveliness of children or
starting new processes) whether the stderr pipe still exists. Once a
child closed its stderr stream, we assume it is terminating very soon,
and use `finish_command()` from the single external process execution
interface to collect the exit status.
By maintaining the strong assumption of stderr being open until the
very end of a child process, we can avoid other hassle such as an
implementation using `waitpid(-1)`, which is not implemented in Windows.
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The new method removes all common signal handlers that were installed
by sigchain_push.
CC: Jeff King <peff@peff.net>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The new call will read from a file descriptor into a strbuf once. The
underlying call xread is just run once. xread only reattempts
reading in case of EINTR, which makes it suitable to use for a
nonblocking read.
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The man page of read(2) says:
EAGAIN The file descriptor fd refers to a file other than a socket
and has been marked nonblocking (O_NONBLOCK), and the read
would block.
EAGAIN or EWOULDBLOCK
The file descriptor fd refers to a socket and has been marked
nonblocking (O_NONBLOCK), and the read would block. POSIX.1-2001
allows either error to be returned for this case, and does not
require these constants to have the same value, so a portable
application should check for both possibilities.
If we get an EAGAIN or EWOULDBLOCK the fd must have set O_NONBLOCK.
As the intent of xread is to read as much as possible either until the
fd is EOF or an actual error occurs, we can ease the feeder of the fd
by not spinning the whole time, but rather wait for it politely by not
busy waiting.
We should not care if the call to poll failed, as we're in an infinite
loop and can only get out with the correct read().
Signed-off-by: Stefan Beller <sbeller@google.com>
Acked-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The "Pushing submodule <foo>" progress output correctly goes to
stderr, but "Fetching submodule <foo>" is going to stdout by
mistake. Fix it to write to stderr.
Noticed while trying to implement a parallel submodule fetch. When
this particular output line went to a different file descriptor, it
was buffered separately, resulting in wrongly interleaved output if
we copied it to the terminal naively.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This back-merges hopefully the last batch of trivially correct fixes
to the 2.6.x maintenance track from the master branch.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
mark_tree_uninteresting() has code to handle the case where it gets
passed a NULL pointer in its 'tree' parameter, but the function had
'object = &tree->object' assignment before checking if tree is
NULL. This gives a compiler an excuse to declare that tree will
never be NULL and apply a wrong optimization. Avoid it.
* sn/null-pointer-arith-in-mark-tree-uninteresting:
revision.c: fix possible null pointer arithmetic
Update "git subtree" (in contrib/) so that it can take whitespaces
in the pathnames, not only in the in-tree pathname but the name of
the directory that the repository is in.
* as/subtree-with-spaces:
contrib/subtree: respect spaces in a repository path
t7900-subtree: test the "space in a subdirectory name" case
Because "test_when_finished" in our test framework queues the
clean-up tasks to be done in a shell variable, it should not be
used inside a subshell. Add a mechanism to allow 'bash' to catch
such uses, and fix the ones that were found.
* jk/test-lint-forbid-when-finished-in-subshell:
test-lib-functions: detect test_when_finished in subshell
t7800: don't use test_config in a subshell
test-lib-functions: support "test_config -C <dir> ..."
t5801: don't use test_when_finished in a subshell
t7610: don't use test_config in a subshell
mark_tree_uninteresting() has code to handle the case where it gets
passed a NULL pointer in its 'tree' parameter, but the function had
'object = &tree->object' assignment before checking if tree is
NULL. This gives a compiler an excuse to declare that tree will
never be NULL and apply a wrong optimization. Avoid it.
* sn/null-pointer-arith-in-mark-tree-uninteresting:
revision.c: fix possible null pointer arithmetic
More transition from "unsigned char[40]" to "struct object_id".
This needed a few merge fixups, but is mostly disentangled from other
topics.
* bc/object-id:
remote: convert functions to struct object_id
Remove get_object_hash.
Convert struct object to object_id
Add several uses of get_object_hash.
object: introduce get_object_hash macro.
ref_newer: convert to use struct object_id
push_refs_with_export: convert to struct object_id
get_remote_heads: convert to struct object_id
parse_fetch: convert to use struct object_id
add_sought_entry_mem: convert to struct object_id
Convert struct ref to use object_id.
sha1_file: introduce has_object_file helper.
The exit code of git-fsck didnot reflect some types of errors found
in packed objects, which has been corrected.
* dt/fsck-verify-pack-error:
verify_pack: do not ignore return value of verification function
The necessary infrastructure to build topics using the free Travis
CI has been added. Developers forking from this topic (and enabling
Travis) can do their own builds, and we can turn on auto-builds for
git/git (including build-status for pull requests that people
open).
* ls/travis-yaml:
Add Travis CI support
Split index related options should appear in the 'SYNOPSIS'
section.
These options are already documented in the 'OPTIONS' section.
Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A build without NO_IPv6 used to use gethostbyname() when guessing
user's hostname, instead of getaddrinfo() that is used in other
codepaths in such a build.
* ep/ident-with-getaddrinfo:
ident.c: add support for IPv6
Fix some racy client/server tests by treating SIGPIPE the same as a
normal non-zero exit.
* ls/test-must-fail-sigpipe:
add "ok=sigpipe" to test_must_fail and use it to fix flaky tests
implement test_might_fail using a refactored test_must_fail
Code preparation for pluggable ref backends.
* dt/refs-backend-pre-vtable:
refs: break out ref conflict checks
files_log_ref_write: new function
initdb: make safe_create_dir public
refs: split filesystem-based refs code into a new file
refs/refs-internal.h: new header file
refname_is_safe(): improve docstring
pack_if_possible_fn(): use ref_type() instead of is_per_worktree_ref()
copy_msg(): rename to copy_reflog_msg()
verify_refname_available(): new function
verify_refname_available(): rename function
Fix some string-matching corner cases when digging in the reflog for
"git status".
* rs/status-detached-head-memcmp:
wt-status: correct and simplify check for detached HEAD
Apple's common crypto implementation of SHA1_Update() does not take
more than 4GB at a time, and we now have a compile-time workaround
for it.
* ad/sha1-update-chunked:
sha1: allow limiting the size of the data passed to SHA1_Update()
sha1: provide another level of indirection for the SHA-1 functions
Produce correct "dirty" marker for shell prompts, even when we
are on an orphan or an unborn branch.
* sg/bash-prompt-dirty-orphan:
bash prompt: indicate dirty index even on orphan branches
bash prompt: remove a redundant 'git diff' option
bash prompt: test dirty index and worktree while on an orphan branch
Allow "git interpret-trailers" to run outside of a Git repository.
* jk/interpret-trailers-outside-a-repository:
interpret-trailers: allow running outside a repository
There was no way to defeat a configured rebase.autostash variable
from the command line, as "git rebase --no-autostash" was missing.
* jk/rebase-no-autostash:
Documentation/git-rebase: fix --no-autostash formatting
rebase: support --no-autostash
mark_tree_uninteresting() dereferences a tree pointer before
checking if the pointer is valid. Fix that by doing the check first.
Signed-off-by: Stefan Naewe <stefan.naewe@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
As CodingGuidelines says, some versions of bash errors out when
$variable substitution is used as the target for redirection without
being quoted (even though POSIX may not require such a quote).
Signed-off-by: Charles Bailey <cbailey32@bloomberg.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The git-submodule(1) is inconsistent. In the synopsis, it says:
git submodule [--quiet] sync [--recursive] [--] [<path>...]
The description of the sync does not mention --recursive, and the
description of --recursive says that it is only available for foreach,
update and status.
The option was introduced (82f49f294c, Teach --recursive to submodule
sync, 2012-10-26) a while ago, so let's document it, too.
Reported-by: Per Cederqvist <cederp@opera.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The code to prepare the working tree side of temporary directory
for the "dir-diff" feature forgot that symbolic links need not be
copied (or symlinked) to the temporary area, as the code already
special cases and overwrites them. Besides, it was wrong to try
computing the object name of the target of symbolic link, which may
not even exist or may be a directory.
* da/difftool:
difftool: ignore symbolic links in use_wt_file
Having a leftover .idx file without corresponding .pack file in
the repository hurts performance; "git gc" learned to prune them.
We may want to do the same for .bitmap (and notice but not prune
.keep) without corresponding .pack, but that can be a separate
topic.
* dk/gc-idx-wo-pack:
gc: remove garbage .idx files from pack dir
t5304: test cleaning pack garbage
prepare_packed_git(): refactor garbage reporting in pack directory
Produce correct "dirty" marker for shell prompts, even when we
are on an orphan or an unborn branch.
* sg/bash-prompt-dirty-orphan:
bash prompt: indicate dirty index even on orphan branches
bash prompt: remove a redundant 'git diff' option
bash prompt: test dirty index and worktree while on an orphan branch
