Commit Graph

41542 Commits

Author SHA1 Message Date
Stefan Beller
72290d6a1d clone: allow an explicit argument for parallel submodule clones
Just pass it along to "git submodule update", which may pick reasonable
defaults if you don't specify an explicit number.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01 11:57:21 -08:00
Stefan Beller
2335b870fa submodule update: expose parallelism to the user
Expose possible parallelism either via the "--jobs" CLI parameter or
the "submodule.fetchJobs" setting.

By having the variable initialized to -1, we make sure 0 can be passed
into the parallel processing machine, which will then pick as many parallel
workers as there are CPUs.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01 11:57:20 -08:00
Stefan Beller
cdc04b65b4 submodule helper: remove double 'fatal: ' prefix
The prefix is added by die(...), so we don't have to do it.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01 11:57:20 -08:00
Stefan Beller
48308681b0 git submodule update: have a dedicated helper for cloning
This introduces a new helper function in git submodule--helper
which takes care of cloning all submodules, which we want to
parallelize eventually.

Some tests (such as empty URL, update_mode=none) are required in the
helper to make the decision for cloning. These checks have been
moved into the C function as well (no need to repeat them in the
shell script).

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01 11:57:20 -08:00
Stefan Beller
aa71049485 run_processes_parallel: rename parameters for the callbacks
The refs code has a similar pattern of passing around 'struct strbuf *err',
which is strictly used for error reporting. This is not the case here,
as the strbuf is used to accumulate all the output (whether it is error
or not) for the user. Rename it to 'out'.

Suggested-by: Jonathan Nieder <jrnieder@gmail.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01 11:57:19 -08:00
Stefan Beller
2dac9b5637 run_processes_parallel: treat output of children as byte array
We do not want the output to be interrupted by a NUL byte, so we
cannot use raw fputs. Introduce strbuf_write to avoid having long
arguments in run-command.c.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01 11:57:19 -08:00
Stefan Beller
8c6b549118 submodule update: direct error message to stderr
Reroute the error message for specified but initialized submodules
to stderr instead of stdout.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01 11:57:18 -08:00
Stefan Beller
a028a1930c fetching submodules: respect submodule.fetchJobs config option
This allows to configure fetching and updating in parallel
without having the command line option.

This moved the responsibility to determine how many parallel processes
to start from builtin/fetch to submodule.c as we need a way to communicate
"The user did not specify the number of parallel processes in the command
line options" in the builtin fetch. The submodule code takes care of
the precedence (CLI > config > default).

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01 11:57:18 -08:00
Stefan Beller
f73da11024 submodule-config: drop check against NULL
Adhere to the common coding style of Git and not check explicitly
for NULL throughout the file. There are still other occurrences in the
code base but that is usually inside of conditions with side effects.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01 11:57:17 -08:00
Stefan Beller
ea2fa5a338 submodule-config: keep update strategy around
Currently submodule.<name>.update is only handled by git-submodule.sh.
C code will start to need to make use of that value as more of the
functionality of git-submodule.sh moves into library code in C.

Add the update field to 'struct submodule' and populate it so it can
be read as sm->update or from sm->update_command.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01 11:57:17 -08:00
Stefan Beller
2a73b3dad0 run-command: do not pass child process data into callbacks
The expected way to pass data into the callback is to pass them via
the customizable callback pointer. The error reporting in
default_{start_failure, task_finished} is not user friendly enough, that
we want to encourage using the child data for such purposes.

Furthermore the struct child data is cleaned by the run-command API,
before we access them in the callbacks, leading to use-after-free
situations.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-03-01 09:42:01 -08:00
Stefan Beller
62104ba14a submodules: allow parallel fetching, add tests and documentation
This enables the work of the previous patches.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Stefan Beller
fe85ee6e23 fetch_populated_submodules: use new parallel job processing
In a later patch we enable parallel processing of submodules, this
only adds the possibility for it. So this change should not change
any user facing behavior.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Stefan Beller
c553c72eed run-command: add an asynchronous parallel child processor
This allows to run external commands in parallel with ordered output
on stderr.

If we run external commands in parallel we cannot pipe the output directly
to the our stdout/err as it would mix up. So each process's output will
flow through a pipe, which we buffer. One subprocess can be directly
piped to out stdout/err for a low latency feedback to the user.

Example:
Let's assume we have 5 submodules A,B,C,D,E and each fetch takes a
different amount of time as the different submodules vary in size, then
the output of fetches in sequential order might look like this:

 time -->
 output: |---A---| |-B-| |-------C-------| |-D-| |-E-|

When we schedule these submodules into maximal two parallel processes,
a schedule and sample output over time may look like this:

process 1: |---A---| |-D-| |-E-|

process 2: |-B-| |-------C-------|

output:    |---A---|B|---C-------|DE

So A will be perceived as it would run normally in the single child
version. As B has finished by the time A is done, we can dump its whole
progress buffer on stderr, such that it looks like it finished in no
time. Once that is done, C is determined to be the visible child and
its progress will be reported in real time.

So this way of output is really good for human consumption, as it only
changes the timing, not the actual output.

For machine consumption the output needs to be prepared in the tasks,
by either having a prefix per line or per block to indicate whose tasks
output is displayed, because the output order may not follow the
original sequential ordering:

 |----A----| |--B--| |-C-|

will be scheduled to be all parallel:

process 1: |----A----|
process 2: |--B--|
process 3: |-C-|
output:    |----A----|CB

This happens because C finished before B did, so it will be queued for
output before B.

To detect when a child has finished executing, we check interleaved
with other actions (such as checking the liveliness of children or
starting new processes) whether the stderr pipe still exists. Once a
child closed its stderr stream, we assume it is terminating very soon,
and use `finish_command()` from the single external process execution
interface to collect the exit status.

By maintaining the strong assumption of stderr being open until the
very end of a child process, we can avoid other hassle such as an
implementation using `waitpid(-1)`, which is not implemented in Windows.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Stefan Beller
bfb6b53c05 sigchain: add command to pop all common signals
The new method removes all common signal handlers that were installed
by sigchain_push.

CC: Jeff King <peff@peff.net>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Stefan Beller
b4e04fb66e strbuf: add strbuf_read_once to read without blocking
The new call will read from a file descriptor into a strbuf once. The
underlying call xread is just run once. xread only reattempts
reading in case of EINTR, which makes it suitable to use for a
nonblocking read.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Stefan Beller
1079c4be0b xread: poll on non blocking fds
The man page of read(2) says:

  EAGAIN The file descriptor fd refers to a file other than a socket
	 and has been marked nonblocking (O_NONBLOCK), and the read
	 would block.

  EAGAIN or EWOULDBLOCK
	 The file descriptor fd refers to a socket and has been marked
	 nonblocking (O_NONBLOCK), and the read would block.  POSIX.1-2001
	 allows either error to be returned for this case, and does not
	 require these constants to have the same value, so a portable
	 application should check for both possibilities.

If we get an EAGAIN or EWOULDBLOCK the fd must have set O_NONBLOCK.
As the intent of xread is to read as much as possible either until the
fd is EOF or an actual error occurs, we can ease the feeder of the fd
by not spinning the whole time, but rather wait for it politely by not
busy waiting.

We should not care if the call to poll failed, as we're in an infinite
loop and can only get out with the correct read().

Signed-off-by: Stefan Beller <sbeller@google.com>
Acked-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Jonathan Nieder
fbf71645d1 submodule.c: write "Fetching submodule <foo>" to stderr
The "Pushing submodule <foo>" progress output correctly goes to
stderr, but "Fetching submodule <foo>" is going to stdout by
mistake.  Fix it to write to stderr.

Noticed while trying to implement a parallel submodule fetch.  When
this particular output line went to a different file descriptor, it
was buffered separately, resulting in wrongly interleaved output if
we copied it to the terminal naively.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Junio C Hamano
4b9ab0ee01 Update release notes to 2.7
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-11 11:20:23 -08:00
Junio C Hamano
52b2e6be99 Merge branch 'maint'
* maint:
  Prepare for 2.6.5
2015-12-11 11:19:43 -08:00
Junio C Hamano
49e863b02a Prepare for 2.6.5
This back-merges hopefully the last batch of trivially correct fixes
to the 2.6.x maintenance track from the master branch.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-11 11:17:47 -08:00
Junio C Hamano
58e3dd21f6 Merge branch 'sn/null-pointer-arith-in-mark-tree-uninteresting' into maint
mark_tree_uninteresting() has code to handle the case where it gets
passed a NULL pointer in its 'tree' parameter, but the function had
'object = &tree->object' assignment before checking if tree is
NULL.  This gives a compiler an excuse to declare that tree will
never be NULL and apply a wrong optimization.  Avoid it.

* sn/null-pointer-arith-in-mark-tree-uninteresting:
  revision.c: fix possible null pointer arithmetic
2015-12-11 11:14:38 -08:00
Junio C Hamano
abca668a93 Merge branch 'sg/lock-file-commit-error' into maint
Cosmetic improvement to lock-file error messages.

* sg/lock-file-commit-error:
  Make error message after failing commit_lock_file() less confusing
2015-12-11 11:14:18 -08:00
Junio C Hamano
76058817e8 Merge branch 'cb/t3404-shellquote' into maint
* cb/t3404-shellquote:
  t3404: fix quoting of redirect for some versions of bash
2015-12-11 11:14:18 -08:00
Junio C Hamano
17e5bcab71 Merge branch 'sb/doc-submodule-sync-recursive' into maint
* sb/doc-submodule-sync-recursive:
  document submodule sync --recursive
2015-12-11 11:14:17 -08:00
Junio C Hamano
63b3db71d8 Merge branch 'nd/doc-check-ref-format-typo' into maint
* nd/doc-check-ref-format-typo:
  git-check-ref-format.txt: typo, s/avoids/avoid/
2015-12-11 11:14:15 -08:00
Junio C Hamano
288fe0cfb6 Merge branch 'rs/show-branch-argv-array' into maint
Code simplification.

* rs/show-branch-argv-array:
  show-branch: use argv_array for default arguments
2015-12-11 11:14:14 -08:00
Junio C Hamano
0af22d6fff Merge branch 'rs/pop-commit' into maint
Code simplification.

* rs/pop-commit:
  use pop_commit() for consuming the first entry of a struct commit_list
2015-12-11 11:14:13 -08:00
Junio C Hamano
8c0a546670 Merge branch 'as/subtree-with-spaces' into maint
Update "git subtree" (in contrib/) so that it can take whitespaces
in the pathnames, not only in the in-tree pathname but the name of
the directory that the repository is in.

* as/subtree-with-spaces:
  contrib/subtree: respect spaces in a repository path
  t7900-subtree: test the "space in a subdirectory name" case
2015-12-11 11:14:11 -08:00
Junio C Hamano
4cb5488fa6 Merge branch 'jk/test-lint-forbid-when-finished-in-subshell' into maint
Because "test_when_finished" in our test framework queues the
clean-up tasks to be done in a shell variable, it should not be
used inside a subshell.  Add a mechanism to allow 'bash' to catch
such uses, and fix the ones that were found.

* jk/test-lint-forbid-when-finished-in-subshell:
  test-lib-functions: detect test_when_finished in subshell
  t7800: don't use test_config in a subshell
  test-lib-functions: support "test_config -C <dir> ..."
  t5801: don't use test_when_finished in a subshell
  t7610: don't use test_config in a subshell
2015-12-11 11:14:10 -08:00
Junio C Hamano
782ca8c44e Merge branch 'sn/null-pointer-arith-in-mark-tree-uninteresting'
mark_tree_uninteresting() has code to handle the case where it gets
passed a NULL pointer in its 'tree' parameter, but the function had
'object = &tree->object' assignment before checking if tree is
NULL.  This gives a compiler an excuse to declare that tree will
never be NULL and apply a wrong optimization.  Avoid it.

* sn/null-pointer-arith-in-mark-tree-uninteresting:
  revision.c: fix possible null pointer arithmetic
2015-12-11 10:41:01 -08:00
Junio C Hamano
fa41b05253 Merge branch 'sb/doc-submodule-sync-recursive'
* sb/doc-submodule-sync-recursive:
  document submodule sync --recursive
2015-12-11 10:41:00 -08:00
Junio C Hamano
c87eec9784 Merge branch 'cb/t3404-shellquote'
* cb/t3404-shellquote:
  t3404: fix quoting of redirect for some versions of bash
2015-12-11 10:40:58 -08:00
Junio C Hamano
e0048d3e0d Merge branch 'sg/lock-file-commit-error'
Cosmetic improvement to lock-file error messages.

* sg/lock-file-commit-error:
  Make error message after failing commit_lock_file() less confusing
2015-12-11 10:40:55 -08:00
Junio C Hamano
7d722536dd Git 2.7-rc0
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-10 12:59:37 -08:00
Junio C Hamano
39e07f77b6 Sync with maint
* maint:
  Documentation/git-update-index: add missing opts to synopsis
2015-12-10 12:45:17 -08:00
Junio C Hamano
86c95ac5d2 Update release notes to 2.7
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-10 12:43:52 -08:00
Junio C Hamano
56d63d0eab Merge branch 'nd/doc-check-ref-format-typo'
* nd/doc-check-ref-format-typo:
  git-check-ref-format.txt: typo, s/avoids/avoid/
2015-12-10 12:36:15 -08:00
Junio C Hamano
844a9ce472 Merge branch 'bc/object-id'
More transition from "unsigned char[40]" to "struct object_id".

This needed a few merge fixups, but is mostly disentangled from other
topics.

* bc/object-id:
  remote: convert functions to struct object_id
  Remove get_object_hash.
  Convert struct object to object_id
  Add several uses of get_object_hash.
  object: introduce get_object_hash macro.
  ref_newer: convert to use struct object_id
  push_refs_with_export: convert to struct object_id
  get_remote_heads: convert to struct object_id
  parse_fetch: convert to use struct object_id
  add_sought_entry_mem: convert to struct object_id
  Convert struct ref to use object_id.
  sha1_file: introduce has_object_file helper.
2015-12-10 12:36:13 -08:00
Junio C Hamano
b12a966eff Merge branch 'dt/fsck-verify-pack-error'
The exit code of git-fsck didnot reflect some types of errors found
in packed objects, which has been corrected.

* dt/fsck-verify-pack-error:
  verify_pack: do not ignore return value of verification function
2015-12-10 12:36:12 -08:00
Junio C Hamano
9eb2449c35 Merge branch 'ls/travis-yaml'
The necessary infrastructure to build topics using the free Travis
CI has been added. Developers forking from this topic (and enabling
Travis) can do their own builds, and we can turn on auto-builds for
git/git (including build-status for pull requests that people
open).

* ls/travis-yaml:
  Add Travis CI support
2015-12-10 12:36:12 -08:00
Christian Couder
bc49712789 Documentation/git-update-index: add missing opts to synopsis
Split index related options should appear in the 'SYNOPSIS'
section.

These options are already documented in the 'OPTIONS' section.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-10 12:21:26 -08:00
Junio C Hamano
545299f822 Merge branch 'ep/ident-with-getaddrinfo'
A build without NO_IPv6 used to use gethostbyname() when guessing
user's hostname, instead of getaddrinfo() that is used in other
codepaths in such a build.

* ep/ident-with-getaddrinfo:
  ident.c: add support for IPv6
2015-12-08 14:14:50 -08:00
Junio C Hamano
2b597f3307 Merge branch 'ls/test-must-fail-sigpipe'
Fix some racy client/server tests by treating SIGPIPE the same as a
normal non-zero exit.

* ls/test-must-fail-sigpipe:
  add "ok=sigpipe" to test_must_fail and use it to fix flaky tests
  implement test_might_fail using a refactored test_must_fail
2015-12-08 14:14:49 -08:00
Junio C Hamano
b1cda70fff Merge branch 'dt/refs-backend-pre-vtable'
Code preparation for pluggable ref backends.

* dt/refs-backend-pre-vtable:
  refs: break out ref conflict checks
  files_log_ref_write: new function
  initdb: make safe_create_dir public
  refs: split filesystem-based refs code into a new file
  refs/refs-internal.h: new header file
  refname_is_safe(): improve docstring
  pack_if_possible_fn(): use ref_type() instead of is_per_worktree_ref()
  copy_msg(): rename to copy_reflog_msg()
  verify_refname_available(): new function
  verify_refname_available(): rename function
2015-12-08 14:14:49 -08:00
Junio C Hamano
fa5f2398e5 Sync with 2.6.4 2015-12-08 14:13:52 -08:00
Junio C Hamano
bdfc6b364a Git 2.6.4
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-08 14:12:45 -08:00
Junio C Hamano
4d0069f277 Merge branch 'rs/status-detached-head-memcmp' into maint
Fix some string-matching corner cases when digging in the reflog for
"git status".

* rs/status-detached-head-memcmp:
  wt-status: correct and simplify check for detached HEAD
2015-12-08 14:11:32 -08:00
Junio C Hamano
b5d2d8eef0 Merge branch 'ad/sha1-update-chunked' into maint
Apple's common crypto implementation of SHA1_Update() does not take
more than 4GB at a time, and we now have a compile-time workaround
for it.

* ad/sha1-update-chunked:
  sha1: allow limiting the size of the data passed to SHA1_Update()
  sha1: provide another level of indirection for the SHA-1 functions
2015-12-08 14:05:03 -08:00
Junio C Hamano
e6ed5a438c Merge branch 'sg/bash-prompt-dirty-orphan' into maint
Produce correct "dirty" marker for shell prompts, even when we
are on an orphan or an unborn branch.

* sg/bash-prompt-dirty-orphan:
  bash prompt: indicate dirty index even on orphan branches
  bash prompt: remove a redundant 'git diff' option
  bash prompt: test dirty index and worktree while on an orphan branch
2015-12-08 14:05:02 -08:00