Since "git fetch" learned "--all" and "--multiple" options, it has become
tempting for users to say "git pull --all". Even though it may fetch from
remotes that do not need to be fetched from for merging with the current
branch, it is handy.
"git fetch" however clears the list of fetched branches every time it
contacts a different remote. Unless the current branch is configured to
merge with a branch from a remote that happens to be the last in the list
of remotes that are contacted, "git pull" that fetches from multiple
remotes will not be able to find the branch it should be merging with.
Make "fetch" clear FETCH_HEAD (unless --append is given) and then append
the list of branches fetched to it (even when --append is not given). That
way, "pull" will be able to find the data for the branch being merged in
FETCH_HEAD no matter where the remote appears in the list of remotes to be
contacted by "git fetch".
Reported-by: Michael Lukashov
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When running a subfetch, the code propagated some options but not others.
Propagate --force, --update-head-ok and --keep options as well.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* sr/vcs-helper:
tests: handle NO_PYTHON setting
builtin-push: don't access freed transport->url
Add Python support library for remote helpers
Basic build infrastructure for Python scripts
Allow helpers to report in "list" command that the ref is unchanged
Fix various memory leaks in transport-helper.c
Allow helper to map private ref names into normal names
Add support for "import" helper command
Allow specifying the remote helper in the url
Add a config option for remotes to specify a foreign vcs
Allow fetch to modify refs
Use a function to determine whether a remote is valid
Allow programs to not depend on remotes having urls
Fix memory leak in helper method for disconnect
Conflicts:
Documentation/git-remote-helpers.txt
Makefile
builtin-ls-remote.c
builtin-push.c
transport-helper.c
* bg/fetch-multi:
Re-implement 'git remote update' using 'git fetch'
builtin-fetch: add --dry-run option
builtin-fetch: add --prune option
teach warn_dangling_symref to take a FILE argument
remote: refactor some logic into get_stale_heads()
Add missing test for 'git remote update --prune'
Add the configuration option skipFetchAll
Teach the --multiple option to 'git fetch'
Teach the --all option to 'git fetch'
* sp/smart-http: (37 commits)
http-backend: Let gcc check the format of more printf-type functions.
http-backend: Fix access beyond end of string.
http-backend: Fix bad treatment of uintmax_t in Content-Length
t5551-http-fetch: Work around broken Accept header in libcurl
t5551-http-fetch: Work around some libcurl versions
http-backend: Protect GIT_PROJECT_ROOT from /../ requests
Git-aware CGI to provide dumb HTTP transport
http-backend: Test configuration options
http-backend: Use http.getanyfile to disable dumb HTTP serving
test smart http fetch and push
http tests: use /dumb/ URL prefix
set httpd port before sourcing lib-httpd
t5540-http-push: remove redundant fetches
Smart HTTP fetch: gzip requests
Smart fetch over HTTP: client side
Smart push over HTTP: client side
Discover refs via smart HTTP server when available
http-backend: more explict LocationMatch
http-backend: add example for gitweb on same URL
http-backend: use mod_alias instead of mod_rewrite
...
Conflicts:
.gitignore
remote-curl.c
* jp/fetch-cull-many-refs:
remote: fix use-after-free error detected by glibc in ref_remove_duplicates
fetch: Speed up fetch of large numbers of refs
remote: Make ref_remove_duplicates faster for large numbers of refs
For fetch and ls-remote, which use the first url of a remote, have
transport_get() determine this by passing a remote and passing NULL
for the url. For push, which uses every url of a remote, use each url
in turn if there are any, and use NULL if there are none.
This will allow the transport code to do something different if the
location is not specified with a url.
Also, have the message for a fetch say "foreign" if there is no url.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Teach fetch --dry-run as users of "git remote prune" switching to "git fetch
--prune" may expect it. Unfortunately OPT__DRY_RUN() cannot be used as fetch
already uses "-n" for something else.
Signed-off-by: Jay Soffian <jaysoffian@gmail.com>
Signed-off-by: Björn Gustavsson <bgustavsson@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Teach fetch to cull stale remote tracking branches after fetching via --prune.
Signed-off-by: Jay Soffian <jaysoffian@gmail.com>
Signed-off-by: Björn Gustavsson <bgustavsson@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Implement the configuration skipFetchAll option to allow
certain remotes to be skipped when doing 'git fetch --all' and
'git remote update'. The existing skipDefaultUpdate variable
is still honored (by 'git fetch --all' and 'git remote update').
(If both are set in the configuration file with different values,
the value of the last occurrence will be used.)
Signed-off-by: Björn Gustavsson <bgustavsson@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add the --multiple option to specify that all arguments are either
groups or remotes. The primary reason for adding this option is
to allow us to re-implement 'git remote update' using fetch.
It would have been nice if this option was not needed, but since
the colon in a refspec is optional, it is in general not possible
to know whether a single, colon-less argument is a remote or a
refspec.
Signed-off-by: Björn Gustavsson <bgustavsson@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
'git remote' is meant for managing remotes and 'git fetch' is meant
for actually fetching data from remote repositories. Therefore, it is
not logical that you must use 'git remote update' to fetch from
more than one repository at once.
Add the --all option to 'git fetch', to tell it to attempt to fetch
from all remotes. Also, if --all is not given, the <repository>
argument is allowed to be the name of a group, to allow fetching
from all repositories in the group.
Other options except -v and -q are silently ignored.
Signed-off-by: Björn Gustavsson <bgustavsson@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Helpers might want a higher level of verbosity than just +1 (the
porcelain default setting) and +2 (-v -v). Expand the field to
allow verbosity in the range -1..3.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
CC: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When there are large numbers of refs, calling read_ref for each ref is
inefficent (and infact downright slow) - so instead use for_each_ref
to build up a string list of all the refs that we currently have,
which significantly improves the volume.
Signed-off-by: Julian Phillips <julian@quantumfyre.co.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When trying to get a list of remote tags to see if we need to fetch
any we were doing a linear search for the matching tag ref for the
tag^{} commit entries. This proves to be incredibly slow for large
numbers of tags. Rewrite the function so that we build up a
string_list of refs to fetch and then process that instead.
As an extreme example, for a repository with 50000 tags (and just a
single commit on a single branch), a fetch that does nothing goes from
~1m50s to ~4.1s.
Signed-off-by: Julian Phillips <julian@quantumfyre.co.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In 2d14d65 (Use a clearer style to issue commands to remote helpers,
2009-09-03) I happened to notice two changes like this:
- write_in_full(helper->in, "list\n", 5);
+
+ strbuf_addstr(&buf, "list\n");
+ write_in_full(helper->in, buf.buf, buf.len);
+ strbuf_reset(&buf);
IMHO, it would be better to define a new function,
static inline ssize_t write_str_in_full(int fd, const char *str)
{
return write_in_full(fd, str, strlen(str));
}
and then use it like this:
- strbuf_addstr(&buf, "list\n");
- write_in_full(helper->in, buf.buf, buf.len);
- strbuf_reset(&buf);
+ write_str_in_full(helper->in, "list\n");
Thus not requiring the added allocation, and still avoiding
the maintenance risk of literal string lengths.
These days, compilers are good enough that strlen("literal")
imposes no run-time cost.
Transformed via this:
perl -pi -e \
's/write_in_full\((.*?), (".*?"), \d+\)/write_str_in_full($1, $2)/'\
$(git grep -l 'write_in_full.*"')
Signed-off-by: Jim Meyering <meyering@redhat.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
quickfetch() calls rev-list to check whether the objects we are about to
fetch are already present in the repo (if so, we can skip the object fetch).
However, when there are many (~1000) refs to be fetched, the rev-list
command line grows larger than the maximum command line size on some systems
(32K in Windows). This causes rev-list to fail, making quickfetch() return
non-zero, which unnecessarily triggers the transport machinery. This somehow
causes fetch to fail with an exit code.
By using the --stdin option to rev-list (and feeding the object list to its
standard input), we prevent the overflow of the rev-list command line,
which causes quickfetch(), and subsequently the overall fetch, to succeed.
However, using rev-list --stdin is not entirely straightforward: rev-list
terminates immediately when encountering an unknown object, which can
trigger SIGPIPE if we are still writing object's to its standard input.
We therefore temporarily ignore SIGPIPE so that the fetch process is not
terminated.
The patch also contains a testcase to verify the fix (note that before
the patch, the testcase would only fail on msysGit).
Signed-off-by: Johan Herland <johan@herland.net>
Improved-by: Johannes Sixt <j6t@kdbg.org>
Improved-by: Alex Riesen <raa.lkml@gmail.com>
Tested-by: Peter Krefting <peter@softwolves.pp.se>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we fail to store a fetched ref, we recommend that the
user try running "git prune" to remove up any old refs that
have been deleted by the remote, which would clear up any DF
conflicts. However, ref storage might fail for other
reasons (e.g., permissions problems) in which case the
advice is useless and misleading.
This patch detects when there is an actual DF situation and
only issues the advice when one is found.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
To give OPT_FILENAME the prefix, we pass the prefix to parse_options()
which passes the prefix to parse_options_start() which sets the prefix
member of parse_opts_ctx accordingly. If there isn't a prefix in the
calling context, passing NULL will suffice.
Signed-off-by: Stephen Boyd <bebarino@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In preparation to be used when the ref object is not available
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
Acked-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The fmt-merge-msg does a strong syntax checking of its input and fails
with if it is incorrect. The LF character is the only character
important for fmt-merge-msg. As the url in FETCH_HEAD plays only
informational role, a quoted representation of the url should be good
and true enough.
The url often comes from either user-editable config or command line,
so it is reasonable to expect all kinds of characters in it, including
the characters which the format of FETCH_HEAD considers special (line
separator in this case).
Noticed and reported by Hugo Mildenberger.
Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When pulling from a remote, the full URL including username
is by default added to the commit message. Since it adds
very little value but could be used by malicious people to
glean valid usernames (with matching hostnames), we're far
better off just stripping the username before storing the
remote URL locally.
Note that this patch has no lasting visible effect when
"git pull" does not create a merge commit. It simply
alters what gets written to .git/FETCH_HEAD, which is used
by "git merge" to automagically create its messages.
Signed-off-by: Andreas Ericsson <ae@op5.se>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Otherwise, it is hard to guess why the fetch failed.
Make sure we at least mention that the repository must be bare.
Also the current branch is printed.
Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When there's no explicitly-named remote, we use the remote specified
for the current branch, which in turn defaults to "origin". But it
this case should require the remote to actually be configured, and not
fall back to the path "origin".
Possibly, the config file's "remote = something" should require the
something to be a configured remote instead of a bare repository URL,
but we actually test with a bare repository URL.
In fetch, we were giving the sensible error message when coming up
with a URL failed, but this wasn't actually reachable, so move that
error up and use it when appropriate.
In push, we need a new error message, because the old one (formerly
unreachable without a lot of help) used the repo name, which was NULL.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The result should be consistent between fetch and push, so we ought to
use the same code in both cases, even though it's short.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The current code is very inconsistent about which signals
are caught for doing cleanup of temporary files and lock
files. Some callsites checked only SIGINT, while others
checked a variety of death-dealing signals.
This patch factors out those signals to a single function,
and then calls it everywhere. For some sites, that means
this is a simple clean up. For others, it is an improvement
in that they will now properly clean themselves up after a
larger variety of signals.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If a piece of code wanted to do some cleanup before exiting
(e.g., cleaning up a lockfile or a tempfile), our usual
strategy was to install a signal handler that did something
like this:
do_cleanup(); /* actual work */
signal(signo, SIG_DFL); /* restore previous behavior */
raise(signo); /* deliver signal, killing ourselves */
For a single handler, this works fine. However, if we want
to clean up two _different_ things, we run into a problem.
The most recently installed handler will run, but when it
removes itself as a handler, it doesn't put back the first
handler.
This patch introduces sigchain, a tiny library for handling
a stack of signal handlers. You sigchain_push each handler,
and use sigchain_pop to restore whoever was before you in
the stack.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
LF at the end of format strings given to die() is redundant because
die already adds one on its own.
Signed-off-by: Alexander Potashev <aspotashev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Implement git-pull --quiet and git-pull --verbose by
adding the options to git-pull and fixing verbosity
handling in git-fetch.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* js/maint-fetch-update-head:
pull: allow "git pull origin $something:$current_branch" into an unborn branch
Fix fetch/pull when run without --update-head-ok
Conflicts:
t/t5510-fetch.sh
With all calls to alloc_ref() gone, we can remove it and then we're free
to give alloc_ref_from_str() the shorter name. It's a much nicer
interface, as the callers always need to have a name string when they
allocate a ref anyway and don't need to calculate and pass its length+1
any more.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Some confusing tutorials suggested that it would be a good idea to fetch
into the current branch with something like this:
git fetch origin master:master
(or even worse: the same command line with "pull" instead of "fetch").
While it might make sense to store what you want to pull, it typically is
plain wrong when the current branch is "master". This should only be
allowed when (an incorrect) "git pull origin master:master" tries to work
around by giving --update-head-ok to underlying "git fetch", and otherwise
we should refuse it, but somewhere along the lines we lost that behavior.
The check for the current branch is now _only_ performed in non-bare
repositories, which is an improvement from the original behaviour.
Some newer tests were depending on the broken behaviour of "git fetch"
this patch fixes, and have been adjusted.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Acked-by: Shawn O. Pearce <spearce@spearce.org>
Acked-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
User notifications are presented as 'git cmd', and code comments
are presented as '"cmd"' or 'git's cmd', rather than 'git-cmd'.
Signed-off-by: Heikki Orsila <heikki.orsila@iki.fi>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The name path_list was correct for the first usage of that data structure,
but it really is a general-purpose string list.
$ perl -i -pe 's/path-list/string-list/g' $(git grep -l path-list)
$ perl -i -pe 's/path_list/string_list/g' $(git grep -l path_list)
$ git mv path-list.h string-list.h
$ git mv path-list.c string-list.c
$ perl -i -pe 's/has_path/has_string/g' $(git grep -l has_path)
$ perl -i -pe 's/path/string/g' string-list.[ch]
$ git mv Documentation/technical/api-path-list.txt \
Documentation/technical/api-string-list.txt
$ perl -i -pe 's/strdup_paths/strdup_strings/g' $(git grep -l strdup_paths)
... and then fix all users of string-list to access the member "string"
instead of "path".
Documentation/technical/api-string-list.txt needed some rewrapping, too.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When you misuse a git command, you are shown the usage string.
But this is currently shown in the dashed form. So if you just
copy what you see, it will not work, when the dashed form
is no longer supported.
This patch makes git commands show the dash-less version.
For shell scripts that do not specify OPTIONS_SPEC, git-sh-setup.sh
generates a dash-less usage string now.
Signed-off-by: Stephan Beyer <s-beyer@gmx.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
There are basically two categories of update failures for
local refs:
1. problems outside of git, like disk full, bad
permissions, etc.
2. D/F conflicts on tracking branch ref names
In either case, there should already have been an error
message. In case '1', hopefully enough information has
already been given that the user can fix it. In the case of
'2', we can hint that the user can clean up their tracking
branch area by using 'git remote prune'.
Note that we don't actually know _which_ case we have, so
the user will receive the hint in case 1, as well. In this
case the suggestion won't do any good, but hopefully the
user is smart enough to figure out that it's just a hint.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Previously, if there was an error while storing a local
tracking ref, the low-level functions would report an error,
but fetch's status output wouldn't indicate any problem.
E.g., imagine you have an old "refs/remotes/origin/foo/bar" but
upstream has deleted "foo/bar" in favor of a new branch
"foo". You would get output like this:
error: there are still refs under 'refs/remotes/origin/foo'
From $url_of_repo
* [new branch] foo -> origin/foo
With this patch, the output takes into account the status of
updating the local ref:
error: there are still refs under 'refs/remotes/origin/foo'
From $url_of_repo
! [new branch] foo -> origin/foo (unable to update local ref)
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Sync with builtin-fetch--tool.c where append_fetch_head()
honors update_local_ref() return value.
This fixes non fast forward fetch exit status,
http://bugzilla.altlinux.org/show_bug.cgi?id=15037
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* db/clone-in-c:
Add test for cloning with "--reference" repo being a subset of source repo
Add a test for another combination of --reference
Test that --reference actually suppresses fetching referenced objects
clone: fall back to copying if hardlinking fails
builtin-clone.c: Need to closedir() in copy_or_link_directory()
builtin-clone: fix initial checkout
Build in clone
Provide API access to init_db()
Add a function to set a non-default work tree
Allow for having for_each_ref() list extra refs
Have a constant extern refspec for "--tags"
Add a library function to add an alternate to the alternates file
Add a lockfile function to append to a file
Mark the list of refs to fetch as const
Conflicts:
cache.h
t/t5700-clone-reference.sh
Also fix an underallocation in walker.c::interpret_target().
Signed-off-by: Krzysztof Kowalczyk <kkowalczyk@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The refspec refs/tags/*:refs/tags/* is sufficiently common and generic
to merit having a constant instead of generating it as needed.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
As reported by Dave Jones:
Since master.kernel.org updated to latest, I noticed that I could crash
git-fetch by doing this..
export KERNEL=/pub/scm/linux/kernel/git/
git fetch $KERNEL/torvalds/linux-2.6 master:linus
(gdb) bt
0 0x000000349fd6d44b in free () from /lib64/libc.so.6
1 0x000000000048f4eb in transport_unlock_pack (transport=0x7ce530) at transport.c:811
2 0x000000349fd31b25 in exit () from /lib64/libc.so.6
3 0x00000000004043d8 in handle_internal_command (argc=3, argv=0x7fffea4449f0) at git.c:379
4 0x0000000000404547 in main (argc=3, argv=0x7fffea4449f0) at git.c:443
5 0x000000349fd1c784 in __libc_start_main () from /lib64/libc.so.6
6 0x0000000000403ef9 in ?? ()
7 0x00007fffea4449d8 in ?? ()
8 0x0000000000000000 in ?? ()
I then remembered, my .bashrc has this..
export MALLOC_PERTURB_=$(($RANDOM % 255 + 1))
which is handy for showing up such bugs.
More info on this glibc feature is at http://udrepper.livejournal.com/11429.html
Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
