Cloning from a shallow repository does not follow the "8 steps for new
.git/shallow" because if it does we need to get through step 6 for all
refs. That means commit walking down to the bottom.
Instead the rule to create .git/shallow is simpler and, more
importantly, cheap: if a shallow commit is found in the pack, it's
probably used (i.e. reachable from some refs), so we add it. Others
are dropped.
One may notice this method seems flawed by the word "probably". A
shallow commit may not be reachable from any refs at all if it's
attached to an object island (a group of objects that are not
reachable by any refs).
If that object island is not complete, a new fetch request may send
more objects to connect it to some ref. At that time, because we
incorrectly installed the shallow commit in this island, the user will
not see anything after that commit (fsck is still ok). This is not
desired.
Given that object islands are rare (C Git never sends such islands for
security reasons) and do not really harm the repository integrity, a
tradeoff is made to surprise the user occasionally but work faster
everyday.
A new option --strict could be added later that follows exactly the 8
steps. "git prune" can also learn to remove dangling objects _and_ the
shallow commits that are attached to them from .git/shallow.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If we borrow objects from another repository, we should also pay
attention to their $GIT_DIR/shallow (and even info/grafts). But
current alternates code does not.
Reject alternate repos that are shallow because we do not do it
right. In future the alternate code may be updated to check
$GIT_DIR/shallow properly so that this restriction could be lifted.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Some progress and diagnostic messages from "git clone" were
incorrectly sent to the standard output stream, not to the standard
error stream.
* jk/clone-progress-to-stderr:
clone: always set transport options
clone: treat "checking connectivity" like other progress
clone: send diagnostic messages to stderr
Since 920b691 (clone: refuse to clone if --branch
points to bogus ref) we refuse to clone with option
"-b" if the specified branch does not exist in the
(non-empty) upstream. If the upstream repository is empty,
the branch doesn't exist, either. So refuse the clone too.
Reported-by: Robert Mitwicki <robert.mitwicki@opensoftware.pl>
Signed-off-by: Ralf Thielow <ralf.thielow@gmail.com>
Acked-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
We have a period in other places after "done" (see e.g. clone_local), so
we should have one here, too.
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
A clone will always create a transport struct, whether we
are cloning locally or using an actual protocol. In the
local case, we only use the transport to get the list of
refs, and then transfer the objects out-of-band.
However, there are many options that we do not bother
setting up in the local case. For the most part, these are
noops, because they only affect the object-fetching stage
(e.g., the --depth option). However, some options do have a
visible impact. For example, giving the path to upload-pack
via "-u" does not currently work for a local clone, even
though we need upload-pack to get the ref list.
We can just drop the conditional entirely and set these
options for both local and non-local clones. Rather than
keep track of which options impact the object versus the ref
fetching stage, we can simply let the noops be noops (and
the cost of setting the options in the first place is not
high).
The one exception is that we also check that the transport
provides both a "get_refs_list" and a "fetch" method. We
will now be checking the former for both cases (which is
good, since a transport that cannot fetch refs would not
work for a local clone), and we tweak the conditional to
check for a "fetch" only when we are non-local.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When stderr does not point to a tty, we typically suppress
"we are now in this phase" progress reporting (e.g., we ask
the server not to send us "counting objects" and the like).
The new "checking connectivity" message is in the same vein,
and should be suppressed. Since clone relies on the
transport code to make the decision, we can simply sneak a
peek at the "progress" field of the transport struct. That
properly takes into account both the verbosity and progress
options we were given, as well as the result of isatty().
Note that we do not set up that progress flag for a local
clone, as we do not fetch using the transport at all. That's
acceptable here, though, because we also do not perform a
connectivity check in that case.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Putting messages like "Cloning into.." and "done" on stdout
is un-Unix and uselessly clutters the stdout channel. Send
them to stderr.
We have to tweak two tests to accommodate this:
1. t5601 checks for doubled output due to forking, and
doesn't actually care where the output goes; adjust it
to check stderr.
2. t5702 is trying to test whether progress output was
sent to stderr, but naively does so by checking
whether stderr produced any output. Instead, have it
look for "%", a token found in progress output but not
elsewhere (and which lets us avoid hard-coding the
progress text in the test).
This should not regress any scripts that try to parse the
current output, as the output is already internationalized
and therefore unstable.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This task emerged from b04ba2bb (parse-options: deprecate OPT_BOOLEAN,
2011-09-27). All occurrences of the respective variables have
been reviewed and none of them relied on the counting up mechanism,
but all of them were using the variable as a true boolean.
This patch does not change semantics of any command intentionally.
Signed-off-by: Stefan Beller <stefanbeller@googlemail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
As of b04ba2bb4 OPTION_BOOLEAN was deprecated.
This commit removes all occurrences of OPTION_BOOLEAN.
In b04ba2bb4 Junio suggested to replace it with either
OPTION_SET_INT or OPTION_COUNTUP instead. However a pattern, which
occurred often with the OPTION_BOOLEAN was a hidden boolean parameter.
So I defined OPT_HIDDEN_BOOL as an additional possible parse option
in parse-options.h to make life easy.
The OPT_HIDDEN_BOOL was used in checkout, clone, commit, show-ref.
The only exception, where there was need to fiddle with OPTION_SET_INT
was log and notes. However in these two files there is also a pattern,
so we could think of introducing OPT_NONEG_BOOL.
Signed-off-by: Stefan Beller <stefanbeller@googlemail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git clone -s/-l" is a filesystem level copy and does not offer any
protection against source repository being corrupt. While the
connectivity validation checks commits and trees being readable, it
made the otherwise instantaneous local modes of clone much more
expensive, without protecting blob data from bitflips.
* jk/maint-clone-shared-no-connectivity-validation:
clone: drop connectivity check for local clones
Commit 0433ad1 (clone: run check_everything_connected,
2013-03-25) added the same connectivity check to clone that
we use for fetching. The intent was to provide enough safety
checks that "git clone git://..." could be counted on to
detect bit errors and other repo corruption, and not
silently propagate them to the clone.
For local clones, this turns out to be a bad idea, for two
reasons:
1. Local clones use hard linking (or even shared object
stores), and so complete far more quickly. The time
spent on the connectivity check is therefore
proportionally much more painful.
2. Local clones do not actually meet our safety guarantee
anyway. The connectivity check makes sure we have all
of the objects we claim to, but it does not check for
bit errors. We will notice bit errors in commits and
trees, but we do not load blob objects at all. Whereas
over the pack transport, we actually recompute the sha1
of each object in the incoming packfile; bit errors
change the sha1 of the object, which is then caught by
the connectivity check.
This patch drops the connectivity check in the local case.
Note that we have to revert the changes from 0433ad1 to
t5710, as we no longer notice the corruption during clone.
We could go a step further and provide a "verify even local
clones" option, but it is probably not worthwhile. You can
already spell that as "cd foo.git && git fsck && git clone ."
or as "git clone --no-local foo.git".
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Fix a typo ("remote remote-tracking") going back to the big cleanup
in 2010 (8b3f3f84 etc). Also, remove some more occurrences of
"tracking" and "remote tracking" in favor of "remote-tracking".
Signed-off-by: Michael Schubert <mschub@elegosoft.com>
Reviewed-by: Johan Herland <johan@herland.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Handle simple transactions for the packed-refs file at the
packed_ref_cache level via new functions lock_packed_refs(),
commit_packed_refs(), and rollback_packed_refs().
Only allow the packed ref cache to be modified (via add_packed_ref())
while the packed refs file is locked.
Change clone to add the new references within a transaction.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Special case "git clone" and use lighter-weight implementation to
check the completeness of the history behind refs.
* nd/clone-connectivity-shortcut:
clone: open a shortcut for connectivity check
index-pack: remove dead code (it should never happen)
fetch-pack: prepare updated shallow file before fetching the pack
clone: let the user know when check_everything_connected is run
"git clone foo/bar:baz" cannot be a request to clone from a remote
over git-over-ssh specified in the scp style. Detect this case and
clone from a local repository at "foo/bar:baz".
* nd/clone-local-with-colon:
clone: allow cloning local paths with colons in them
Update reading and updating packed-refs file, correcting corner case
bugs.
* mh/packed-refs-various: (33 commits)
refs: handle the main ref_cache specially
refs: change do_for_each_*() functions to take ref_cache arguments
pack_one_ref(): do some cheap tests before a more expensive one
pack_one_ref(): use write_packed_entry() to do the writing
pack_one_ref(): use function peel_entry()
refs: inline function do_not_prune()
pack_refs(): change to use do_for_each_entry()
refs: use same lock_file object for both ref-packing functions
pack_one_ref(): rename "path" parameter to "refname"
pack-refs: merge code from pack-refs.{c,h} into refs.{c,h}
pack-refs: rename handle_one_ref() to pack_one_ref()
refs: extract a function write_packed_entry()
repack_without_ref(): write peeled refs in the rewritten file
t3211: demonstrate loss of peeled refs if a packed ref is deleted
refs: change how packed refs are deleted
search_ref_dir(): return an index rather than a pointer
repack_without_ref(): silence errors for dangling packed refs
t3210: test for spurious error messages for dangling packed refs
refs: change the internal reference-iteration API
refs: extract a function peel_entry()
...
In order to make sure the cloned repository is good, we run "rev-list
--objects --not --all $new_refs" on the repository. This is expensive
on large repositories. This patch attempts to mitigate the impact in
this special case.
In the "good" clone case, we only have one pack. If all of the
following are met, we can be sure that all objects reachable from the
new refs exist, which is the intention of running "rev-list ...":
- all refs point to an object in the pack
- there are no dangling pointers in any object in the pack
- no objects in the pack point to objects outside the pack
The second and third checks can be done with the help of index-pack as
a slight variation of --strict check (which introduces a new condition
for the shortcut: pack transfer must be used and the number of objects
large enough to call index-pack). The first is checked in
check_everything_connected after we get an "ok" from index-pack.
"index-pack + new checks" is still faster than the current "index-pack
+ rev-list", which is the whole point of this patch. If any of the
conditions fail, we fall back to the good old but expensive "rev-list
..". In that case it's even more expensive because we have to pay for
the new checks in index-pack. But that should only happen when the
other side is either buggy or malicious.
Cloning linux-2.6 over file://
before after
real 3m25.693s 2m53.050s
user 5m2.037s 4m42.396s
sys 0m13.750s 0m16.574s
A more realistic test with ssh:// over wireless
before after
real 11m26.629s 10m4.213s
user 5m43.196s 5m19.444s
sys 0m35.812s 0m37.630s
This shortcut is not applied to shallow clones, partly because shallow
clones should have no more objects than a usual fetch and the cost of
rev-list is acceptable, partly to avoid dealing with corner cases when
grafting is involved.
This shortcut does not apply to unpack-objects code path either
because the number of objects must be small in order to trigger that
code path.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
check_everything_connected could take a long time, especially in the
clone case where the whole DAG is traversed. The user deserves to know
what's going on.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Usually "foo:bar" is interpreted as an ssh url. This patch allows to
clone from such paths by putting at least one slash before the colon
(i.e. /path/to/foo:bar or just ./foo:bar).
file://foo:bar should also work, but local optimizations are off in
that case, which may be unwanted. While at there, warn the users about
--local being ignored in this case.
Reported-by: William Giokas <1007380@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
pack-refs.c doesn't contain much code, and the code it does contain is
closely related to reference handling. Moreover, there is some
duplication between pack_refs() and repack_without_ref(). Therefore,
merge pack-refs.c into refs.c and pack-refs.h into refs.h.
The code duplication will be addressed in future commits.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Sparse issues an "'junk_mode' not declared. Should it be static?"
warning. In order to suppress the warning, since this symbol does
not need more than file visibility, we simply add the static
modifier to its declaration.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Acked-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Try reading gitfile files when processing --reference options to clone.
This will allow, among other things, using a submodule checked out with
a recent version of git as a reference repository without requiring the
user to have internal knowledge of submodule layout.
Signed-off-by: Aaron Schrab <aaron@schrab.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Do not report that an argument to clone's --reference option is not a
local directory. Nothing checks for the existence or type of the path
as supplied by the user; checks are only done for particular contents of
the supposed directory, so we have no way to know the status of the
supplied path. Telling the user that a directory doesn't exist when
that isn't actually known may lead him or her on the wrong path to
finding the problem.
Instead just state that the entered path is not a local repository which
is really all that is known about it. It could be more helpful to state
the actual paths which were checked, but I believe that giving a good
description of that would be too verbose for a simple error message and
would be too dependent on implementation details.
Signed-off-by: Aaron Schrab <aaron@schrab.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If we manage to clone a remote repository but run into an
error in the checkout, it is probably sane to leave the repo
directory in place. That lets the user examine the situation
without spending time to re-clone from the remote (which may
be a lengthy process).
Rather than try to convert each die() from the checkout code
path into an error(), we simply set a flag that tells the
"remove_junk" atexit function to print a helpful message and
leave the repo in place.
Note that the test added in this patch actually passes
without the code change. The reason is that the cleanup code
is buggy; we chdir into the working tree for the checkout,
but still may use relative paths to remove the directories
(which means if you cloned into "foo", we would accidentally
remove "foo" from the working tree!). There's no point in
fixing it now, since this patch means we will never try to
remove anything after the chdir, anyway.
[jc: replaced the message with a more succinct version from
Jonathan]
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we fetch from a remote, we do a revision walk to make
sure that what we received is connected to our existing
history. We do not do the same check for clone, which should
be able to check that we received an intact history graph.
The upside of this patch is that it will make clone more
resilient against propagating repository corruption. The
downside is that we will now traverse "rev-list --objects
--all" down to the roots, which may take some time (it is
especially noticeable for a "--local --bare" clone).
Note that we need to adjust t5710, which tries to make such
a bogus clone. Rather than checking after the fact that our
clone is bogus, we can simplify it to just make sure "git
clone" reports failure.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When clone is populating the working tree, it ignores the
return status from unpack_trees; this means we may report a
successful clone, even when the checkout fails.
When checkout fails, we may want to leave the $GIT_DIR in
place, as it might be possible to recover the data through
further use of "git checkout" (e.g., if the checkout failed
due to a transient error, disk full, etc). However, we
already die on a number of other checkout-related errors, so
this patch follows that pattern.
In addition to marking a now-passing test, we need to adjust
t5710, which blindly assumed it could make bogus clones of
very deep alternates hierarchies. By using "--bare", we can
avoid it actually touching any objects.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We stopped paying attention to $GIT_CONFIG environment that points
at a single configuration file from any command other than "git config"
quite a while ago, but "git clone" internally set, exported, and
then unexported the variable during its operation unnecessarily.
* jc/no-git-config-in-clone:
clone: do not export and unexport GIT_CONFIG
When "git clone --separate-git-dir=$over_there" is interrupted, it
failed to remove the real location of the $GIT_DIR it created. This
was most visible when interrupting a submodule update.
* jl/interrupt-clone-remove-separate-git-dir:
clone: support atomic operation with --separate-git-dir
Earlier, dc87183 (use GIT_CONFIG only in "git config", not other
programs, 2008-06-30) made sure that the environment variable is
never used outside "git config", but "git clone", after creating a
directory for the new repository and until the init_db() function
populates its .git/ directory, exported the variable for no good
reason. No hook will run from init_db() and more importantly no
hook can run until init_db() finishes creation of the new
repository, so it cannot be used by any invocation of "git config"
by definition.
Stop doing the useless export/unexport.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The --separate-git-dir option was introduced to make it simple to put
the git directory somewhere outside the worktree, for example when
cloning a repository for use as a submodule.
It was not intended for use when creating a bare repository. In that
case there is no worktree and it is more natural to directly clone the
repository and create a .git file as separate steps:
git clone --bare /path/to/repo.git bar.git
printf 'gitdir: bar.git\n' >foo.git
Forbid the combination, making the command easier to explain.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When "git clone --separate-git-dir" is interrupted, we failed to
remove the real location we created the repository.
* jl/interrupt-clone-remove-separate-git-dir:
clone: support atomic operation with --separate-git-dir
Since b57fb80a7d (init, clone: support --separate-git-dir for .git file)
git clone supports the --separate-git-dir option to create the git dir
outside the work tree. But when that option is used, the git dir won't be
deleted in case the clone fails like it would be without this option. This
makes clone lose its atomicity as in case of a failure a partly set up git
dir is left behind. A real world example where this leads to problems is
when "git submodule update" fails to clone a submodule and later calls to
"git submodule update" stumble over the partially set up git dir and try
to revive the submodule from there, which then fails with a not very user
friendly error message.
Fix that by updating the junk_git_dir variable (used to remember if and
what git dir should be removed in case of failure) to the new value given
with the --seperate-git-dir option. Also add a test for this to t5600 (and
while at it fix the former last test to not cd into a directory to test
for its existence but use "test -d" instead).
Reported-by: Manlio Perillo <manlio.perillo@gmail.com>
Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A repository created with "git clone --single" had its fetch
refspecs set up just like a clone without "--single", leading the
subsequent "git fetch" to slurp all the other branches, defeating
the whole point of specifying "only this branch".
* rt/maint-clone-single:
clone --single: limit the fetch refspec to fetched branch
Running "git fetch" in a repository made with "git clone --single"
slurps all the branches, defeating the point of "--single".
* rt/maint-clone-single:
clone --single: limit the fetch refspec to fetched branch
After running "git clone --single", the resulting repository has the
usual default "+refs/heads/*:refs/remotes/origin/*" wildcard fetch
refspec installed, which means that a subsequent "git fetch" will
end up grabbing all the other branches.
Update the fetch refspec to cover only the singly cloned ref instead
to correct this.
That means:
If "--single" is used without "--branch" or "--mirror", the
fetch refspec covers the branch on which remote's HEAD points to.
If "--single" is used with "--branch", it'll cover only the branch
specified in the "--branch" option.
If "--single" is combined with "--mirror", then it'll cover all
refs of the cloned repository.
If "--single" is used with "--branch" that specifies a tag, then
it'll cover only the ref for this tag.
Signed-off-by: Ralf Thielow <ralf.thielow@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* rj/path-cleanup:
Call mkpathdup() rather than xstrdup(mkpath(...))
Call git_pathdup() rather than xstrdup(git_path("..."))
path.c: Use vsnpath() in the implementation of git_path()
path.c: Don't discard the return value of vsnpath()
path.c: Remove the 'git_' prefix from a file scope function
In addition to updating the xstrdup(mkpath(...)) call sites with
mkpathdup(), we also fix a memory leak (in merge_3way()) caused by
neglecting to free the memory allocated to the 'base_name' variable.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A handful of files and directories we create had tighter than
necessary permission bits when the user wanted to have group
writability (e.g. by setting "umask 002").
* ar/clone-honor-umask-at-top:
add: create ADD_EDIT.patch with mode 0666
rerere: make rr-cache fanout directory honor umask
Restore umasks influence on the permissions of work tree created by clone
A handful of files and directories we create had tighter than
necessary permission bits when the user wanted to have group
writability (e.g. by setting "umask 002").
* ar/clone-honor-umask-at-top:
add: create ADD_EDIT.patch with mode 0666
rerere: make rr-cache fanout directory honor umask
Restore umasks influence on the permissions of work tree created by clone
The original version of the git-clone just used mkdir(1) to create
the working directories. The version rewritten in C creates all
directories inside the working tree by using the mode argument of
0777 when calling mkdir(2) to let the umask take effect.
But the top-level directory of the working tree is created by
passing the mode argument of 0755 to mkdir(2), which results in an
overly tight restriction if the user wants to make directories group
writable with a looser umask like 002.
Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git clone --single-branch" to clone a single branch did not limit
the cloning to the specified branch.
* nd/clone-single-fix:
clone: fix ref selection in --single-branch --branch=xxx
- do not fetch HEAD
- do not also fetch refs following "xxx"
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This is basically the same as using "file://", but is a
little less subtle for the end user. It also allows relative
paths to be specified.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
