A new flag -q makes underlying pack-objects less chatty.
A new flag -f forces delta to be recomputed from scratch.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This introduces --no-reuse-delta option to disable reusing of
existing delta, which is a large part of the optimization
introduced by this series. This may become necessary if
repeated repacking makes delta chain too long. With this, the
output of the command becomes identical to that of the older
implementation. But the performance suffers greatly.
It still allows reusing non-deltified representations; there is
no point uncompressing and recompressing the whole text.
It also adds a couple more statistics output, while squelching
it under -q flag, which the last round forgot to do.
$ time old-git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects....................
real 12m8.530s user 11m1.450s sys 0m57.920s
$ time git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 138297), reused 178833 (delta 134081)
real 0m59.549s user 0m56.670s sys 0m2.400s
$ time git-pack-objects --stdout --no-reuse-delta >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 134833), reused 47904 (delta 0)
real 11m13.830s user 9m45.240s sys 0m44.330s
There is one remaining issue when --no-reuse-delta option is not
used. It can create delta chains that are deeper than specified.
A<--B<--C<--D E F G
Suppose we have a delta chain A to D (A is stored in full either
in a pack or as a loose object. B is depth1 delta relative to A,
C is depth2 delta relative to B...) with loose objects E, F, G.
And we are going to pack all of them.
B, C and D are left as delta against A, B and C respectively.
So A, E, F, and G are examined for deltification, and let's say
we decided to keep E expanded, and store the rest as deltas like
this:
E<--F<--G<--A
Oops. We ended up making D a bit too deep, didn't we? B, C and
D form a chain on top of A!
This is because we did not know what the final depth of A would
be, when we checked objects and decided to keep the existing
delta. Unfortunately, deferring the decision until just before
the deltification is not an option. To be able to make B, C,
and D candidates for deltification with the rest, we need to
know the type and final unexpanded size of them, but the major
part of the optimization comes from the fact that we do not read
the delta data to do so -- getting the final size is quite an
expensive operation.
To prevent this from happening, we should keep A from being
deltified. But how would we tell that, cheaply?
To do this most precisely, after check_object() runs, each
object that is used as the base object of some existing delta
needs to be marked with the maximum depth of the objects we
decided to keep deltified (in this case, D is depth 3 relative
to A, so if no other delta chain that is longer than 3 based on
A exists, mark A with 3). Then when attempting to deltify A, we
would take that number into account to see if the final delta
chain that leads to D becomes too deep.
However, this is a bit cumbersome to compute, so we would cheat
and reduce the maximum depth for A arbitrarily to depth/4 in
this implementation.
Signed-off-by: Junio C Hamano <junkio@cox.net>
When generating a new pack, notice if we have already needed
objects in existing packs. If an object is stored deltified,
and its base object is also what we are going to pack, then
reuse the existing deltified representation unconditionally,
bypassing all the expensive find_deltas() and try_deltas()
calls.
Also, notice if what we are going to write out exactly match
what is already in an existing pack (either deltified or just
compressed). In such a case, we can just copy it instead of
going through the usual uncompressing & recompressing cycle.
Without this patch, in linux-2.6 repository with about 1500
loose objects and a single mega pack:
$ git-rev-list --objects v2.6.16-rc3 >RL
$ wc -l RL
184141 RL
$ time git-pack-objects p <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects....................
a1fc7b3e537fcb9b3c46b7505df859f0a11e79d2
real 12m4.323s
user 11m2.560s
sys 0m55.950s
With this patch, the same input:
$ time ../git.junio/git-pack-objects q <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
a1fc7b3e537fcb9b3c46b7505df859f0a11e79d2
Total 184141, written 184141, reused 182441
real 1m2.608s
user 0m55.090s
sys 0m1.830s
Signed-off-by: Junio C Hamano <junkio@cox.net>
The real problem triggered an earlier fix was that an alternate
entry was pointing at a removed directory. Complaining on
object/pack directory that cannot be opendir-ed produces noise
in an ancient repository that does not have object/pack
directory and has never been packed.
Detect the real user error and report it. Also if opendir
failed for other reasons (e.g. no read permissions), report that
as well.
Spotted by Andrew Vasquez <andrew.vasquez@qlogic.com>.
Signed-off-by: Junio C Hamano <junkio@cox.net>
It turns out that the git-push documentation didn't describe what it
would do when not given a refspec, (not on the command line, nor in a
remotes file). This is fairly important for the user who is trying to
understand operations such as:
git clone git://something/some/where
# hack, hack, hack
git push origin
I tracked the mystery behavior down to git-send-pack and lifted the
relevant portion of its documentation up to git-push, (namely that all
refs existing both locally and remotely are updated).
Signed-off-by: Carl Worth <cworth@cworth.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This adds support to git-add to allow the common -- to separate
command-line options and file names. It adds documentation and a new
git-add test case as well.
[jc: this should apply to 1.2.X maintenance series, so I reworked
git-ls-files --error-unmatch test. ]
Signed-off-by: Junio C Hamano <junkio@cox.net>
Fixed a couple of bugs in recovering from broken connections:
The _line() method now returns undef correctly when the connection
is broken instead of falling off the function and returning garbage.
Retries are now reported to stderr and the eventual partially
downloaded file is discarded instead of being appended to.
The "Server gone away" test has been removed, because it was
reachable only if the garbage return bug bit.
Signed-off-by: Martin Mares <mj@ucw.cz>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This fixes a bug when importing where a directory gets removed/renamed
but is immediately replaced by a file of the same name in the same
changeset.
This fix only applies to the accurate (default) strategy the moment.
This patch should also fix the fast strategy if/when it is updated
to handle the cases that would've triggered this bug.
This bug was originally found in git-svn, but I remembered I did the
same thing with archimport as well.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Signed-off-by: Junio C Hamano <junkio@cox.net>
When git-reset --hard is used and a subdirectory becomes
empty (as it contains no tracked files in the target tree)
the empty subdirectory should be removed. This matches
the behavior of git-checkout-index and git-read-tree -m
which would not have created the subdirectory or would
have deleted it when updating the working directory.
Subdirectories which are not empty will be left behind.
This may happen if the subdirectory still contains object
files from the user's build process (for example).
[jc: simplified the logic a bit, while keeping the test script.]
If Git is compiled with NO_CURL=YesPlease and one tries to
clone a http repository, git-clone tries to call the curl
binary. This trivial patch prints an error instead in such
situation.
Signed-off-by: Fernando J. Pereda <ferdy@gentoo.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
When we refused to switch branches, we incorrectly showed
differences from the branch we would have switched to.
Signed-off-by: Junio C Hamano <junkio@cox.net>
The raw format "git-diff-files -c" to show unmerged state forgot
to initialize the status fields from parents, causing NUL
characters to be emitted.
Signed-off-by: Junio C Hamano <junkio@cox.net>
When showing a conflicted merge from index stages and working
tree file, we did not fetch the mode from the working tree,
and mistook that as a deleted file. Also if the manual
resolution (or automated resolution by git rerere) ended up
taking either parent's version, we did not show _anything_ for
that path. Either was quite bad and confusing.
Signed-off-by: Junio C Hamano <junkio@cox.net>
With the current Makefile we don't use the shell chosen by the
platform specific defines when we invoke GIT-VERSION-GEN.
Signed-off-by: Fredrik Kuivinen <freku045@student.liu.se>
Signed-off-by: Junio C Hamano <junkio@cox.net>
I noticed that we forgot to clean this file and kept it that
way, while trying to help with Andrew's bisect problem.
Signed-off-by: Junio C Hamano <junkio@cox.net>
The documentation was mistakenly describing the --only semantics to
be default. The 1.2.0 release and its maintenance series 1.2.X will
keep the traditional --include semantics as the default. Clarify the
situation.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Earlier, when we switched a branch we used diff-files to show
paths that are dirty in the working tree. But we allow switching
branches with updated index ("read-tree -m -u $old $new" works that
way), and only showing paths that have differences in the working
tree but not paths that are different in index was confusing.
This shows both as modified from the top commit of the branch we
just have switched to.
Signed-off-by: Junio C Hamano <junkio@cox.net>
You could give -q to squelch it, but currently no tool does it.
This would make 'git clone host:repo here' over ssh not silent
again.
Signed-off-by: Junio C Hamano <junkio@cox.net>
FreeBSD 4.11 being one example: the built-in echo doesn't have -e,
and the installed /bin/echo does not do "-e" as well.
"printf" works, laking just "\e" and "\xAB'.
Signed-off-by: Junio C Hamano <junkio@cox.net>
The hashed object lookup had a subtle bug in re-hashing: it did
for (i = 0; i < count; i++)
if (objs[i]) {
.. rehash ..
where "count" was the old hash couny. Oon the face of it is obvious, since
it clearly re-hashes all the old objects.
However, it's wrong.
If the last old hash entry before re-hashing was in use (or became in use
by the re-hashing), then when re-hashing could have inserted an object
into the hash entries with idx >= count due to overflow. When we then
rehash the last old entry, that old entry might become empty, which means
that the overflow entries should be re-hashed again.
In other words, the loop has to be fixed to either traverse the whole
array, rather than just the old count.
(There's room for a slight optimization: instead of counting all the way
up, we can break when we see the first empty slot that is above the old
"count". At that point we know we don't have any collissions that we might
have to fix up any more. This patch only does the trivial fix)
[jc: with trivial fix on trivial fix]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Calling hashtable_index from find_object before objs is created
would result in division by zero failure. Avoid it.
Also the given object name may not be aligned suitably for
unsigned int; avoid dereferencing casted pointer.
Signed-off-by: Junio C Hamano <junkio@cox.net>
In a simple test, this brings down the CPU time from 47 sec to 22 sec.
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This howto consists of a footnote from an email by JC to the git
mailing list (<7vfyms0x4p.fsf@assigned-by-dhcp.cox.net>).
Signed-off-by: Kent Engstrom <kent@lysator.liu.se>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This is to be nicer to people with unusable GECOS field.
"git-var -l" is currently broken in that when used by a user who
does not have a usable GECOS field and has not corrected it by
exporting GIT_COMMITTER_NAME environment variable it dies when
it tries to output GIT_COMMITTER_IDENT (same thing for AUTHOR).
"git-pull" used "git-var -l" only because it needed to get a
configuration variable before "git-repo-config --get" was
introduced. Use the latter tool designed exactly for this
purpose.
"git-sh-setup" used "git-var GIT_AUTHOR_IDENT" without actually
wanting to use its value. The only purpose was to cause the
command to check and barf if the repository format version
recorded in the $GIT_DIR/config file is too new for us to deal
with correctly. Instead, use "repo-config --get" on a random
property and see if it die()s, and check if the exit status is
128 (comes from die -- missing variable is reported with exit
status 1, so we can tell that case apart).
Signed-off-by: Junio C Hamano <junkio@cox.net>
Currently, git-repo-config will just return the raw value of option
as specified in the config file; this makes things difficult for scripts
calling it, especially if the value is supposed to be boolean.
This patch makes it possible to ask git-repo-config to check if the option
is of the given type (int or bool) and write out the value in its
canonical form. If you do not pass --int or --bool, the behaviour stays
unchanged and the raw value is emitted.
This also incidentally fixes the segfault when option with no value is
encountered.
[jc: tweaked the option parsing a bit to make it easier to see
that the patch does not change anything but the type stuff in
the diff output. Also changed to avoid "foo ? : bar" construct. ]
Signed-off-by: Petr Baudis <pasky@suse.cz>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Marco says it breaks qgit. This makes the flags a bit more
orthogonal.
$ git-diff-tree -r --abbrev ca18
No output from this command because you asked to skip merge by
not having -m there.
$ git-diff-tree -r -m --abbrev ca18
ca182053c7
:100644 100644 538d21d... 59042d1... M Makefile
:100644 100644 410b758... 6c47c3a... M entry.c
ca182053c7
:100644 100644 30479b4... 59042d1... M Makefile
The same "independent sets of diff" as before without -c.
$ git-diff-tree -r -m -c --abbrev ca18
ca182053c7
::100644 100644 100644 538d21d... 30479b4... 59042d1... MM Makefile
Combined.
$ git-diff-tree -r -c --abbrev ca18
ca182053c7
::100644 100644 100644 538d21d... 30479b4... 59042d1... MM Makefile
Asking for combined without -m does not make sense, so -c
implies -m.
We need to supply -c as default to whatchanged, which is a
one-liner.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Now pack-object is not as chatty when its stderr is not connected
to a terminal, so the test needs to be adjusted for that.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Johannes noticed that git-rerere depends on Digest.pm, and if
one does not use the command, one can live without it.
Signed-off-by: Junio C Hamano <junkio@cox.net>
The absolute path (with the leading slash) breaks SVN importing,
because it then looks for /trunk/... instead of /svn/trunk/...
(in my case, the repository URL was https://servername/svn/)
Signed-off-by: Christian Biesinger <cbiesinger@web.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
We shouldn't fail a fetch just because a signal might have interrupted
the read.
Normally, we don't install any signal handlers, so EINTR really shouldn't
happen. That said, really old versions of Linux will interrupt an
interruptible system call even for signals that turn out to be ignored
(SIGWINCH is the classic example - resizing your xterm would cause it).
The same might well be true elsewhere too.
Also, since receive_keep_pack() doesn't control the caller, it can't know
that no signal handlers exist.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Average it out over a few events to make the numbers stable, and fix the
silly usec->binary-ms conversion.
Yeah, yeah, it's arguably eye-candy to keep the user calm, but let's do
that right.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
It used to be that "git-unpack-objects" would give nice percentages, but
now that we don't unpack the initial clone pack any more, it doesn't. And
I'd love to do that nice percentage view in the pack objects downloader
too, but the thing doesn't even read the pack header, much less know how
much it's going to get, so I was lazy and didn't.
Instead, it at least prints out how much data it's gotten, and what the
packing speed is. Which makes the user realize that it's actually doing
something useful instead of sitting there silently (and if the recipient
knows how large the final result is, he can at least make a guess about
when it migt be done).
So with this patch, I get something like this on my DSL line:
[torvalds@g5 ~]$ time git clone master.kernel.org:/pub/scm/linux/kernel/git/torvalds/linux-2.6 clone-test
Packing 188543 objects
48.398MB (154 kB/s)
where even the speed approximation seems to be roughtly correct (even
though my algorithm is a truly stupid one, and only really gives "speed in
the last half second or so").
Anyway, _something_ like this is definitely needed. It could certainly be
better (if it showed the same kind of thing that git-unpack-objects did,
that would be much nicer, but would require parsing the object stream as
it comes in). But this is big step forward, I think.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Without these, running tests with an account with empty gecos
field would fail.
We might want to loosen error from "git-var -l" (but not
"git-var GIT_AUTHOR_NAME") later, but that is more or less an
independent issue.
Signed-off-by: Junio C Hamano <junkio@cox.net>
* lt/diff-tree:
combine-diff: Record diff status a bit more faithfully
find_unique_abbrev() simplification.
combine-diff: move formatting logic to show_combined_diff()
combined-diff: use diffcore before intersecting paths.
diff-tree -c raw output
It was pointed out that otherwise more important summary
information prefixed with '#' would become prone to be missed.
Also instead of chopping at the first '^---$' line, stop at the
first 'diff --git a/' line.
Signed-off-by: Junio C Hamano <junkio@cox.net>
When we prettyprint commit log messages, merge parent names were
often very long and there was no way to abbreviate it.
This changes them to be abbreviated by default, and non-default
abbreviations can be specified with --no-abbrev or --abbrev=<n>
options.
Note that this affects only the prettyprinted parent names. The
output from --show-parents is meant for machine consumption and
is not affected by this flag.
