A new flag -q makes underlying pack-objects less chatty.
A new flag -f forces delta to be recomputed from scratch.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This introduces --no-reuse-delta option to disable reusing of
existing delta, which is a large part of the optimization
introduced by this series. This may become necessary if
repeated repacking makes delta chain too long. With this, the
output of the command becomes identical to that of the older
implementation. But the performance suffers greatly.
It still allows reusing non-deltified representations; there is
no point uncompressing and recompressing the whole text.
It also adds a couple more statistics output, while squelching
it under -q flag, which the last round forgot to do.
$ time old-git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects....................
real 12m8.530s user 11m1.450s sys 0m57.920s
$ time git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 138297), reused 178833 (delta 134081)
real 0m59.549s user 0m56.670s sys 0m2.400s
$ time git-pack-objects --stdout --no-reuse-delta >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 134833), reused 47904 (delta 0)
real 11m13.830s user 9m45.240s sys 0m44.330s
There is one remaining issue when --no-reuse-delta option is not
used. It can create delta chains that are deeper than specified.
A<--B<--C<--D E F G
Suppose we have a delta chain A to D (A is stored in full either
in a pack or as a loose object. B is depth1 delta relative to A,
C is depth2 delta relative to B...) with loose objects E, F, G.
And we are going to pack all of them.
B, C and D are left as delta against A, B and C respectively.
So A, E, F, and G are examined for deltification, and let's say
we decided to keep E expanded, and store the rest as deltas like
this:
E<--F<--G<--A
Oops. We ended up making D a bit too deep, didn't we? B, C and
D form a chain on top of A!
This is because we did not know what the final depth of A would
be, when we checked objects and decided to keep the existing
delta. Unfortunately, deferring the decision until just before
the deltification is not an option. To be able to make B, C,
and D candidates for deltification with the rest, we need to
know the type and final unexpanded size of them, but the major
part of the optimization comes from the fact that we do not read
the delta data to do so -- getting the final size is quite an
expensive operation.
To prevent this from happening, we should keep A from being
deltified. But how would we tell that, cheaply?
To do this most precisely, after check_object() runs, each
object that is used as the base object of some existing delta
needs to be marked with the maximum depth of the objects we
decided to keep deltified (in this case, D is depth 3 relative
to A, so if no other delta chain that is longer than 3 based on
A exists, mark A with 3). Then when attempting to deltify A, we
would take that number into account to see if the final delta
chain that leads to D becomes too deep.
However, this is a bit cumbersome to compute, so we would cheat
and reduce the maximum depth for A arbitrarily to depth/4 in
this implementation.
Signed-off-by: Junio C Hamano <junkio@cox.net>
When generating a new pack, notice if we have already needed
objects in existing packs. If an object is stored deltified,
and its base object is also what we are going to pack, then
reuse the existing deltified representation unconditionally,
bypassing all the expensive find_deltas() and try_deltas()
calls.
Also, notice if what we are going to write out exactly match
what is already in an existing pack (either deltified or just
compressed). In such a case, we can just copy it instead of
going through the usual uncompressing & recompressing cycle.
Without this patch, in linux-2.6 repository with about 1500
loose objects and a single mega pack:
$ git-rev-list --objects v2.6.16-rc3 >RL
$ wc -l RL
184141 RL
$ time git-pack-objects p <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects....................
a1fc7b3e537fcb9b3c46b7505df859f0a11e79d2
real 12m4.323s
user 11m2.560s
sys 0m55.950s
With this patch, the same input:
$ time ../git.junio/git-pack-objects q <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
a1fc7b3e537fcb9b3c46b7505df859f0a11e79d2
Total 184141, written 184141, reused 182441
real 1m2.608s
user 0m55.090s
sys 0m1.830s
Signed-off-by: Junio C Hamano <junkio@cox.net>
We run svn log against a URL without a working copy for the first fetch,
so we end up a log that's sorted from highest to lowest. That's bad, we
always want lowest to highest. Just default to --revision 0:HEAD now if
-r isn't specified for the first fetch.
Also sort the revisions after we get them just in case somebody
accidentally reverses the argument to --revision for whatever reason.
Thanks again to Emmanuel Guerin for helping me find this.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Systems using some uClibc versions do not properly support
iconv stuff. This patch allows Git to be built on those
systems by passing NO_ICONV=YesPlease to make. The only
drawback is mailinfo won't do charset conversion in those
systems.
Signed-off-by: Fernando J. Pereda <ferdy@gentoo.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
It's actually very useful for other things too. Notably, we could do the
combined diff a lot more efficiently with this.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
On Tue, 14 Feb 2006, Linus Torvalds wrote:
>
> Here, btw, is the trivial diff to turn my previous "tree-resolve" into a
> "resolve tree relative to the current branch".
Gaah. It was trivial, and it happened to work fine for my test-case, but
when I started looking at not doing that extremely aggressive subdirectory
merging, that showed a few other issues...
So in case people want to try, here's a third patch. Oh, and it's against
my _original_ path, not incremental to the middle one (ie both patches two
and three are against patch #1, it's not a nice series).
Now I'm really done, and won't be sending out any more patches today.
Sorry for the noise.
Linus
Signed-off-by: Junio C Hamano <junkio@cox.net>
On Tue, 14 Feb 2006, Junio C Hamano wrote:
> Linus Torvalds <torvalds@osdl.org> writes:
>
> > If somebody is interested in making the "lots of filename changes" case go
> > fast, I'd be more than happy to walk them through what they'd need to
> > change. I'm just not horribly motivated to do it myself. Hint, hint.
>
> In case anybody is wondering, I share the same feeling. I
> cannot say I'd be "more than happy to" clean up potential
> breakages during the development of such changes, but if the
> change eventually would help certain use cases, I can be
> persuaded to help debugging such a mess ;-).
Actually, I got interested in seeing how hard this is, and wrote a simple
first cut at doing a tree-optimized merger.
Let me shout a bit first:
THIS IS WORKING CODE, BUT BE CAREFUL: IT'S A TECHNOLOGY DEMONSTRATION
RATHER THAN THE FINAL PRODUCT!
With that out of the way, let me descibe what this does (and then describe
the missing parts).
This is basically a three-way merge that works entirely on the "tree"
level, rather than on the index. A lot of the _concepts_ are the same,
though, and if you're familiar with the results of an index merge, some of
the output will make more sense.
You give it three trees: the base tree (tree 0), and the two branches to
be merged (tree 1 and tree 2 respectively). It will then walk these three
trees, and resolve them as it goes along.
The interesting part is:
- it can resolve whole sub-directories in one go, without actually even
looking recursively at them. A whole subdirectory will resolve the same
way as any individual files will (although that may need some
modification, see later).
- if it has a "content conflict", for subdirectories that means "try to
do a recursive tree merge", while for non-subdirectories it's just a
content conflict and we'll output the stage 1/2/3 information.
- a successful merge will output a single stage 0 ("merged") entry,
potentially for a whole subdirectory.
- it outputs all the resolve information on stdout, so something like the
recursive resolver can pretty easily parse it all.
Now, the caveats:
- we probably need to be more careful about subdirectory resolves. The
trivial case (both branches have the exact same subdirectory) is a
trivial resolve, but the other cases ("branch1 matches base, branch2 is
different" probably can't be silently just resolved to the "branch2"
subdirectory state, since it might involve renames into - or out of -
that subdirectory)
- we do not track the current index file at all, so this does not do the
"check that index matches branch1" logic that the three-way merge in
git-read-tree does. The theory is that we'd do a full three-way merge
(ignoring the index and working directory), and then to update the
working tree, we'd do a two-way "git-read-tree branch1->result"
- I didn't actually make it do all the trivial resolve cases that
git-read-tree does. It's a technology demonstration.
Finally (a more serious caveat):
- doing things through stdout may end up being so expensive that we'd
need to do something else. In particular, it's likely that I should
not actually output the "merge results", but instead output a "merge
results as they _differ_ from branch1"
However, I think this patch is already interesting enough that people who
are interested in merging trees might want to look at it. Please keep in
mind that tech _demo_ part, and in particular, keep in mind the final
"serious caveat" part.
In many ways, the really _interesting_ part of a merge is not the result,
but how it _changes_ the branch we're merging into. That's particularly
important as it should hopefully also mean that the output size for any
reasonable case is minimal (and tracks what we actually need to do to the
current state to create the final result).
The code very much is organized so that doing the result as a "diff
against branch1" should be quite easy/possible. I was actually going to do
it, but I decided that it probably makes the output harder to read. I
dunno.
Anyway, let's think about this kind of approach.. Note how the code itself
is actually quite small and short, although it's prbably pretty "dense".
As an interesting test-case, I'd suggest this merge in the kernel:
git-merge-tree $(git-merge-base 4cbf876 7d2babc) 4cbf876 7d2babc
which resolves beautifully (there are no actual file-level conflicts), and
you can look at the output of that command to start thinking about what
it does.
The interesting part (perhaps) is that timing that command for me shows
that it takes all of 0.004 seconds.. (the git-merge-base thing takes
considerably more ;)
The point is, we _can_ do the actual merge part really really quickly.
Linus
PS. Final note: when I say that it is "WORKING CODE", that is obviously by
my standards. IOW, I tested it once and it gave reasonable results - so it
must be perfect.
Whether it works for anybody else, or indeed for any other test-case, is
not my problem ;)
Signed-off-by: Junio C Hamano <junkio@cox.net>
This adds --date-order to rev-list; it is similar to topo order
in the sense that no parent comes before all of its children,
but otherwise things are still ordered in the commit timestamp
order.
The same flag is also added to show-branch.
Signed-off-by: Junio C Hamano <junkio@cox.net>
If Git is compiled with NO_CURL=YesPlease and one tries to
clone a http repository, git-clone tries to call the curl
binary. This trivial patch prints an error instead in such
situation.
Signed-off-by: Fernando J. Pereda <ferdy@gentoo.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This is in the same spirit as an earlier patch for git-commit.
It does an extra ls-files to avoid complaining when a fully
tracked directory name is given on the command line (otherwise
--others restriction would say the pathspec does not match).
Signed-off-by: Junio C Hamano <junkio@cox.net>
Earlier patch mistakenly used prefix_len when it meant
prefix_offset. The latter is to strip the leading directories
when run from a subdirectory.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This allows you to rewrite history a bit more flexibly, by
separating the other branch name and new branch point. By
default, the new branch point is the same as the tip of the
other branch as before, but you can specify where you graft the
rebased branch onto.
When you have this ancestry graph:
A---B---C topic
/
D---E---F---G master
$ git rebase --onto master~1 master topic
would rewrite the history to look like this:
A'\''--B'\''--C'\'' topic
/
D---E---F---G master
Signed-off-by: Junio C Hamano <junkio@cox.net>
When we refused to switch branches, we incorrectly showed
differences from the branch we would have switched to.
Signed-off-by: Junio C Hamano <junkio@cox.net>
When you say "git commit Documentaiton" to make partial commit
for the files only in that directory, we did not detect that as
a misspelled pathname and attempted to commit index without
change. If nothing matched, there is no harm done, but if the
index gets modified otherwise by having another valid pathspec
or after an explicit update-index, a user will not notice
without paying attention to the "git status" preview.
This introduces --error-unmatch option to ls-files, and uses it
to detect this common user error.
Signed-off-by: Junio C Hamano <junkio@cox.net>
New -r flag for prepending the corresponding Subversion revision
number to each commit message.
Signed-off-by: Karl Hasselström <kha@treskal.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
The raw format "git-diff-files -c" to show unmerged state forgot
to initialize the status fields from parents, causing NUL
characters to be emitted.
Signed-off-by: Junio C Hamano <junkio@cox.net>
When showing a conflicted merge from index stages and working
tree file, we did not fetch the mode from the working tree,
and mistook that as a deleted file. Also if the manual
resolution (or automated resolution by git rerere) ended up
taking either parent's version, we did not show _anything_ for
that path. Either was quite bad and confusing.
Signed-off-by: Junio C Hamano <junkio@cox.net>
With the current Makefile we don't use the shell chosen by the
platform specific defines when we invoke GIT-VERSION-GEN.
Signed-off-by: Fredrik Kuivinen <freku045@student.liu.se>
Signed-off-by: Junio C Hamano <junkio@cox.net>
I noticed that we forgot to clean this file and kept it that
way, while trying to help with Andrew's bisect problem.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Since Junio used this in an example, and I've personally tried to use it, I
suppose the option should actually exist.
Signed-off-by: Ryan Anderson <ryan@michonline.com>
The documentation was mistakenly describing the --only semantics to
be default. The 1.2.0 release and its maintenance series 1.2.X will
keep the traditional --include semantics as the default. Clarify the
situation.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This lets a hook to interfere a rebase and help prevent certain
branches from being rebased by mistake. A sample hook to show
how to prevent a topic branch that has already been merged into
publish branch.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This changes the "git commit paths..." to default to --only
semantics from traditional --include semantics, as agreed on the
list.
Signed-off-by: Junio C Hamano <junkio@cox.net>
git-bisect reset without an argument would return to master even
if the bisecting started at a non-master branch. This patch makes
it save the original branch name to .git/head-name and restore it
afterwards.
This is also compatible with Cogito and cg-seek, so cg-status will
show that we are seeked on the bisect branch and cg-reset will
properly restore the original branch.
git-bisect start will refuse to work if it is not on a bisect but
.git/head-name exists; this is to protect against conflicts with
other seeking tools.
Signed-off-by: Petr Baudis <pasky@suse.cz>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Earlier, when we switched a branch we used diff-files to show
paths that are dirty in the working tree. But we allow switching
branches with updated index ("read-tree -m -u $old $new" works that
way), and only showing paths that have differences in the working
tree but not paths that are different in index was confusing.
This shows both as modified from the top commit of the branch we
just have switched to.
Signed-off-by: Junio C Hamano <junkio@cox.net>
You could give -q to squelch it, but currently no tool does it.
This would make 'git clone host:repo here' over ssh not silent
again.
Signed-off-by: Junio C Hamano <junkio@cox.net>
FreeBSD 4.11 being one example: the built-in echo doesn't have -e,
and the installed /bin/echo does not do "-e" as well.
"printf" works, laking just "\e" and "\xAB'.
Signed-off-by: Junio C Hamano <junkio@cox.net>
The hashed object lookup had a subtle bug in re-hashing: it did
for (i = 0; i < count; i++)
if (objs[i]) {
.. rehash ..
where "count" was the old hash couny. Oon the face of it is obvious, since
it clearly re-hashes all the old objects.
However, it's wrong.
If the last old hash entry before re-hashing was in use (or became in use
by the re-hashing), then when re-hashing could have inserted an object
into the hash entries with idx >= count due to overflow. When we then
rehash the last old entry, that old entry might become empty, which means
that the overflow entries should be re-hashed again.
In other words, the loop has to be fixed to either traverse the whole
array, rather than just the old count.
(There's room for a slight optimization: instead of counting all the way
up, we can break when we see the first empty slot that is above the old
"count". At that point we know we don't have any collissions that we might
have to fix up any more. This patch only does the trivial fix)
[jc: with trivial fix on trivial fix]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
