2005-07-14 09:08:05 +02:00
|
|
|
git-pack-objects(1)
|
|
|
|
|
===================
|
|
|
|
|
|
|
|
|
|
NAME
|
|
|
|
|
----
|
2006-03-09 17:24:50 +01:00
|
|
|
git-pack-objects - Create a packed archive of objects
|
2005-07-14 09:08:05 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
SYNOPSIS
|
|
|
|
|
--------
|
pack-objects: finishing touches.
This introduces --no-reuse-delta option to disable reusing of
existing delta, which is a large part of the optimization
introduced by this series. This may become necessary if
repeated repacking makes delta chain too long. With this, the
output of the command becomes identical to that of the older
implementation. But the performance suffers greatly.
It still allows reusing non-deltified representations; there is
no point uncompressing and recompressing the whole text.
It also adds a couple more statistics output, while squelching
it under -q flag, which the last round forgot to do.
$ time old-git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects....................
real 12m8.530s user 11m1.450s sys 0m57.920s
$ time git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 138297), reused 178833 (delta 134081)
real 0m59.549s user 0m56.670s sys 0m2.400s
$ time git-pack-objects --stdout --no-reuse-delta >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 134833), reused 47904 (delta 0)
real 11m13.830s user 9m45.240s sys 0m44.330s
There is one remaining issue when --no-reuse-delta option is not
used. It can create delta chains that are deeper than specified.
A<--B<--C<--D E F G
Suppose we have a delta chain A to D (A is stored in full either
in a pack or as a loose object. B is depth1 delta relative to A,
C is depth2 delta relative to B...) with loose objects E, F, G.
And we are going to pack all of them.
B, C and D are left as delta against A, B and C respectively.
So A, E, F, and G are examined for deltification, and let's say
we decided to keep E expanded, and store the rest as deltas like
this:
E<--F<--G<--A
Oops. We ended up making D a bit too deep, didn't we? B, C and
D form a chain on top of A!
This is because we did not know what the final depth of A would
be, when we checked objects and decided to keep the existing
delta. Unfortunately, deferring the decision until just before
the deltification is not an option. To be able to make B, C,
and D candidates for deltification with the rest, we need to
know the type and final unexpanded size of them, but the major
part of the optimization comes from the fact that we do not read
the delta data to do so -- getting the final size is quite an
expensive operation.
To prevent this from happening, we should keep A from being
deltified. But how would we tell that, cheaply?
To do this most precisely, after check_object() runs, each
object that is used as the base object of some existing delta
needs to be marked with the maximum depth of the objects we
decided to keep deltified (in this case, D is depth 3 relative
to A, so if no other delta chain that is longer than 3 based on
A exists, mark A with 3). Then when attempting to deltify A, we
would take that number into account to see if the final delta
chain that leads to D becomes too deep.
However, this is a bit cumbersome to compute, so we would cheat
and reduce the maximum depth for A arbitrarily to depth/4 in
this implementation.
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-02-16 20:55:51 +01:00
|
|
|
[verse]
|
2006-10-10 10:06:20 +02:00
|
|
|
'git-pack-objects' [-q] [--no-reuse-delta] [--delta-base-offset] [--non-empty]
|
pack-objects: finishing touches.
This introduces --no-reuse-delta option to disable reusing of
existing delta, which is a large part of the optimization
introduced by this series. This may become necessary if
repeated repacking makes delta chain too long. With this, the
output of the command becomes identical to that of the older
implementation. But the performance suffers greatly.
It still allows reusing non-deltified representations; there is
no point uncompressing and recompressing the whole text.
It also adds a couple more statistics output, while squelching
it under -q flag, which the last round forgot to do.
$ time old-git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects....................
real 12m8.530s user 11m1.450s sys 0m57.920s
$ time git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 138297), reused 178833 (delta 134081)
real 0m59.549s user 0m56.670s sys 0m2.400s
$ time git-pack-objects --stdout --no-reuse-delta >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 134833), reused 47904 (delta 0)
real 11m13.830s user 9m45.240s sys 0m44.330s
There is one remaining issue when --no-reuse-delta option is not
used. It can create delta chains that are deeper than specified.
A<--B<--C<--D E F G
Suppose we have a delta chain A to D (A is stored in full either
in a pack or as a loose object. B is depth1 delta relative to A,
C is depth2 delta relative to B...) with loose objects E, F, G.
And we are going to pack all of them.
B, C and D are left as delta against A, B and C respectively.
So A, E, F, and G are examined for deltification, and let's say
we decided to keep E expanded, and store the rest as deltas like
this:
E<--F<--G<--A
Oops. We ended up making D a bit too deep, didn't we? B, C and
D form a chain on top of A!
This is because we did not know what the final depth of A would
be, when we checked objects and decided to keep the existing
delta. Unfortunately, deferring the decision until just before
the deltification is not an option. To be able to make B, C,
and D candidates for deltification with the rest, we need to
know the type and final unexpanded size of them, but the major
part of the optimization comes from the fact that we do not read
the delta data to do so -- getting the final size is quite an
expensive operation.
To prevent this from happening, we should keep A from being
deltified. But how would we tell that, cheaply?
To do this most precisely, after check_object() runs, each
object that is used as the base object of some existing delta
needs to be marked with the maximum depth of the objects we
decided to keep deltified (in this case, D is depth 3 relative
to A, so if no other delta chain that is longer than 3 based on
A exists, mark A with 3). Then when attempting to deltify A, we
would take that number into account to see if the final delta
chain that leads to D becomes too deep.
However, this is a bit cumbersome to compute, so we would cheat
and reduce the maximum depth for A arbitrarily to depth/4 in
this implementation.
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-02-16 20:55:51 +01:00
|
|
|
[--local] [--incremental] [--window=N] [--depth=N]
|
2006-09-13 07:59:15 +02:00
|
|
|
[--revs [--unpacked | --all]*] [--stdout | base-name] < object-list
|
2005-07-14 09:08:05 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
DESCRIPTION
|
|
|
|
|
-----------
|
|
|
|
|
Reads list of objects from the standard input, and writes a packed
|
|
|
|
|
archive with specified base-name, or to the standard output.
|
|
|
|
|
|
|
|
|
|
A packed archive is an efficient way to transfer set of objects
|
|
|
|
|
between two repositories, and also is an archival format which
|
|
|
|
|
is efficient to access. The packed archive format (.pack) is
|
|
|
|
|
designed to be unpackable without having anything else, but for
|
|
|
|
|
random access, accompanied with the pack index file (.idx).
|
|
|
|
|
|
|
|
|
|
'git-unpack-objects' command can read the packed archive and
|
|
|
|
|
expand the objects contained in the pack into "one-file
|
|
|
|
|
one-object" format; this is typically done by the smart-pull
|
|
|
|
|
commands when a pack is created on-the-fly for efficient network
|
|
|
|
|
transport by their peers.
|
|
|
|
|
|
2005-08-05 17:05:02 +02:00
|
|
|
Placing both in the pack/ subdirectory of $GIT_OBJECT_DIRECTORY (or
|
2005-07-14 09:08:05 +02:00
|
|
|
any of the directories on $GIT_ALTERNATE_OBJECT_DIRECTORIES)
|
2005-10-11 01:01:31 +02:00
|
|
|
enables git to read from such an archive.
|
2005-07-14 09:08:05 +02:00
|
|
|
|
pack-objects: finishing touches.
This introduces --no-reuse-delta option to disable reusing of
existing delta, which is a large part of the optimization
introduced by this series. This may become necessary if
repeated repacking makes delta chain too long. With this, the
output of the command becomes identical to that of the older
implementation. But the performance suffers greatly.
It still allows reusing non-deltified representations; there is
no point uncompressing and recompressing the whole text.
It also adds a couple more statistics output, while squelching
it under -q flag, which the last round forgot to do.
$ time old-git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects....................
real 12m8.530s user 11m1.450s sys 0m57.920s
$ time git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 138297), reused 178833 (delta 134081)
real 0m59.549s user 0m56.670s sys 0m2.400s
$ time git-pack-objects --stdout --no-reuse-delta >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 134833), reused 47904 (delta 0)
real 11m13.830s user 9m45.240s sys 0m44.330s
There is one remaining issue when --no-reuse-delta option is not
used. It can create delta chains that are deeper than specified.
A<--B<--C<--D E F G
Suppose we have a delta chain A to D (A is stored in full either
in a pack or as a loose object. B is depth1 delta relative to A,
C is depth2 delta relative to B...) with loose objects E, F, G.
And we are going to pack all of them.
B, C and D are left as delta against A, B and C respectively.
So A, E, F, and G are examined for deltification, and let's say
we decided to keep E expanded, and store the rest as deltas like
this:
E<--F<--G<--A
Oops. We ended up making D a bit too deep, didn't we? B, C and
D form a chain on top of A!
This is because we did not know what the final depth of A would
be, when we checked objects and decided to keep the existing
delta. Unfortunately, deferring the decision until just before
the deltification is not an option. To be able to make B, C,
and D candidates for deltification with the rest, we need to
know the type and final unexpanded size of them, but the major
part of the optimization comes from the fact that we do not read
the delta data to do so -- getting the final size is quite an
expensive operation.
To prevent this from happening, we should keep A from being
deltified. But how would we tell that, cheaply?
To do this most precisely, after check_object() runs, each
object that is used as the base object of some existing delta
needs to be marked with the maximum depth of the objects we
decided to keep deltified (in this case, D is depth 3 relative
to A, so if no other delta chain that is longer than 3 based on
A exists, mark A with 3). Then when attempting to deltify A, we
would take that number into account to see if the final delta
chain that leads to D becomes too deep.
However, this is a bit cumbersome to compute, so we would cheat
and reduce the maximum depth for A arbitrarily to depth/4 in
this implementation.
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-02-16 20:55:51 +01:00
|
|
|
In a packed archive, an object is either stored as a compressed
|
|
|
|
|
whole, or as a difference from some other object. The latter is
|
|
|
|
|
often called a delta.
|
|
|
|
|
|
2005-07-14 09:08:05 +02:00
|
|
|
|
|
|
|
|
OPTIONS
|
|
|
|
|
-------
|
|
|
|
|
base-name::
|
|
|
|
|
Write into a pair of files (.pack and .idx), using
|
|
|
|
|
<base-name> to determine the name of the created file.
|
|
|
|
|
When this option is used, the two files are written in
|
|
|
|
|
<base-name>-<SHA1>.{pack,idx} files. <SHA1> is a hash
|
2006-11-01 23:34:47 +01:00
|
|
|
of the sorted object names to make the resulting filename
|
|
|
|
|
based on the pack content, and written to the standard
|
2005-07-14 09:08:05 +02:00
|
|
|
output of the command.
|
|
|
|
|
|
|
|
|
|
--stdout::
|
2005-12-29 10:20:06 +01:00
|
|
|
Write the pack contents (what would have been written to
|
2005-07-14 09:08:05 +02:00
|
|
|
.pack file) out to the standard output.
|
|
|
|
|
|
2006-09-13 07:59:15 +02:00
|
|
|
--revs::
|
|
|
|
|
Read the revision arguments from the standard input, instead of
|
|
|
|
|
individual object names. The revision arguments are processed
|
|
|
|
|
the same way as gitlink:git-rev-list[1] with `--objects` flag
|
|
|
|
|
uses its `commit` arguments to build the list of objects it
|
|
|
|
|
outputs. The objects on the resulting list are packed.
|
|
|
|
|
|
|
|
|
|
--unpacked::
|
|
|
|
|
This implies `--revs`. When processing the list of
|
|
|
|
|
revision arguments read from the standard input, limit
|
|
|
|
|
the objects packed to those that are not already packed.
|
|
|
|
|
|
|
|
|
|
--all::
|
|
|
|
|
This implies `--revs`. In addition to the list of
|
|
|
|
|
revision arguments read from the standard input, pretend
|
2006-10-06 11:15:03 +02:00
|
|
|
as if all refs under `$GIT_DIR/refs` are specified to be
|
2006-09-13 07:59:15 +02:00
|
|
|
included.
|
|
|
|
|
|
2006-10-06 11:15:03 +02:00
|
|
|
--window=[N], --depth=[N]::
|
|
|
|
|
These two options affect how the objects contained in
|
2005-07-14 09:08:05 +02:00
|
|
|
the pack are stored using delta compression. The
|
|
|
|
|
objects are first internally sorted by type, size and
|
|
|
|
|
optionally names and compared against the other objects
|
|
|
|
|
within --window to see if using delta compression saves
|
|
|
|
|
space. --depth limits the maximum delta depth; making
|
|
|
|
|
it too deep affects the performance on the unpacker
|
|
|
|
|
side, because delta data needs to be applied that many
|
|
|
|
|
times to get to the necessary object.
|
2006-10-06 11:15:03 +02:00
|
|
|
The default value for both --window and --depth is 10.
|
2005-07-14 09:08:05 +02:00
|
|
|
|
|
|
|
|
--incremental::
|
|
|
|
|
This flag causes an object already in a pack ignored
|
|
|
|
|
even if it appears in the standard input.
|
|
|
|
|
|
2005-10-30 10:14:33 +01:00
|
|
|
--local::
|
|
|
|
|
This flag is similar to `--incremental`; instead of
|
|
|
|
|
ignoring all packed objects, it only ignores objects
|
|
|
|
|
that are packed and not in the local object store
|
|
|
|
|
(i.e. borrowed from an alternate).
|
2005-07-14 09:08:05 +02:00
|
|
|
|
2005-12-09 00:28:05 +01:00
|
|
|
--non-empty::
|
|
|
|
|
Only create a packed archive if it would contain at
|
|
|
|
|
least one object.
|
|
|
|
|
|
pack-objects: finishing touches.
This introduces --no-reuse-delta option to disable reusing of
existing delta, which is a large part of the optimization
introduced by this series. This may become necessary if
repeated repacking makes delta chain too long. With this, the
output of the command becomes identical to that of the older
implementation. But the performance suffers greatly.
It still allows reusing non-deltified representations; there is
no point uncompressing and recompressing the whole text.
It also adds a couple more statistics output, while squelching
it under -q flag, which the last round forgot to do.
$ time old-git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects....................
real 12m8.530s user 11m1.450s sys 0m57.920s
$ time git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 138297), reused 178833 (delta 134081)
real 0m59.549s user 0m56.670s sys 0m2.400s
$ time git-pack-objects --stdout --no-reuse-delta >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 134833), reused 47904 (delta 0)
real 11m13.830s user 9m45.240s sys 0m44.330s
There is one remaining issue when --no-reuse-delta option is not
used. It can create delta chains that are deeper than specified.
A<--B<--C<--D E F G
Suppose we have a delta chain A to D (A is stored in full either
in a pack or as a loose object. B is depth1 delta relative to A,
C is depth2 delta relative to B...) with loose objects E, F, G.
And we are going to pack all of them.
B, C and D are left as delta against A, B and C respectively.
So A, E, F, and G are examined for deltification, and let's say
we decided to keep E expanded, and store the rest as deltas like
this:
E<--F<--G<--A
Oops. We ended up making D a bit too deep, didn't we? B, C and
D form a chain on top of A!
This is because we did not know what the final depth of A would
be, when we checked objects and decided to keep the existing
delta. Unfortunately, deferring the decision until just before
the deltification is not an option. To be able to make B, C,
and D candidates for deltification with the rest, we need to
know the type and final unexpanded size of them, but the major
part of the optimization comes from the fact that we do not read
the delta data to do so -- getting the final size is quite an
expensive operation.
To prevent this from happening, we should keep A from being
deltified. But how would we tell that, cheaply?
To do this most precisely, after check_object() runs, each
object that is used as the base object of some existing delta
needs to be marked with the maximum depth of the objects we
decided to keep deltified (in this case, D is depth 3 relative
to A, so if no other delta chain that is longer than 3 based on
A exists, mark A with 3). Then when attempting to deltify A, we
would take that number into account to see if the final delta
chain that leads to D becomes too deep.
However, this is a bit cumbersome to compute, so we would cheat
and reduce the maximum depth for A arbitrarily to depth/4 in
this implementation.
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-02-16 20:55:51 +01:00
|
|
|
-q::
|
|
|
|
|
This flag makes the command not to report its progress
|
|
|
|
|
on the standard error stream.
|
|
|
|
|
|
|
|
|
|
--no-reuse-delta::
|
|
|
|
|
When creating a packed archive in a repository that
|
|
|
|
|
has existing packs, the command reuses existing deltas.
|
|
|
|
|
This sometimes results in a slightly suboptimal pack.
|
|
|
|
|
This flag tells the command not to reuse existing deltas
|
|
|
|
|
but compute them from scratch.
|
|
|
|
|
|
2006-10-10 10:06:20 +02:00
|
|
|
--delta-base-offset::
|
|
|
|
|
A packed archive can express base object of a delta as
|
|
|
|
|
either 20-byte object name or as an offset in the
|
|
|
|
|
stream, but older version of git does not understand the
|
|
|
|
|
latter. By default, git-pack-objects only uses the
|
|
|
|
|
former format for better compatibility. This option
|
|
|
|
|
allows the command to use the latter format for
|
|
|
|
|
compactness. Depending on the average delta chain
|
|
|
|
|
length, this option typically shrinks the resulting
|
|
|
|
|
packfile by 3-5 per-cent.
|
|
|
|
|
|
pack-objects: finishing touches.
This introduces --no-reuse-delta option to disable reusing of
existing delta, which is a large part of the optimization
introduced by this series. This may become necessary if
repeated repacking makes delta chain too long. With this, the
output of the command becomes identical to that of the older
implementation. But the performance suffers greatly.
It still allows reusing non-deltified representations; there is
no point uncompressing and recompressing the whole text.
It also adds a couple more statistics output, while squelching
it under -q flag, which the last round forgot to do.
$ time old-git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects....................
real 12m8.530s user 11m1.450s sys 0m57.920s
$ time git-pack-objects --stdout >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 138297), reused 178833 (delta 134081)
real 0m59.549s user 0m56.670s sys 0m2.400s
$ time git-pack-objects --stdout --no-reuse-delta >/dev/null <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
Total 184141, written 184141 (delta 134833), reused 47904 (delta 0)
real 11m13.830s user 9m45.240s sys 0m44.330s
There is one remaining issue when --no-reuse-delta option is not
used. It can create delta chains that are deeper than specified.
A<--B<--C<--D E F G
Suppose we have a delta chain A to D (A is stored in full either
in a pack or as a loose object. B is depth1 delta relative to A,
C is depth2 delta relative to B...) with loose objects E, F, G.
And we are going to pack all of them.
B, C and D are left as delta against A, B and C respectively.
So A, E, F, and G are examined for deltification, and let's say
we decided to keep E expanded, and store the rest as deltas like
this:
E<--F<--G<--A
Oops. We ended up making D a bit too deep, didn't we? B, C and
D form a chain on top of A!
This is because we did not know what the final depth of A would
be, when we checked objects and decided to keep the existing
delta. Unfortunately, deferring the decision until just before
the deltification is not an option. To be able to make B, C,
and D candidates for deltification with the rest, we need to
know the type and final unexpanded size of them, but the major
part of the optimization comes from the fact that we do not read
the delta data to do so -- getting the final size is quite an
expensive operation.
To prevent this from happening, we should keep A from being
deltified. But how would we tell that, cheaply?
To do this most precisely, after check_object() runs, each
object that is used as the base object of some existing delta
needs to be marked with the maximum depth of the objects we
decided to keep deltified (in this case, D is depth 3 relative
to A, so if no other delta chain that is longer than 3 based on
A exists, mark A with 3). Then when attempting to deltify A, we
would take that number into account to see if the final delta
chain that leads to D becomes too deep.
However, this is a bit cumbersome to compute, so we would cheat
and reduce the maximum depth for A arbitrarily to depth/4 in
this implementation.
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-02-16 20:55:51 +01:00
|
|
|
|
2005-07-14 09:08:05 +02:00
|
|
|
Author
|
|
|
|
|
------
|
|
|
|
|
Written by Linus Torvalds <torvalds@osdl.org>
|
|
|
|
|
|
|
|
|
|
Documentation
|
|
|
|
|
-------------
|
|
|
|
|
Documentation by Junio C Hamano
|
|
|
|
|
|
2006-03-05 07:20:07 +01:00
|
|
|
See Also
|
2005-08-16 00:48:47 +02:00
|
|
|
--------
|
2006-09-13 07:59:15 +02:00
|
|
|
gitlink:git-rev-list[1]
|
2005-10-03 19:16:30 +02:00
|
|
|
gitlink:git-repack[1]
|
|
|
|
|
gitlink:git-prune-packed[1]
|
2005-08-16 00:48:47 +02:00
|
|
|
|
2005-07-14 09:08:05 +02:00
|
|
|
GIT
|
|
|
|
|
---
|
2005-09-19 12:10:51 +02:00
|
|
|
Part of the gitlink:git[7] suite
|
2005-07-14 09:08:05 +02:00
|
|
|
|