[PATCH] Remove "delta" object representation.

Packed delta files created by git-pack-objects seems to be the
way to go, and existing "delta" object handling code has exposed
the object representation details to too many places.  Remove it
while we refactor code to come up with a proper interface in
sha1_file.c.

Signed-off-by: Junio C Hamano <junkio@cox.net>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
This commit is contained in:
Junio C Hamano 2005-06-27 03:33:33 -07:00 committed by Linus Torvalds
parent ee85cbc688
commit c4584ae3fd
25 changed files with 22 additions and 948 deletions

View File

@ -9,7 +9,7 @@ git-fsck-cache - Verifies the connectivity and validity of the objects in the da
SYNOPSIS SYNOPSIS
-------- --------
'git-fsck-cache' [--tags] [--root] [--delta-depth] [--unreachable] [--cache] [<object>*] 'git-fsck-cache' [--tags] [--root] [--unreachable] [--cache] [<object>*]
DESCRIPTION DESCRIPTION
----------- -----------
@ -37,9 +37,6 @@ OPTIONS
Consider any object recorded in the cache also as a head node for Consider any object recorded in the cache also as a head node for
an unreachability trace. an unreachability trace.
--delta-depth::
Report back the length of the longest delta chain found.
It tests SHA1 and general object sanity, and it does full tracking of It tests SHA1 and general object sanity, and it does full tracking of
the resulting reachability and everything else. It prints out any the resulting reachability and everything else. It prints out any
corruption it finds (missing or bad objects), and if you use the corruption it finds (missing or bad objects), and if you use the

View File

@ -21,13 +21,6 @@ Downloads a remote GIT repository via HTTP.
Get trees associated with the commit objects. Get trees associated with the commit objects.
-a:: -a::
Get all the objects. Get all the objects.
-d::
Do not check for delta base objects (use this option
only when you know the remote repository is not
deltified).
--recover::
Check dependency of deltified object more carefully than
usual, to recover after earlier pull that was interrupted.
-v:: -v::
Report what is downloaded. Report what is downloaded.
-w:: -w::

View File

@ -23,13 +23,6 @@ OPTIONS
Get trees associated with the commit objects. Get trees associated with the commit objects.
-a:: -a::
Get all the objects. Get all the objects.
-d::
Do not check for delta base objects (use this option
only when you know the remote repository is not
deltified).
--recover::
Check dependency of deltified object more carefully than
usual, to recover after earlier pull that was interrupted.
-v:: -v::
Report what is downloaded. Report what is downloaded.
-w:: -w::

View File

@ -1,45 +0,0 @@
git-mkdelta(1)
==============
May 2005
NAME
----
git-mkdelta - Creates a delta object
SYNOPSIS
--------
'git-mkdelta' [-v] [-d N | --max-depth=N ] <reference_object> <target_object> [ <next_object> ... ]
DESCRIPTION
-----------
Creates a delta object to replace <reference_object> by using an
ordered list of potential objects to deltafy against earlier objects
in the list.
A cap on the depth of delta references can be provided as well,
otherwise the default is to not have any limit. A limit of 0 will
also undeltafy a given object.
OPTIONS
-------
-v::
Verbose
-d|--max-depth::
limit the number of delta references in a chain
If 0 then all objects are undeltafied.
Author
------
Git is written by Linus Torvalds <torvalds@osdl.org> and the git-list <git@vger.kernel.org>.
Documentation
--------------
Documentation by David Greaves, Junio C Hamano and the git-list <git@vger.kernel.org>.
GIT
---
Part of the link:git.html[git] suite

View File

@ -31,13 +31,6 @@ commit-id::
Get trees associated with the commit objects. Get trees associated with the commit objects.
-a:: -a::
Get all the objects. Get all the objects.
-d::
Do not check for delta base objects (use this option
only when you know the remote repository is not
deltified).
--recover::
Check dependency of deltified object more carefully than
usual, to recover after earlier pull that was interrupted.
-v:: -v::
Report what is downloaded. Report what is downloaded.
-w:: -w::

View File

@ -28,13 +28,6 @@ commit-id::
Get tree associated with the requested commit object. Get tree associated with the requested commit object.
-a:: -a::
Get all the objects. Get all the objects.
-d::
Do not check for delta base objects (use this option
only when you know the local repository is not
deltified).
--recover::
Check dependency of deltified object more carefully than
usual, to recover after earlier push that was interrupted.
-v:: -v::
Report what is uploaded. Report what is uploaded.
-w:: -w::

View File

@ -51,9 +51,6 @@ link:git-init-db.html[git-init-db]::
link:git-merge-base.html[git-merge-base]:: link:git-merge-base.html[git-merge-base]::
Finds as good a common ancestor as possible for a merge Finds as good a common ancestor as possible for a merge
link:git-mkdelta.html[git-mkdelta]::
Creates a delta object
link:git-mktag.html[git-mktag]:: link:git-mktag.html[git-mktag]::
Creates a tag object Creates a tag object

View File

@ -22,7 +22,7 @@ INSTALL=install
SCRIPTS=git git-apply-patch-script git-merge-one-file-script git-prune-script \ SCRIPTS=git git-apply-patch-script git-merge-one-file-script git-prune-script \
git-pull-script git-tag-script git-resolve-script git-whatchanged \ git-pull-script git-tag-script git-resolve-script git-whatchanged \
git-deltafy-script git-fetch-script git-status-script git-commit-script \ git-fetch-script git-status-script git-commit-script \
git-log-script git-shortlog git-cvsimport-script git-diff-script \ git-log-script git-shortlog git-cvsimport-script git-diff-script \
git-reset-script git-add-script git-checkout-script git-clone-script \ git-reset-script git-add-script git-checkout-script git-clone-script \
gitk git-cherry git-rebase-script git-relink-script gitk git-cherry git-rebase-script git-relink-script
@ -34,7 +34,7 @@ PROG= git-update-cache git-diff-files git-init-db git-write-tree \
git-unpack-file git-export git-diff-cache git-convert-cache \ git-unpack-file git-export git-diff-cache git-convert-cache \
git-http-pull git-ssh-push git-ssh-pull git-rev-list git-mktag \ git-http-pull git-ssh-push git-ssh-pull git-rev-list git-mktag \
git-diff-helper git-tar-tree git-local-pull git-write-blob \ git-diff-helper git-tar-tree git-local-pull git-write-blob \
git-get-tar-commit-id git-mkdelta git-apply git-stripspace \ git-get-tar-commit-id git-apply git-stripspace \
git-cvs2git git-diff-stages git-rev-parse git-patch-id \ git-cvs2git git-diff-stages git-rev-parse git-patch-id \
git-pack-objects git-unpack-objects git-pack-objects git-unpack-objects
@ -44,7 +44,7 @@ install: $(PROG) $(SCRIPTS)
$(INSTALL) $(PROG) $(SCRIPTS) $(dest)$(bin) $(INSTALL) $(PROG) $(SCRIPTS) $(dest)$(bin)
LIB_OBJS=read-cache.o sha1_file.o usage.o object.o commit.o tree.o blob.o \ LIB_OBJS=read-cache.o sha1_file.o usage.o object.o commit.o tree.o blob.o \
tag.o delta.o date.o index.o diff-delta.o patch-delta.o entry.o \ tag.o date.o index.o diff-delta.o patch-delta.o entry.o \
epoch.o refs.o csum-file.o epoch.o refs.o csum-file.o
LIB_FILE=libgit.a LIB_FILE=libgit.a
LIB_H=cache.h object.h blob.h tree.h commit.h tag.h delta.h epoch.h csum-file.h LIB_H=cache.h object.h blob.h tree.h commit.h tag.h delta.h epoch.h csum-file.h
@ -117,7 +117,6 @@ git-mktag: mktag.c
git-diff-helper: diff-helper.c git-diff-helper: diff-helper.c
git-tar-tree: tar-tree.c git-tar-tree: tar-tree.c
git-write-blob: write-blob.c git-write-blob: write-blob.c
git-mkdelta: mkdelta.c
git-stripspace: stripspace.c git-stripspace: stripspace.c
git-cvs2git: cvs2git.c git-cvs2git: cvs2git.c
git-diff-stages: diff-stages.c git-diff-stages: diff-stages.c

35
README
View File

@ -32,8 +32,8 @@ build up a hierarchy of objects.
All objects have a statically determined "type" aka "tag", which is All objects have a statically determined "type" aka "tag", which is
determined at object creation time, and which identifies the format of determined at object creation time, and which identifies the format of
the object (i.e. how it is used, and how it can refer to other the object (i.e. how it is used, and how it can refer to other
objects). There are currently five different object types: "blob", objects). There are currently four different object types: "blob",
"tree", "commit", "tag" and "delta" "tree", "commit" and "tag".
A "blob" object cannot refer to any other object, and is, like the tag A "blob" object cannot refer to any other object, and is, like the tag
implies, a pure storage object containing some user data. It is used to implies, a pure storage object containing some user data. It is used to
@ -62,16 +62,12 @@ A "tag" object symbolically identifies and can be used to sign other
objects. It contains the identifier and type of another object, a objects. It contains the identifier and type of another object, a
symbolic name (of course!) and, optionally, a signature. symbolic name (of course!) and, optionally, a signature.
A "delta" object is used internally by the object database to minimise
disk usage. Instead of storing the entire contents of a revision, git
can behave in a similar manner to RCS et al and simply store a delta.
Regardless of object type, all objects share the following Regardless of object type, all objects share the following
characteristics: they are all deflated with zlib, and have a header characteristics: they are all deflated with zlib, and have a header
that not only specifies their tag, but also provides size information that not only specifies their tag, but also provides size information
about the data in the object. It's worth noting that the SHA1 hash about the data in the object. It's worth noting that the SHA1 hash
that is used to name the object is the hash of the original data or that is used to name the object is the hash of the original data.
the delta. (Historical note: in the dawn of the age of git the hash (Historical note: in the dawn of the age of git the hash
was the sha1 of the _compressed_ object) was the sha1 of the _compressed_ object)
As a result, the general consistency of an object can always be tested As a result, the general consistency of an object can always be tested
@ -219,29 +215,6 @@ verification) has to come from outside.
A tag is created with link:git-mktag.html[git-mktag] and A tag is created with link:git-mktag.html[git-mktag] and
it's data can be accessed by link:git-cat-file.html[git-cat-file] it's data can be accessed by link:git-cat-file.html[git-cat-file]
Delta Object
~~~~~~~~~~~~
The "delta" object is used internally by the object database to
minimise storage usage by using xdeltas (byte level diffs). Deltas can
form chains of arbitrary length as RCS does (although this is
configureable at creation time). Most operations won't see or even be
aware of delta objects as they are automatically 'applied' and appear
as 'real' git objects In other words, if you write your own routines
to look at the contents of the object database then you need to know
about this - otherwise you don't. Actually, that's not quite true -
one important area where deltas are likely to prove very valuable is
in reducing bandwidth loads - so the more sophisticated network tools
for git repositories will be aware of them too.
Finally, git repositories can (and must) be deltafied in the
background - the work to calculate the differences does not take place
automatically at commit time.
A delta can be created (or undeltafied) with
link:git-mkdelta.html[git-mkdelta] it's raw data cannot be accessed at
present.
The "index" aka "Current Directory Cache" The "index" aka "Current Directory Cache"
----------------------------------------- -----------------------------------------

View File

@ -162,7 +162,6 @@ extern char *sha1_file_name(const unsigned char *sha1);
extern void * map_sha1_file(const unsigned char *sha1, unsigned long *size); extern void * map_sha1_file(const unsigned char *sha1, unsigned long *size);
extern int unpack_sha1_header(z_stream *stream, void *map, unsigned long mapsize, void *buffer, unsigned long size); extern int unpack_sha1_header(z_stream *stream, void *map, unsigned long mapsize, void *buffer, unsigned long size);
extern int parse_sha1_header(char *hdr, char *type, unsigned long *sizep); extern int parse_sha1_header(char *hdr, char *type, unsigned long *sizep);
extern int sha1_delta_base(const unsigned char *, unsigned char *);
extern int sha1_file_size(const unsigned char *, unsigned long *); extern int sha1_file_size(const unsigned char *, unsigned long *);
extern void * unpack_sha1_file(void *map, unsigned long mapsize, char *type, unsigned long *size); extern void * unpack_sha1_file(void *map, unsigned long mapsize, char *type, unsigned long *size);
extern void * read_sha1_file(const unsigned char *sha1, char *type, unsigned long *size); extern void * read_sha1_file(const unsigned char *sha1, char *type, unsigned long *size);

114
delta.c
View File

@ -1,114 +0,0 @@
#include "object.h"
#include "blob.h"
#include "tree.h"
#include "commit.h"
#include "tag.h"
#include "delta.h"
#include "cache.h"
/* the delta object definition (it can alias any other object) */
struct delta {
union {
struct object object;
struct blob blob;
struct tree tree;
struct commit commit;
struct tag tag;
} u;
};
struct delta *lookup_delta(const unsigned char *sha1)
{
struct object *obj = lookup_object(sha1);
if (!obj) {
struct delta *ret = xmalloc(sizeof(struct delta));
memset(ret, 0, sizeof(struct delta));
created_object(sha1, &ret->u.object);
return ret;
}
return (struct delta *) obj;
}
int parse_delta_buffer(struct delta *item, void *buffer, unsigned long size)
{
struct object *reference;
struct object_list *p;
if (item->u.object.delta)
return 0;
item->u.object.delta = 1;
if (size <= 20)
return -1;
reference = lookup_object(buffer);
if (!reference) {
struct delta *ref = xmalloc(sizeof(struct delta));
memset(ref, 0, sizeof(struct delta));
created_object(buffer, &ref->u.object);
reference = &ref->u.object;
}
p = xmalloc(sizeof(*p));
p->item = &item->u.object;
p->next = reference->attached_deltas;
reference->attached_deltas = p;
return 0;
}
int process_deltas(void *src, unsigned long src_size, const char *src_type,
struct object_list *delta_list)
{
int deepest = 0;
do {
struct object *obj = delta_list->item;
static char type[10];
void *map, *delta, *buf;
unsigned long map_size, delta_size, buf_size;
map = map_sha1_file(obj->sha1, &map_size);
if (!map)
continue;
delta = unpack_sha1_file(map, map_size, type, &delta_size);
munmap(map, map_size);
if (!delta)
continue;
if (strcmp(type, "delta") || delta_size <= 20) {
free(delta);
continue;
}
buf = patch_delta(src, src_size,
delta+20, delta_size-20,
&buf_size);
free(delta);
if (!buf)
continue;
if (check_sha1_signature(obj->sha1, buf, buf_size, src_type) < 0)
printf("sha1 mismatch for delta %s\n", sha1_to_hex(obj->sha1));
if (obj->type && obj->type != src_type) {
error("got %s when expecting %s for delta %s",
src_type, obj->type, sha1_to_hex(obj->sha1));
free(buf);
continue;
}
obj->type = src_type;
if (src_type == blob_type) {
parse_blob_buffer((struct blob *)obj, buf, buf_size);
} else if (src_type == tree_type) {
parse_tree_buffer((struct tree *)obj, buf, buf_size);
} else if (src_type == commit_type) {
parse_commit_buffer((struct commit *)obj, buf, buf_size);
} else if (src_type == tag_type) {
parse_tag_buffer((struct tag *)obj, buf, buf_size);
} else {
error("unknown object type %s", src_type);
free(buf);
continue;
}
if (obj->attached_deltas) {
int depth = process_deltas(buf, buf_size, src_type,
obj->attached_deltas);
if (deepest < depth)
deepest = depth;
}
free(buf);
} while ((delta_list = delta_list->next));
return deepest + 1;
}

View File

@ -9,13 +9,4 @@ extern void *patch_delta(void *src_buf, unsigned long src_size,
void *delta_buf, unsigned long delta_size, void *delta_buf, unsigned long delta_size,
unsigned long *dst_size); unsigned long *dst_size);
/* handling of delta objects */
struct delta;
struct object_list;
extern struct delta *lookup_delta(const unsigned char *sha1);
extern int parse_delta_buffer(struct delta *item, void *buffer, unsigned long size);
extern int parse_delta(struct delta *item, unsigned char sha1);
extern int process_deltas(void *src, unsigned long src_size,
const char *src_type, struct object_list *delta);
#endif #endif

View File

@ -6,46 +6,15 @@
#include "tree.h" #include "tree.h"
#include "blob.h" #include "blob.h"
#include "tag.h" #include "tag.h"
#include "delta.h"
#define REACHABLE 0x0001 #define REACHABLE 0x0001
static int show_root = 0; static int show_root = 0;
static int show_tags = 0; static int show_tags = 0;
static int show_unreachable = 0; static int show_unreachable = 0;
static int show_max_delta_depth = 0;
static int keep_cache_objects = 0; static int keep_cache_objects = 0;
static unsigned char head_sha1[20]; static unsigned char head_sha1[20];
static void expand_deltas(void)
{
int i, max_depth = 0;
/*
* To be as efficient as possible we look for delta heads and
* recursively process them going backward, and parsing
* resulting objects along the way. This allows for processing
* each delta objects only once regardless of the delta depth.
*/
for (i = 0; i < nr_objs; i++) {
struct object *obj = objs[i];
if (obj->parsed && !obj->delta && obj->attached_deltas) {
int depth = 0;
char type[10];
unsigned long size;
void *buf = read_sha1_file(obj->sha1, type, &size);
if (!buf)
continue;
depth = process_deltas(buf, size, obj->type,
obj->attached_deltas);
if (max_depth < depth)
max_depth = depth;
}
}
if (show_max_delta_depth)
printf("maximum delta depth = %d\n", max_depth);
}
static void check_connectivity(void) static void check_connectivity(void)
{ {
int i; int i;
@ -56,12 +25,8 @@ static void check_connectivity(void)
struct object_list *refs; struct object_list *refs;
if (!obj->parsed) { if (!obj->parsed) {
if (obj->delta) printf("missing %s %s\n",
printf("unresolved delta %s\n", obj->type, sha1_to_hex(obj->sha1));
sha1_to_hex(obj->sha1));
else
printf("missing %s %s\n",
obj->type, sha1_to_hex(obj->sha1));
continue; continue;
} }
@ -75,12 +40,8 @@ static void check_connectivity(void)
} }
if (show_unreachable && !(obj->flags & REACHABLE)) { if (show_unreachable && !(obj->flags & REACHABLE)) {
if (obj->attached_deltas) printf("unreachable %s %s\n",
printf("foreign delta reference %s\n", obj->type, sha1_to_hex(obj->sha1));
sha1_to_hex(obj->sha1));
else
printf("unreachable %s %s\n",
obj->type, sha1_to_hex(obj->sha1));
continue; continue;
} }
@ -240,8 +201,6 @@ static int fsck_sha1(unsigned char *sha1)
return fsck_commit((struct commit *) obj); return fsck_commit((struct commit *) obj);
if (obj->type == tag_type) if (obj->type == tag_type)
return fsck_tag((struct tag *) obj); return fsck_tag((struct tag *) obj);
if (!obj->type && obj->delta)
return 0;
return -1; return -1;
} }
@ -427,10 +386,6 @@ int main(int argc, char **argv)
show_root = 1; show_root = 1;
continue; continue;
} }
if (!strcmp(arg, "--delta-depth")) {
show_max_delta_depth = 1;
continue;
}
if (!strcmp(arg, "--cache")) { if (!strcmp(arg, "--cache")) {
keep_cache_objects = 1; keep_cache_objects = 1;
continue; continue;
@ -447,8 +402,6 @@ int main(int argc, char **argv)
} }
fsck_sha1_list(); fsck_sha1_list();
expand_deltas();
heads = 0; heads = 0;
for (i = 1; i < argc; i++) { for (i = 1; i < argc; i++) {
const char *arg = argv[i]; const char *arg = argv[i];

View File

@ -1,81 +0,0 @@
#!/bin/bash
# Example script to deltify an entire GIT repository based on the commit list.
# The most recent version of a file is the reference and previous versions
# are made delta against the best earlier version available. And so on for
# successive versions going back in time. This way the increasing delta
# overhead is pushed towards older versions of any given file.
#
# The -d argument allows to provide a limit on the delta chain depth.
# If 0 is passed then everything is undeltafied. Limiting the delta
# depth is meaningful for subsequent access performance to old revisions.
# A value of 16 might be a good compromize between performance and good
# space saving. Current default is unbounded.
#
# The --max-behind=30 argument is passed to git-mkdelta so to keep
# combinations and memory usage bounded a bit. If you have lots of memory
# and CPU power you may remove it (or set to 0) to let git-mkdelta find the
# best delta match regardless of the number of revisions for a given file.
# You can also make the value smaller to make it faster and less
# memory hungry. A value of 5 ought to still give pretty good results.
# When set to 0 or ommitted then look behind is unbounded. Note that
# git-mkdelta might die with a segmentation fault in that case if it
# runs out of memory. Note that the GIT repository will still be consistent
# even if git-mkdelta dies unexpectedly.
set -e
max_depth=
[ "$1" == "-d" ] && max_depth="--max-depth=$2" && shift 2
overlap=30
max_behind="--max-behind=$overlap"
function process_list() {
if [ "$list" ]; then
echo "Processing $curr_file"
echo "$list" | xargs git-mkdelta $max_depth $max_behind -v
fi
}
rev_list=""
curr_file=""
git-rev-list HEAD |
while true; do
# Let's batch revisions into groups of 1000 to give it a chance to
# scale with repositories containing long revision lists. We also
# overlap with the previous batch the size of mkdelta's look behind
# value in order to account for the processing discontinuity.
rev_list="$(echo -e -n "$rev_list" | tail --lines=$overlap)"
for i in $(seq 1000); do
read rev || break
rev_list="$rev_list$rev\n"
done
echo -e -n "$rev_list" |
git-diff-tree -r -t --stdin |
awk '/^:/ { if ($5 == "M") printf "%s %s\n%s %s\n", $4, $6, $3, $6 }' |
LC_ALL=C sort -s -k 2 | uniq |
while read sha1 file; do
if [ "$file" == "$curr_file" ]; then
list="$list $sha1"
else
process_list
curr_file="$file"
list="$sha1"
fi
done
[ "$rev" ] || break
done
process_list
curr_file="root directory"
list="$(
git-rev-list HEAD |
while read commit; do
git-cat-file commit $commit |
sed -n 's/tree //p;Q'
done
)"
process_list

View File

@ -150,10 +150,6 @@ int main(int argc, char **argv)
get_tree = 1; get_tree = 1;
} else if (argv[arg][1] == 'c') { } else if (argv[arg][1] == 'c') {
get_history = 1; get_history = 1;
} else if (argv[arg][1] == 'd') {
get_delta = 0;
} else if (!strcmp(argv[arg], "--recover")) {
get_delta = 2;
} else if (argv[arg][1] == 'a') { } else if (argv[arg][1] == 'a') {
get_all = 1; get_all = 1;
get_tree = 1; get_tree = 1;

View File

@ -120,10 +120,6 @@ int main(int argc, char **argv)
get_tree = 1; get_tree = 1;
else if (argv[arg][1] == 'c') else if (argv[arg][1] == 'c')
get_history = 1; get_history = 1;
else if (argv[arg][1] == 'd')
get_delta = 0;
else if (!strcmp(argv[arg], "--recover"))
get_delta = 2;
else if (argv[arg][1] == 'a') { else if (argv[arg][1] == 'a') {
get_all = 1; get_all = 1;
get_tree = 1; get_tree = 1;

364
mkdelta.c
View File

@ -1,364 +0,0 @@
/*
* Deltafication of a GIT database.
*
* (C) 2005 Nicolas Pitre <nico@cam.org>
*
* This code is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
#include "cache.h"
#include "delta.h"
static int replace_object(char *buf, unsigned long size, unsigned char *sha1)
{
char tmpfile[PATH_MAX];
int fd;
snprintf(tmpfile, sizeof(tmpfile), "%s/obj_XXXXXX", get_object_directory());
fd = mkstemp(tmpfile);
if (fd < 0)
return error("%s: %s\n", tmpfile, strerror(errno));
if (write(fd, buf, size) != size) {
perror("unable to write file");
close(fd);
unlink(tmpfile);
return -1;
}
fchmod(fd, 0444);
close(fd);
if (rename(tmpfile, sha1_file_name(sha1))) {
perror("unable to replace original object");
unlink(tmpfile);
return -1;
}
return 0;
}
static void *create_object(unsigned char *buf, unsigned long len,
char *hdr, int hdrlen, unsigned long *retsize)
{
unsigned char *compressed;
unsigned long size;
z_stream stream;
/* Set it up */
memset(&stream, 0, sizeof(stream));
deflateInit(&stream, Z_BEST_COMPRESSION);
size = deflateBound(&stream, len+hdrlen);
compressed = xmalloc(size);
/* Compress it */
stream.next_out = compressed;
stream.avail_out = size;
/* First header.. */
stream.next_in = (unsigned char *)hdr;
stream.avail_in = hdrlen;
while (deflate(&stream, 0) == Z_OK)
/* nothing */;
/* Then the data itself.. */
stream.next_in = buf;
stream.avail_in = len;
while (deflate(&stream, Z_FINISH) == Z_OK)
/* nothing */;
deflateEnd(&stream);
*retsize = stream.total_out;
return compressed;
}
static int restore_original_object(unsigned char *buf, unsigned long len,
char *type, unsigned char *sha1)
{
char hdr[50];
int hdrlen, ret;
void *compressed;
unsigned long size;
hdrlen = sprintf(hdr, "%s %lu", type, len)+1;
compressed = create_object(buf, len, hdr, hdrlen, &size);
ret = replace_object(compressed, size, sha1);
free(compressed);
return ret;
}
static void *create_delta_object(unsigned char *buf, unsigned long len,
unsigned char *sha1_ref, unsigned long *size)
{
char hdr[50];
int hdrlen;
/* Generate the header + sha1 of reference for delta */
hdrlen = sprintf(hdr, "delta %lu", len+20)+1;
memcpy(hdr + hdrlen, sha1_ref, 20);
hdrlen += 20;
return create_object(buf, len, hdr, hdrlen, size);
}
static void *get_buffer(unsigned char *sha1, char *type,
unsigned long *size, unsigned long *compsize)
{
unsigned long mapsize;
void *map = map_sha1_file(sha1, &mapsize);
if (map) {
void *buffer = unpack_sha1_file(map, mapsize, type, size);
munmap(map, mapsize);
if (compsize)
*compsize = mapsize;
if (buffer)
return buffer;
}
error("unable to get object %s", sha1_to_hex(sha1));
return NULL;
}
static void *expand_delta(void *delta, unsigned long *size, char *type,
unsigned int *depth, unsigned char **links)
{
void *buf = NULL;
unsigned int level = (*depth)++;
if (*size < 20) {
error("delta object is bad");
free(delta);
} else {
unsigned long ref_size;
void *ref = get_buffer(delta, type, &ref_size, NULL);
if (ref && !strcmp(type, "delta"))
ref = expand_delta(ref, &ref_size, type, depth, links);
else if (ref)
{
*links = xmalloc(*depth * 20);
}
if (ref) {
buf = patch_delta(ref, ref_size, delta+20, *size-20, size);
free(ref);
if (buf)
memcpy(*links + level*20, delta, 20);
else
free(*links);
}
free(delta);
}
return buf;
}
static char *mkdelta_usage =
"mkdelta [--max-depth=N] [--max-behind=N] <reference_sha1> <target_sha1> [<next_sha1> ...]";
struct delta {
unsigned char sha1[20]; /* object sha1 */
unsigned long size; /* object size */
void *buf; /* object content */
unsigned char *links; /* delta reference links */
unsigned int depth; /* delta depth */
};
int main(int argc, char **argv)
{
struct delta *ref, trg;
char ref_type[20], trg_type[20], *skip_reason;
void *best_buf;
unsigned long best_size, orig_size, orig_compsize;
unsigned int r, orig_ref, best_ref, nb_refs, next_ref, max_refs = 0;
unsigned int i, duplicate, skip_lvl, verbose = 0, quiet = 0;
unsigned int max_depth = -1;
for (i = 1; i < argc; i++) {
if (!strcmp(argv[i], "-v")) {
verbose = 1;
quiet = 0;
} else if (!strcmp(argv[i], "-q")) {
quiet = 1;
verbose = 0;
} else if (!strcmp(argv[i], "-d") && i+1 < argc) {
max_depth = atoi(argv[++i]);
} else if (!strncmp(argv[i], "--max-depth=", 12)) {
max_depth = atoi(argv[i]+12);
} else if (!strcmp(argv[i], "-b") && i+1 < argc) {
max_refs = atoi(argv[++i]);
} else if (!strncmp(argv[i], "--max-behind=", 13)) {
max_refs = atoi(argv[i]+13);
} else
break;
}
if (i + (max_depth != 0) >= argc)
usage(mkdelta_usage);
if (!max_refs || max_refs > argc - i)
max_refs = argc - i;
ref = xmalloc(max_refs * sizeof(*ref));
for (r = 0; r < max_refs; r++)
ref[r].buf = ref[r].links = NULL;
next_ref = nb_refs = 0;
do {
if (get_sha1(argv[i], trg.sha1))
die("bad sha1 %s", argv[i]);
trg.buf = get_buffer(trg.sha1, trg_type, &trg.size, &orig_compsize);
if (trg.buf && !trg.size) {
if (verbose)
printf("skip %s (object is empty)\n", argv[i]);
continue;
}
orig_size = trg.size;
orig_ref = -1;
trg.depth = 0;
trg.links = NULL;
if (trg.buf && !strcmp(trg_type, "delta")) {
for (r = 0; r < nb_refs; r++)
if (!memcmp(trg.buf, ref[r].sha1, 20))
break;
if (r < nb_refs) {
/* no need to reload the reference object */
trg.depth = ref[r].depth + 1;
trg.links = xmalloc(trg.depth*20);
memcpy(trg.links, trg.buf, 20);
memcpy(trg.links+20, ref[r].links, ref[r].depth*20);
trg.buf = patch_delta(ref[r].buf, ref[r].size,
trg.buf+20, trg.size-20,
&trg.size);
strcpy(trg_type, ref_type);
orig_ref = r;
} else {
trg.buf = expand_delta(trg.buf, &trg.size, trg_type,
&trg.depth, &trg.links);
}
}
if (!trg.buf)
die("unable to read target object %s", argv[i]);
if (!nb_refs) {
strcpy(ref_type, trg_type);
} else if (max_depth && strcmp(ref_type, trg_type)) {
die("type mismatch for object %s", argv[i]);
}
duplicate = 0;
best_buf = NULL;
best_size = -1;
best_ref = -1;
skip_lvl = 0;
skip_reason = NULL;
for (r = 0; max_depth && r < nb_refs; r++) {
void *delta_buf, *comp_buf;
unsigned long delta_size, comp_size;
unsigned int l;
duplicate = !memcmp(trg.sha1, ref[r].sha1, 20);
if (duplicate) {
skip_reason = "already seen";
break;
}
if (ref[r].depth >= max_depth) {
if (skip_lvl < 1) {
skip_reason = "exceeding max link depth";
skip_lvl = 1;
}
continue;
}
for (l = 0; l < ref[r].depth; l++)
if (!memcmp(trg.sha1, ref[r].links + l*20, 20))
break;
if (l != ref[r].depth) {
if (skip_lvl < 2) {
skip_reason = "would create a loop";
skip_lvl = 2;
}
continue;
}
if (trg.depth < max_depth && r == orig_ref) {
if (skip_lvl < 3) {
skip_reason = "delta already in place";
skip_lvl = 3;
}
continue;
}
delta_buf = diff_delta(ref[r].buf, ref[r].size,
trg.buf, trg.size,
&delta_size, ~0UL);
if (!delta_buf)
die("out of memory");
if (trg.depth < max_depth &&
delta_size+20 >= orig_size) {
/* no need to even try to compress if original
object is smaller than this delta */
free(delta_buf);
if (skip_lvl < 4) {
skip_reason = "no size reduction";
skip_lvl = 4;
}
continue;
}
comp_buf = create_delta_object(delta_buf, delta_size,
ref[r].sha1, &comp_size);
if (!comp_buf)
die("out of memory");
free(delta_buf);
if (trg.depth < max_depth &&
comp_size >= orig_compsize) {
free(comp_buf);
if (skip_lvl < 5) {
skip_reason = "no size reduction";
skip_lvl = 5;
}
continue;
}
if ((comp_size < best_size) ||
(comp_size == best_size &&
ref[r].depth < ref[best_ref].depth)) {
free(best_buf);
best_buf = comp_buf;
best_size = comp_size;
best_ref = r;
}
}
if (best_buf) {
if (replace_object(best_buf, best_size, trg.sha1))
die("unable to write delta for %s", argv[i]);
free(best_buf);
free(trg.links);
trg.depth = ref[best_ref].depth + 1;
trg.links = xmalloc(trg.depth*20);
memcpy(trg.links, ref[best_ref].sha1, 20);
memcpy(trg.links+20, ref[best_ref].links, ref[best_ref].depth*20);
if (!quiet)
printf("delta %s (size=%ld.%02ld%% depth=%d dist=%d)\n",
argv[i], best_size*100 / orig_compsize,
(best_size*10000 / orig_compsize)%100,
trg.depth,
(next_ref - best_ref + max_refs)
% (max_refs + 1) + 1);
} else if (trg.depth > max_depth) {
if (restore_original_object(trg.buf, trg.size, trg_type, trg.sha1))
die("unable to restore %s", argv[i]);
if (!quiet)
printf("undelta %s (depth was %d)\n",
argv[i], trg.depth);
trg.depth = 0;
free(trg.links);
trg.links = NULL;
} else if (skip_reason && verbose) {
printf("skip %s (%s)\n", argv[i], skip_reason);
}
if (!duplicate) {
free(ref[next_ref].buf);
free(ref[next_ref].links);
ref[next_ref] = trg;
if (++next_ref > nb_refs)
nb_refs = next_ref;
if (next_ref == max_refs)
next_ref = 0;
} else {
free(trg.buf);
free(trg.links);
}
} while (++i < argc);
return 0;
}

View File

@ -4,7 +4,6 @@
#include "commit.h" #include "commit.h"
#include "cache.h" #include "cache.h"
#include "tag.h" #include "tag.h"
#include "delta.h"
struct object **objs; struct object **objs;
int nr_objs; int nr_objs;
@ -116,25 +115,14 @@ struct object *lookup_object_type(const unsigned char *sha1, const char *type)
struct object *parse_object(const unsigned char *sha1) struct object *parse_object(const unsigned char *sha1)
{ {
unsigned long mapsize; unsigned long size;
void *map = map_sha1_file(sha1, &mapsize); char type[20];
if (map) { void *buffer = read_sha1_file(sha1, type, &size);
int is_delta; if (buffer) {
struct object *obj; struct object *obj;
char type[100]; if (check_sha1_signature(sha1, buffer, size, type) < 0)
unsigned long size;
void *buffer = unpack_sha1_file(map, mapsize, type, &size);
munmap(map, mapsize);
if (!buffer)
return NULL;
is_delta = !strcmp(type, "delta");
if (!is_delta && check_sha1_signature(sha1, buffer, size, type) < 0)
printf("sha1 mismatch %s\n", sha1_to_hex(sha1)); printf("sha1 mismatch %s\n", sha1_to_hex(sha1));
if (is_delta) { if (!strcmp(type, "blob")) {
struct delta *delta = lookup_delta(sha1);
parse_delta_buffer(delta, buffer, size);
obj = (struct object *) delta;
} else if (!strcmp(type, "blob")) {
struct blob *blob = lookup_blob(sha1); struct blob *blob = lookup_blob(sha1);
parse_blob_buffer(blob, buffer, size); parse_blob_buffer(blob, buffer, size);
obj = &blob->object; obj = &blob->object;

View File

@ -10,12 +10,10 @@ struct object_list {
struct object { struct object {
unsigned parsed : 1; unsigned parsed : 1;
unsigned used : 1; unsigned used : 1;
unsigned delta : 1;
unsigned int flags; unsigned int flags;
unsigned char sha1[20]; unsigned char sha1[20];
const char *type; const char *type;
struct object_list *refs; struct object_list *refs;
struct object_list *attached_deltas;
void *util; void *util;
}; };

View File

@ -11,7 +11,7 @@ enum object_type {
OBJ_COMMIT, OBJ_COMMIT,
OBJ_TREE, OBJ_TREE,
OBJ_BLOB, OBJ_BLOB,
OBJ_DELTA // NOTE! This is _not_ the same as a "delta" object in the filesystem OBJ_DELTA
}; };
struct object_entry { struct object_entry {

11
pull.c
View File

@ -13,8 +13,6 @@ const unsigned char *current_ref = NULL;
int get_tree = 0; int get_tree = 0;
int get_history = 0; int get_history = 0;
/* 1 means "get delta", 2 means "really check delta harder */
int get_delta = 1;
int get_all = 0; int get_all = 0;
int get_verbosely = 0; int get_verbosely = 0;
static unsigned char current_commit_sha1[20]; static unsigned char current_commit_sha1[20];
@ -47,15 +45,6 @@ static int make_sure_we_have_it(const char *what, unsigned char *sha1)
if (status && what) if (status && what)
report_missing(what, sha1); report_missing(what, sha1);
} }
else if (get_delta < 2)
return 0;
if (get_delta) {
unsigned char delta_sha1[20];
status = sha1_delta_base(sha1, delta_sha1);
if (0 < status)
status = make_sure_we_have_it(what, delta_sha1);
}
return status; return status;
} }

5
pull.h
View File

@ -30,11 +30,6 @@ extern int get_history;
/* Set to fetch the trees in the commit history. */ /* Set to fetch the trees in the commit history. */
extern int get_all; extern int get_all;
/* Set to zero to skip the check for delta object base;
* set to two to check delta dependency even for objects we already have.
*/
extern int get_delta;
/* Set to be verbose */ /* Set to be verbose */
extern int get_verbosely; extern int get_verbosely;

View File

@ -7,7 +7,6 @@
* creation etc. * creation etc.
*/ */
#include "cache.h" #include "cache.h"
#include "delta.h"
#ifndef O_NOATIME #ifndef O_NOATIME
#if defined(__linux__) && (defined(__i386__) || defined(__PPC__)) #if defined(__linux__) && (defined(__i386__) || defined(__PPC__))
@ -410,37 +409,6 @@ void * unpack_sha1_file(void *map, unsigned long mapsize, char *type, unsigned l
return unpack_sha1_rest(&stream, hdr, *size); return unpack_sha1_rest(&stream, hdr, *size);
} }
int sha1_delta_base(const unsigned char *sha1, unsigned char *base_sha1)
{
int ret;
unsigned long mapsize, size;
void *map;
z_stream stream;
char hdr[64], type[20];
void *delta_data_head;
map = map_sha1_file(sha1, &mapsize);
if (!map)
return -1;
ret = unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr));
if (ret < Z_OK || parse_sha1_header(hdr, type, &size) < 0) {
ret = -1;
goto out;
}
if (strcmp(type, "delta")) {
ret = 0;
goto out;
}
delta_data_head = hdr + strlen(hdr) + 1;
ret = 1;
memcpy(base_sha1, delta_data_head, 20);
out:
inflateEnd(&stream);
munmap(map, mapsize);
return ret;
}
int sha1_file_size(const unsigned char *sha1, unsigned long *sizep) int sha1_file_size(const unsigned char *sha1, unsigned long *sizep)
{ {
int ret, status; int ret, status;
@ -448,54 +416,17 @@ int sha1_file_size(const unsigned char *sha1, unsigned long *sizep)
void *map; void *map;
z_stream stream; z_stream stream;
char hdr[64], type[20]; char hdr[64], type[20];
const unsigned char *data;
unsigned char cmd;
int i;
map = map_sha1_file(sha1, &mapsize); map = map_sha1_file(sha1, &mapsize);
if (!map) if (!map)
return -1; return -1;
ret = unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr)); ret = unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr));
status = -1;
if (ret < Z_OK || parse_sha1_header(hdr, type, &size) < 0) if (ret < Z_OK || parse_sha1_header(hdr, type, &size) < 0)
goto out; status = -1;
if (strcmp(type, "delta")) { else {
*sizep = size;
status = 0; status = 0;
goto out; *sizep = size;
} }
/* We are dealing with a delta object. Inflated, the first
* 20 bytes hold the base object SHA1, and delta data follows
* immediately after it.
*
* The initial part of the delta starts at delta_data_head +
* 20. Borrow code from patch-delta to read the result size.
*/
data = (unsigned char *)(hdr + strlen(hdr) + 1 + 20);
/* Skip over the source size; we are not interested in
* it and we cannot verify it because we do not want
* to read the base object.
*/
cmd = *data++;
while (cmd) {
if (cmd & 1)
data++;
cmd >>= 1;
}
/* Read the result size */
size = i = 0;
cmd = *data++;
while (cmd) {
if (cmd & 1)
size |= *data++ << i;
i += 8;
cmd >>= 1;
}
*sizep = size;
status = 0;
out:
inflateEnd(&stream); inflateEnd(&stream);
munmap(map, mapsize); munmap(map, mapsize);
return status; return status;
@ -510,19 +441,6 @@ void * read_sha1_file(const unsigned char *sha1, char *type, unsigned long *size
if (map) { if (map) {
buf = unpack_sha1_file(map, mapsize, type, size); buf = unpack_sha1_file(map, mapsize, type, size);
munmap(map, mapsize); munmap(map, mapsize);
if (buf && !strcmp(type, "delta")) {
void *ref = NULL, *delta = buf;
unsigned long ref_size, delta_size = *size;
buf = NULL;
if (delta_size > 20)
ref = read_sha1_file(delta, type, &ref_size);
if (ref)
buf = patch_delta(ref, ref_size,
delta+20, delta_size-20,
size);
free(delta);
free(ref);
}
return buf; return buf;
} }
return NULL; return NULL;

View File

@ -65,10 +65,6 @@ int main(int argc, char **argv)
get_tree = 1; get_tree = 1;
} else if (argv[arg][1] == 'c') { } else if (argv[arg][1] == 'c') {
get_history = 1; get_history = 1;
} else if (argv[arg][1] == 'd') {
get_delta = 0;
} else if (!strcmp(argv[arg], "--recover")) {
get_delta = 2;
} else if (argv[arg][1] == 'a') { } else if (argv[arg][1] == 'a') {
get_all = 1; get_all = 1;
get_tree = 1; get_tree = 1;

View File

@ -1,79 +0,0 @@
#!/bin/sh
#
# Copyright (c) 2005 Junio C Hamano
#
test_description='Test pulling deltified objects
'
. ./test-lib.sh
locate_obj='s|\(..\)|.git/objects/\1/|'
test_expect_success \
setup \
'cat ../README >a &&
git-update-cache --add a &&
a0=`git-ls-files --stage |
sed -e '\''s/^[0-7]* \([0-9a-f]*\) .*/\1/'\''` &&
sed -e 's/test/TEST/g' ../README >a &&
git-update-cache a &&
a1=`git-ls-files --stage |
sed -e '\''s/^[0-7]* \([0-9a-f]*\) .*/\1/'\''` &&
tree=`git-write-tree` &&
commit=`git-commit-tree $tree </dev/null` &&
a0f=`echo "$a0" | sed -e "$locate_obj"` &&
a1f=`echo "$a1" | sed -e "$locate_obj"` &&
echo commit $commit &&
echo a0 $a0 &&
echo a1 $a1 &&
ls -l $a0f $a1f &&
echo $commit >.git/HEAD &&
git-mkdelta -v $a0 $a1 &&
ls -l $a0f $a1f'
# Now commit has a tree that records delitified "a" whose SHA1 is a1.
# Create a new repo and pull this commit into it.
test_expect_success \
'setup and cd into new repo' \
'mkdir dest && cd dest && rm -fr .git && git-init-db'
test_expect_success \
'pull from deltified repo into a new repo without -d' \
'rm -fr .git a && git-init-db &&
git-local-pull -v -a $commit ../.git/ &&
git-cat-file blob $a1 >a &&
diff -u a ../a'
test_expect_failure \
'pull from deltified repo into a new repo with -d' \
'rm -fr .git a && git-init-db &&
git-local-pull -v -a -d $commit ../.git/ &&
git-cat-file blob $a1 >a &&
diff -u a ../a'
test_expect_failure \
'pull from deltified repo after delta failure without --recover' \
'rm -f a &&
git-local-pull -v -a $commit ../.git/ &&
git-cat-file blob $a1 >a &&
diff -u a ../a'
test_expect_success \
'pull from deltified repo after delta failure with --recover' \
'rm -f a &&
git-local-pull -v -a --recover $commit ../.git/ &&
git-cat-file blob $a1 >a &&
diff -u a ../a'
test_expect_success \
'missing-tree or missing-blob should be re-fetched without --recover' \
'rm -f a $a0f $a1f &&
git-local-pull -v -a $commit ../.git/ &&
git-cat-file blob $a1 >a &&
diff -u a ../a'
test_done