665d70ad03
The commit-graph format reserved a byte among the header of the file to
store a "hash version". During the SHA-256 work, this was not modified
because file formats are not necessarily intended to work across hash
versions. If a repository has SHA-256 as its hash algorithm, it
automatically up-shifts the lengths of object names in all necessary
formats.
However, since we have this byte available for adjusting the version, we
can make the file formats more obviously incompatible instead of relying
on other context from the repository.
Update the oid_version() method in commit-graph.c to add a new value, 2,
for sha-256. This automatically writes the new value in a SHA-256
repository _and_ verifies the value is correct. This is a breaking
change relative to the current 'master' branch since 092b677 (Merge
branch 'bc/sha-256-cvs-svn-updates', 2020-08-13) but it is not breaking
relative to any released version of Git.
The test impact is relatively minor: the output of 'test-tool
read-graph' lists the header information, so those instances of '1' need
to be replaced with a variable determined by GIT_TEST_DEFAULT_HASH. A
more careful test is added that specifically creates a repository of
each type then swaps the commit-graph files. The important value here is
that the "git log" command succeeds while writing a message to stderr.
Helped-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Reviewed-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
140 lines
6.0 KiB
Plaintext
140 lines
6.0 KiB
Plaintext
Git commit graph format
|
|
=======================
|
|
|
|
The Git commit graph stores a list of commit OIDs and some associated
|
|
metadata, including:
|
|
|
|
- The generation number of the commit. Commits with no parents have
|
|
generation number 1; commits with parents have generation number
|
|
one more than the maximum generation number of its parents. We
|
|
reserve zero as special, and can be used to mark a generation
|
|
number invalid or as "not computed".
|
|
|
|
- The root tree OID.
|
|
|
|
- The commit date.
|
|
|
|
- The parents of the commit, stored using positional references within
|
|
the graph file.
|
|
|
|
- The Bloom filter of the commit carrying the paths that were changed between
|
|
the commit and its first parent, if requested.
|
|
|
|
These positional references are stored as unsigned 32-bit integers
|
|
corresponding to the array position within the list of commit OIDs. Due
|
|
to some special constants we use to track parents, we can store at most
|
|
(1 << 30) + (1 << 29) + (1 << 28) - 1 (around 1.8 billion) commits.
|
|
|
|
== Commit graph files have the following format:
|
|
|
|
In order to allow extensions that add extra data to the graph, we organize
|
|
the body into "chunks" and provide a binary lookup table at the beginning
|
|
of the body. The header includes certain values, such as number of chunks
|
|
and hash type.
|
|
|
|
All multi-byte numbers are in network byte order.
|
|
|
|
HEADER:
|
|
|
|
4-byte signature:
|
|
The signature is: {'C', 'G', 'P', 'H'}
|
|
|
|
1-byte version number:
|
|
Currently, the only valid version is 1.
|
|
|
|
1-byte Hash Version
|
|
We infer the hash length (H) from this value:
|
|
1 => SHA-1
|
|
2 => SHA-256
|
|
If the hash type does not match the repository's hash algorithm, the
|
|
commit-graph file should be ignored with a warning presented to the
|
|
user.
|
|
|
|
1-byte number (C) of "chunks"
|
|
|
|
1-byte number (B) of base commit-graphs
|
|
We infer the length (H*B) of the Base Graphs chunk
|
|
from this value.
|
|
|
|
CHUNK LOOKUP:
|
|
|
|
(C + 1) * 12 bytes listing the table of contents for the chunks:
|
|
First 4 bytes describe the chunk id. Value 0 is a terminating label.
|
|
Other 8 bytes provide the byte-offset in current file for chunk to
|
|
start. (Chunks are ordered contiguously in the file, so you can infer
|
|
the length using the next chunk position if necessary.) Each chunk
|
|
ID appears at most once.
|
|
|
|
The remaining data in the body is described one chunk at a time, and
|
|
these chunks may be given in any order. Chunks are required unless
|
|
otherwise specified.
|
|
|
|
CHUNK DATA:
|
|
|
|
OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)
|
|
The ith entry, F[i], stores the number of OIDs with first
|
|
byte at most i. Thus F[255] stores the total
|
|
number of commits (N).
|
|
|
|
OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)
|
|
The OIDs for all commits in the graph, sorted in ascending order.
|
|
|
|
Commit Data (ID: {'C', 'D', 'A', 'T' }) (N * (H + 16) bytes)
|
|
* The first H bytes are for the OID of the root tree.
|
|
* The next 8 bytes are for the positions of the first two parents
|
|
of the ith commit. Stores value 0x7000000 if no parent in that
|
|
position. If there are more than two parents, the second value
|
|
has its most-significant bit on and the other bits store an array
|
|
position into the Extra Edge List chunk.
|
|
* The next 8 bytes store the generation number of the commit and
|
|
the commit time in seconds since EPOCH. The generation number
|
|
uses the higher 30 bits of the first 4 bytes, while the commit
|
|
time uses the 32 bits of the second 4 bytes, along with the lowest
|
|
2 bits of the lowest byte, storing the 33rd and 34th bit of the
|
|
commit time.
|
|
|
|
Extra Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]
|
|
This list of 4-byte values store the second through nth parents for
|
|
all octopus merges. The second parent value in the commit data stores
|
|
an array position within this list along with the most-significant bit
|
|
on. Starting at that array position, iterate through this list of commit
|
|
positions for the parents until reaching a value with the most-significant
|
|
bit on. The other bits correspond to the position of the last parent.
|
|
|
|
Bloom Filter Index (ID: {'B', 'I', 'D', 'X'}) (N * 4 bytes) [Optional]
|
|
* The ith entry, BIDX[i], stores the number of bytes in all Bloom filters
|
|
from commit 0 to commit i (inclusive) in lexicographic order. The Bloom
|
|
filter for the i-th commit spans from BIDX[i-1] to BIDX[i] (plus header
|
|
length), where BIDX[-1] is 0.
|
|
* The BIDX chunk is ignored if the BDAT chunk is not present.
|
|
|
|
Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]
|
|
* It starts with header consisting of three unsigned 32-bit integers:
|
|
- Version of the hash algorithm being used. We currently only support
|
|
value 1 which corresponds to the 32-bit version of the murmur3 hash
|
|
implemented exactly as described in
|
|
https://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double
|
|
hashing technique using seed values 0x293ae76f and 0x7e646e2 as
|
|
described in https://doi.org/10.1007/978-3-540-30494-4_26 "Bloom Filters
|
|
in Probabilistic Verification"
|
|
- The number of times a path is hashed and hence the number of bit positions
|
|
that cumulatively determine whether a file is present in the commit.
|
|
- The minimum number of bits 'b' per entry in the Bloom filter. If the filter
|
|
contains 'n' entries, then the filter size is the minimum number of 64-bit
|
|
words that contain n*b bits.
|
|
* The rest of the chunk is the concatenation of all the computed Bloom
|
|
filters for the commits in lexicographic order.
|
|
* Note: Commits with no changes or more than 512 changes have Bloom filters
|
|
of length zero.
|
|
* The BDAT chunk is present if and only if BIDX is present.
|
|
|
|
Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]
|
|
This list of H-byte hashes describe a set of B commit-graph files that
|
|
form a commit-graph chain. The graph position for the ith commit in this
|
|
file's OID Lookup chunk is equal to i plus the number of commits in all
|
|
base graphs. If B is non-zero, this chunk must exist.
|
|
|
|
TRAILER:
|
|
|
|
H-byte HASH-checksum of all of the above.
|