Merge branch 'ac/bitmap-format-doc'
Adjust technical/bitmap-format to be formatted by AsciiDoc, and add some missing information to the documentation. * ac/bitmap-format-doc: bitmap-format.txt: add information for trailing checksum bitmap-format.txt: fix some formatting issues bitmap-format.txt: feed the file to asciidoc to generate html
This commit is contained in:
commit
5dbbdaac79
@ -94,6 +94,7 @@ TECH_DOCS += MyFirstContribution
|
|||||||
TECH_DOCS += MyFirstObjectWalk
|
TECH_DOCS += MyFirstObjectWalk
|
||||||
TECH_DOCS += SubmittingPatches
|
TECH_DOCS += SubmittingPatches
|
||||||
TECH_DOCS += ToolsForGit
|
TECH_DOCS += ToolsForGit
|
||||||
|
TECH_DOCS += technical/bitmap-format
|
||||||
TECH_DOCS += technical/bundle-format
|
TECH_DOCS += technical/bundle-format
|
||||||
TECH_DOCS += technical/cruft-packs
|
TECH_DOCS += technical/cruft-packs
|
||||||
TECH_DOCS += technical/hash-function-transition
|
TECH_DOCS += technical/hash-function-transition
|
||||||
|
@ -25,7 +25,7 @@ An object is uniquely described by its bit position within a bitmap:
|
|||||||
is defined as follows:
|
is defined as follows:
|
||||||
|
|
||||||
o1 <= o2 <==> pack(o1) <= pack(o2) /\ offset(o1) <= offset(o2)
|
o1 <= o2 <==> pack(o1) <= pack(o2) /\ offset(o1) <= offset(o2)
|
||||||
|
+
|
||||||
The ordering between packs is done according to the MIDX's .rev file.
|
The ordering between packs is done according to the MIDX's .rev file.
|
||||||
Notably, the preferred pack sorts ahead of all other packs.
|
Notably, the preferred pack sorts ahead of all other packs.
|
||||||
|
|
||||||
@ -39,19 +39,22 @@ MIDXs, both the bit-cache and rev-cache extensions are required.
|
|||||||
|
|
||||||
== On-disk format
|
== On-disk format
|
||||||
|
|
||||||
- A header appears at the beginning:
|
* A header appears at the beginning:
|
||||||
|
|
||||||
4-byte signature: {'B', 'I', 'T', 'M'}
|
4-byte signature: :: {'B', 'I', 'T', 'M'}
|
||||||
|
|
||||||
|
2-byte version number (network byte order): ::
|
||||||
|
|
||||||
2-byte version number (network byte order)
|
|
||||||
The current implementation only supports version 1
|
The current implementation only supports version 1
|
||||||
of the bitmap index (the same one as JGit).
|
of the bitmap index (the same one as JGit).
|
||||||
|
|
||||||
2-byte flags (network byte order)
|
2-byte flags (network byte order): ::
|
||||||
|
|
||||||
The following flags are supported:
|
The following flags are supported:
|
||||||
|
|
||||||
- BITMAP_OPT_FULL_DAG (0x1) REQUIRED
|
** {empty}
|
||||||
|
BITMAP_OPT_FULL_DAG (0x1) REQUIRED: :::
|
||||||
|
|
||||||
This flag must always be present. It implies that the
|
This flag must always be present. It implies that the
|
||||||
bitmap index has been generated for a packfile or
|
bitmap index has been generated for a packfile or
|
||||||
multi-pack index (MIDX) with full closure (i.e. where
|
multi-pack index (MIDX) with full closure (i.e. where
|
||||||
@ -61,75 +64,83 @@ MIDXs, both the bit-cache and rev-cache extensions are required.
|
|||||||
JGit, that greatly reduces the complexity of the
|
JGit, that greatly reduces the complexity of the
|
||||||
implementation.
|
implementation.
|
||||||
|
|
||||||
- BITMAP_OPT_HASH_CACHE (0x4)
|
** {empty}
|
||||||
|
BITMAP_OPT_HASH_CACHE (0x4): :::
|
||||||
|
|
||||||
If present, the end of the bitmap file contains
|
If present, the end of the bitmap file contains
|
||||||
`N` 32-bit name-hash values, one per object in the
|
`N` 32-bit name-hash values, one per object in the
|
||||||
pack/MIDX. The format and meaning of the name-hash is
|
pack/MIDX. The format and meaning of the name-hash is
|
||||||
described below.
|
described below.
|
||||||
|
|
||||||
4-byte entry count (network byte order)
|
4-byte entry count (network byte order): ::
|
||||||
|
|
||||||
The total count of entries (bitmapped commits) in this bitmap index.
|
The total count of entries (bitmapped commits) in this bitmap index.
|
||||||
|
|
||||||
20-byte checksum
|
20-byte checksum: ::
|
||||||
|
|
||||||
The SHA1 checksum of the pack/MIDX this bitmap index
|
The SHA1 checksum of the pack/MIDX this bitmap index
|
||||||
belongs to.
|
belongs to.
|
||||||
|
|
||||||
- 4 EWAH bitmaps that act as type indexes
|
* 4 EWAH bitmaps that act as type indexes
|
||||||
|
+
|
||||||
Type indexes are serialized after the hash cache in the shape
|
Type indexes are serialized after the hash cache in the shape
|
||||||
of four EWAH bitmaps stored consecutively (see Appendix A for
|
of four EWAH bitmaps stored consecutively (see Appendix A for
|
||||||
the serialization format of an EWAH bitmap).
|
the serialization format of an EWAH bitmap).
|
||||||
|
+
|
||||||
There is a bitmap for each Git object type, stored in the following
|
There is a bitmap for each Git object type, stored in the following
|
||||||
order:
|
order:
|
||||||
|
+
|
||||||
- Commits
|
- Commits
|
||||||
- Trees
|
- Trees
|
||||||
- Blobs
|
- Blobs
|
||||||
- Tags
|
- Tags
|
||||||
|
|
||||||
|
+
|
||||||
In each bitmap, the `n`th bit is set to true if the `n`th object
|
In each bitmap, the `n`th bit is set to true if the `n`th object
|
||||||
in the packfile or multi-pack index is of that type.
|
in the packfile or multi-pack index is of that type.
|
||||||
|
+
|
||||||
The obvious consequence is that the OR of all 4 bitmaps will result
|
The obvious consequence is that the OR of all 4 bitmaps will result
|
||||||
in a full set (all bits set), and the AND of all 4 bitmaps will
|
in a full set (all bits set), and the AND of all 4 bitmaps will
|
||||||
result in an empty bitmap (no bits set).
|
result in an empty bitmap (no bits set).
|
||||||
|
|
||||||
- N entries with compressed bitmaps, one for each indexed commit
|
* N entries with compressed bitmaps, one for each indexed commit
|
||||||
|
+
|
||||||
Where `N` is the total amount of entries in this bitmap index.
|
Where `N` is the total amount of entries in this bitmap index.
|
||||||
Each entry contains the following:
|
Each entry contains the following:
|
||||||
|
|
||||||
- 4-byte object position (network byte order)
|
** {empty}
|
||||||
|
4-byte object position (network byte order): ::
|
||||||
The position **in the index for the packfile or
|
The position **in the index for the packfile or
|
||||||
multi-pack index** where the bitmap for this commit is
|
multi-pack index** where the bitmap for this commit is
|
||||||
found.
|
found.
|
||||||
|
|
||||||
- 1-byte XOR-offset
|
** {empty}
|
||||||
|
1-byte XOR-offset: ::
|
||||||
The xor offset used to compress this bitmap. For an entry
|
The xor offset used to compress this bitmap. For an entry
|
||||||
in position `x`, a XOR offset of `y` means that the actual
|
in position `x`, a XOR offset of `y` means that the actual
|
||||||
bitmap representing this commit is composed by XORing the
|
bitmap representing this commit is composed by XORing the
|
||||||
bitmap for this entry with the bitmap in entry `x-y` (i.e.
|
bitmap for this entry with the bitmap in entry `x-y` (i.e.
|
||||||
the bitmap `y` entries before this one).
|
the bitmap `y` entries before this one).
|
||||||
|
+
|
||||||
Note that this compression can be recursive. In order to
|
NOTE: This compression can be recursive. In order to
|
||||||
XOR this entry with a previous one, the previous entry needs
|
XOR this entry with a previous one, the previous entry needs
|
||||||
to be decompressed first, and so on.
|
to be decompressed first, and so on.
|
||||||
|
+
|
||||||
The hard-limit for this offset is 160 (an entry can only be
|
The hard-limit for this offset is 160 (an entry can only be
|
||||||
xor'ed against one of the 160 entries preceding it). This
|
xor'ed against one of the 160 entries preceding it). This
|
||||||
number is always positive, and hence entries are always xor'ed
|
number is always positive, and hence entries are always xor'ed
|
||||||
with **previous** bitmaps, not bitmaps that will come afterwards
|
with **previous** bitmaps, not bitmaps that will come afterwards
|
||||||
in the index.
|
in the index.
|
||||||
|
|
||||||
- 1-byte flags for this bitmap
|
** {empty}
|
||||||
|
1-byte flags for this bitmap: ::
|
||||||
At the moment the only available flag is `0x1`, which hints
|
At the moment the only available flag is `0x1`, which hints
|
||||||
that this bitmap can be re-used when rebuilding bitmap indexes
|
that this bitmap can be re-used when rebuilding bitmap indexes
|
||||||
for the repository.
|
for the repository.
|
||||||
|
|
||||||
- The compressed bitmap itself, see Appendix A.
|
** The compressed bitmap itself, see Appendix A.
|
||||||
|
|
||||||
|
* {empty}
|
||||||
|
TRAILER: ::
|
||||||
|
Trailing checksum of the preceding contents.
|
||||||
|
|
||||||
== Appendix A: Serialization format for an EWAH bitmap
|
== Appendix A: Serialization format for an EWAH bitmap
|
||||||
|
|
||||||
@ -142,7 +153,7 @@ implementation:
|
|||||||
- 4-byte number of words of the COMPRESSED bitmap, when stored
|
- 4-byte number of words of the COMPRESSED bitmap, when stored
|
||||||
|
|
||||||
- N x 8-byte words, as specified by the previous field
|
- N x 8-byte words, as specified by the previous field
|
||||||
|
+
|
||||||
This is the actual content of the compressed bitmap.
|
This is the actual content of the compressed bitmap.
|
||||||
|
|
||||||
- 4-byte position of the current RLW for the compressed
|
- 4-byte position of the current RLW for the compressed
|
||||||
|
Loading…
x
Reference in New Issue
Block a user