f32dde8c12
The previous changes to the line-log machinery focused on making the
first result appear faster. This was achieved by no longer walking the
entire commit history before returning the early results. There is still
another way to improve the performance: walk most commits much faster.
Let's use the changed-path Bloom filters to reduce time spent computing
diffs.
Since the line-log computation requires opening blobs and checking the
content-diff, there is still a lot of necessary computation that cannot
be replaced with changed-path Bloom filters. The part that we can reduce
is most effective when checking the history of a file that is deep in
several directories and those directories are modified frequently. In
this case, the computation to check if a commit is TREESAME to its first
parent takes a large fraction of the time. That is ripe for improvement
with changed-path Bloom filters.
We must ensure that prepare_to_use_bloom_filters() is called in
revision.c so that the bloom_filter_settings are loaded into the struct
rev_info from the commit-graph. Of course, some cases are still
forbidden, but in the line-log case the pathspec is provided in a
different way than normal.
Since multiple paths and segments could be requested, we compute the
struct bloom_key data dynamically during the commit walk. This could
likely be improved, but adds code complexity that is not valuable at
this time.
There are two cases to care about: merge commits and "ordinary" commits.
Merge commits have multiple parents, but if we are TREESAME to our first
parent in every range, then pass the blame for all ranges to the first
parent. Ordinary commits have the same condition, but each is done
slightly differently in the process_ranges_[merge|ordinary]_commit()
methods. By checking if the changed-path Bloom filter can guarantee
TREESAME, we can avoid that tree-diff cost. If the filter says "probably
changed", then we need to run the tree-diff and then the blob-diff if
there was a real edit.
The Linux kernel repository is a good testing ground for the performance
improvements claimed here. There are two different cases to test. The
first is the "entire history" case, where we output the entire history
to /dev/null to see how long it would take to compute the full line-log
history. The second is the "first result" case, where we find how long
it takes to show the first value, which is an indicator of how quickly a
user would see responses when waiting at a terminal.
To test, I selected the paths that were changed most frequently in the
top 10,000 commits using this command (stolen from StackOverflow [1]):
git log --pretty=format: --name-only -n 10000 | sort | \
uniq -c | sort -rg | head -10
which results in
121 MAINTAINERS
63 fs/namei.c
60 arch/x86/kvm/cpuid.c
59 fs/io_uring.c
58 arch/x86/kvm/vmx/vmx.c
51 arch/x86/kvm/x86.c
45 arch/x86/kvm/svm.c
42 fs/btrfs/disk-io.c
42 Documentation/scsi/index.rst
(along with a bogus first result). It appears that the path
arch/x86/kvm/svm.c was renamed, so we ignore that entry. This leaves the
following results for the real command time:
| | Entire History | First Result |
| Path | Before | After | Before | After |
|------------------------------|--------|--------|--------|--------|
| MAINTAINERS | 4.26 s | 3.87 s | 0.41 s | 0.39 s |
| fs/namei.c | 1.99 s | 0.99 s | 0.42 s | 0.21 s |
| arch/x86/kvm/cpuid.c | 5.28 s | 1.12 s | 0.16 s | 0.09 s |
| fs/io_uring.c | 4.34 s | 0.99 s | 0.94 s | 0.27 s |
| arch/x86/kvm/vmx/vmx.c | 5.01 s | 1.34 s | 0.21 s | 0.12 s |
| arch/x86/kvm/x86.c | 2.24 s | 1.18 s | 0.21 s | 0.14 s |
| fs/btrfs/disk-io.c | 1.82 s | 1.01 s | 0.06 s | 0.05 s |
| Documentation/scsi/index.rst | 3.30 s | 0.89 s | 1.46 s | 0.03 s |
It is worth noting that the least speedup comes for the MAINTAINERS file
which is
* edited frequently,
* low in the directory heirarchy, and
* quite a large file.
All of those points lead to spending more time doing the blob diff and
less time doing the tree diff. Still, we see some improvement in that
case and significant improvement in other cases. A 2-4x speedup is
likely the more typical case as opposed to the small 5% change for that
file.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
91 lines
2.4 KiB
C
91 lines
2.4 KiB
C
#ifndef BLOOM_H
|
|
#define BLOOM_H
|
|
|
|
struct commit;
|
|
struct repository;
|
|
|
|
struct bloom_filter_settings {
|
|
/*
|
|
* The version of the hashing technique being used.
|
|
* We currently only support version = 1 which is
|
|
* the seeded murmur3 hashing technique implemented
|
|
* in bloom.c.
|
|
*/
|
|
uint32_t hash_version;
|
|
|
|
/*
|
|
* The number of times a path is hashed, i.e. the
|
|
* number of bit positions tht cumulatively
|
|
* determine whether a path is present in the
|
|
* Bloom filter.
|
|
*/
|
|
uint32_t num_hashes;
|
|
|
|
/*
|
|
* The minimum number of bits per entry in the Bloom
|
|
* filter. If the filter contains 'n' entries, then
|
|
* filter size is the minimum number of 8-bit words
|
|
* that contain n*b bits.
|
|
*/
|
|
uint32_t bits_per_entry;
|
|
};
|
|
|
|
#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10 }
|
|
#define BITS_PER_WORD 8
|
|
#define BLOOMDATA_CHUNK_HEADER_SIZE 3 * sizeof(uint32_t)
|
|
|
|
/*
|
|
* A bloom_filter struct represents a data segment to
|
|
* use when testing hash values. The 'len' member
|
|
* dictates how many entries are stored in
|
|
* 'data'.
|
|
*/
|
|
struct bloom_filter {
|
|
unsigned char *data;
|
|
size_t len;
|
|
};
|
|
|
|
/*
|
|
* A bloom_key represents the k hash values for a
|
|
* given string. These can be precomputed and
|
|
* stored in a bloom_key for re-use when testing
|
|
* against a bloom_filter. The number of hashes is
|
|
* given by the Bloom filter settings and is the same
|
|
* for all Bloom filters and keys interacting with
|
|
* the loaded version of the commit graph file and
|
|
* the Bloom data chunks.
|
|
*/
|
|
struct bloom_key {
|
|
uint32_t *hashes;
|
|
};
|
|
|
|
/*
|
|
* Calculate the murmur3 32-bit hash value for the given data
|
|
* using the given seed.
|
|
* Produces a uniformly distributed hash value.
|
|
* Not considered to be cryptographically secure.
|
|
* Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm
|
|
*/
|
|
uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);
|
|
|
|
void fill_bloom_key(const char *data,
|
|
size_t len,
|
|
struct bloom_key *key,
|
|
const struct bloom_filter_settings *settings);
|
|
void clear_bloom_key(struct bloom_key *key);
|
|
|
|
void add_key_to_filter(const struct bloom_key *key,
|
|
struct bloom_filter *filter,
|
|
const struct bloom_filter_settings *settings);
|
|
|
|
void init_bloom_filters(void);
|
|
|
|
struct bloom_filter *get_bloom_filter(struct repository *r,
|
|
struct commit *c,
|
|
int compute_if_not_present);
|
|
|
|
int bloom_filter_contains(const struct bloom_filter *filter,
|
|
const struct bloom_key *key,
|
|
const struct bloom_filter_settings *settings);
|
|
|
|
#endif |