f1294eaf7f
Introduce the constructs for Bloom filters, Bloom filter keys
and Bloom filter settings.
For details on what Bloom filters are and how they work, refer
to Dr. Derrick Stolee's blog post [1]. It provides a concise
explanation of the adoption of Bloom filters as described in
[2] and [3].
Implementation specifics:
1. We currently use 7 and 10 for the number of hashes and the
size of each entry respectively. They served as great starting
values, the mathematical details behind this choice are
described in [1] and [4]. The implementation, while not
completely open to it at the moment, is flexible enough to allow
for tweaking these settings in the future.
Note: The performance gains we have observed with these values
are significant enough that we did not need to tweak these
settings. The performance numbers are included in the cover letter
of this series and in the commit message of the subsequent commit
where we use Bloom filters to speed up `git log -- path`.
2. As described in [1] and [3], we do not need 7 independent hashing
functions. We use the Murmur3 hashing scheme, seed it twice and
then combine those to procure an arbitrary number of hash values.
3. The filters will be sized according to the number of changes in
each commit, in multiples of 8 bit words.
[1] Derrick Stolee
"Supercharging the Git Commit Graph IV: Bloom Filters"
https://devblogs.microsoft.com/devops/super-charging-the-git-commit-graph-iv-Bloom-filters/
[2] Flavio Bonomi, Michael Mitzenmacher, Rina Panigrahy, Sushil Singh, George Varghese
"An Improved Construction for Counting Bloom Filters"
http://theory.stanford.edu/~rinap/papers/esa2006b.pdf
https://doi.org/10.1007/11841036_61
[3] Peter C. Dillinger and Panagiotis Manolios
"Bloom Filters in Probabilistic Verification"
http://www.ccs.neu.edu/home/pete/pub/Bloom-filters-verification.pdf
https://doi.org/10.1007/978-3-540-30494-4_26
[4] Thomas Mueller Graf, Daniel Lemire
"Xor Filters: Faster and Smaller Than Bloom and Cuckoo Filters"
https://arxiv.org/abs/1912.08258
Helped-by: Derrick Stolee <dstolee@microsoft.com>
Reviewed-by: Jakub Narębski <jnareb@gmail.com>
Signed-off-by: Garima Singh <garima.singh@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
70 lines
1.9 KiB
Bash
Executable File
70 lines
1.9 KiB
Bash
Executable File
#!/bin/sh
|
|
|
|
test_description='Testing the various Bloom filter computations in bloom.c'
|
|
. ./test-lib.sh
|
|
|
|
test_expect_success 'compute unseeded murmur3 hash for empty string' '
|
|
cat >expect <<-\EOF &&
|
|
Murmur3 Hash with seed=0:0x00000000
|
|
EOF
|
|
test-tool bloom get_murmur3 "" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute unseeded murmur3 hash for test string 1' '
|
|
cat >expect <<-\EOF &&
|
|
Murmur3 Hash with seed=0:0x627b0c2c
|
|
EOF
|
|
test-tool bloom get_murmur3 "Hello world!" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute unseeded murmur3 hash for test string 2' '
|
|
cat >expect <<-\EOF &&
|
|
Murmur3 Hash with seed=0:0x2e4ff723
|
|
EOF
|
|
test-tool bloom get_murmur3 "The quick brown fox jumps over the lazy dog" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute bloom key for empty string' '
|
|
cat >expect <<-\EOF &&
|
|
Hashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|
|
|
Filter_Length:2
|
|
Filter_Data:11|11|
|
|
EOF
|
|
test-tool bloom generate_filter "" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute bloom key for whitespace' '
|
|
cat >expect <<-\EOF &&
|
|
Hashes:0xf178874c|0x5f3d6eb6|0xcd025620|0x3ac73d8a|0xa88c24f4|0x16510c5e|0x8415f3c8|
|
|
Filter_Length:2
|
|
Filter_Data:51|55|
|
|
EOF
|
|
test-tool bloom generate_filter " " >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute bloom key for test string 1' '
|
|
cat >expect <<-\EOF &&
|
|
Hashes:0xb270de9b|0x1bb6f26e|0x84fd0641|0xee431a14|0x57892de7|0xc0cf41ba|0x2a15558d|
|
|
Filter_Length:2
|
|
Filter_Data:92|6c|
|
|
EOF
|
|
test-tool bloom generate_filter "Hello world!" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_expect_success 'compute bloom key for test string 2' '
|
|
cat >expect <<-\EOF &&
|
|
Hashes:0x20ab385b|0xf5237fe2|0xc99bc769|0x9e140ef0|0x728c5677|0x47049dfe|0x1b7ce585|
|
|
Filter_Length:2
|
|
Filter_Data:a5|4a|
|
|
EOF
|
|
test-tool bloom generate_filter "file.txt" >actual &&
|
|
test_cmp expect actual
|
|
'
|
|
|
|
test_done |