2011-09-03 01:33:22 +02:00
|
|
|
#ifndef CONNECTED_H
|
|
|
|
|
#define CONNECTED_H
|
|
|
|
|
|
2018-08-15 19:54:05 +02:00
|
|
|
struct object_id;
|
2013-05-26 03:16:17 +02:00
|
|
|
struct transport;
|
|
|
|
|
|
2011-09-03 01:33:22 +02:00
|
|
|
/*
|
|
|
|
|
* Take callback data, and return next object name in the buffer.
|
|
|
|
|
* When called after returning the name for the last object, return -1
|
|
|
|
|
* to signal EOF, otherwise return 0.
|
|
|
|
|
*/
|
2021-09-01 15:09:50 +02:00
|
|
|
typedef const struct object_id *(*oid_iterate_fn)(void *);
|
2011-09-03 01:33:22 +02:00
|
|
|
|
check_everything_connected: use a struct with named options
The number of variants of check_everything_connected has
grown over the years, so that the "real" function takes
several possibly-zero, possibly-NULL arguments. We hid the
complexity behind some wrapper functions, but this doesn't
scale well when we want to add new options.
If we add more wrapper variants to handle the new options,
then we can get a combinatorial explosion when those options
might be used together (right now nobody wants to use both
"shallow" and "transport" together, so we get by with just a
few wrappers).
If instead we add new parameters to each function, each of
which can have a default value, then callers who want the
defaults end up with confusing invocations like:
check_everything_connected(fn, 0, data, -1, 0, NULL);
where it is unclear which parameter is which (and every
caller needs updated when we add new options).
Instead, let's add a struct to hold all of the optional
parameters. This is a little more verbose for the callers
(who have to declare the struct and fill it in), but it
makes their code much easier to follow, because every option
is named as it is set (and unused options do not have to be
mentioned at all).
Note that we could also stick the iteration function and its
callback data into the option struct, too. But since those
are required for each call, by avoiding doing so, we can let
very simple callers just pass "NULL" for the options and not
worry about the struct at all.
While we're touching each site, let's also rename the
function to check_connected(). The existing name was quite
long, and not all of the wrappers even used the full name.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-15 12:30:40 +02:00
|
|
|
/*
|
|
|
|
|
* Named-arguments struct for check_connected. All arguments are
|
|
|
|
|
* optional, and can be left to defaults as set by CHECK_CONNECTED_INIT.
|
|
|
|
|
*/
|
|
|
|
|
struct check_connected_options {
|
|
|
|
|
/* Avoid printing any errors to stderr. */
|
|
|
|
|
int quiet;
|
|
|
|
|
|
|
|
|
|
/* --shallow-file to pass to rev-list sub-process */
|
|
|
|
|
const char *shallow_file;
|
|
|
|
|
|
|
|
|
|
/* Transport whose objects we are checking, if available. */
|
|
|
|
|
struct transport *transport;
|
2016-07-15 12:32:03 +02:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If non-zero, send error messages to this descriptor rather
|
|
|
|
|
* than stderr. The descriptor is closed before check_connected
|
|
|
|
|
* returns.
|
|
|
|
|
*/
|
|
|
|
|
int err_fd;
|
2016-07-15 12:32:28 +02:00
|
|
|
|
|
|
|
|
/* If non-zero, show progress as we traverse the objects. */
|
|
|
|
|
int progress;
|
2016-10-03 22:49:08 +02:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Insert these variables into the environment of the child process.
|
|
|
|
|
*/
|
|
|
|
|
const char **env;
|
fetch-pack: write shallow, then check connectivity
When fetching, connectivity is checked after the shallow file is
updated. There are 2 issues with this: (1) the connectivity check is
only performed up to ancestors of existing refs (which is not thorough
enough if we were deepening an existing ref in the first place), and (2)
there is no rollback of the shallow file if the connectivity check
fails.
To solve (1), update the connectivity check to check the ancestry chain
completely in the case of a deepening fetch by refraining from passing
"--not --all" when invoking rev-list in connected.c.
To solve (2), have fetch_pack() perform its own connectivity check
before updating the shallow file. To support existing use cases in which
"git fetch-pack" is used to download objects without much regard as to
the connectivity of the resulting objects with respect to the existing
repository, the connectivity check is only done if necessary (that is,
the fetch is not a clone, and the fetch involves shallow/deepen
functionality). "git fetch" still performs its own connectivity check,
preserving correctness but sometimes performing redundant work. This
redundancy is mitigated by the fact that fetch_pack() reports if it has
performed a connectivity check itself, and if the transport supports
connect or stateless-connect, it will bubble up that report so that "git
fetch" knows not to perform the connectivity check in such a case.
This was noticed when a user tried to deepen an existing repository by
fetching with --no-shallow from a server that did not send all necessary
objects - the connectivity check as run by "git fetch" succeeded, but a
subsequent "git fsck" failed.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-03 00:08:43 +02:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If non-zero, check the ancestry chain completely, not stopping at
|
|
|
|
|
* any existing ref. This is necessary when deepening existing refs
|
|
|
|
|
* during a fetch.
|
|
|
|
|
*/
|
|
|
|
|
unsigned is_deepening_fetch : 1;
|
receive-pack: only use visible refs for connectivity check
When serving a push, git-receive-pack(1) needs to verify that the
packfile sent by the client contains all objects that are required by
the updated references. This connectivity check works by marking all
preexisting references as uninteresting and using the new reference tips
as starting point for a graph walk.
Marking all preexisting references as uninteresting can be a problem
when it comes to performance. Git forges tend to do internal bookkeeping
to keep alive sets of objects for internal use or make them easy to find
via certain references. These references are typically hidden away from
the user so that they are neither advertised nor writeable. At GitLab,
we have one particular repository that contains a total of 7 million
references, of which 6.8 million are indeed internal references. With
the current connectivity check we are forced to load all these
references in order to mark them as uninteresting, and this alone takes
around 15 seconds to compute.
We can optimize this by only taking into account the set of visible refs
when marking objects as uninteresting. This means that we may now walk
more objects until we hit any object that is marked as uninteresting.
But it is rather unlikely that clients send objects that make large
parts of objects reachable that have previously only ever been hidden,
whereas the common case is to push incremental changes that build on top
of the visible object graph.
This provides a huge boost to performance in the mentioned repository,
where the vast majority of its refs hidden. Pushing a new commit into
this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs
as it is configured in Gitaly leads to a 4.5-fold speedup:
Benchmark 1: main
Time (mean ± σ): 30.977 s ± 0.157 s [User: 30.226 s, System: 1.083 s]
Range (min … max): 30.796 s … 31.071 s 3 runs
Benchmark 2: pks-connectivity-check-hide-refs
Time (mean ± σ): 6.799 s ± 0.063 s [User: 6.803 s, System: 0.354 s]
Range (min … max): 6.729 s … 6.850 s 3 runs
Summary
'pks-connectivity-check-hide-refs' ran
4.56 ± 0.05 times faster than 'main'
As we mostly go through the same codepaths even in the case where there
are no hidden refs at all compared to the code before there is no change
in performance when no refs are hidden:
Benchmark 1: main
Time (mean ± σ): 48.188 s ± 0.432 s [User: 49.326 s, System: 5.009 s]
Range (min … max): 47.706 s … 48.539 s 3 runs
Benchmark 2: pks-connectivity-check-hide-refs
Time (mean ± σ): 48.027 s ± 0.500 s [User: 48.934 s, System: 5.025 s]
Range (min … max): 47.504 s … 48.500 s 3 runs
Summary
'pks-connectivity-check-hide-refs' ran
1.00 ± 0.01 times faster than 'main'
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
2022-11-17 06:47:04 +01:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If not NULL, use `--exclude-hidden=$section` to exclude all refs
|
|
|
|
|
* hidden via the `$section.hideRefs` config from the set of
|
|
|
|
|
* already-reachable refs.
|
|
|
|
|
*/
|
|
|
|
|
const char *exclude_hidden_refs_section;
|
check_everything_connected: use a struct with named options
The number of variants of check_everything_connected has
grown over the years, so that the "real" function takes
several possibly-zero, possibly-NULL arguments. We hid the
complexity behind some wrapper functions, but this doesn't
scale well when we want to add new options.
If we add more wrapper variants to handle the new options,
then we can get a combinatorial explosion when those options
might be used together (right now nobody wants to use both
"shallow" and "transport" together, so we get by with just a
few wrappers).
If instead we add new parameters to each function, each of
which can have a default value, then callers who want the
defaults end up with confusing invocations like:
check_everything_connected(fn, 0, data, -1, 0, NULL);
where it is unclear which parameter is which (and every
caller needs updated when we add new options).
Instead, let's add a struct to hold all of the optional
parameters. This is a little more verbose for the callers
(who have to declare the struct and fill it in), but it
makes their code much easier to follow, because every option
is named as it is set (and unused options do not have to be
mentioned at all).
Note that we could also stick the iteration function and its
callback data into the option struct, too. But since those
are required for each call, by avoiding doing so, we can let
very simple callers just pass "NULL" for the options and not
worry about the struct at all.
While we're touching each site, let's also rename the
function to check_connected(). The existing name was quite
long, and not all of the wrappers even used the full name.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-15 12:30:40 +02:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
#define CHECK_CONNECTED_INIT { 0 }
|
|
|
|
|
|
2011-09-03 01:33:22 +02:00
|
|
|
/*
|
2018-09-21 20:22:37 +02:00
|
|
|
* Make sure that all given objects and all objects reachable from them
|
|
|
|
|
* either exist in our object store or (if the repository is a partial
|
|
|
|
|
* clone) are promised to be available.
|
2011-09-03 01:33:22 +02:00
|
|
|
*
|
|
|
|
|
* Return 0 if Ok, non zero otherwise (i.e. some missing objects)
|
check_everything_connected: use a struct with named options
The number of variants of check_everything_connected has
grown over the years, so that the "real" function takes
several possibly-zero, possibly-NULL arguments. We hid the
complexity behind some wrapper functions, but this doesn't
scale well when we want to add new options.
If we add more wrapper variants to handle the new options,
then we can get a combinatorial explosion when those options
might be used together (right now nobody wants to use both
"shallow" and "transport" together, so we get by with just a
few wrappers).
If instead we add new parameters to each function, each of
which can have a default value, then callers who want the
defaults end up with confusing invocations like:
check_everything_connected(fn, 0, data, -1, 0, NULL);
where it is unclear which parameter is which (and every
caller needs updated when we add new options).
Instead, let's add a struct to hold all of the optional
parameters. This is a little more verbose for the callers
(who have to declare the struct and fill it in), but it
makes their code much easier to follow, because every option
is named as it is set (and unused options do not have to be
mentioned at all).
Note that we could also stick the iteration function and its
callback data into the option struct, too. But since those
are required for each call, by avoiding doing so, we can let
very simple callers just pass "NULL" for the options and not
worry about the struct at all.
While we're touching each site, let's also rename the
function to check_connected(). The existing name was quite
long, and not all of the wrappers even used the full name.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-15 12:30:40 +02:00
|
|
|
*
|
|
|
|
|
* If "opt" is NULL, behaves as if CHECK_CONNECTED_INIT was passed.
|
2011-09-03 01:33:22 +02:00
|
|
|
*/
|
2017-10-16 00:06:54 +02:00
|
|
|
int check_connected(oid_iterate_fn fn, void *cb_data,
|
check_everything_connected: use a struct with named options
The number of variants of check_everything_connected has
grown over the years, so that the "real" function takes
several possibly-zero, possibly-NULL arguments. We hid the
complexity behind some wrapper functions, but this doesn't
scale well when we want to add new options.
If we add more wrapper variants to handle the new options,
then we can get a combinatorial explosion when those options
might be used together (right now nobody wants to use both
"shallow" and "transport" together, so we get by with just a
few wrappers).
If instead we add new parameters to each function, each of
which can have a default value, then callers who want the
defaults end up with confusing invocations like:
check_everything_connected(fn, 0, data, -1, 0, NULL);
where it is unclear which parameter is which (and every
caller needs updated when we add new options).
Instead, let's add a struct to hold all of the optional
parameters. This is a little more verbose for the callers
(who have to declare the struct and fill it in), but it
makes their code much easier to follow, because every option
is named as it is set (and unused options do not have to be
mentioned at all).
Note that we could also stick the iteration function and its
callback data into the option struct, too. But since those
are required for each call, by avoiding doing so, we can let
very simple callers just pass "NULL" for the options and not
worry about the struct at all.
While we're touching each site, let's also rename the
function to check_connected(). The existing name was quite
long, and not all of the wrappers even used the full name.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-15 12:30:40 +02:00
|
|
|
struct check_connected_options *opt);
|
2011-09-03 01:33:22 +02:00
|
|
|
|
|
|
|
|
#endif /* CONNECTED_H */
|