2007-03-14 02:58:22 +01:00
|
|
|
/*
|
|
|
|
* git gc builtin command
|
|
|
|
*
|
|
|
|
* Cleanup unreachable files and optimize the repository.
|
|
|
|
*
|
|
|
|
* Copyright (c) 2007 James Bowes
|
|
|
|
*
|
|
|
|
* Based on git-gc.sh, which is
|
|
|
|
*
|
|
|
|
* Copyright (c) 2006 Shawn O. Pearce
|
|
|
|
*/
|
|
|
|
|
2007-07-15 01:14:45 +02:00
|
|
|
#include "builtin.h"
|
2018-03-23 18:20:59 +01:00
|
|
|
#include "repository.h"
|
2017-06-14 20:07:36 +02:00
|
|
|
#include "config.h"
|
2015-08-10 11:47:49 +02:00
|
|
|
#include "tempfile.h"
|
2014-10-01 12:28:42 +02:00
|
|
|
#include "lockfile.h"
|
2007-11-02 02:02:27 +01:00
|
|
|
#include "parse-options.h"
|
2007-03-14 02:58:22 +01:00
|
|
|
#include "run-command.h"
|
gc: remove gc.pid file at end of execution
This file isn't really harmful, but isn't useful either, and can create
minor annoyance for the user:
* It's confusing, as the presence of a *.pid file often implies that a
process is currently running. A user running "ls .git/" and finding
this file may incorrectly guess that a "git gc" is currently running.
* Leaving this file means that a "git gc" in an already gc-ed repo is
no-longer a no-op. A user running "git gc" in a set of repositories,
and then synchronizing this set (e.g. rsync -av, unison, ...) will see
all the gc.pid files as changed, which creates useless noise.
This patch unlinks the file after the garbage collection is done, so that
gc.pid is actually present only during execution.
Future versions of Git may want to use the information left in the gc.pid
file (e.g. for policies like "don't attempt to run a gc if one has
already been ran less than X hours ago"). If so, this patch can safely be
reverted. For now, let's not bother the users.
Explained-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Improved-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-17 01:11:46 +02:00
|
|
|
#include "sigchain.h"
|
2020-07-28 22:23:39 +02:00
|
|
|
#include "strvec.h"
|
2013-12-05 14:02:54 +01:00
|
|
|
#include "commit.h"
|
2018-06-27 15:24:46 +02:00
|
|
|
#include "commit-graph.h"
|
2017-08-19 00:20:26 +02:00
|
|
|
#include "packfile.h"
|
2018-03-23 18:20:59 +01:00
|
|
|
#include "object-store.h"
|
2018-04-15 17:36:17 +02:00
|
|
|
#include "pack.h"
|
|
|
|
#include "pack-objects.h"
|
|
|
|
#include "blob.h"
|
|
|
|
#include "tree.h"
|
2019-06-25 15:40:31 +02:00
|
|
|
#include "promisor-remote.h"
|
2007-03-14 02:58:22 +01:00
|
|
|
|
|
|
|
#define FAILED_RUN "failed to run %s"
|
|
|
|
|
2007-11-02 02:02:27 +01:00
|
|
|
static const char * const builtin_gc_usage[] = {
|
2015-01-13 08:44:47 +01:00
|
|
|
N_("git gc [<options>]"),
|
2007-11-02 02:02:27 +01:00
|
|
|
NULL
|
|
|
|
};
|
2007-03-14 02:58:22 +01:00
|
|
|
|
Make "git gc" pack all refs by default
I've taught myself to use "git gc" instead of doing the repack explicitly,
but it doesn't actually do what I think it should do.
We've had packed refs for a long time now, and I think it just makes sense
to pack normal branches too. So I end up having to do
git pack-refs --all --prune
in order to get a nice git repo that doesn't have any unnecessary files.
So why not just do that in "git gc"? It's not as if there really is any
downside to packing branches, even if they end up changing later. Quite
often they don't, and even if they do, so what?
Also, make the default for refs packing just be an unambiguous "do it",
rather than "do it by default only for non-bare repositories". If you want
that behaviour, you can always just add a
[gc]
packrefs = notbare
in your ~/.gitconfig file, but I don't actually see why bare would be any
different (except for the broken reason that http-fetching used to be
totally broken, and not doing it just meant that it didn't even get
fixed in a timely manner!).
So here's a trivial patch to make "git gc" do a better job. Hmm?
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-24 20:41:39 +02:00
|
|
|
static int pack_refs = 1;
|
2014-05-25 02:38:29 +02:00
|
|
|
static int prune_reflogs = 1;
|
gc: default aggressive depth to 50
This commit message is long and has lots of background and
numbers. The summary is: the current default of 250 doesn't
save much space, and costs CPU. It's not a good tradeoff.
Read on for details.
The "--aggressive" flag to git-gc does three things:
1. use "-f" to throw out existing deltas and recompute from
scratch
2. use "--window=250" to look harder for deltas
3. use "--depth=250" to make longer delta chains
Items (1) and (2) are good matches for an "aggressive"
repack. They ask the repack to do more computation work in
the hopes of getting a better pack. You pay the costs during
the repack, and other operations see only the benefit.
Item (3) is not so clear. Allowing longer chains means fewer
restrictions on the deltas, which means potentially finding
better ones and saving some space. But it also means that
operations which access the deltas have to follow longer
chains, which affects their performance. So it's a tradeoff,
and it's not clear that the tradeoff is even a good one.
The existing "250" numbers for "--aggressive" come
originally from this thread:
http://public-inbox.org/git/alpine.LFD.0.9999.0712060803430.13796@woody.linux-foundation.org/
where Linus says:
So when I said "--depth=250 --window=250", I chose those
numbers more as an example of extremely aggressive
packing, and I'm not at all sure that the end result is
necessarily wonderfully usable. It's going to save disk
space (and network bandwidth - the delta's will be re-used
for the network protocol too!), but there are definitely
downsides too, and using long delta chains may
simply not be worth it in practice.
There are some numbers in that thread, but they're mostly
focused on the improved window size, and measure the
improvement from --depth=250 and --window=250 together.
E.g.:
http://public-inbox.org/git/9e4733910712062006l651571f3w7f76ce64c6650dff@mail.gmail.com/
talks about the improved run-time of "git-blame", which
comes from the reduced pack size. But most of that reduction
is coming from --window=250, whereas most of the extra costs
come from --depth=250. There's a link in that thread showing
that increasing the depth beyond 50 doesn't seem to help
much with the size:
https://vcscompare.blogspot.com/2008/06/git-repack-parameters.html
but again, no discussion of the timing impact.
In an earlier thread from Ted Ts'o which discussed setting
the non-aggressive default (from 10 to 50):
http://public-inbox.org/git/20070509134958.GA21489%40thunk.org/
we have more numbers, with the conclusion that going past 50
does not help size much, and hurts the speed of normal
operations.
So from that, we might guess that 50 is actually a sweet
spot, even for aggressive, if we interpret aggressive to
"spend time now to make a better pack". It is not clear that
"--depth=250" is actually a better pack. It may be slightly
_smaller_, but it carries a run-time penalty.
Here are some more recent timings I did to verify that. They
show three things:
- the size of the resulting pack (so disk saved to store,
bandwidth saved on clones/fetches)
- the cost of "rev-list --objects --all", which shows the
effect of the delta chains on trees (commits typically
don't delta, and the command doesn't touch the blobs at
all)
- the cost of "log -Sfoo", which will additionally access
each blob
All cases were repacked with "git repack -adf --depth=$d
--window=250" (so basically, what would happen if we tweaked
the "gc --aggressive" default depth).
The timings are all wall-clock best-of-3. The machine itself
has plenty of RAM compared to the repositories (which is
probably typical of most workstations these days), so we're
really measuring CPU usage, as the whole thing will be in
disk cache after the first run.
The core.deltaBaseCacheLimit is at its default of 96MiB.
It's possible that tweaking it would have some impact on the
tests, as some of them (especially "log -S" on a large repo)
are likely to overflow that. But bumping that carries a
run-time memory cost, so for these tests, I focused on what
we could do just with the on-disk pack tradeoffs.
Each test is done for four depths: 250 (the current value),
50 (the current default that tested well previously), 100
(to show something on the larger side, which previous tests
showed was not a good tradeoff), and 10 (the very old
default, which previous tests showed was worse than 50).
Here are the numbers for linux.git:
depth | size | % | rev-list | % | log -Sfoo | %
-------+-------+-------+----------+--------+-----------+-------
250 | 967MB | n/a | 48.159s | n/a | 378.088 | n/a
100 | 971MB | +0.4% | 41.471s | -13.9% | 342.060 | -9.5%
50 | 979MB | +1.2% | 37.778s | -21.6% | 311.040s | -17.7%
10 | 1.1GB | +6.6% | 32.518s | -32.5% | 279.890s | -25.9%
and for git.git:
depth | size | % | rev-list | % | log -Sfoo | %
-------+-------+-------+----------+--------+-----------+-------
250 | 48MB | n/a | 2.215s | n/a | 20.922s | n/a
100 | 49MB | +0.5% | 2.140s | -3.4% | 17.736s | -15.2%
50 | 49MB | +1.7% | 2.099s | -5.2% | 15.418s | -26.3%
10 | 53MB | +9.3% | 2.001s | -9.7% | 12.677s | -39.4%
You can see that that the CPU savings for regular operations improves as we
decrease the depth. The savings are less for "rev-list" on a smaller repository
than they are for blob-accessing operations, or even rev-list on a larger
repository. This may mean that a larger delta cache would help (though setting
core.deltaBaseCacheLimit by itself doesn't).
But we can also see that the space savings are not that great as the depth goes
higher. Saving 5-10% between 10 and 50 is probably worth the CPU tradeoff.
Saving 1% to go from 50 to 100, or another 0.5% to go from 100 to 250 is
probably not.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-11 18:13:09 +02:00
|
|
|
static int aggressive_depth = 50;
|
2007-12-06 13:03:38 +01:00
|
|
|
static int aggressive_window = 250;
|
2007-09-05 22:01:37 +02:00
|
|
|
static int gc_auto_threshold = 6700;
|
2008-03-23 08:04:48 +01:00
|
|
|
static int gc_auto_pack_limit = 50;
|
2014-02-08 08:08:52 +01:00
|
|
|
static int detach_auto = 1;
|
2017-04-26 21:29:31 +02:00
|
|
|
static timestamp_t gc_log_expire_time;
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
static const char *gc_log_expire = "1.day.ago";
|
2008-09-30 22:28:58 +02:00
|
|
|
static const char *prune_expire = "2.weeks.ago";
|
2014-11-30 09:24:53 +01:00
|
|
|
static const char *prune_worktrees_expire = "3.months.ago";
|
2018-04-15 17:36:15 +02:00
|
|
|
static unsigned long big_pack_threshold;
|
2018-04-15 17:36:17 +02:00
|
|
|
static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;
|
2007-03-14 02:58:22 +01:00
|
|
|
|
2020-07-28 22:24:27 +02:00
|
|
|
static struct strvec pack_refs_cmd = STRVEC_INIT;
|
|
|
|
static struct strvec reflog = STRVEC_INIT;
|
|
|
|
static struct strvec repack = STRVEC_INIT;
|
|
|
|
static struct strvec prune = STRVEC_INIT;
|
|
|
|
static struct strvec prune_worktrees = STRVEC_INIT;
|
|
|
|
static struct strvec rerere = STRVEC_INIT;
|
2007-03-14 02:58:22 +01:00
|
|
|
|
tempfile: auto-allocate tempfiles on heap
The previous commit taught the tempfile code to give up
ownership over tempfiles that have been renamed or deleted.
That makes it possible to use a stack variable like this:
struct tempfile t;
create_tempfile(&t, ...);
...
if (!err)
rename_tempfile(&t, ...);
else
delete_tempfile(&t);
But doing it this way has a high potential for creating
memory errors. The tempfile we pass to create_tempfile()
ends up on a global linked list, and it's not safe for it to
go out of scope until we've called one of those two
deactivation functions.
Imagine that we add an early return from the function that
forgets to call delete_tempfile(). With a static or heap
tempfile variable, the worst case is that the tempfile hangs
around until the program exits (and some functions like
setup_shallow_temporary rely on this intentionally, creating
a tempfile and then leaving it for later cleanup).
But with a stack variable as above, this is a serious memory
error: the variable goes out of scope and may be filled with
garbage by the time the tempfile code looks at it. Let's
see if we can make it harder to get this wrong.
Since many callers need to allocate arbitrary numbers of
tempfiles, we can't rely on static storage as a general
solution. So we need to turn to the heap. We could just ask
all callers to pass us a heap variable, but that puts the
burden on them to call free() at the right time.
Instead, let's have the tempfile code handle the heap
allocation _and_ the deallocation (when the tempfile is
deactivated and removed from the list).
This changes the return value of all of the creation
functions. For the cleanup functions (delete and rename),
we'll add one extra bit of safety: instead of taking a
tempfile pointer, we'll take a pointer-to-pointer and set it
to NULL after freeing the object. This makes it safe to
double-call functions like delete_tempfile(), as the second
call treats the NULL input as a noop. Several callsites
follow this pattern.
The resulting patch does have a fair bit of noise, as each
caller needs to be converted to handle:
1. Storing a pointer instead of the struct itself.
2. Passing the pointer instead of taking the struct
address.
3. Handling a "struct tempfile *" return instead of a file
descriptor.
We could play games to make this less noisy. For example, by
defining the tempfile like this:
struct tempfile {
struct heap_allocated_part_of_tempfile {
int fd;
...etc
} *actual_data;
}
Callers would continue to have a "struct tempfile", and it
would be "active" only when the inner pointer was non-NULL.
But that just makes things more awkward in the long run.
There aren't that many callers, so we can simply bite
the bullet and adjust all of them. And the compiler makes it
easy for us to find them all.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-05 14:15:08 +02:00
|
|
|
static struct tempfile *pidfile;
|
2015-09-19 07:13:23 +02:00
|
|
|
static struct lock_file log_lock;
|
gc: remove gc.pid file at end of execution
This file isn't really harmful, but isn't useful either, and can create
minor annoyance for the user:
* It's confusing, as the presence of a *.pid file often implies that a
process is currently running. A user running "ls .git/" and finding
this file may incorrectly guess that a "git gc" is currently running.
* Leaving this file means that a "git gc" in an already gc-ed repo is
no-longer a no-op. A user running "git gc" in a set of repositories,
and then synchronizing this set (e.g. rsync -av, unison, ...) will see
all the gc.pid files as changed, which creates useless noise.
This patch unlinks the file after the garbage collection is done, so that
gc.pid is actually present only during execution.
Future versions of Git may want to use the information left in the gc.pid
file (e.g. for policies like "don't attempt to run a gc if one has
already been ran less than X hours ago"). If so, this patch can safely be
reverted. For now, let's not bother the users.
Explained-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Improved-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-17 01:11:46 +02:00
|
|
|
|
2015-11-04 04:05:08 +01:00
|
|
|
static struct string_list pack_garbage = STRING_LIST_INIT_DUP;
|
|
|
|
|
|
|
|
static void clean_pack_garbage(void)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
for (i = 0; i < pack_garbage.nr; i++)
|
|
|
|
unlink_or_warn(pack_garbage.items[i].string);
|
|
|
|
string_list_clear(&pack_garbage, 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void report_pack_garbage(unsigned seen_bits, const char *path)
|
|
|
|
{
|
|
|
|
if (seen_bits == PACKDIR_FILE_IDX)
|
|
|
|
string_list_append(&pack_garbage, path);
|
|
|
|
}
|
|
|
|
|
2015-09-19 07:13:23 +02:00
|
|
|
static void process_log_file(void)
|
|
|
|
{
|
|
|
|
struct stat st;
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
if (fstat(get_lock_file_fd(&log_lock), &st)) {
|
|
|
|
/*
|
|
|
|
* Perhaps there was an i/o error or another
|
|
|
|
* unlikely situation. Try to make a note of
|
|
|
|
* this in gc.log along with any existing
|
|
|
|
* messages.
|
|
|
|
*/
|
|
|
|
int saved_errno = errno;
|
|
|
|
fprintf(stderr, _("Failed to fstat %s: %s"),
|
tempfile: auto-allocate tempfiles on heap
The previous commit taught the tempfile code to give up
ownership over tempfiles that have been renamed or deleted.
That makes it possible to use a stack variable like this:
struct tempfile t;
create_tempfile(&t, ...);
...
if (!err)
rename_tempfile(&t, ...);
else
delete_tempfile(&t);
But doing it this way has a high potential for creating
memory errors. The tempfile we pass to create_tempfile()
ends up on a global linked list, and it's not safe for it to
go out of scope until we've called one of those two
deactivation functions.
Imagine that we add an early return from the function that
forgets to call delete_tempfile(). With a static or heap
tempfile variable, the worst case is that the tempfile hangs
around until the program exits (and some functions like
setup_shallow_temporary rely on this intentionally, creating
a tempfile and then leaving it for later cleanup).
But with a stack variable as above, this is a serious memory
error: the variable goes out of scope and may be filled with
garbage by the time the tempfile code looks at it. Let's
see if we can make it harder to get this wrong.
Since many callers need to allocate arbitrary numbers of
tempfiles, we can't rely on static storage as a general
solution. So we need to turn to the heap. We could just ask
all callers to pass us a heap variable, but that puts the
burden on them to call free() at the right time.
Instead, let's have the tempfile code handle the heap
allocation _and_ the deallocation (when the tempfile is
deactivated and removed from the list).
This changes the return value of all of the creation
functions. For the cleanup functions (delete and rename),
we'll add one extra bit of safety: instead of taking a
tempfile pointer, we'll take a pointer-to-pointer and set it
to NULL after freeing the object. This makes it safe to
double-call functions like delete_tempfile(), as the second
call treats the NULL input as a noop. Several callsites
follow this pattern.
The resulting patch does have a fair bit of noise, as each
caller needs to be converted to handle:
1. Storing a pointer instead of the struct itself.
2. Passing the pointer instead of taking the struct
address.
3. Handling a "struct tempfile *" return instead of a file
descriptor.
We could play games to make this less noisy. For example, by
defining the tempfile like this:
struct tempfile {
struct heap_allocated_part_of_tempfile {
int fd;
...etc
} *actual_data;
}
Callers would continue to have a "struct tempfile", and it
would be "active" only when the inner pointer was non-NULL.
But that just makes things more awkward in the long run.
There aren't that many callers, so we can simply bite
the bullet and adjust all of them. And the compiler makes it
easy for us to find them all.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-05 14:15:08 +02:00
|
|
|
get_tempfile_path(log_lock.tempfile),
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
strerror(saved_errno));
|
|
|
|
fflush(stderr);
|
2015-09-19 07:13:23 +02:00
|
|
|
commit_lock_file(&log_lock);
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
errno = saved_errno;
|
|
|
|
} else if (st.st_size) {
|
|
|
|
/* There was some error recorded in the lock file */
|
|
|
|
commit_lock_file(&log_lock);
|
|
|
|
} else {
|
|
|
|
/* No error, clean up any old gc.log */
|
|
|
|
unlink(git_path("gc.log"));
|
2015-09-19 07:13:23 +02:00
|
|
|
rollback_lock_file(&log_lock);
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
}
|
2015-09-19 07:13:23 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
static void process_log_file_at_exit(void)
|
|
|
|
{
|
|
|
|
fflush(stderr);
|
|
|
|
process_log_file();
|
|
|
|
}
|
|
|
|
|
|
|
|
static void process_log_file_on_signal(int signo)
|
|
|
|
{
|
|
|
|
process_log_file();
|
|
|
|
sigchain_pop(signo);
|
|
|
|
raise(signo);
|
|
|
|
}
|
|
|
|
|
gc: handle & check gc.reflogExpire config
Don't redundantly run "git reflog expire --all" when gc.reflogExpire
and gc.reflogExpireUnreachable are set to "never", and die immediately
if those configuration valuer are bad.
As an earlier "assert lack of early exit" change to the tests for "git
reflog expire" shows, an early check of gc.reflogExpire{Unreachable,}
isn't wanted in general for "git reflog expire", but it makes sense
for "gc" because:
1) Similarly to 8ab5aa4bd8 ("parseopt: handle malformed --expire
arguments more nicely", 2018-04-21) we'll now die early if the
config variables are set to invalid values.
We run "pack-refs" before "reflog expire", which can take a while,
only to then die on an invalid gc.reflogExpire{Unreachable,}
configuration.
2) Not invoking the command at all means it won't show up in trace
output, which makes what's going on more obvious when the two are
set to "never".
3) As a later change documents we lock the refs when looping over the
refs to expire, even in cases where we end up doing nothing due to
this config.
For the reasons noted in the earlier "assert lack of early exit"
change I don't think it's worth it to bend over backwards in "git
reflog expire" itself to carefully detect if we'll really do
nothing given the combination of all its possible options and skip
that locking, but that's easy to detect here in "gc" where we'll
only run "reflog expire" in a relatively simple mode.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-28 17:14:34 +01:00
|
|
|
static int gc_config_is_timestamp_never(const char *var)
|
|
|
|
{
|
|
|
|
const char *value;
|
|
|
|
timestamp_t expire;
|
|
|
|
|
|
|
|
if (!git_config_get_value(var, &value) && value) {
|
|
|
|
if (parse_expiry_date(value, &expire))
|
|
|
|
die(_("failed to parse '%s' value '%s'"), var, value);
|
|
|
|
return expire == 0;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2014-08-07 18:21:22 +02:00
|
|
|
static void gc_config(void)
|
2007-03-14 02:58:22 +01:00
|
|
|
{
|
2014-08-07 18:21:22 +02:00
|
|
|
const char *value;
|
|
|
|
|
|
|
|
if (!git_config_get_value("gc.packrefs", &value)) {
|
2008-02-08 15:26:18 +01:00
|
|
|
if (value && !strcmp(value, "notbare"))
|
2007-03-14 02:58:22 +01:00
|
|
|
pack_refs = -1;
|
|
|
|
else
|
2014-08-07 18:21:22 +02:00
|
|
|
pack_refs = git_config_bool("gc.packrefs", value);
|
2007-09-17 09:55:13 +02:00
|
|
|
}
|
2014-08-07 18:21:22 +02:00
|
|
|
|
gc: handle & check gc.reflogExpire config
Don't redundantly run "git reflog expire --all" when gc.reflogExpire
and gc.reflogExpireUnreachable are set to "never", and die immediately
if those configuration valuer are bad.
As an earlier "assert lack of early exit" change to the tests for "git
reflog expire" shows, an early check of gc.reflogExpire{Unreachable,}
isn't wanted in general for "git reflog expire", but it makes sense
for "gc" because:
1) Similarly to 8ab5aa4bd8 ("parseopt: handle malformed --expire
arguments more nicely", 2018-04-21) we'll now die early if the
config variables are set to invalid values.
We run "pack-refs" before "reflog expire", which can take a while,
only to then die on an invalid gc.reflogExpire{Unreachable,}
configuration.
2) Not invoking the command at all means it won't show up in trace
output, which makes what's going on more obvious when the two are
set to "never".
3) As a later change documents we lock the refs when looping over the
refs to expire, even in cases where we end up doing nothing due to
this config.
For the reasons noted in the earlier "assert lack of early exit"
change I don't think it's worth it to bend over backwards in "git
reflog expire" itself to carefully detect if we'll really do
nothing given the combination of all its possible options and skip
that locking, but that's easy to detect here in "gc" where we'll
only run "reflog expire" in a relatively simple mode.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-28 17:14:34 +01:00
|
|
|
if (gc_config_is_timestamp_never("gc.reflogexpire") &&
|
|
|
|
gc_config_is_timestamp_never("gc.reflogexpireunreachable"))
|
|
|
|
prune_reflogs = 0;
|
|
|
|
|
2014-08-07 18:21:22 +02:00
|
|
|
git_config_get_int("gc.aggressivewindow", &aggressive_window);
|
|
|
|
git_config_get_int("gc.aggressivedepth", &aggressive_depth);
|
|
|
|
git_config_get_int("gc.auto", &gc_auto_threshold);
|
|
|
|
git_config_get_int("gc.autopacklimit", &gc_auto_pack_limit);
|
|
|
|
git_config_get_bool("gc.autodetach", &detach_auto);
|
2017-02-27 19:00:13 +01:00
|
|
|
git_config_get_expiry("gc.pruneexpire", &prune_expire);
|
|
|
|
git_config_get_expiry("gc.worktreepruneexpire", &prune_worktrees_expire);
|
2017-03-17 21:50:23 +01:00
|
|
|
git_config_get_expiry("gc.logexpiry", &gc_log_expire);
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
|
2018-04-15 17:36:15 +02:00
|
|
|
git_config_get_ulong("gc.bigpackthreshold", &big_pack_threshold);
|
2018-04-15 17:36:17 +02:00
|
|
|
git_config_get_ulong("pack.deltacachesize", &max_delta_cache_size);
|
2018-04-15 17:36:15 +02:00
|
|
|
|
2014-08-07 18:21:22 +02:00
|
|
|
git_config(git_default_config, NULL);
|
2007-03-14 02:58:22 +01:00
|
|
|
}
|
|
|
|
|
2007-09-17 09:44:17 +02:00
|
|
|
static int too_many_loose_objects(void)
|
2007-09-05 22:01:37 +02:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Quickly check if a "gc" is needed, by estimating how
|
|
|
|
* many loose objects there are. Because SHA-1 is evenly
|
|
|
|
* distributed, we can check only one and get a reasonable
|
|
|
|
* estimate.
|
|
|
|
*/
|
|
|
|
DIR *dir;
|
|
|
|
struct dirent *ent;
|
|
|
|
int auto_threshold;
|
|
|
|
int num_loose = 0;
|
|
|
|
int needed = 0;
|
2019-03-15 16:59:53 +01:00
|
|
|
const unsigned hexsz_loose = the_hash_algo->hexsz - 2;
|
2007-09-05 22:01:37 +02:00
|
|
|
|
2017-03-28 21:47:03 +02:00
|
|
|
dir = opendir(git_path("objects/17"));
|
2007-09-05 22:01:37 +02:00
|
|
|
if (!dir)
|
|
|
|
return 0;
|
|
|
|
|
2017-07-08 12:35:35 +02:00
|
|
|
auto_threshold = DIV_ROUND_UP(gc_auto_threshold, 256);
|
2007-09-05 22:01:37 +02:00
|
|
|
while ((ent = readdir(dir)) != NULL) {
|
2019-03-15 16:59:53 +01:00
|
|
|
if (strspn(ent->d_name, "0123456789abcdef") != hexsz_loose ||
|
|
|
|
ent->d_name[hexsz_loose] != '\0')
|
2007-09-05 22:01:37 +02:00
|
|
|
continue;
|
|
|
|
if (++num_loose > auto_threshold) {
|
|
|
|
needed = 1;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
closedir(dir);
|
|
|
|
return needed;
|
|
|
|
}
|
|
|
|
|
2018-04-15 17:36:17 +02:00
|
|
|
static struct packed_git *find_base_packs(struct string_list *packs,
|
|
|
|
unsigned long limit)
|
2018-04-15 17:36:14 +02:00
|
|
|
{
|
|
|
|
struct packed_git *p, *base = NULL;
|
|
|
|
|
2018-08-20 18:52:04 +02:00
|
|
|
for (p = get_all_packs(the_repository); p; p = p->next) {
|
2018-04-15 17:36:14 +02:00
|
|
|
if (!p->pack_local)
|
|
|
|
continue;
|
2018-04-15 17:36:15 +02:00
|
|
|
if (limit) {
|
|
|
|
if (p->pack_size >= limit)
|
|
|
|
string_list_append(packs, p->pack_name);
|
|
|
|
} else if (!base || base->pack_size < p->pack_size) {
|
2018-04-15 17:36:14 +02:00
|
|
|
base = p;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (base)
|
|
|
|
string_list_append(packs, base->pack_name);
|
2018-04-15 17:36:17 +02:00
|
|
|
|
|
|
|
return base;
|
2018-04-15 17:36:14 +02:00
|
|
|
}
|
|
|
|
|
2007-09-17 09:55:13 +02:00
|
|
|
static int too_many_packs(void)
|
|
|
|
{
|
|
|
|
struct packed_git *p;
|
|
|
|
int cnt;
|
|
|
|
|
|
|
|
if (gc_auto_pack_limit <= 0)
|
|
|
|
return 0;
|
|
|
|
|
2018-08-20 18:52:04 +02:00
|
|
|
for (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {
|
2007-09-17 09:55:13 +02:00
|
|
|
if (!p->pack_local)
|
|
|
|
continue;
|
2008-11-12 18:59:07 +01:00
|
|
|
if (p->pack_keep)
|
2007-09-17 09:55:13 +02:00
|
|
|
continue;
|
|
|
|
/*
|
|
|
|
* Perhaps check the size of the pack and count only
|
|
|
|
* very small ones here?
|
|
|
|
*/
|
|
|
|
cnt++;
|
|
|
|
}
|
2016-06-25 08:46:47 +02:00
|
|
|
return gc_auto_pack_limit < cnt;
|
2007-09-17 09:55:13 +02:00
|
|
|
}
|
|
|
|
|
2018-04-15 17:36:17 +02:00
|
|
|
static uint64_t total_ram(void)
|
|
|
|
{
|
|
|
|
#if defined(HAVE_SYSINFO)
|
|
|
|
struct sysinfo si;
|
|
|
|
|
|
|
|
if (!sysinfo(&si))
|
|
|
|
return si.totalram;
|
|
|
|
#elif defined(HAVE_BSD_SYSCTL) && (defined(HW_MEMSIZE) || defined(HW_PHYSMEM))
|
|
|
|
int64_t physical_memory;
|
|
|
|
int mib[2];
|
|
|
|
size_t length;
|
|
|
|
|
|
|
|
mib[0] = CTL_HW;
|
|
|
|
# if defined(HW_MEMSIZE)
|
|
|
|
mib[1] = HW_MEMSIZE;
|
|
|
|
# else
|
|
|
|
mib[1] = HW_PHYSMEM;
|
|
|
|
# endif
|
|
|
|
length = sizeof(int64_t);
|
|
|
|
if (!sysctl(mib, 2, &physical_memory, &length, NULL, 0))
|
|
|
|
return physical_memory;
|
|
|
|
#elif defined(GIT_WINDOWS_NATIVE)
|
|
|
|
MEMORYSTATUSEX memInfo;
|
|
|
|
|
|
|
|
memInfo.dwLength = sizeof(MEMORYSTATUSEX);
|
|
|
|
if (GlobalMemoryStatusEx(&memInfo))
|
|
|
|
return memInfo.ullTotalPhys;
|
|
|
|
#endif
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static uint64_t estimate_repack_memory(struct packed_git *pack)
|
|
|
|
{
|
|
|
|
unsigned long nr_objects = approximate_object_count();
|
|
|
|
size_t os_cache, heap;
|
|
|
|
|
|
|
|
if (!pack || !nr_objects)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* First we have to scan through at least one pack.
|
|
|
|
* Assume enough room in OS file cache to keep the entire pack
|
|
|
|
* or we may accidentally evict data of other processes from
|
|
|
|
* the cache.
|
|
|
|
*/
|
|
|
|
os_cache = pack->pack_size + pack->index_size;
|
|
|
|
/* then pack-objects needs lots more for book keeping */
|
|
|
|
heap = sizeof(struct object_entry) * nr_objects;
|
|
|
|
/*
|
|
|
|
* internal rev-list --all --objects takes up some memory too,
|
|
|
|
* let's say half of it is for blobs
|
|
|
|
*/
|
|
|
|
heap += sizeof(struct blob) * nr_objects / 2;
|
|
|
|
/*
|
|
|
|
* and the other half is for trees (commits and tags are
|
|
|
|
* usually insignificant)
|
|
|
|
*/
|
|
|
|
heap += sizeof(struct tree) * nr_objects / 2;
|
|
|
|
/* and then obj_hash[], underestimated in fact */
|
|
|
|
heap += sizeof(struct object *) * nr_objects;
|
|
|
|
/* revindex is used also */
|
|
|
|
heap += sizeof(struct revindex_entry) * nr_objects;
|
|
|
|
/*
|
|
|
|
* read_sha1_file() (either at delta calculation phase, or
|
|
|
|
* writing phase) also fills up the delta base cache
|
|
|
|
*/
|
|
|
|
heap += delta_base_cache_limit;
|
|
|
|
/* and of course pack-objects has its own delta cache */
|
|
|
|
heap += max_delta_cache_size;
|
|
|
|
|
|
|
|
return os_cache + heap;
|
|
|
|
}
|
|
|
|
|
2018-04-15 17:36:14 +02:00
|
|
|
static int keep_one_pack(struct string_list_item *item, void *data)
|
|
|
|
{
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_pushf(&repack, "--keep-pack=%s", basename(item->string));
|
2018-04-15 17:36:14 +02:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void add_repack_all_option(struct string_list *keep_pack)
|
2012-04-07 12:30:09 +02:00
|
|
|
{
|
|
|
|
if (prune_expire && !strcmp(prune_expire, "now"))
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_push(&repack, "-a");
|
2012-04-07 12:30:09 +02:00
|
|
|
else {
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_push(&repack, "-A");
|
2012-04-18 23:10:19 +02:00
|
|
|
if (prune_expire)
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_pushf(&repack, "--unpack-unreachable=%s", prune_expire);
|
2012-04-07 12:30:09 +02:00
|
|
|
}
|
2018-04-15 17:36:14 +02:00
|
|
|
|
|
|
|
if (keep_pack)
|
|
|
|
for_each_string_list(keep_pack, keep_one_pack, NULL);
|
2012-04-07 12:30:09 +02:00
|
|
|
}
|
|
|
|
|
2016-12-28 23:45:41 +01:00
|
|
|
static void add_repack_incremental_option(void)
|
|
|
|
{
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_push(&repack, "--no-write-bitmap-index");
|
2016-12-28 23:45:41 +01:00
|
|
|
}
|
|
|
|
|
2007-09-17 09:44:17 +02:00
|
|
|
static int need_to_gc(void)
|
|
|
|
{
|
|
|
|
/*
|
2008-03-19 22:53:20 +01:00
|
|
|
* Setting gc.auto to 0 or negative can disable the
|
|
|
|
* automatic gc.
|
2007-09-17 09:44:17 +02:00
|
|
|
*/
|
2008-03-19 22:53:20 +01:00
|
|
|
if (gc_auto_threshold <= 0)
|
2007-09-17 09:48:39 +02:00
|
|
|
return 0;
|
|
|
|
|
2007-09-17 09:55:13 +02:00
|
|
|
/*
|
|
|
|
* If there are too many loose objects, but not too many
|
|
|
|
* packs, we run "repack -d -l". If there are too many packs,
|
|
|
|
* we run "repack -A -d -l". Otherwise we tell the caller
|
|
|
|
* there is no need.
|
|
|
|
*/
|
2018-04-15 17:36:15 +02:00
|
|
|
if (too_many_packs()) {
|
|
|
|
struct string_list keep_pack = STRING_LIST_INIT_NODUP;
|
|
|
|
|
2018-04-15 17:36:16 +02:00
|
|
|
if (big_pack_threshold) {
|
2018-04-15 17:36:15 +02:00
|
|
|
find_base_packs(&keep_pack, big_pack_threshold);
|
2018-04-15 17:36:16 +02:00
|
|
|
if (keep_pack.nr >= gc_auto_pack_limit) {
|
|
|
|
big_pack_threshold = 0;
|
|
|
|
string_list_clear(&keep_pack, 0);
|
|
|
|
find_base_packs(&keep_pack, 0);
|
|
|
|
}
|
2018-04-15 17:36:17 +02:00
|
|
|
} else {
|
|
|
|
struct packed_git *p = find_base_packs(&keep_pack, 0);
|
|
|
|
uint64_t mem_have, mem_want;
|
|
|
|
|
|
|
|
mem_have = total_ram();
|
|
|
|
mem_want = estimate_repack_memory(p);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Only allow 1/2 of memory for pack-objects, leave
|
|
|
|
* the rest for the OS and other processes in the
|
|
|
|
* system.
|
|
|
|
*/
|
|
|
|
if (!mem_have || mem_want < mem_have / 2)
|
|
|
|
string_list_clear(&keep_pack, 0);
|
2018-04-15 17:36:16 +02:00
|
|
|
}
|
2018-04-15 17:36:15 +02:00
|
|
|
|
|
|
|
add_repack_all_option(&keep_pack);
|
|
|
|
string_list_clear(&keep_pack, 0);
|
|
|
|
} else if (too_many_loose_objects())
|
2016-12-28 23:45:41 +01:00
|
|
|
add_repack_incremental_option();
|
|
|
|
else
|
2007-09-17 09:55:13 +02:00
|
|
|
return 0;
|
2008-04-02 21:34:38 +02:00
|
|
|
|
2014-03-18 11:00:53 +01:00
|
|
|
if (run_hook_le(NULL, "pre-auto-gc", NULL))
|
2008-04-02 21:34:38 +02:00
|
|
|
return 0;
|
2007-09-17 09:48:39 +02:00
|
|
|
return 1;
|
2007-09-17 09:44:17 +02:00
|
|
|
}
|
|
|
|
|
2013-08-08 13:05:38 +02:00
|
|
|
/* return NULL on success, else hostname running the gc */
|
|
|
|
static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
|
|
|
|
{
|
2018-05-09 22:55:38 +02:00
|
|
|
struct lock_file lock = LOCK_INIT;
|
2017-04-18 23:57:42 +02:00
|
|
|
char my_host[HOST_NAME_MAX + 1];
|
2013-08-08 13:05:38 +02:00
|
|
|
struct strbuf sb = STRBUF_INIT;
|
|
|
|
struct stat st;
|
|
|
|
uintmax_t pid;
|
|
|
|
FILE *fp;
|
2014-01-29 17:59:37 +01:00
|
|
|
int fd;
|
2015-08-10 11:47:48 +02:00
|
|
|
char *pidfile_path;
|
2013-08-08 13:05:38 +02:00
|
|
|
|
tempfile: auto-allocate tempfiles on heap
The previous commit taught the tempfile code to give up
ownership over tempfiles that have been renamed or deleted.
That makes it possible to use a stack variable like this:
struct tempfile t;
create_tempfile(&t, ...);
...
if (!err)
rename_tempfile(&t, ...);
else
delete_tempfile(&t);
But doing it this way has a high potential for creating
memory errors. The tempfile we pass to create_tempfile()
ends up on a global linked list, and it's not safe for it to
go out of scope until we've called one of those two
deactivation functions.
Imagine that we add an early return from the function that
forgets to call delete_tempfile(). With a static or heap
tempfile variable, the worst case is that the tempfile hangs
around until the program exits (and some functions like
setup_shallow_temporary rely on this intentionally, creating
a tempfile and then leaving it for later cleanup).
But with a stack variable as above, this is a serious memory
error: the variable goes out of scope and may be filled with
garbage by the time the tempfile code looks at it. Let's
see if we can make it harder to get this wrong.
Since many callers need to allocate arbitrary numbers of
tempfiles, we can't rely on static storage as a general
solution. So we need to turn to the heap. We could just ask
all callers to pass us a heap variable, but that puts the
burden on them to call free() at the right time.
Instead, let's have the tempfile code handle the heap
allocation _and_ the deallocation (when the tempfile is
deactivated and removed from the list).
This changes the return value of all of the creation
functions. For the cleanup functions (delete and rename),
we'll add one extra bit of safety: instead of taking a
tempfile pointer, we'll take a pointer-to-pointer and set it
to NULL after freeing the object. This makes it safe to
double-call functions like delete_tempfile(), as the second
call treats the NULL input as a noop. Several callsites
follow this pattern.
The resulting patch does have a fair bit of noise, as each
caller needs to be converted to handle:
1. Storing a pointer instead of the struct itself.
2. Passing the pointer instead of taking the struct
address.
3. Handling a "struct tempfile *" return instead of a file
descriptor.
We could play games to make this less noisy. For example, by
defining the tempfile like this:
struct tempfile {
struct heap_allocated_part_of_tempfile {
int fd;
...etc
} *actual_data;
}
Callers would continue to have a "struct tempfile", and it
would be "active" only when the inner pointer was non-NULL.
But that just makes things more awkward in the long run.
There aren't that many callers, so we can simply bite
the bullet and adjust all of them. And the compiler makes it
easy for us to find them all.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-05 14:15:08 +02:00
|
|
|
if (is_tempfile_active(pidfile))
|
gc: remove gc.pid file at end of execution
This file isn't really harmful, but isn't useful either, and can create
minor annoyance for the user:
* It's confusing, as the presence of a *.pid file often implies that a
process is currently running. A user running "ls .git/" and finding
this file may incorrectly guess that a "git gc" is currently running.
* Leaving this file means that a "git gc" in an already gc-ed repo is
no-longer a no-op. A user running "git gc" in a set of repositories,
and then synchronizing this set (e.g. rsync -av, unison, ...) will see
all the gc.pid files as changed, which creates useless noise.
This patch unlinks the file after the garbage collection is done, so that
gc.pid is actually present only during execution.
Future versions of Git may want to use the information left in the gc.pid
file (e.g. for policies like "don't attempt to run a gc if one has
already been ran less than X hours ago"). If so, this patch can safely be
reverted. For now, let's not bother the users.
Explained-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Improved-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-17 01:11:46 +02:00
|
|
|
/* already locked */
|
|
|
|
return NULL;
|
|
|
|
|
2017-04-18 23:57:43 +02:00
|
|
|
if (xgethostname(my_host, sizeof(my_host)))
|
2015-09-24 23:06:08 +02:00
|
|
|
xsnprintf(my_host, sizeof(my_host), "unknown");
|
2013-08-08 13:05:38 +02:00
|
|
|
|
2015-08-10 11:47:48 +02:00
|
|
|
pidfile_path = git_pathdup("gc.pid");
|
|
|
|
fd = hold_lock_file_for_update(&lock, pidfile_path,
|
2013-08-08 13:05:38 +02:00
|
|
|
LOCK_DIE_ON_ERROR);
|
|
|
|
if (!force) {
|
2017-04-18 23:57:42 +02:00
|
|
|
static char locking_host[HOST_NAME_MAX + 1];
|
|
|
|
static char *scan_fmt;
|
2014-01-29 17:59:37 +01:00
|
|
|
int should_exit;
|
2017-04-18 23:57:42 +02:00
|
|
|
|
|
|
|
if (!scan_fmt)
|
2017-09-17 05:16:55 +02:00
|
|
|
scan_fmt = xstrfmt("%s %%%ds", "%"SCNuMAX, HOST_NAME_MAX);
|
2015-08-10 11:47:48 +02:00
|
|
|
fp = fopen(pidfile_path, "r");
|
2013-08-08 13:05:38 +02:00
|
|
|
memset(locking_host, 0, sizeof(locking_host));
|
|
|
|
should_exit =
|
|
|
|
fp != NULL &&
|
|
|
|
!fstat(fileno(fp), &st) &&
|
|
|
|
/*
|
|
|
|
* 12 hour limit is very generous as gc should
|
|
|
|
* never take that long. On the other hand we
|
|
|
|
* don't really need a strict limit here,
|
|
|
|
* running gc --auto one day late is not a big
|
|
|
|
* problem. --force can be used in manual gc
|
|
|
|
* after the user verifies that no gc is
|
|
|
|
* running.
|
|
|
|
*/
|
|
|
|
time(NULL) - st.st_mtime <= 12 * 3600 &&
|
2017-04-18 23:57:42 +02:00
|
|
|
fscanf(fp, scan_fmt, &pid, locking_host) == 2 &&
|
2013-08-08 13:05:38 +02:00
|
|
|
/* be gentle to concurrent "gc" on remote hosts */
|
2013-12-31 13:07:39 +01:00
|
|
|
(strcmp(locking_host, my_host) || !kill(pid, 0) || errno == EPERM);
|
2013-08-08 13:05:38 +02:00
|
|
|
if (fp != NULL)
|
|
|
|
fclose(fp);
|
|
|
|
if (should_exit) {
|
|
|
|
if (fd >= 0)
|
|
|
|
rollback_lock_file(&lock);
|
|
|
|
*ret_pid = pid;
|
2015-08-10 11:47:48 +02:00
|
|
|
free(pidfile_path);
|
2013-08-08 13:05:38 +02:00
|
|
|
return locking_host;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
strbuf_addf(&sb, "%"PRIuMAX" %s",
|
|
|
|
(uintmax_t) getpid(), my_host);
|
|
|
|
write_in_full(fd, sb.buf, sb.len);
|
|
|
|
strbuf_release(&sb);
|
|
|
|
commit_lock_file(&lock);
|
tempfile: auto-allocate tempfiles on heap
The previous commit taught the tempfile code to give up
ownership over tempfiles that have been renamed or deleted.
That makes it possible to use a stack variable like this:
struct tempfile t;
create_tempfile(&t, ...);
...
if (!err)
rename_tempfile(&t, ...);
else
delete_tempfile(&t);
But doing it this way has a high potential for creating
memory errors. The tempfile we pass to create_tempfile()
ends up on a global linked list, and it's not safe for it to
go out of scope until we've called one of those two
deactivation functions.
Imagine that we add an early return from the function that
forgets to call delete_tempfile(). With a static or heap
tempfile variable, the worst case is that the tempfile hangs
around until the program exits (and some functions like
setup_shallow_temporary rely on this intentionally, creating
a tempfile and then leaving it for later cleanup).
But with a stack variable as above, this is a serious memory
error: the variable goes out of scope and may be filled with
garbage by the time the tempfile code looks at it. Let's
see if we can make it harder to get this wrong.
Since many callers need to allocate arbitrary numbers of
tempfiles, we can't rely on static storage as a general
solution. So we need to turn to the heap. We could just ask
all callers to pass us a heap variable, but that puts the
burden on them to call free() at the right time.
Instead, let's have the tempfile code handle the heap
allocation _and_ the deallocation (when the tempfile is
deactivated and removed from the list).
This changes the return value of all of the creation
functions. For the cleanup functions (delete and rename),
we'll add one extra bit of safety: instead of taking a
tempfile pointer, we'll take a pointer-to-pointer and set it
to NULL after freeing the object. This makes it safe to
double-call functions like delete_tempfile(), as the second
call treats the NULL input as a noop. Several callsites
follow this pattern.
The resulting patch does have a fair bit of noise, as each
caller needs to be converted to handle:
1. Storing a pointer instead of the struct itself.
2. Passing the pointer instead of taking the struct
address.
3. Handling a "struct tempfile *" return instead of a file
descriptor.
We could play games to make this less noisy. For example, by
defining the tempfile like this:
struct tempfile {
struct heap_allocated_part_of_tempfile {
int fd;
...etc
} *actual_data;
}
Callers would continue to have a "struct tempfile", and it
would be "active" only when the inner pointer was non-NULL.
But that just makes things more awkward in the long run.
There aren't that many callers, so we can simply bite
the bullet and adjust all of them. And the compiler makes it
easy for us to find them all.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-05 14:15:08 +02:00
|
|
|
pidfile = register_tempfile(pidfile_path);
|
2015-08-10 11:47:49 +02:00
|
|
|
free(pidfile_path);
|
2013-08-08 13:05:38 +02:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
gc: do not return error for prior errors in daemonized mode
Some build machines started consistently failing to fetch updated
source using "repo sync", with error
error: The last gc run reported the following. Please correct the root cause
and remove /build/.repo/projects/tools/git.git/gc.log.
Automatic cleanup will not be performed until the file is removed.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
The cause takes some time to describe.
In v2.0.0-rc0~145^2 (gc: config option for running --auto in
background, 2014-02-08), "git gc --auto" learned to run in the
background instead of blocking the invoking command. In this mode, it
closed stderr to avoid interleaving output with any subsequent
commands, causing warnings like the above to be swallowed; v2.6.3~24^2
(gc: save log from daemonized gc --auto and print it next time,
2015-09-19) addressed that by storing any diagnostic output in
.git/gc.log and allowing the next "git gc --auto" run to print it.
To avoid wasteful repeated fruitless gcs, when gc.log is present, the
subsequent "gc --auto" would die after printing its contents. Most
git commands, such as "git fetch", ignore the exit status from "git gc
--auto" so all is well at this point: the user gets to see the error
message, and the fetch succeeds, without a wasteful additional attempt
at an automatic gc.
External tools like repo[1], though, do care about the exit status
from "git gc --auto". In non-daemonized mode, the exit status is
straightforward: if there is an error, it is nonzero, but after a
warning like the above, the status is zero. The daemonized mode, as a
side effect of the other properties provided, offers a very strange
exit code convention:
- if no housekeeping was required, the exit status is 0
- the first real run, after forking into the background, returns exit
status 0 unconditionally. The parent process has no way to know
whether gc will succeed.
- if there is any diagnostic output in gc.log, subsequent runs return
a nonzero exit status to indicate that gc was not triggered.
There's nothing for the calling program to act on on the basis of that
error. Use status 0 consistently instead, to indicate that we decided
not to run a gc (just like if no housekeeping was required). This
way, repo and similar tools can get the benefit of the same behavior
as tools like "git fetch" that ignore the exit status from gc --auto.
Once the period of time described by gc.pruneExpire elapses, the
unreachable loose objects will be removed by "git gc --auto"
automatically.
[1] https://gerrit-review.googlesource.com/c/git-repo/+/10598/
Reported-by: Andrii Dehtiarov <adehtiarov@google.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-17 08:57:40 +02:00
|
|
|
/*
|
|
|
|
* Returns 0 if there was no previous error and gc can proceed, 1 if
|
|
|
|
* gc should not proceed due to an error in the last run. Prints a
|
2019-11-05 18:07:23 +01:00
|
|
|
* message and returns -1 if an error occurred while reading gc.log
|
gc: do not return error for prior errors in daemonized mode
Some build machines started consistently failing to fetch updated
source using "repo sync", with error
error: The last gc run reported the following. Please correct the root cause
and remove /build/.repo/projects/tools/git.git/gc.log.
Automatic cleanup will not be performed until the file is removed.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
The cause takes some time to describe.
In v2.0.0-rc0~145^2 (gc: config option for running --auto in
background, 2014-02-08), "git gc --auto" learned to run in the
background instead of blocking the invoking command. In this mode, it
closed stderr to avoid interleaving output with any subsequent
commands, causing warnings like the above to be swallowed; v2.6.3~24^2
(gc: save log from daemonized gc --auto and print it next time,
2015-09-19) addressed that by storing any diagnostic output in
.git/gc.log and allowing the next "git gc --auto" run to print it.
To avoid wasteful repeated fruitless gcs, when gc.log is present, the
subsequent "gc --auto" would die after printing its contents. Most
git commands, such as "git fetch", ignore the exit status from "git gc
--auto" so all is well at this point: the user gets to see the error
message, and the fetch succeeds, without a wasteful additional attempt
at an automatic gc.
External tools like repo[1], though, do care about the exit status
from "git gc --auto". In non-daemonized mode, the exit status is
straightforward: if there is an error, it is nonzero, but after a
warning like the above, the status is zero. The daemonized mode, as a
side effect of the other properties provided, offers a very strange
exit code convention:
- if no housekeeping was required, the exit status is 0
- the first real run, after forking into the background, returns exit
status 0 unconditionally. The parent process has no way to know
whether gc will succeed.
- if there is any diagnostic output in gc.log, subsequent runs return
a nonzero exit status to indicate that gc was not triggered.
There's nothing for the calling program to act on on the basis of that
error. Use status 0 consistently instead, to indicate that we decided
not to run a gc (just like if no housekeeping was required). This
way, repo and similar tools can get the benefit of the same behavior
as tools like "git fetch" that ignore the exit status from gc --auto.
Once the period of time described by gc.pruneExpire elapses, the
unreachable loose objects will be removed by "git gc --auto"
automatically.
[1] https://gerrit-review.googlesource.com/c/git-repo/+/10598/
Reported-by: Andrii Dehtiarov <adehtiarov@google.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-17 08:57:40 +02:00
|
|
|
*/
|
|
|
|
static int report_last_gc_error(void)
|
2015-09-19 07:13:23 +02:00
|
|
|
{
|
|
|
|
struct strbuf sb = STRBUF_INIT;
|
gc: do not return error for prior errors in daemonized mode
Some build machines started consistently failing to fetch updated
source using "repo sync", with error
error: The last gc run reported the following. Please correct the root cause
and remove /build/.repo/projects/tools/git.git/gc.log.
Automatic cleanup will not be performed until the file is removed.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
The cause takes some time to describe.
In v2.0.0-rc0~145^2 (gc: config option for running --auto in
background, 2014-02-08), "git gc --auto" learned to run in the
background instead of blocking the invoking command. In this mode, it
closed stderr to avoid interleaving output with any subsequent
commands, causing warnings like the above to be swallowed; v2.6.3~24^2
(gc: save log from daemonized gc --auto and print it next time,
2015-09-19) addressed that by storing any diagnostic output in
.git/gc.log and allowing the next "git gc --auto" run to print it.
To avoid wasteful repeated fruitless gcs, when gc.log is present, the
subsequent "gc --auto" would die after printing its contents. Most
git commands, such as "git fetch", ignore the exit status from "git gc
--auto" so all is well at this point: the user gets to see the error
message, and the fetch succeeds, without a wasteful additional attempt
at an automatic gc.
External tools like repo[1], though, do care about the exit status
from "git gc --auto". In non-daemonized mode, the exit status is
straightforward: if there is an error, it is nonzero, but after a
warning like the above, the status is zero. The daemonized mode, as a
side effect of the other properties provided, offers a very strange
exit code convention:
- if no housekeeping was required, the exit status is 0
- the first real run, after forking into the background, returns exit
status 0 unconditionally. The parent process has no way to know
whether gc will succeed.
- if there is any diagnostic output in gc.log, subsequent runs return
a nonzero exit status to indicate that gc was not triggered.
There's nothing for the calling program to act on on the basis of that
error. Use status 0 consistently instead, to indicate that we decided
not to run a gc (just like if no housekeeping was required). This
way, repo and similar tools can get the benefit of the same behavior
as tools like "git fetch" that ignore the exit status from gc --auto.
Once the period of time described by gc.pruneExpire elapses, the
unreachable loose objects will be removed by "git gc --auto"
automatically.
[1] https://gerrit-review.googlesource.com/c/git-repo/+/10598/
Reported-by: Andrii Dehtiarov <adehtiarov@google.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-17 08:57:40 +02:00
|
|
|
int ret = 0;
|
2018-07-17 08:53:21 +02:00
|
|
|
ssize_t len;
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
struct stat st;
|
|
|
|
char *gc_log_path = git_pathdup("gc.log");
|
2015-09-19 07:13:23 +02:00
|
|
|
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
if (stat(gc_log_path, &st)) {
|
|
|
|
if (errno == ENOENT)
|
|
|
|
goto done;
|
|
|
|
|
gc: do not return error for prior errors in daemonized mode
Some build machines started consistently failing to fetch updated
source using "repo sync", with error
error: The last gc run reported the following. Please correct the root cause
and remove /build/.repo/projects/tools/git.git/gc.log.
Automatic cleanup will not be performed until the file is removed.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
The cause takes some time to describe.
In v2.0.0-rc0~145^2 (gc: config option for running --auto in
background, 2014-02-08), "git gc --auto" learned to run in the
background instead of blocking the invoking command. In this mode, it
closed stderr to avoid interleaving output with any subsequent
commands, causing warnings like the above to be swallowed; v2.6.3~24^2
(gc: save log from daemonized gc --auto and print it next time,
2015-09-19) addressed that by storing any diagnostic output in
.git/gc.log and allowing the next "git gc --auto" run to print it.
To avoid wasteful repeated fruitless gcs, when gc.log is present, the
subsequent "gc --auto" would die after printing its contents. Most
git commands, such as "git fetch", ignore the exit status from "git gc
--auto" so all is well at this point: the user gets to see the error
message, and the fetch succeeds, without a wasteful additional attempt
at an automatic gc.
External tools like repo[1], though, do care about the exit status
from "git gc --auto". In non-daemonized mode, the exit status is
straightforward: if there is an error, it is nonzero, but after a
warning like the above, the status is zero. The daemonized mode, as a
side effect of the other properties provided, offers a very strange
exit code convention:
- if no housekeeping was required, the exit status is 0
- the first real run, after forking into the background, returns exit
status 0 unconditionally. The parent process has no way to know
whether gc will succeed.
- if there is any diagnostic output in gc.log, subsequent runs return
a nonzero exit status to indicate that gc was not triggered.
There's nothing for the calling program to act on on the basis of that
error. Use status 0 consistently instead, to indicate that we decided
not to run a gc (just like if no housekeeping was required). This
way, repo and similar tools can get the benefit of the same behavior
as tools like "git fetch" that ignore the exit status from gc --auto.
Once the period of time described by gc.pruneExpire elapses, the
unreachable loose objects will be removed by "git gc --auto"
automatically.
[1] https://gerrit-review.googlesource.com/c/git-repo/+/10598/
Reported-by: Andrii Dehtiarov <adehtiarov@google.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-17 08:57:40 +02:00
|
|
|
ret = error_errno(_("cannot stat '%s'"), gc_log_path);
|
|
|
|
goto done;
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
if (st.st_mtime < gc_log_expire_time)
|
|
|
|
goto done;
|
|
|
|
|
2018-07-17 08:53:21 +02:00
|
|
|
len = strbuf_read_file(&sb, gc_log_path, 0);
|
|
|
|
if (len < 0)
|
gc: do not return error for prior errors in daemonized mode
Some build machines started consistently failing to fetch updated
source using "repo sync", with error
error: The last gc run reported the following. Please correct the root cause
and remove /build/.repo/projects/tools/git.git/gc.log.
Automatic cleanup will not be performed until the file is removed.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
The cause takes some time to describe.
In v2.0.0-rc0~145^2 (gc: config option for running --auto in
background, 2014-02-08), "git gc --auto" learned to run in the
background instead of blocking the invoking command. In this mode, it
closed stderr to avoid interleaving output with any subsequent
commands, causing warnings like the above to be swallowed; v2.6.3~24^2
(gc: save log from daemonized gc --auto and print it next time,
2015-09-19) addressed that by storing any diagnostic output in
.git/gc.log and allowing the next "git gc --auto" run to print it.
To avoid wasteful repeated fruitless gcs, when gc.log is present, the
subsequent "gc --auto" would die after printing its contents. Most
git commands, such as "git fetch", ignore the exit status from "git gc
--auto" so all is well at this point: the user gets to see the error
message, and the fetch succeeds, without a wasteful additional attempt
at an automatic gc.
External tools like repo[1], though, do care about the exit status
from "git gc --auto". In non-daemonized mode, the exit status is
straightforward: if there is an error, it is nonzero, but after a
warning like the above, the status is zero. The daemonized mode, as a
side effect of the other properties provided, offers a very strange
exit code convention:
- if no housekeeping was required, the exit status is 0
- the first real run, after forking into the background, returns exit
status 0 unconditionally. The parent process has no way to know
whether gc will succeed.
- if there is any diagnostic output in gc.log, subsequent runs return
a nonzero exit status to indicate that gc was not triggered.
There's nothing for the calling program to act on on the basis of that
error. Use status 0 consistently instead, to indicate that we decided
not to run a gc (just like if no housekeeping was required). This
way, repo and similar tools can get the benefit of the same behavior
as tools like "git fetch" that ignore the exit status from gc --auto.
Once the period of time described by gc.pruneExpire elapses, the
unreachable loose objects will be removed by "git gc --auto"
automatically.
[1] https://gerrit-review.googlesource.com/c/git-repo/+/10598/
Reported-by: Andrii Dehtiarov <adehtiarov@google.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-17 08:57:40 +02:00
|
|
|
ret = error_errno(_("cannot read '%s'"), gc_log_path);
|
|
|
|
else if (len > 0) {
|
|
|
|
/*
|
|
|
|
* A previous gc failed. Report the error, and don't
|
|
|
|
* bother with an automatic gc run since it is likely
|
|
|
|
* to fail in the same way.
|
|
|
|
*/
|
|
|
|
warning(_("The last gc run reported the following. "
|
2015-09-19 07:13:23 +02:00
|
|
|
"Please correct the root cause\n"
|
|
|
|
"and remove %s.\n"
|
|
|
|
"Automatic cleanup will not be performed "
|
|
|
|
"until the file is removed.\n\n"
|
|
|
|
"%s"),
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
gc_log_path, sb.buf);
|
gc: do not return error for prior errors in daemonized mode
Some build machines started consistently failing to fetch updated
source using "repo sync", with error
error: The last gc run reported the following. Please correct the root cause
and remove /build/.repo/projects/tools/git.git/gc.log.
Automatic cleanup will not be performed until the file is removed.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
The cause takes some time to describe.
In v2.0.0-rc0~145^2 (gc: config option for running --auto in
background, 2014-02-08), "git gc --auto" learned to run in the
background instead of blocking the invoking command. In this mode, it
closed stderr to avoid interleaving output with any subsequent
commands, causing warnings like the above to be swallowed; v2.6.3~24^2
(gc: save log from daemonized gc --auto and print it next time,
2015-09-19) addressed that by storing any diagnostic output in
.git/gc.log and allowing the next "git gc --auto" run to print it.
To avoid wasteful repeated fruitless gcs, when gc.log is present, the
subsequent "gc --auto" would die after printing its contents. Most
git commands, such as "git fetch", ignore the exit status from "git gc
--auto" so all is well at this point: the user gets to see the error
message, and the fetch succeeds, without a wasteful additional attempt
at an automatic gc.
External tools like repo[1], though, do care about the exit status
from "git gc --auto". In non-daemonized mode, the exit status is
straightforward: if there is an error, it is nonzero, but after a
warning like the above, the status is zero. The daemonized mode, as a
side effect of the other properties provided, offers a very strange
exit code convention:
- if no housekeeping was required, the exit status is 0
- the first real run, after forking into the background, returns exit
status 0 unconditionally. The parent process has no way to know
whether gc will succeed.
- if there is any diagnostic output in gc.log, subsequent runs return
a nonzero exit status to indicate that gc was not triggered.
There's nothing for the calling program to act on on the basis of that
error. Use status 0 consistently instead, to indicate that we decided
not to run a gc (just like if no housekeeping was required). This
way, repo and similar tools can get the benefit of the same behavior
as tools like "git fetch" that ignore the exit status from gc --auto.
Once the period of time described by gc.pruneExpire elapses, the
unreachable loose objects will be removed by "git gc --auto"
automatically.
[1] https://gerrit-review.googlesource.com/c/git-repo/+/10598/
Reported-by: Andrii Dehtiarov <adehtiarov@google.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-17 08:57:40 +02:00
|
|
|
ret = 1;
|
|
|
|
}
|
2015-09-19 07:13:23 +02:00
|
|
|
strbuf_release(&sb);
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
done:
|
|
|
|
free(gc_log_path);
|
gc: do not return error for prior errors in daemonized mode
Some build machines started consistently failing to fetch updated
source using "repo sync", with error
error: The last gc run reported the following. Please correct the root cause
and remove /build/.repo/projects/tools/git.git/gc.log.
Automatic cleanup will not be performed until the file is removed.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
The cause takes some time to describe.
In v2.0.0-rc0~145^2 (gc: config option for running --auto in
background, 2014-02-08), "git gc --auto" learned to run in the
background instead of blocking the invoking command. In this mode, it
closed stderr to avoid interleaving output with any subsequent
commands, causing warnings like the above to be swallowed; v2.6.3~24^2
(gc: save log from daemonized gc --auto and print it next time,
2015-09-19) addressed that by storing any diagnostic output in
.git/gc.log and allowing the next "git gc --auto" run to print it.
To avoid wasteful repeated fruitless gcs, when gc.log is present, the
subsequent "gc --auto" would die after printing its contents. Most
git commands, such as "git fetch", ignore the exit status from "git gc
--auto" so all is well at this point: the user gets to see the error
message, and the fetch succeeds, without a wasteful additional attempt
at an automatic gc.
External tools like repo[1], though, do care about the exit status
from "git gc --auto". In non-daemonized mode, the exit status is
straightforward: if there is an error, it is nonzero, but after a
warning like the above, the status is zero. The daemonized mode, as a
side effect of the other properties provided, offers a very strange
exit code convention:
- if no housekeeping was required, the exit status is 0
- the first real run, after forking into the background, returns exit
status 0 unconditionally. The parent process has no way to know
whether gc will succeed.
- if there is any diagnostic output in gc.log, subsequent runs return
a nonzero exit status to indicate that gc was not triggered.
There's nothing for the calling program to act on on the basis of that
error. Use status 0 consistently instead, to indicate that we decided
not to run a gc (just like if no housekeeping was required). This
way, repo and similar tools can get the benefit of the same behavior
as tools like "git fetch" that ignore the exit status from gc --auto.
Once the period of time described by gc.pruneExpire elapses, the
unreachable loose objects will be removed by "git gc --auto"
automatically.
[1] https://gerrit-review.googlesource.com/c/git-repo/+/10598/
Reported-by: Andrii Dehtiarov <adehtiarov@google.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-17 08:57:40 +02:00
|
|
|
return ret;
|
2015-09-19 07:13:23 +02:00
|
|
|
}
|
|
|
|
|
2018-07-17 08:54:16 +02:00
|
|
|
static void gc_before_repack(void)
|
2014-05-25 02:38:29 +02:00
|
|
|
{
|
2019-03-15 16:59:54 +01:00
|
|
|
/*
|
|
|
|
* We may be called twice, as both the pre- and
|
|
|
|
* post-daemonized phases will call us, but running these
|
|
|
|
* commands more than once is pointless and wasteful.
|
|
|
|
*/
|
|
|
|
static int done = 0;
|
|
|
|
if (done++)
|
|
|
|
return;
|
|
|
|
|
2020-07-29 02:37:20 +02:00
|
|
|
if (pack_refs && run_command_v_opt(pack_refs_cmd.v, RUN_GIT_CMD))
|
|
|
|
die(FAILED_RUN, pack_refs_cmd.v[0]);
|
2014-05-25 02:38:29 +02:00
|
|
|
|
2020-07-29 02:37:20 +02:00
|
|
|
if (prune_reflogs && run_command_v_opt(reflog.v, RUN_GIT_CMD))
|
|
|
|
die(FAILED_RUN, reflog.v[0]);
|
2014-05-25 02:38:29 +02:00
|
|
|
}
|
|
|
|
|
2007-03-14 02:58:22 +01:00
|
|
|
int cmd_gc(int argc, const char **argv, const char *prefix)
|
|
|
|
{
|
2007-11-02 02:02:27 +01:00
|
|
|
int aggressive = 0;
|
2007-09-05 22:01:37 +02:00
|
|
|
int auto_gc = 0;
|
2008-02-29 22:53:39 +01:00
|
|
|
int quiet = 0;
|
2013-08-08 13:05:38 +02:00
|
|
|
int force = 0;
|
|
|
|
const char *name;
|
|
|
|
pid_t pid;
|
2015-09-19 07:13:23 +02:00
|
|
|
int daemonized = 0;
|
2018-04-15 17:36:14 +02:00
|
|
|
int keep_base_pack = -1;
|
2018-04-21 05:13:13 +02:00
|
|
|
timestamp_t dummy;
|
2007-03-14 02:58:22 +01:00
|
|
|
|
2007-11-02 02:02:27 +01:00
|
|
|
struct option builtin_gc_options[] = {
|
2012-08-20 14:32:14 +02:00
|
|
|
OPT__QUIET(&quiet, N_("suppress progress reporting")),
|
|
|
|
{ OPTION_STRING, 0, "prune", &prune_expire, N_("date"),
|
|
|
|
N_("prune unreferenced objects"),
|
2009-02-14 23:10:10 +01:00
|
|
|
PARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },
|
2013-08-03 13:51:19 +02:00
|
|
|
OPT_BOOL(0, "aggressive", &aggressive, N_("be more thorough (increased runtime)")),
|
2018-02-09 12:01:58 +01:00
|
|
|
OPT_BOOL_F(0, "auto", &auto_gc, N_("enable auto-gc mode"),
|
|
|
|
PARSE_OPT_NOCOMPLETE),
|
|
|
|
OPT_BOOL_F(0, "force", &force,
|
|
|
|
N_("force running gc even if there may be another gc running"),
|
|
|
|
PARSE_OPT_NOCOMPLETE),
|
2018-04-15 17:36:14 +02:00
|
|
|
OPT_BOOL(0, "keep-largest-pack", &keep_base_pack,
|
|
|
|
N_("repack all other packs except the largest pack")),
|
2007-11-02 02:02:27 +01:00
|
|
|
OPT_END()
|
|
|
|
};
|
|
|
|
|
2010-10-22 08:47:19 +02:00
|
|
|
if (argc == 2 && !strcmp(argv[1], "-h"))
|
|
|
|
usage_with_options(builtin_gc_usage, builtin_gc_options);
|
|
|
|
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_pushl(&pack_refs_cmd, "pack-refs", "--all", "--prune", NULL);
|
|
|
|
strvec_pushl(&reflog, "reflog", "expire", "--all", NULL);
|
|
|
|
strvec_pushl(&repack, "repack", "-d", "-l", NULL);
|
|
|
|
strvec_pushl(&prune, "prune", "--expire", NULL);
|
|
|
|
strvec_pushl(&prune_worktrees, "worktree", "prune", "--expire", NULL);
|
|
|
|
strvec_pushl(&rerere, "rerere", "gc", NULL);
|
2012-04-18 23:10:19 +02:00
|
|
|
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
/* default expiry time, overwritten in gc_config */
|
2014-08-07 18:21:22 +02:00
|
|
|
gc_config();
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
if (parse_expiry_date(gc_log_expire, &gc_log_expire_time))
|
2018-04-23 15:36:14 +02:00
|
|
|
die(_("failed to parse gc.logexpiry value %s"), gc_log_expire);
|
2007-03-14 02:58:22 +01:00
|
|
|
|
|
|
|
if (pack_refs < 0)
|
|
|
|
pack_refs = !is_bare_repository();
|
|
|
|
|
2009-05-23 20:53:12 +02:00
|
|
|
argc = parse_options(argc, argv, prefix, builtin_gc_options,
|
|
|
|
builtin_gc_usage, 0);
|
2007-11-02 02:02:27 +01:00
|
|
|
if (argc > 0)
|
|
|
|
usage_with_options(builtin_gc_usage, builtin_gc_options);
|
|
|
|
|
2018-04-21 05:13:13 +02:00
|
|
|
if (prune_expire && parse_expiry_date(prune_expire, &dummy))
|
|
|
|
die(_("failed to parse prune expiry value %s"), prune_expire);
|
|
|
|
|
2007-11-02 02:02:27 +01:00
|
|
|
if (aggressive) {
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_push(&repack, "-f");
|
gc --aggressive: make --depth configurable
When 1c192f3 (gc --aggressive: make it really aggressive - 2007-12-06)
made --depth=250 the default value, it didn't really explain the
reason behind, especially the pros and cons of --depth=250.
An old mail from Linus below explains it at length. Long story short,
--depth=250 is a disk saver and a performance killer. Not everybody
agrees on that aggressiveness. Let the user configure it.
From: Linus Torvalds <torvalds@linux-foundation.org>
Subject: Re: [PATCH] gc --aggressive: make it really aggressive
Date: Thu, 6 Dec 2007 08:19:24 -0800 (PST)
Message-ID: <alpine.LFD.0.9999.0712060803430.13796@woody.linux-foundation.org>
Gmane-URL: http://article.gmane.org/gmane.comp.gcc.devel/94637
On Thu, 6 Dec 2007, Harvey Harrison wrote:
>
> 7:41:25elapsed 86%CPU
Heh. And this is why you want to do it exactly *once*, and then just
export the end result for others ;)
> -r--r--r-- 1 hharrison hharrison 324094684 2007-12-06 07:26 pack-1d46...pack
But yeah, especially if you allow longer delta chains, the end result can
be much smaller (and what makes the one-time repack more expensive is the
window size, not the delta chain - you could make the delta chains longer
with no cost overhead at packing time)
HOWEVER.
The longer delta chains do make it potentially much more expensive to then
use old history. So there's a trade-off. And quite frankly, a delta depth
of 250 is likely going to cause overflows in the delta cache (which is
only 256 entries in size *and* it's a hash, so it's going to start having
hash conflicts long before hitting the 250 depth limit).
So when I said "--depth=250 --window=250", I chose those numbers more as
an example of extremely aggressive packing, and I'm not at all sure that
the end result is necessarily wonderfully usable. It's going to save disk
space (and network bandwidth - the delta's will be re-used for the network
protocol too!), but there are definitely downsides too, and using long
delta chains may simply not be worth it in practice.
(And some of it might just want to have git tuning, ie if people think
that long deltas are worth it, we could easily just expand on the delta
hash, at the cost of some more memory used!)
That said, the good news is that working with *new* history will not be
affected negatively, and if you want to be _really_ sneaky, there are ways
to say "create a pack that contains the history up to a version one year
ago, and be very aggressive about those old versions that we still want to
have around, but do a separate pack for newer stuff using less aggressive
parameters"
So this is something that can be tweaked, although we don't really have
any really nice interfaces for stuff like that (ie the git delta cache
size is hardcoded in the sources and cannot be set in the config file, and
the "pack old history more aggressively" involves some manual scripting
and knowing how "git pack-objects" works rather than any nice simple
command line switch).
So the thing to take away from this is:
- git is certainly flexible as hell
- .. but to get the full power you may need to tweak things
- .. happily you really only need to have one person to do the tweaking,
and the tweaked end results will be available to others that do not
need to know/care.
And whether the difference between 320MB and 500MB is worth any really
involved tweaking (considering the potential downsides), I really don't
know. Only testing will tell.
Linus
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-03-16 14:35:03 +01:00
|
|
|
if (aggressive_depth > 0)
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_pushf(&repack, "--depth=%d", aggressive_depth);
|
2012-04-18 23:10:19 +02:00
|
|
|
if (aggressive_window > 0)
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_pushf(&repack, "--window=%d", aggressive_window);
|
2007-03-14 02:58:22 +01:00
|
|
|
}
|
2008-02-29 22:53:39 +01:00
|
|
|
if (quiet)
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_push(&repack, "-q");
|
2007-03-14 02:58:22 +01:00
|
|
|
|
2007-09-05 22:01:37 +02:00
|
|
|
if (auto_gc) {
|
|
|
|
/*
|
|
|
|
* Auto-gc should be least intrusive as possible.
|
|
|
|
*/
|
|
|
|
if (!need_to_gc())
|
|
|
|
return 0;
|
2014-02-08 08:08:52 +01:00
|
|
|
if (!quiet) {
|
|
|
|
if (detach_auto)
|
|
|
|
fprintf(stderr, _("Auto packing the repository in background for optimum performance.\n"));
|
|
|
|
else
|
|
|
|
fprintf(stderr, _("Auto packing the repository for optimum performance.\n"));
|
|
|
|
fprintf(stderr, _("See \"git help gc\" for manual housekeeping.\n"));
|
|
|
|
}
|
2014-05-25 02:38:29 +02:00
|
|
|
if (detach_auto) {
|
gc: do not return error for prior errors in daemonized mode
Some build machines started consistently failing to fetch updated
source using "repo sync", with error
error: The last gc run reported the following. Please correct the root cause
and remove /build/.repo/projects/tools/git.git/gc.log.
Automatic cleanup will not be performed until the file is removed.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
The cause takes some time to describe.
In v2.0.0-rc0~145^2 (gc: config option for running --auto in
background, 2014-02-08), "git gc --auto" learned to run in the
background instead of blocking the invoking command. In this mode, it
closed stderr to avoid interleaving output with any subsequent
commands, causing warnings like the above to be swallowed; v2.6.3~24^2
(gc: save log from daemonized gc --auto and print it next time,
2015-09-19) addressed that by storing any diagnostic output in
.git/gc.log and allowing the next "git gc --auto" run to print it.
To avoid wasteful repeated fruitless gcs, when gc.log is present, the
subsequent "gc --auto" would die after printing its contents. Most
git commands, such as "git fetch", ignore the exit status from "git gc
--auto" so all is well at this point: the user gets to see the error
message, and the fetch succeeds, without a wasteful additional attempt
at an automatic gc.
External tools like repo[1], though, do care about the exit status
from "git gc --auto". In non-daemonized mode, the exit status is
straightforward: if there is an error, it is nonzero, but after a
warning like the above, the status is zero. The daemonized mode, as a
side effect of the other properties provided, offers a very strange
exit code convention:
- if no housekeeping was required, the exit status is 0
- the first real run, after forking into the background, returns exit
status 0 unconditionally. The parent process has no way to know
whether gc will succeed.
- if there is any diagnostic output in gc.log, subsequent runs return
a nonzero exit status to indicate that gc was not triggered.
There's nothing for the calling program to act on on the basis of that
error. Use status 0 consistently instead, to indicate that we decided
not to run a gc (just like if no housekeeping was required). This
way, repo and similar tools can get the benefit of the same behavior
as tools like "git fetch" that ignore the exit status from gc --auto.
Once the period of time described by gc.pruneExpire elapses, the
unreachable loose objects will be removed by "git gc --auto"
automatically.
[1] https://gerrit-review.googlesource.com/c/git-repo/+/10598/
Reported-by: Andrii Dehtiarov <adehtiarov@google.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-17 08:57:40 +02:00
|
|
|
int ret = report_last_gc_error();
|
|
|
|
if (ret < 0)
|
2019-11-05 18:07:23 +01:00
|
|
|
/* an I/O error occurred, already reported */
|
gc: do not return error for prior errors in daemonized mode
Some build machines started consistently failing to fetch updated
source using "repo sync", with error
error: The last gc run reported the following. Please correct the root cause
and remove /build/.repo/projects/tools/git.git/gc.log.
Automatic cleanup will not be performed until the file is removed.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
The cause takes some time to describe.
In v2.0.0-rc0~145^2 (gc: config option for running --auto in
background, 2014-02-08), "git gc --auto" learned to run in the
background instead of blocking the invoking command. In this mode, it
closed stderr to avoid interleaving output with any subsequent
commands, causing warnings like the above to be swallowed; v2.6.3~24^2
(gc: save log from daemonized gc --auto and print it next time,
2015-09-19) addressed that by storing any diagnostic output in
.git/gc.log and allowing the next "git gc --auto" run to print it.
To avoid wasteful repeated fruitless gcs, when gc.log is present, the
subsequent "gc --auto" would die after printing its contents. Most
git commands, such as "git fetch", ignore the exit status from "git gc
--auto" so all is well at this point: the user gets to see the error
message, and the fetch succeeds, without a wasteful additional attempt
at an automatic gc.
External tools like repo[1], though, do care about the exit status
from "git gc --auto". In non-daemonized mode, the exit status is
straightforward: if there is an error, it is nonzero, but after a
warning like the above, the status is zero. The daemonized mode, as a
side effect of the other properties provided, offers a very strange
exit code convention:
- if no housekeeping was required, the exit status is 0
- the first real run, after forking into the background, returns exit
status 0 unconditionally. The parent process has no way to know
whether gc will succeed.
- if there is any diagnostic output in gc.log, subsequent runs return
a nonzero exit status to indicate that gc was not triggered.
There's nothing for the calling program to act on on the basis of that
error. Use status 0 consistently instead, to indicate that we decided
not to run a gc (just like if no housekeeping was required). This
way, repo and similar tools can get the benefit of the same behavior
as tools like "git fetch" that ignore the exit status from gc --auto.
Once the period of time described by gc.pruneExpire elapses, the
unreachable loose objects will be removed by "git gc --auto"
automatically.
[1] https://gerrit-review.googlesource.com/c/git-repo/+/10598/
Reported-by: Andrii Dehtiarov <adehtiarov@google.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-17 08:57:40 +02:00
|
|
|
exit(128);
|
|
|
|
if (ret == 1)
|
|
|
|
/* Last gc --auto failed. Skip this one. */
|
|
|
|
return 0;
|
2015-09-19 07:13:23 +02:00
|
|
|
|
gc: run pre-detach operations under lock
We normally try to avoid having two auto-gc operations run
at the same time, because it wastes resources. This was done
long ago in 64a99eb47 (gc: reject if another gc is running,
unless --force is given, 2013-08-08).
When we do a detached auto-gc, we run the ref-related
commands _before_ detaching, to avoid confusing lock
contention. This was done by 62aad1849 (gc --auto: do not
lock refs in the background, 2014-05-25).
These two features do not interact well. The pre-detach
operations are run before we check the gc.pid lock, meaning
that on a busy repository we may run many of them
concurrently. Ideally we'd take the lock before spawning any
operations, and hold it for the duration of the program.
This is tricky, though, with the way the pid-file interacts
with the daemonize() process. Other processes will check
that the pid recorded in the pid-file still exists. But
detaching causes us to fork and continue running under a
new pid. So if we take the lock before detaching, the
pid-file will have a bogus pid in it. We'd have to go back
and update it with the new pid after detaching. We'd also
have to play some tricks with the tempfile subsystem to
tweak the "owner" field, so that the parent process does not
clean it up on exit, but the child process does.
Instead, we can do something a bit simpler: take the lock
only for the duration of the pre-detach work, then detach,
then take it again for the post-detach work. Technically,
this means that the post-detach lock could lose to another
process doing pre-detach work. But in the long run this
works out.
That second process would then follow-up by doing
post-detach work. Unless it was in turn blocked by a third
process doing pre-detach work, and so on. This could in
theory go on indefinitely, as the pre-detach work does not
repack, and so need_to_gc() will continue to trigger. But
in each round we are racing between the pre- and post-detach
locks. Eventually, one of the post-detach locks will win the
race and complete the full gc. So in the worst case, we may
racily repeat the pre-detach work, but we would never do so
simultaneously (it would happen via a sequence of serialized
race-wins).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-07-11 11:06:35 +02:00
|
|
|
if (lock_repo_for_gc(force, &pid))
|
|
|
|
return 0;
|
2018-07-17 08:54:16 +02:00
|
|
|
gc_before_repack(); /* dies on failure */
|
gc: run pre-detach operations under lock
We normally try to avoid having two auto-gc operations run
at the same time, because it wastes resources. This was done
long ago in 64a99eb47 (gc: reject if another gc is running,
unless --force is given, 2013-08-08).
When we do a detached auto-gc, we run the ref-related
commands _before_ detaching, to avoid confusing lock
contention. This was done by 62aad1849 (gc --auto: do not
lock refs in the background, 2014-05-25).
These two features do not interact well. The pre-detach
operations are run before we check the gc.pid lock, meaning
that on a busy repository we may run many of them
concurrently. Ideally we'd take the lock before spawning any
operations, and hold it for the duration of the program.
This is tricky, though, with the way the pid-file interacts
with the daemonize() process. Other processes will check
that the pid recorded in the pid-file still exists. But
detaching causes us to fork and continue running under a
new pid. So if we take the lock before detaching, the
pid-file will have a bogus pid in it. We'd have to go back
and update it with the new pid after detaching. We'd also
have to play some tricks with the tempfile subsystem to
tweak the "owner" field, so that the parent process does not
clean it up on exit, but the child process does.
Instead, we can do something a bit simpler: take the lock
only for the duration of the pre-detach work, then detach,
then take it again for the post-detach work. Technically,
this means that the post-detach lock could lose to another
process doing pre-detach work. But in the long run this
works out.
That second process would then follow-up by doing
post-detach work. Unless it was in turn blocked by a third
process doing pre-detach work, and so on. This could in
theory go on indefinitely, as the pre-detach work does not
repack, and so need_to_gc() will continue to trigger. But
in each round we are racing between the pre- and post-detach
locks. Eventually, one of the post-detach locks will win the
race and complete the full gc. So in the worst case, we may
racily repeat the pre-detach work, but we would never do so
simultaneously (it would happen via a sequence of serialized
race-wins).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-07-11 11:06:35 +02:00
|
|
|
delete_tempfile(&pidfile);
|
|
|
|
|
2014-02-08 08:08:52 +01:00
|
|
|
/*
|
|
|
|
* failure to daemonize is ok, we'll continue
|
|
|
|
* in foreground
|
|
|
|
*/
|
2015-09-19 07:13:23 +02:00
|
|
|
daemonized = !daemonize();
|
2014-05-25 02:38:29 +02:00
|
|
|
}
|
2018-04-15 17:36:14 +02:00
|
|
|
} else {
|
|
|
|
struct string_list keep_pack = STRING_LIST_INIT_NODUP;
|
|
|
|
|
|
|
|
if (keep_base_pack != -1) {
|
|
|
|
if (keep_base_pack)
|
2018-04-15 17:36:15 +02:00
|
|
|
find_base_packs(&keep_pack, 0);
|
|
|
|
} else if (big_pack_threshold) {
|
|
|
|
find_base_packs(&keep_pack, big_pack_threshold);
|
2018-04-15 17:36:14 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
add_repack_all_option(&keep_pack);
|
|
|
|
string_list_clear(&keep_pack, 0);
|
|
|
|
}
|
2007-09-05 22:01:37 +02:00
|
|
|
|
2013-08-08 13:05:38 +02:00
|
|
|
name = lock_repo_for_gc(force, &pid);
|
|
|
|
if (name) {
|
|
|
|
if (auto_gc)
|
|
|
|
return 0; /* be quiet on --auto */
|
|
|
|
die(_("gc is already running on machine '%s' pid %"PRIuMAX" (use --force if not)"),
|
|
|
|
name, (uintmax_t)pid);
|
|
|
|
}
|
|
|
|
|
2015-09-19 07:13:23 +02:00
|
|
|
if (daemonized) {
|
|
|
|
hold_lock_file_for_update(&log_lock,
|
|
|
|
git_path("gc.log"),
|
|
|
|
LOCK_DIE_ON_ERROR);
|
2015-10-16 00:43:32 +02:00
|
|
|
dup2(get_lock_file_fd(&log_lock), 2);
|
2015-09-19 07:13:23 +02:00
|
|
|
sigchain_push_common(process_log_file_on_signal);
|
|
|
|
atexit(process_log_file_at_exit);
|
|
|
|
}
|
|
|
|
|
2018-07-17 08:54:16 +02:00
|
|
|
gc_before_repack();
|
2007-03-14 02:58:22 +01:00
|
|
|
|
2015-06-23 12:54:11 +02:00
|
|
|
if (!repository_format_precious_objects) {
|
2019-05-17 20:41:49 +02:00
|
|
|
close_object_store(the_repository->objects);
|
2020-07-29 02:37:20 +02:00
|
|
|
if (run_command_v_opt(repack.v, RUN_GIT_CMD))
|
|
|
|
die(FAILED_RUN, repack.v[0]);
|
2015-06-23 12:54:11 +02:00
|
|
|
|
|
|
|
if (prune_expire) {
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_push(&prune, prune_expire);
|
2015-06-23 12:54:11 +02:00
|
|
|
if (quiet)
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_push(&prune, "--no-progress");
|
2019-06-25 15:40:31 +02:00
|
|
|
if (has_promisor_remote())
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_push(&prune,
|
strvec: fix indentation in renamed calls
Code which split an argv_array call across multiple lines, like:
argv_array_pushl(&args, "one argument",
"another argument", "and more",
NULL);
was recently mechanically renamed to use strvec, which results in
mis-matched indentation like:
strvec_pushl(&args, "one argument",
"another argument", "and more",
NULL);
Let's fix these up to align the arguments with the opening paren. I did
this manually by sifting through the results of:
git jump grep 'strvec_.*,$'
and liberally applying my editor's auto-format. Most of the changes are
of the form shown above, though I also normalized a few that had
originally used a single-tab indentation (rather than our usual style of
aligning with the open paren). I also rewrapped a couple of obvious
cases (e.g., where previously too-long lines became short enough to fit
on one), but I wasn't aggressive about it. In cases broken to three or
more lines, the grouping of arguments is sometimes meaningful, and it
wasn't worth my time or reviewer time to ponder each case individually.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-28 22:26:31 +02:00
|
|
|
"--exclude-promisor-objects");
|
2020-07-29 02:37:20 +02:00
|
|
|
if (run_command_v_opt(prune.v, RUN_GIT_CMD))
|
|
|
|
die(FAILED_RUN, prune.v[0]);
|
2015-06-23 12:54:11 +02:00
|
|
|
}
|
2009-02-14 23:10:10 +01:00
|
|
|
}
|
2007-03-14 02:58:22 +01:00
|
|
|
|
2014-11-30 09:24:53 +01:00
|
|
|
if (prune_worktrees_expire) {
|
2020-07-28 22:24:27 +02:00
|
|
|
strvec_push(&prune_worktrees, prune_worktrees_expire);
|
2020-07-29 02:37:20 +02:00
|
|
|
if (run_command_v_opt(prune_worktrees.v, RUN_GIT_CMD))
|
|
|
|
die(FAILED_RUN, prune_worktrees.v[0]);
|
2014-11-30 09:24:53 +01:00
|
|
|
}
|
|
|
|
|
2020-07-29 02:37:20 +02:00
|
|
|
if (run_command_v_opt(rerere.v, RUN_GIT_CMD))
|
|
|
|
die(FAILED_RUN, rerere.v[0]);
|
2007-03-14 02:58:22 +01:00
|
|
|
|
2015-11-04 04:05:08 +01:00
|
|
|
report_garbage = report_pack_garbage;
|
2018-03-23 18:45:21 +01:00
|
|
|
reprepare_packed_git(the_repository);
|
2018-12-15 23:04:01 +01:00
|
|
|
if (pack_garbage.nr > 0) {
|
2019-05-17 20:41:49 +02:00
|
|
|
close_object_store(the_repository->objects);
|
2015-11-04 04:05:08 +01:00
|
|
|
clean_pack_garbage();
|
2018-12-15 23:04:01 +01:00
|
|
|
}
|
2015-11-04 04:05:08 +01:00
|
|
|
|
2019-08-13 20:37:43 +02:00
|
|
|
prepare_repo_settings(the_repository);
|
|
|
|
if (the_repository->settings.gc_write_commit_graph == 1)
|
2020-02-04 06:51:50 +01:00
|
|
|
write_commit_graph_reachable(the_repository->objects->odb,
|
2019-09-09 21:26:36 +02:00
|
|
|
!quiet && !daemonized ? COMMIT_GRAPH_WRITE_PROGRESS : 0,
|
2019-08-13 20:37:43 +02:00
|
|
|
NULL);
|
2018-06-27 15:24:46 +02:00
|
|
|
|
2007-09-17 09:44:17 +02:00
|
|
|
if (auto_gc && too_many_loose_objects())
|
2011-02-23 00:42:24 +01:00
|
|
|
warning(_("There are too many unreachable loose objects; "
|
|
|
|
"run 'git prune' to remove them."));
|
2007-09-17 09:44:17 +02:00
|
|
|
|
gc: ignore old gc.log files
A server can end up in a state where there are lots of unreferenced
loose objects (say, because many users are doing a bunch of rebasing
and pushing their rebased branches). Running "git gc --auto" in
this state would cause a gc.log file to be created, preventing
future auto gcs, causing pack files to pile up. Since many git
operations are O(n) in the number of pack files, this would lead to
poor performance.
Git should never get itself into a state where it refuses to do any
maintenance, just because at some point some piece of the maintenance
didn't make progress.
Teach Git to ignore gc.log files which are older than (by default)
one day old, which can be tweaked via the gc.logExpiry configuration
variable. That way, these pack files will get cleaned up, if
necessary, at least once per day. And operators who find a need for
more-frequent gcs can adjust gc.logExpiry to meet their needs.
There is also some cleanup: a successful manual gc, or a
warning-free auto gc with an old log file, will remove any old
gc.log files.
It might still happen that manual intervention is required
(e.g. because the repo is corrupt), but at the very least it won't
be because Git is too dumb to try again.
Signed-off-by: David Turner <dturner@twosigma.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-10 22:28:22 +01:00
|
|
|
if (!daemonized)
|
|
|
|
unlink(git_path("gc.log"));
|
|
|
|
|
2007-03-14 02:58:22 +01:00
|
|
|
return 0;
|
|
|
|
}
|
maintenance: create basic maintenance runner
The 'gc' builtin is our current entrypoint for automatically maintaining
a repository. This one tool does many operations, such as repacking the
repository, packing refs, and rewriting the commit-graph file. The name
implies it performs "garbage collection" which means several different
things, and some users may not want to use this operation that rewrites
the entire object database.
Create a new 'maintenance' builtin that will become a more general-
purpose command. To start, it will only support the 'run' subcommand,
but will later expand to add subcommands for scheduling maintenance in
the background.
For now, the 'maintenance' builtin is a thin shim over the 'gc' builtin.
In fact, the only option is the '--auto' toggle, which is handed
directly to the 'gc' builtin. The current change is isolated to this
simple operation to prevent more interesting logic from being lost in
all of the boilerplate of adding a new builtin.
Use existing builtin/gc.c file because we want to share code between the
two builtins. It is possible that we will have 'maintenance' replace the
'gc' builtin entirely at some point, leaving 'git gc' as an alias for
some specific arguments to 'git maintenance run'.
Create a new test_subcommand helper that allows us to test if a certain
subcommand was run. It requires storing the GIT_TRACE2_EVENT logs in a
file. A negation mode is available that will be used in later tests.
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-17 20:11:42 +02:00
|
|
|
|
|
|
|
static const char * const builtin_maintenance_run_usage[] = {
|
2020-09-17 20:11:43 +02:00
|
|
|
N_("git maintenance run [--auto] [--[no-]quiet]"),
|
maintenance: create basic maintenance runner
The 'gc' builtin is our current entrypoint for automatically maintaining
a repository. This one tool does many operations, such as repacking the
repository, packing refs, and rewriting the commit-graph file. The name
implies it performs "garbage collection" which means several different
things, and some users may not want to use this operation that rewrites
the entire object database.
Create a new 'maintenance' builtin that will become a more general-
purpose command. To start, it will only support the 'run' subcommand,
but will later expand to add subcommands for scheduling maintenance in
the background.
For now, the 'maintenance' builtin is a thin shim over the 'gc' builtin.
In fact, the only option is the '--auto' toggle, which is handed
directly to the 'gc' builtin. The current change is isolated to this
simple operation to prevent more interesting logic from being lost in
all of the boilerplate of adding a new builtin.
Use existing builtin/gc.c file because we want to share code between the
two builtins. It is possible that we will have 'maintenance' replace the
'gc' builtin entirely at some point, leaving 'git gc' as an alias for
some specific arguments to 'git maintenance run'.
Create a new test_subcommand helper that allows us to test if a certain
subcommand was run. It requires storing the GIT_TRACE2_EVENT logs in a
file. A negation mode is available that will be used in later tests.
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-17 20:11:42 +02:00
|
|
|
NULL
|
|
|
|
};
|
|
|
|
|
|
|
|
struct maintenance_run_opts {
|
|
|
|
int auto_flag;
|
2020-09-17 20:11:43 +02:00
|
|
|
int quiet;
|
maintenance: create basic maintenance runner
The 'gc' builtin is our current entrypoint for automatically maintaining
a repository. This one tool does many operations, such as repacking the
repository, packing refs, and rewriting the commit-graph file. The name
implies it performs "garbage collection" which means several different
things, and some users may not want to use this operation that rewrites
the entire object database.
Create a new 'maintenance' builtin that will become a more general-
purpose command. To start, it will only support the 'run' subcommand,
but will later expand to add subcommands for scheduling maintenance in
the background.
For now, the 'maintenance' builtin is a thin shim over the 'gc' builtin.
In fact, the only option is the '--auto' toggle, which is handed
directly to the 'gc' builtin. The current change is isolated to this
simple operation to prevent more interesting logic from being lost in
all of the boilerplate of adding a new builtin.
Use existing builtin/gc.c file because we want to share code between the
two builtins. It is possible that we will have 'maintenance' replace the
'gc' builtin entirely at some point, leaving 'git gc' as an alias for
some specific arguments to 'git maintenance run'.
Create a new test_subcommand helper that allows us to test if a certain
subcommand was run. It requires storing the GIT_TRACE2_EVENT logs in a
file. A negation mode is available that will be used in later tests.
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-17 20:11:42 +02:00
|
|
|
};
|
|
|
|
|
|
|
|
static int maintenance_task_gc(struct maintenance_run_opts *opts)
|
|
|
|
{
|
|
|
|
struct child_process child = CHILD_PROCESS_INIT;
|
|
|
|
|
|
|
|
child.git_cmd = 1;
|
|
|
|
strvec_push(&child.args, "gc");
|
|
|
|
|
|
|
|
if (opts->auto_flag)
|
|
|
|
strvec_push(&child.args, "--auto");
|
2020-09-17 20:11:43 +02:00
|
|
|
if (opts->quiet)
|
|
|
|
strvec_push(&child.args, "--quiet");
|
|
|
|
else
|
|
|
|
strvec_push(&child.args, "--no-quiet");
|
maintenance: create basic maintenance runner
The 'gc' builtin is our current entrypoint for automatically maintaining
a repository. This one tool does many operations, such as repacking the
repository, packing refs, and rewriting the commit-graph file. The name
implies it performs "garbage collection" which means several different
things, and some users may not want to use this operation that rewrites
the entire object database.
Create a new 'maintenance' builtin that will become a more general-
purpose command. To start, it will only support the 'run' subcommand,
but will later expand to add subcommands for scheduling maintenance in
the background.
For now, the 'maintenance' builtin is a thin shim over the 'gc' builtin.
In fact, the only option is the '--auto' toggle, which is handed
directly to the 'gc' builtin. The current change is isolated to this
simple operation to prevent more interesting logic from being lost in
all of the boilerplate of adding a new builtin.
Use existing builtin/gc.c file because we want to share code between the
two builtins. It is possible that we will have 'maintenance' replace the
'gc' builtin entirely at some point, leaving 'git gc' as an alias for
some specific arguments to 'git maintenance run'.
Create a new test_subcommand helper that allows us to test if a certain
subcommand was run. It requires storing the GIT_TRACE2_EVENT logs in a
file. A negation mode is available that will be used in later tests.
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-17 20:11:42 +02:00
|
|
|
|
|
|
|
close_object_store(the_repository->objects);
|
|
|
|
return run_command(&child);
|
|
|
|
}
|
|
|
|
|
2020-09-17 20:11:45 +02:00
|
|
|
typedef int maintenance_task_fn(struct maintenance_run_opts *opts);
|
|
|
|
|
|
|
|
struct maintenance_task {
|
|
|
|
const char *name;
|
|
|
|
maintenance_task_fn *fn;
|
|
|
|
unsigned enabled:1;
|
|
|
|
};
|
|
|
|
|
|
|
|
enum maintenance_task_label {
|
|
|
|
TASK_GC,
|
|
|
|
|
|
|
|
/* Leave as final value */
|
|
|
|
TASK__COUNT
|
|
|
|
};
|
|
|
|
|
|
|
|
static struct maintenance_task tasks[] = {
|
|
|
|
[TASK_GC] = {
|
|
|
|
"gc",
|
|
|
|
maintenance_task_gc,
|
|
|
|
1,
|
|
|
|
},
|
|
|
|
};
|
|
|
|
|
|
|
|
static int maintenance_run_tasks(struct maintenance_run_opts *opts)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
int result = 0;
|
|
|
|
|
|
|
|
for (i = 0; i < TASK__COUNT; i++) {
|
|
|
|
if (!tasks[i].enabled)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
if (tasks[i].fn(opts)) {
|
|
|
|
error(_("task '%s' failed"), tasks[i].name);
|
|
|
|
result = 1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
maintenance: create basic maintenance runner
The 'gc' builtin is our current entrypoint for automatically maintaining
a repository. This one tool does many operations, such as repacking the
repository, packing refs, and rewriting the commit-graph file. The name
implies it performs "garbage collection" which means several different
things, and some users may not want to use this operation that rewrites
the entire object database.
Create a new 'maintenance' builtin that will become a more general-
purpose command. To start, it will only support the 'run' subcommand,
but will later expand to add subcommands for scheduling maintenance in
the background.
For now, the 'maintenance' builtin is a thin shim over the 'gc' builtin.
In fact, the only option is the '--auto' toggle, which is handed
directly to the 'gc' builtin. The current change is isolated to this
simple operation to prevent more interesting logic from being lost in
all of the boilerplate of adding a new builtin.
Use existing builtin/gc.c file because we want to share code between the
two builtins. It is possible that we will have 'maintenance' replace the
'gc' builtin entirely at some point, leaving 'git gc' as an alias for
some specific arguments to 'git maintenance run'.
Create a new test_subcommand helper that allows us to test if a certain
subcommand was run. It requires storing the GIT_TRACE2_EVENT logs in a
file. A negation mode is available that will be used in later tests.
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-17 20:11:42 +02:00
|
|
|
static int maintenance_run(int argc, const char **argv, const char *prefix)
|
|
|
|
{
|
|
|
|
struct maintenance_run_opts opts;
|
|
|
|
struct option builtin_maintenance_run_options[] = {
|
|
|
|
OPT_BOOL(0, "auto", &opts.auto_flag,
|
|
|
|
N_("run tasks based on the state of the repository")),
|
2020-09-17 20:11:43 +02:00
|
|
|
OPT_BOOL(0, "quiet", &opts.quiet,
|
|
|
|
N_("do not report progress or other information over stderr")),
|
maintenance: create basic maintenance runner
The 'gc' builtin is our current entrypoint for automatically maintaining
a repository. This one tool does many operations, such as repacking the
repository, packing refs, and rewriting the commit-graph file. The name
implies it performs "garbage collection" which means several different
things, and some users may not want to use this operation that rewrites
the entire object database.
Create a new 'maintenance' builtin that will become a more general-
purpose command. To start, it will only support the 'run' subcommand,
but will later expand to add subcommands for scheduling maintenance in
the background.
For now, the 'maintenance' builtin is a thin shim over the 'gc' builtin.
In fact, the only option is the '--auto' toggle, which is handed
directly to the 'gc' builtin. The current change is isolated to this
simple operation to prevent more interesting logic from being lost in
all of the boilerplate of adding a new builtin.
Use existing builtin/gc.c file because we want to share code between the
two builtins. It is possible that we will have 'maintenance' replace the
'gc' builtin entirely at some point, leaving 'git gc' as an alias for
some specific arguments to 'git maintenance run'.
Create a new test_subcommand helper that allows us to test if a certain
subcommand was run. It requires storing the GIT_TRACE2_EVENT logs in a
file. A negation mode is available that will be used in later tests.
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-17 20:11:42 +02:00
|
|
|
OPT_END()
|
|
|
|
};
|
|
|
|
memset(&opts, 0, sizeof(opts));
|
|
|
|
|
2020-09-17 20:11:43 +02:00
|
|
|
opts.quiet = !isatty(2);
|
|
|
|
|
maintenance: create basic maintenance runner
The 'gc' builtin is our current entrypoint for automatically maintaining
a repository. This one tool does many operations, such as repacking the
repository, packing refs, and rewriting the commit-graph file. The name
implies it performs "garbage collection" which means several different
things, and some users may not want to use this operation that rewrites
the entire object database.
Create a new 'maintenance' builtin that will become a more general-
purpose command. To start, it will only support the 'run' subcommand,
but will later expand to add subcommands for scheduling maintenance in
the background.
For now, the 'maintenance' builtin is a thin shim over the 'gc' builtin.
In fact, the only option is the '--auto' toggle, which is handed
directly to the 'gc' builtin. The current change is isolated to this
simple operation to prevent more interesting logic from being lost in
all of the boilerplate of adding a new builtin.
Use existing builtin/gc.c file because we want to share code between the
two builtins. It is possible that we will have 'maintenance' replace the
'gc' builtin entirely at some point, leaving 'git gc' as an alias for
some specific arguments to 'git maintenance run'.
Create a new test_subcommand helper that allows us to test if a certain
subcommand was run. It requires storing the GIT_TRACE2_EVENT logs in a
file. A negation mode is available that will be used in later tests.
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-17 20:11:42 +02:00
|
|
|
argc = parse_options(argc, argv, prefix,
|
|
|
|
builtin_maintenance_run_options,
|
|
|
|
builtin_maintenance_run_usage,
|
|
|
|
PARSE_OPT_STOP_AT_NON_OPTION);
|
|
|
|
|
|
|
|
if (argc != 0)
|
|
|
|
usage_with_options(builtin_maintenance_run_usage,
|
|
|
|
builtin_maintenance_run_options);
|
2020-09-17 20:11:45 +02:00
|
|
|
return maintenance_run_tasks(&opts);
|
maintenance: create basic maintenance runner
The 'gc' builtin is our current entrypoint for automatically maintaining
a repository. This one tool does many operations, such as repacking the
repository, packing refs, and rewriting the commit-graph file. The name
implies it performs "garbage collection" which means several different
things, and some users may not want to use this operation that rewrites
the entire object database.
Create a new 'maintenance' builtin that will become a more general-
purpose command. To start, it will only support the 'run' subcommand,
but will later expand to add subcommands for scheduling maintenance in
the background.
For now, the 'maintenance' builtin is a thin shim over the 'gc' builtin.
In fact, the only option is the '--auto' toggle, which is handed
directly to the 'gc' builtin. The current change is isolated to this
simple operation to prevent more interesting logic from being lost in
all of the boilerplate of adding a new builtin.
Use existing builtin/gc.c file because we want to share code between the
two builtins. It is possible that we will have 'maintenance' replace the
'gc' builtin entirely at some point, leaving 'git gc' as an alias for
some specific arguments to 'git maintenance run'.
Create a new test_subcommand helper that allows us to test if a certain
subcommand was run. It requires storing the GIT_TRACE2_EVENT logs in a
file. A negation mode is available that will be used in later tests.
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-17 20:11:42 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
static const char builtin_maintenance_usage[] = N_("git maintenance run [<options>]");
|
|
|
|
|
|
|
|
int cmd_maintenance(int argc, const char **argv, const char *prefix)
|
|
|
|
{
|
|
|
|
if (argc < 2 ||
|
|
|
|
(argc == 2 && !strcmp(argv[1], "-h")))
|
|
|
|
usage(builtin_maintenance_usage);
|
|
|
|
|
|
|
|
if (!strcmp(argv[1], "run"))
|
|
|
|
return maintenance_run(argc - 1, argv + 1, prefix);
|
|
|
|
|
|
|
|
die(_("invalid subcommand: %s"), argv[1]);
|
|
|
|
}
|