Multi-ref transaction support we merged a few releases ago
unnecessarily kept many file descriptors open, risking to fail with
resource exhaustion. This is for 2.4.x track.
* mh/write-refs-sooner-2.4:
ref_transaction_commit(): fix atomicity and avoid fd exhaustion
ref_transaction_commit(): remove the local flags variable
ref_transaction_commit(): inline call to write_ref_sha1()
rename_ref(): inline calls to write_ref_sha1() from this function
commit_ref_update(): new function, extracted from write_ref_sha1()
write_ref_to_lockfile(): new function, extracted from write_ref_sha1()
t7004: rename ULIMIT test prerequisite to ULIMIT_STACK_SIZE
update-ref: test handling large transactions properly
ref_transaction_commit(): fix atomicity and avoid fd exhaustion
ref_transaction_commit(): remove the local flags variable
ref_transaction_commit(): inline call to write_ref_sha1()
rename_ref(): inline calls to write_ref_sha1() from this function
commit_ref_update(): new function, extracted from write_ref_sha1()
write_ref_to_lockfile(): new function, extracted from write_ref_sha1()
t7004: rename ULIMIT test prerequisite to ULIMIT_STACK_SIZE
update-ref: test handling large transactions properly
The old code was roughly
for update in updates:
acquire locks and check old_sha
for update in updates:
if changing value:
write_ref_to_lockfile()
commit_ref_update()
for update in updates:
if deleting value:
unlink()
rewrite packed-refs file
for update in updates:
if reference still locked:
unlock_ref()
This has two problems.
Non-atomic updates
==================
The atomicity of the reference transaction depends on all pre-checks
being done in the first loop, before any changes have started being
committed in the second loop. The problem is that
write_ref_to_lockfile() (previously part of write_ref_sha1()), which
is called from the second loop, contains two more checks:
* It verifies that new_sha1 is a valid object
* If the reference being updated is a branch, it verifies that
new_sha1 points at a commit object (as opposed to a tag, tree, or
blob).
If either of these checks fails, the "transaction" is aborted during
the second loop. But this might happen after some reference updates
have already been permanently committed. In other words, the
all-or-nothing promise of "git update-ref --stdin" could be violated.
So these checks have to be moved to the first loop.
File descriptor exhaustion
==========================
The old code locked all of the references in the first loop, leaving
all of the lockfiles open until later loops. Since we might be
updating a lot of references, this could result in file descriptor
exhaustion.
The solution
============
After this patch, the code looks like
for update in updates:
acquire locks and check old_sha
if changing value:
write_ref_to_lockfile()
else:
close_ref()
for update in updates:
if changing value:
commit_ref_update()
for update in updates:
if deleting value:
unlink()
rewrite packed-refs file
for update in updates:
if reference still locked:
unlock_ref()
This fixes both problems:
1. The pre-checks in write_ref_to_lockfile() are now done in the first
loop, before any changes have been committed. If any of the checks
fails, the whole transaction can now be rolled back correctly.
2. All lockfiles are closed in the first loop immediately after they
are created (either by write_ref_to_lockfile() or by close_ref()).
This means that there is never more than one open lockfile at a
time, preventing file descriptor exhaustion.
To simplify the bookkeeping across loops, add a new REF_NEEDS_COMMIT
bit to update->flags, which keeps track of whether the corresponding
lockfile needs to be committed, as opposed to just unlocked. (Since
"struct ref_update" is internal to the refs module, this change is not
visible to external callers.)
This change fixes two tests in t1400.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
While we are in the area, let's remove a superfluous definite article
from the error message that is emitted when the reference cannot be
locked. This improves how it reads and makes it a bit shorter.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
If "git update-ref --stdin" was given a "verify" command with no
"<newvalue>" at all (not even zeros), the code was mistakenly setting
have_old=0 (and leaving old_sha1 uninitialized). But this is
incorrect: this command is supposed to verify that the reference
doesn't exist. So in this case we really need old_sha1 to be set to
null_sha1 and have_old to be set to 1.
Moreover, since have_old was being set to zero, *no* check of the old
value was being done, so the new value of the reference was being set
unconditionally to the value in new_sha1. new_sha1, in turn, was set
to null_sha1 in the expectation that that was the old value and it
shouldn't be changed. But because the precondition was not being
checked, the result was that the reference was being deleted
unconditionally.
So, if <oldvalue> is missing, set have_old unconditionally and set
old_sha1 to null_sha1.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Acked-by: Brad King <brad.king@kitware.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Two of the tests fail because
verify refs/heads/foo
with no argument (not even zeros) actually *deletes* refs/heads/foo.
This problem will be fixed in the next commit.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
There's no straightforward way to grep for all tests dealing with
invalid refs. Put them in a single test script so it is easy to see
what functionality has not been exercised with bad ref names yet.
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If a repository gets in a broken state with too much symref nesting,
it cannot be repaired with "git branch -d":
$ git symbolic-ref refs/heads/nonsense refs/heads/nonsense
$ git branch -d nonsense
error: branch 'nonsense' not found.
Worse, "git update-ref --no-deref -d" doesn't work for such repairs
either:
$ git update-ref -d refs/heads/nonsense
error: unable to resolve reference refs/heads/nonsense: Too many levels of symbolic links
Fix both by teaching resolve_ref_unsafe a new RESOLVE_REF_NO_RECURSE
flag and passing it when appropriate.
Callers can still read the value of a symref (for example to print a
message about it) with that flag set --- resolve_ref_unsafe will
resolve one level of symrefs and stop there.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Reviewed-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Update "update-ref --stdin [-z]" and then introduce a transactional
support for (multi-)reference updates.
* mh/ref-transaction: (27 commits)
ref_transaction_commit(): work with transaction->updates in place
struct ref_update: add a type field
struct ref_update: add a lock field
ref_transaction_commit(): simplify code using temporary variables
struct ref_update: store refname as a FLEX_ARRAY
struct ref_update: rename field "ref_name" to "refname"
refs: remove API function update_refs()
update-ref --stdin: reimplement using reference transactions
refs: add a concept of a reference transaction
update-ref --stdin: harmonize error messages
update-ref --stdin: improve the error message for unexpected EOF
t1400: test one mistake at a time
update-ref --stdin -z: deprecate interpreting the empty string as zeros
update-ref.c: extract a new function, parse_next_sha1()
t1400: test that stdin -z update treats empty <newvalue> as zeros
update-ref --stdin: simplify error messages for missing oldvalues
update-ref --stdin: make error messages more consistent
update-ref --stdin: improve error messages for invalid values
update-ref.c: extract a new function, parse_refname()
parse_cmd_verify(): copy old_sha1 instead of evaluating <oldvalue> twice
...
read_ref_at has its own parsing of the reflog file for no really good reason
so lets change this to use the existing reflog iterators. This removes one
instance where we manually unmarshall the reflog file format.
Remove the now redundant ref_msg function.
Log messages for errors are changed slightly. We no longer print the file
name for the reflog, instead we refer to it as 'Log for ref <refname>'.
This might be a minor useability regression, but I don't really think so, since
experienced users would know where the log is anyway and inexperienced users
would not know what to do about/how to repair 'Log ... has gap ...' anyway.
Adapt the t1400 test to handle the change in log messages.
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Make (most of) the error messages for invalid input have the same
format [1]:
$COMMAND [SP $REFNAME]: $MESSAGE
Update the tests accordingly.
[1] A few error messages are left with their old form, because
$COMMAND and $REFNAME aren't passed all the way down the call
stack. Maybe those sites should be changed some day, too.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Distinguish this error from the error that an argument is missing for
another reason. Update the tests accordingly.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This case wants to test passing a bad refname to the "update" command.
But it also passes too few arguments to "update", which muddles the
situation: which error should be diagnosed? So split this test into
two:
* One that passes too few arguments to update
* One that passes all three arguments to "update", but with a bad
refname.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the original version of this command, for the single case of the
"update" command's <newvalue>, the empty string was interpreted as
being equivalent to 40 "0"s. This shorthand is unnecessary (binary
input will usually be generated programmatically anyway), and it
complicates the parser and the documentation.
So gently deprecate this usage: remove its description from the
documentation and emit a warning if it is found. But for reasons of
backwards compatibility, continue to accept it.
Helped-by: Brad King <brad.king@kitware.com>
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Replace three functions, update_store_new_sha1(),
update_store_old_sha1(), and parse_next_arg(), with a single function,
parse_next_sha1(). The new function takes care of a whole argument,
including checking whether it is there, converting it to an SHA-1, and
emitting errors on EOF or for invalid values. The return value
indicates whether the argument was present or absent, which requires
a bit of intelligence because absent values are represented
differently depending on whether "-z" was used.
The new interface means that the calling functions, parse_cmd_*(),
don't have to interpret the result differently based on the
line_termination mode that is in effect. It also means that
parse_cmd_create() can distinguish unambiguously between an empty new
value and a zeros new value, which fixes a failure in t1400.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This is the (slightly inconsistent) status quo; make sure it doesn't
change by accident.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Instead of, for example,
fatal: update refs/heads/master missing [<oldvalue>] NUL
emit
fatal: update refs/heads/master missing <oldvalue>
Update the tests accordingly.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The old error messages emitted for invalid input sometimes said
"<oldvalue>"/"<newvalue>" and sometimes said "old value"/"new value".
Convert them all to the former. Update the tests accordingly.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If an invalid value is passed to "update-ref --stdin" as <oldvalue> or
<newvalue>, include the command and the name of the reference at the
beginning of the error message. Update the tests accordingly.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Previously there were no good tests of C-quoted arguments.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The old parse_arg(), when fed an argument
"refs/heads/a"master
parsed 'refs/heads/a' off of the front of the argument and considered
itself successful. It was only when parse_next_arg() tried to parse
the *next* argument that a problem was noticed. But in fact, the
definition of the input format requires arguments to be terminated by
SP or NUL, so *this* argument is already erroneous and parse_arg()
should diagnose the problem.
So teach parse_arg() to verify that C-quoted arguments are terminated
correctly. If not, emit a more specific error message.
There is no corresponding error case of a non-C-quoted argument that
is not terminated correctly, because the end of a non-quoted argument
is *by definition* a space or NUL, so there is no way to insert other
junk between the "end" of the argument and the argument terminator.
Adjust the tests to expect the new error message. Add a docstring to
the function, incorporating the comments that were formerly within the
function plus some added information.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The old version was passing (among other things)
update SP refs/heads/c NUL NUL 0{40} NUL
to "git update-ref -z --stdin" to test whether the old-value check for
c is working. But the <newvalue> is empty, which is a bit off the
beaten track.
So, to be sure that we are testing what we want to test, provide an
actual <newvalue> on the "update" line.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The test
stdin -z create ref fails with zero new value
actually passes an empty new value, not a zero new value. So rename
the test s/zero/empty/, and change the expected error from
fatal: create $c given zero new value
to
fatal: create $c missing <newvalue>
Of course, this makes the test fail now, because although "git
update-ref" tries to distinguish between these two errors, it does not
succeed in this situation. Fixing it is more than a one-liner, so
mark the test test_expect_failure for now. The failure will be fixed
later in this patch series.
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Aman Gupta <aman@tmm1.net>
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Acked-by: Brad King <brad.king@kitware.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Extend t/t1400-update-ref.sh to cover cases using the --stdin option.
Signed-off-by: Brad King <brad.king@kitware.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When deleting a ref through a symref (e.g. using 'git update-ref -d HEAD'
to delete refs/heads/master), we would remove the loose ref, but a packed
version of the same ref would remain, the end result being that instead of
deleting refs/heads/master we would appear to reset it to its state as of
the last repack.
This patch fixes the issue, by making sure we pass the correct ref name
when invoking repack_without_ref() from within delete_ref().
Signed-off-by: Johan Herland <johan@herland.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When deleting a ref through a symref (e.g. using 'git update-ref -d HEAD'
to delete refs/heads/master), we currently fail to remove the packed
version of that ref. This testcase demonstrates the bug.
Signed-off-by: Johan Herland <johan@herland.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* jc/fix-diff-files-unmerged:
diff-files: show unmerged entries correctly
diff: remove often unused parameters from diff_unmerge()
diff.c: return filepair from diff_unmerge()
test: use $_z40 from test-lib
There is no need to duplicate the definition of $_z40 and $_x40 that
test-lib.sh supplies the test scripts.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
As t/README explains:
When a gitcommand dies due to a segfault, test_must_fail
diagnoses it as an error; "! git <command>" treats it as
just another expected failure, which would let such a bug
go unnoticed.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Same rules as before: this patch only adds " &&" to the end of
some lines in the test suite.
Intended to be applied on top of or squashed with the last
batch if they look okay.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Several old tests were written before test_cmp was introduced, convert
these to test_cmp.
If were are at it, fix the order of the arguments where necessary to
make expected come first, so the command shows how the test result
deviates from the correct output.
Signed-off-by: Miklos Vajna <vmiklos@frugalware.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In this case we did nothing in the past, but we should delete the
reference in fact.
The problem was that when the symref is not packed but the referenced
ref is packed, then we assumed that the symref is packed as well, but
symrefs are never packed.
Signed-off-by: Miklos Vajna <vmiklos@frugalware.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Till now --no-deref was just ignored when deleting refs, fix this.
Signed-off-by: Miklos Vajna <vmiklos@frugalware.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This patch changes every occurrence of "! git" -- with the meaning
that a git call has to gracefully fail -- into "test_must_fail git".
This is useful to
- make sure the test does not fail because of a signal,
e.g. SIGSEGV, and
- advertise the use of "test_must_fail" for new tests.
Signed-off-by: Stephan Beyer <s-beyer@gmx.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When the user specifies a ref by a reflog entry older than
one we have (e.g., "HEAD@{20 years ago"}), we issue a
warning and give them the "from" value of the oldest reflog
entry. That is, we say "we don't know what happened before
this entry, but before this we know we had some particular
SHA1".
However, the oldest reflog entry is often a creation event
such as clone or branch creation. In this case, the entry
claims that the ref went from "00000..." (the null sha1) to
the new value, and the reflog lookup returns the null sha1.
While this is technically correct (the entry tells us that
the ref didn't exist at the specified time) it is not
terribly useful to the end user. What they probably want
instead is "the oldest useful sha1 that this ref ever had".
This patch changes the behavior such that if the oldest ref
would return the null sha1, it instead returns the first
value the ref ever had.
We never discovered this problem in the test scripts because
we created "fake" reflogs that had only a specified segment
of history. This patch updates the tests with a creation
event at the beginning of history.
Signed-off-by: Jeff King <peff@peff.net>
Acked-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Giving the old sha1 is already optional when changing a ref, and it's
quite handy when running update-ref manually. So make it optional for
deleting a ref too.
Signed-off-by: Karl Hasselström <kha@treskal.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Originally, test_expect_failure was designed to be the opposite
of test_expect_success, but this was a bad decision. Most tests
run a series of commands that leads to the single command that
needs to be tested, like this:
test_expect_{success,failure} 'test title' '
setup1 &&
setup2 &&
setup3 &&
what is to be tested
'
And expecting a failure exit from the whole sequence misses the
point of writing tests. Your setup$N that are supposed to
succeed may have failed without even reaching what you are
trying to test. The only valid use of test_expect_failure is to
check a trivial single command that is expected to fail, which
is a minority in tests of Porcelain-ish commands.
This large-ish patch rewrites all uses of test_expect_failure to
use test_expect_success and rewrites the condition of what is
tested, like this:
test_expect_success 'test title' '
setup1 &&
setup2 &&
setup3 &&
! this command should fail
'
test_expect_failure is redefined to serve as a reminder that
that test *should* succeed but due to a known breakage in git it
currently does not pass. So if git-foo command should create a
file 'bar' but you discovered a bug that it doesn't, you can
write a test like this:
test_expect_failure 'git-foo should create bar' '
rm -f bar &&
git foo &&
test -f bar
'
This construct acts similar to test_expect_success, but instead
of reporting "ok/FAIL" like test_expect_success does, the
outcome is reported as "FIXED/still broken".
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This makes write_ref_sha1() more careful: it actually checks the SHA1 of
the ref it is updating, and refuses to update a ref with an object that it
cannot find.
Perhaps more importantly, it also refuses to update a branch head with a
non-commit object. I don't quite know *how* the stable series maintainers
were able to corrupt their repository to have a HEAD that pointed to a tag
rather than a commit object, but they did. Which results in a totally
broken repository that cannot be cloned or committed on.
So make it harder for people to shoot themselves in the foot like that.
The test t1400-update-ref.sh is fixed at the same time, as it
assumed that the commands involved in the particular test would
not care about corrupted repositories whose refs point at
nonexistant bogus objects. That assumption does not hold true
anymore.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The last rm in the test was lacking an "&&" before it,
which caused the errors in the commands be silently hidden.
Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Kristian Høgsberg pointed out that the two file modifications
we were doing during the 'creating initial files' step are not even
used within the test suite. This was actually confusing as we do
not even need these changes for the tests to pass. All that really
matters here is the specific commit dates are used so that these
appear in the branch's reflog, and that the dates are different so
that the branch will update when asked and the reflog entry is
also updated. There is no need for the file modification.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
You can pass an extra argument to the function to receive the
reflog message information. Also when the log does not go back
beyond the point the user asked, the cut-off time and count are
given back to the caller for emitting the error messages as
appropriately.
We could later add configuration for get_sha1_basic() to make it
an error instead of it being just a warning.
Signed-off-by: Junio C Hamano <junkio@cox.net>
New and experienced Git users alike are finding out too late that
they forgot to enable reflogs in the current repository, and cannot
use the information stored within it to recover from an incorrectly
entered command such as `git reset --hard HEAD^^^` when they really
meant HEAD^^ (aka HEAD~2).
So enable reflogs by default in all future versions of Git, unless
the user specifically disables it with:
[core]
logAllRefUpdates = false
in their .git/config or ~/.gitconfig.
We only enable reflogs in repositories that have a working directory
associated with them, as shared/bare repositories do not have
an easy means to prune away old log entries, or may fail logging
entirely if the user's gecos information is not valid during a push.
This heuristic was suggested on the mailing list by Junio.
Documentation was also updated to indicate the new default behavior.
We probably should start to teach usuing the reflog to recover
from mistakes in some of the tutorial material, as new users are
likely to make a few along the way and will feel better knowing
they can recover from them quickly and easily, without fsck-objects'
lost+found features.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
It depended on specific error messages to detect failure but the
implementation changed and broke the test. This fixes the breakage
minimally.
Signed-off-by: Junio C Hamano <junkio@cox.net>
I had a hard time figuring out why this test was failing with
the packed-refs update without running it under "sh -x". This
makes output from "sh t1400-update-ref.sh -v" more descriptive.
Updating other tests would be a good janitorial task.
Signed-off-by: Junio C Hamano <junkio@cox.net>
