Now that we can read packet data from memory as easily as a
descriptor, get_remote_heads can take either one as a
source. This will allow further refactoring in remote-curl.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Most of the callers of packet_read_line just read into a
static 1000-byte buffer (callers which handle arbitrary
binary data already use LARGE_PACKET_MAX). This works fine
in practice, because:
1. The only variable-sized data in these lines is a ref
name, and refs tend to be a lot shorter than 1000
characters.
2. When sending ref lines, git-core always limits itself
to 1000 byte packets.
However, the only limit given in the protocol specification
in Documentation/technical/protocol-common.txt is
LARGE_PACKET_MAX; the 1000 byte limit is mentioned only in
pack-protocol.txt, and then only describing what we write,
not as a specific limit for readers.
This patch lets us bump the 1000-byte limit to
LARGE_PACKET_MAX. Even though git-core will never write a
packet where this makes a difference, there are two good
reasons to do this:
1. Other git implementations may have followed
protocol-common.txt and used a larger maximum size. We
don't bump into it in practice because it would involve
very long ref names.
2. We may want to increase the 1000-byte limit one day.
Since packets are transferred before any capabilities,
it's difficult to do this in a backwards-compatible
way. But if we bump the size of buffer the readers can
handle, eventually older versions of git will be
obsolete enough that we can justify bumping the
writers, as well. We don't have plans to do this
anytime soon, but there is no reason not to start the
clock ticking now.
Just bumping all of the reading bufs to LARGE_PACKET_MAX
would waste memory. Instead, since most readers just read
into a temporary buffer anyway, let's provide a single
static buffer that all callers can use. We can further wrap
this detail away by having the packet_read_line wrapper just
use the buffer transparently and return a pointer to the
static storage. That covers most of the cases, and the
remaining ones already read into their own LARGE_PACKET_MAX
buffers.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The packets sent during ref negotiation are all terminated
by newline; even though the code to chomp these newlines is
short, we end up doing it in a lot of places.
This patch teaches packet_read_line to auto-chomp the
trailing newline; this lets us get rid of a lot of inline
chomping code.
As a result, some call-sites which are not reading
line-oriented data (e.g., when reading chunks of packfiles
alongside sideband) transition away from packet_read_line to
the generic packet_read interface. This patch converts all
of the existing callsites.
Since the function signature of packet_read_line does not
change (but its behavior does), there is a possibility of
new callsites being introduced in later commits, silently
introducing an incompatibility. However, since a later
patch in this series will change the signature, such a
commit would have to be merged directly into this commit,
not to the tip of the series; we can therefore ignore the
issue.
This is an internal cleanup and should produce no change of
behavior in the normal case. However, there is one corner
case to note. Callers of packet_read_line have never been
able to tell the difference between a flush packet ("0000")
and an empty packet ("0004"), as both cause packet_read_line
to return a length of 0. Readers treat them identically,
even though Documentation/technical/protocol-common.txt says
we must not; it also says that implementations should not
send an empty pkt-line.
By stripping out the newline before the result gets to the
caller, we will now treat the newline-only packet ("0005\n")
the same as an empty packet, which in turn gets treated like
a flush packet. In practice this doesn't matter, as neither
empty nor newline-only packets are part of git's protocols
(at least not for the line-oriented bits, and readers who
are not expecting line-oriented packets will be calling
packet_read directly, anyway). But even if we do decide to
care about the distinction later, it is orthogonal to this
patch. The right place to tighten would be to stop treating
empty packets as flush packets, and this change does not
make doing so any harder.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This is just write_or_die by another name. The one
distinction is that write_or_die will treat EPIPE specially
by suppressing error messages. That's fine, as we die by
SIGPIPE anyway (and in the off chance that it is disabled,
write_or_die will simulate it).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The current parsing scheme for upload-archive is to pack
arguments into a fixed-size buffer, separated by NULs, and
put a pointer to each argument in the buffer into a
fixed-size argv array.
This works fine, and the limits are high enough that nobody
reasonable is going to hit them, but it makes the code hard
to follow. Instead, let's just stuff the arguments into an
argv_array, which is much simpler. That lifts the "all
arguments must fit inside 4K together" limit.
We could also trivially lift the MAX_ARGS limitation (in
fact, we have to keep extra code to enforce it). But that
would mean a client could force us to allocate an arbitrary
amount of memory simply by sending us "argument" lines. By
limiting the MAX_ARGS, we limit an attacker to about 4
megabytes (64 times a maximum 64K packet buffer). That may
sound like a lot compared to the 4K limit, but it's not a
big deal compared to what git-archive will actually allocate
while working (e.g., to load blobs into memory). The
important thing is that it is bounded.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
According to the comment, enter_repo will modify its input.
However, this has not been the case since 1c64b48
(enter_repo: do not modify input, 2011-10-04). Drop the
now-useless copy.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Allow the server side to redact the refs/ namespace it shows to the
client.
Will merge to 'master'.
* jc/hidden-refs:
upload/receive-pack: allow hiding ref hierarchies
upload-pack: simplify request validation
upload-pack: share more code
Instead of requiring the full 40-hex object names on the index
line, we can read submodule commit object names from the textual
diff when synthesizing a fake ancestore tree for "git am -3".
* jc/extended-fake-ancestor-for-gitlink:
apply: verify submodule commit object name better
pathspec is the most widely used term, and is the one defined in
gitglossary.txt. <filepattern> was used only in the synopsys for git-add
and git-commit, and in git-add.txt. Get rid of it.
This patch is obtained with by running:
perl -pi -e 's/filepattern/pathspec/' `git grep -l filepattern`
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Make sure the similarity value shown in the "apply --summary"
output is sensible, even when the input had a bogus value.
* jk/apply-similaritly-parsing:
builtin/apply: tighten (dis)similarity index parsing
Rebasing the history of superproject with change in the submodule
has been broken since v1.7.12.
* jc/fake-ancestor-with-non-blobs:
apply: diagnose incomplete submodule object name better
apply: simplify build_fake_ancestor()
git-am: record full index line in the patch used while rebasing
Fix various error messages and conditions in "git branch", e.g. we
advertised "branch -d/-D" to remove one or more branches but actually
implemented removal of zero or more branches---request to remove no
branches was not rejected.
* nd/branch-error-cases:
branch: let branch filters imply --list
docs: clarify git-branch --list behavior
branch: mark more strings for translation
branch: give a more helpful message on redundant arguments
branch: reject -D/-d without branch name
A repository may have refs that are only used for its internal
bookkeeping purposes that should not be exposed to the others that
come over the network.
Teach upload-pack to omit some refs from its initial advertisement
by paying attention to the uploadpack.hiderefs multi-valued
configuration variable. Do the same to receive-pack via the
receive.hiderefs variable. As a convenient short-hand, allow using
transfer.hiderefs to set the value to both of these variables.
Any ref that is under the hierarchies listed on the value of these
variable is excluded from responses to requests made by "ls-remote",
"fetch", etc. (for upload-pack) and "push" (for receive-pack).
Because these hidden refs do not count as OUR_REF, an attempt to
fetch objects at the tip of them will be rejected, and because these
refs do not get advertised, "git push :" will not see local branches
that have the same name as them as "matching" ones to be sent.
An attempt to update/delete these hidden refs with an explicit
refspec, e.g. "git push origin :refs/hidden/22", is rejected. This
is not a new restriction. To the pusher, it would appear that there
is no such ref, so its push request will conclude with "Now that I
sent you all the data, it is time for you to update the refs. I saw
that the ref did not exist when I started pushing, and I want the
result to point at this commit". The receiving end will apply the
compare-and-swap rule to this request and rejects the push with
"Well, your update request conflicts with somebody else; I see there
is such a ref.", which is the right thing to do. Otherwise a push to
a hidden ref will always be "the last one wins", which is not a good
default.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Rebasing the history of superproject with change in the submodule
was broken since v1.7.12.
* jc/fake-ancestor-with-non-blobs:
apply: diagnose incomplete submodule object name better
apply: simplify build_fake_ancestor()
git-am: record full index line in the patch used while rebasing
A textual patch also records the submodule commit object name in
full. Make the parsing more robust by reading from there and
verifying the (possibly abbreviated) name on the index line matches.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Clarify the ownership rule for commit->buffer field, which some
callers incorrectly accessed without making sure it is populated.
* jk/read-commit-buffer-data-after-free:
logmsg_reencode: lazily load missing commit buffers
logmsg_reencode: never return NULL
commit: drop useless xstrdup of commit message
Forbid "git add -u" and "git add -A" without pathspec run from a
subdirectory, to train people to type "." (or ":/") to make the
choice of default does not matter.
* mm/add-u-A-sans-pathspec:
add: warn when -u or -A is used without pathspec
Improve error and advice messages given locally when "git push"
refuses when it cannot compute fast-forwardness by separating these
cases from the normal "not a fast-forward; merge first and push
again" case.
* jc/push-reject-reasons:
push: finishing touches to explain REJECT_ALREADY_EXISTS better
push: introduce REJECT_FETCH_FIRST and REJECT_NEEDS_FORCE
push: further simplify the logic to assign rejection reason
push: further clean up fields of "struct ref"
Configuration parsing for tar.* configuration variables were
broken. Introduce a new config-keyname parser API to make the
callers much less error prone.
* jk/config-parsing-cleanup:
reflog: use parse_config_key in config callback
help: use parse_config_key for man config
submodule: simplify memory handling in config parsing
submodule: use parse_config_key when parsing config
userdiff: drop parse_driver function
convert some config callbacks to parse_config_key
archive-tar: use parse_config_key when parsing config
config: add helper function for parsing key names
Allow a configuration variable core.commentchar to customize the
character used to comment out the hint lines in the edited text from
the default '#'.
* jc/custom-comment-char:
Allow custom "comment char"
Attempt to "branch --edit-description" an existing branch, while
being on a detached HEAD, errored out.
* nd/edit-branch-desc-while-detached:
branch: no detached HEAD check when editing another branch's description
* jc/merge-blobs:
Makefile: Replace merge-file.h with merge-blobs.h in LIB_H
merge-tree: fix d/f conflicts
merge-tree: add comments to clarify what these functions are doing
merge-tree: lose unused "resolve_directories"
merge-tree: lose unused "flags" from merge_list
Which merge_file() function do you mean?
This was prompted by an incorrect warning issued by clang [1], and a
suggestion by Linus to restrict the range to check for values greater
than INT_MAX since these will give bogus output after casting to int.
In fact the (dis)similarity index is a percentage, so reject values
greater than 100.
[1] http://article.gmane.org/gmane.comp.version-control.git/213857
Signed-off-by: John Keeping <john@keeping.me.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Attempt to "branch --edit-description" an existing branch, while
being on a detached HEAD, errored out.
* nd/edit-branch-desc-while-detached:
branch: no detached HEAD check when editing another branch's description
Help "fetch only" repositories that do not trigger "gc --auto"
often enough.
* jk/gc-auto-after-fetch:
fetch-pack: avoid repeatedly re-scanning pack directory
fetch: run gc --auto after fetching
We stopped paying attention to $GIT_CONFIG environment that points
at a single configuration file from any command other than "git config"
quite a while ago, but "git clone" internally set, exported, and
then unexported the variable during its operation unnecessarily.
* jc/no-git-config-in-clone:
clone: do not export and unexport GIT_CONFIG
"git fetch --depth" was broken in at least three ways. The
resulting history was deeper than specified by one commit, it was
unclear how to wipe the shallowness of the repository with the
command, and documentation was misleading.
* nd/fetch-depth-is-broken:
fetch: elaborate --depth action
upload-pack: fix off-by-one depth calculation in shallow clone
fetch: add --unshallow for turning shallow repo into complete one
"git am -3" uses this function to build a tree that records how the
preimage the patch was created from would have looked like. An
abbreviated object name on the index line is ordinarily sufficient
for us to figure out the object name the preimage tree would have
contained, but a change to a submodule by definition shows an object
name of a submodule commit which our repository should not have, and
get_sha1_blob() is not an appropriate way to read it (or get_sha1()
for that matter).
Use get_sha1_hex() and complain if we do not find a full object name
there.
We could read from the payload part of the patch to learn the full
object name of the commit, but the primary user "git rebase" has
been fixed to give us a full object name, so this should suffice
for now.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The local variable sha1_ptr in the build_fake_ancestor() function
used to either point at the null_sha1[] (if the ancestor did not
have the path) or at sha1[] (if we read the object name into the
local array), but 7a98869 (apply: get rid of --index-info in favor
of --build-fake-ancestor, 2007-09-17) made the "missing in the
ancestor" case unnecessary, hence sha1_ptr, when used, always points
at the local array.
Get rid of the unneeded variable, and restructure the if/else
cascade a bit to make it easier to read. There should be no
behaviour change.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Currently, a branch filter like `--contains`, `--merged`, or
`--no-merged` is ignored when we are not in listing mode.
For example:
git branch --contains=foo bar
will create the branch "bar" from the current HEAD, ignoring
the `--contains` argument entirely. This is not very
helpful. There are two reasonable behaviors for git here:
1. Flag an error; the arguments do not make sense.
2. Implicitly go into `--list` mode
This patch chooses the latter, as it is more convenient, and
there should not be any ambiguity with attempting to create
a branch; using `--contains` and not wanting to list is
nonsensical.
That leaves the case where an explicit modification option
like `-d` is given. We already catch the case where
`--list` is given alongside `-d` and flag an error. With
this patch, we will also catch the use of `--contains` and
other filter options alongside `-d`.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When giving arguments without "--" disambiguation, object names
that come earlier on the command line must not be interpretable as
pathspecs and pathspecs that come later on the command line must
not be interpretable as object names. Tweak the disambiguation
rule so that ":/" (no other string before or after) is always
interpreted as a pathspec, to avoid having to say "git cmd -- :/".
* nd/magic-pathspec-from-root:
grep: avoid accepting ambiguous revision
Update :/abc ambiguity check
A header file that has the definition of a static array was
included in two places, wasting the space.
* jc/help:
help: include <common-cmds.h> only in one file
Most Git commands that can be used with or without pathspec operate
tree-wide by default, the pathspec being used to restrict their
scope. A few exceptions are: 'git grep', 'git clean', 'git add -u'
and 'git add -A'. When run in a subdirectory without pathspec, they
operate only on paths in the current directory.
The inconsistency of 'git add -u' and 'git add -A' is particularly
problematic since other 'git add' subcommands (namely 'git add -p'
and 'git add -e') are tree-wide by default. It also means that "git
add -u && git commit" will record a state that is different from
what is recorded with "git commit -a".
Flipping the default now is unacceptable, so let's start training
users to type 'git add -u|-A :/' or 'git add -u|-A .' explicitly, to
prepare for the next steps:
* forbid 'git add -u|-A' without pathspec (like 'git add' without
option)
* much later, maybe, re-allow 'git add -u|-A' without pathspec, that
will add all tracked and modified files, or all files, tree-wide.
A nice side effect of this patch is that it makes the :/ magic
pathspec easier to discover for users.
When the command is called from the root of the tree, there is no
ambiguity and no need to change the behavior, hence no need to warn.
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we look up a sha1 object for reading via parse_object() =>
read_sha1_file() => read_object() callpath, we first check
packfiles, and then loose objects. If we still haven't found it, we
re-scan the list of packfiles in `objects/pack`. This final step
ensures that we can co-exist with a simultaneous repack process
which creates a new pack and then prunes the old object.
This extra re-scan usually does not have a performance impact for
two reasons:
1. If an object is missing, then typically the re-scan will find a
new pack, then no more misses will occur. Or if it truly is
missing, then our next step is usually to die().
2. Re-scanning is cheap enough that we do not even notice.
However, these do not always hold. The assumption in (1) is that the
caller is expecting to find the object. This is usually the case,
but the call to `parse_object` in `everything_local` does not follow
this pattern. It is looking to see whether we have objects that the
remote side is advertising, not something we expect to
have. Therefore if we are fetching from a remote which has many refs
pointing to objects we do not have, we may end up re-scanning the
pack directory many times.
Even with this extra re-scanning, the impact is often not noticeable
due to (2); we just readdir() the packs directory and skip any packs
that are already loaded. However, if there are a large number of
packs, even enumerating the directory can be expensive, especially
if we do it repeatedly.
Having this many packs is a good sign the user should run `git gc`,
but it would still be nice to avoid having to scan the directory at
all.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We generally try to run "gc --auto" after any commands that
might introduce a large number of new objects. An obvious
place to do so is after running "fetch", which may introduce
new loose objects or packs (depending on the size of the
fetch).
While an active developer repository will probably
eventually trigger a "gc --auto" on another action (e.g.,
git-rebase), there are two good reasons why it is nicer to
do it at fetch time:
1. Read-only repositories which track an upstream (e.g., a
continuous integration server which fetches and builds,
but never makes new commits) will accrue loose objects
and small packs, but never coalesce them into a more
efficient larger pack.
2. Fetching is often already perceived to be slow to the
user, since they have to wait on the network. It's much
more pleasant to include a potentially slow auto-gc as
part of the already-long network fetch than in the
middle of productive work with git-rebase or similar.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Usually a commit that makes it to logmsg_reencode will have
been parsed, and the commit->buffer struct member will be
valid. However, some code paths will free commit buffers
after having used them (for example, the log traversal
machinery will do so to keep memory usage down).
Most of the time this is fine; log should only show a commit
once, and then exits. However, there are some code paths
where this does not work. At least two are known:
1. A commit may be shown as part of a regular ref, and
then it may be shown again as part of a submodule diff
(e.g., if a repo contains refs to both the superproject
and subproject).
2. A notes-cache commit may be shown during "log --all",
and then later used to access a textconv cache during a
diff.
Lazily loading in logmsg_reencode does not necessarily catch
all such cases, but it should catch most of them. Users of
the commit buffer tend to be either parsing for structure
(in which they will call parse_commit, and either we will
already have parsed, or we will load commit->buffer lazily
there), or outputting (either to the user, or fetching a
part of the commit message via format_commit_message). In
the latter case, we should always be using logmsg_reencode
anyway (and typically we do so via the pretty-print
machinery).
If there are any cases that this misses, we can fix them up
to use logmsg_reencode (or handle them on a case-by-case
basis if that is inappropriate).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The logmsg_reencode function will return the reencoded
commit buffer, or NULL if reencoding failed or no reencoding
was necessary. Since every caller then ends up checking for NULL
and just using the commit's original buffer, anyway, we can
be a bit more helpful and just return that buffer when we
would have returned NULL.
Since the resulting string may or may not need to be freed,
we introduce a logmsg_free, which checks whether the buffer
came from the commit object or not (callers either
implemented the same check already, or kept two separate
pointers, one to mark the buffer to be used, and one for the
to-be-freed string).
Pushing this logic into logmsg_* simplifies the callers, and
will let future patches lazily load the commit buffer in a
single place.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When git-commit is asked to reuse a commit message via "-c",
we call read_commit_message, which looks up the commit and
hands back either the re-encoded result, or a copy of the
original. We make a copy in the latter case so that the
ownership semantics of the return value are clear (in either
case, it can be freed).
However, since we return a "const char *", and since the
resulting buffer's lifetime is the same as that of the whole
program, we never bother to free it at all.
Let's just drop the copy. That saves us a copy in the common
case. While it does mean we leak in the re-encode case, it
doesn't matter, since we are relying on program exit to free
the memory anyway.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Now that "already exists" errors are given only when a push tries to
update an existing ref in refs/tags/ hierarchy, we can say "the
tag", instead of "the destination reference", and that is far easier
to understand.
Pointed out by Chris Rorvick.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we push to update an existing ref, if:
* the object at the tip of the remote is not a commit; or
* the object we are pushing is not a commit,
it won't be correct to suggest to fetch, integrate and push again,
as the old and new objects will not "merge". We should explain that
the push must be forced when there is a non-committish object is
involved in such a case.
If we do not have the current object at the tip of the remote, we do
not even know that object, when fetched, is something that can be
merged. In such a case, suggesting to pull first just like
non-fast-forward case may not be technically correct, but in
practice, most such failures are seen when you try to push your work
to a branch without knowing that somebody else already pushed to
update the same branch since you forked, so "pull first" would work
as a suggestion most of the time. And if the object at the tip is
not a commit, "pull first" will fail, without making any permanent
damage. As a side effect, it also makes the error message the user
will get during the next "push" attempt easier to understand, now
the user is aware that a non-commit object is involved.
In these cases, the current code already rejects such a push on the
client end, but we used the same error and advice messages as the
ones used when rejecting a non-fast-forward push, i.e. pull from
there and integrate before pushing again.
Introduce new rejection reasons and reword the messages
appropriately.
[jc: with help by Peff on message details]
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add an extra hook so that "git push" that is run without making
sure what is being pushed is sane can be checked and rejected (as
opposed to the user deciding not pushing).
* as/pre-push-hook:
Add sample pre-push hook script
push: Add support for pre-push hooks
hooks: Add function to check if a hook exists
Add a new command "git check-ignore" for debugging .gitignore
files.
The variable names may want to get cleaned up but that can be done
in-tree.
* as/check-ignore:
clean.c, ls-files.c: respect encapsulation of exclude_list_groups
t0008: avoid brace expansion
add git-check-ignore sub-command
setup.c: document get_pathspec()
add.c: extract new die_if_path_beyond_symlink() for reuse
add.c: extract check_path_for_gitlink() from treat_gitlinks() for reuse
pathspec.c: rename newly public functions for clarity
add.c: move pathspec matchers into new pathspec.c for reuse
add.c: remove unused argument from validate_pathspec()
dir.c: improve docs for match_pathspec() and match_pathspec_depth()
dir.c: provide clear_directory() for reclaiming dir_struct memory
dir.c: keep track of where patterns came from
dir.c: use a single struct exclude_list per source of excludes
Conflicts:
builtin/ls-files.c
dir.c
