Commit Graph

2188 Commits

Author SHA1 Message Date
Junio C Hamano
8377f34540 Merge branch 'jh/memihash-opt'
Hotfix for a topic that is already in 'master'.

* jh/memihash-opt:
  p0004: make perf test executable
  t3008: skip lazy-init test on a single-core box
  test-online-cpus: helper to return cpu count
  name-hash: fix buffer overrun
2017-04-19 21:37:25 -07:00
Junio C Hamano
5ab8f2261f Merge branch 'nd/files-backend-git-dir'
The "submodule" specific field in the ref_store structure is
replaced with a more generic "gitdir" that can later be used also
when dealing with ref_store that represents the set of refs visible
from the other worktrees.

* nd/files-backend-git-dir: (28 commits)
  refs.h: add a note about sorting order of for_each_ref_*
  t1406: new tests for submodule ref store
  t1405: some basic tests on main ref store
  t/helper: add test-ref-store to test ref-store functions
  refs: delete pack_refs() in favor of refs_pack_refs()
  files-backend: avoid ref api targeting main ref store
  refs: new transaction related ref-store api
  refs: add new ref-store api
  refs: rename get_ref_store() to get_submodule_ref_store() and make it public
  files-backend: replace submodule_allowed check in files_downcast()
  refs: move submodule code out of files-backend.c
  path.c: move some code out of strbuf_git_path_submodule()
  refs.c: make get_main_ref_store() public and use it
  refs.c: kill register_ref_store(), add register_submodule_ref_store()
  refs.c: flatten get_ref_store() a bit
  refs: rename lookup_ref_store() to lookup_submodule_ref_store()
  refs.c: introduce get_main_ref_store()
  files-backend: remove the use of git_path()
  files-backend: add and use files_ref_path()
  files-backend: add and use files_reflog_path()
  ...
2017-04-19 21:37:19 -07:00
Junio C Hamano
3817d631de Merge branch 'ab/regen-perl-mak-with-different-perl'
Update the build dependency so that an update to /usr/bin/perl
etc. result in recomputation of perl.mak file.

* ab/regen-perl-mak-with-different-perl:
  perl: regenerate perl.mak if perl -V changes
2017-04-16 23:29:33 -07:00
Michael Haggerty
958f964691 refs: split ref_cache code into separate files
The `ref_cache` code is currently too tightly coupled to
`files-backend`, making the code harder to understand and making it
awkward for new code to use `ref_cache` (as we indeed have planned).
Start loosening that coupling by splitting `ref_cache` into a separate
module.

This commit moves code, adds declarations, and changes the visibility
of some functions, but doesn't change any code.

The modules are still too tightly coupled, but the situation will be
improved in subsequent commits.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-16 21:32:45 -07:00
Jeff Hostetler
a6db3fbb6e read-cache: add strcmp_offset function
Add strcmp_offset() function to also return the offset of the
first change.

Add unit test and helper to verify.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-15 02:21:12 -07:00
Nguyễn Thái Ngọc Duy
80f2a6097c t/helper: add test-ref-store to test ref-store functions
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-14 03:53:25 -07:00
Jeff Hostetler
e3482ccf27 test-online-cpus: helper to return cpu count
Created helper executable to print the value of online_cpus()
allowing multi-threaded tests to be skipped when appropriate.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-12 23:17:19 -07:00
Junio C Hamano
876eb616d3 Merge branch 'jk/make-coccicheck-detect-errors'
Build fix.

* jk/make-coccicheck-detect-errors:
  Makefile: detect errors in running spatch
2017-03-30 14:07:19 -07:00
Ævar Arnfjörð Bjarmason
c59c4939c2 perl: regenerate perl.mak if perl -V changes
Change the perl/perl.mak build process so that the file is regenerated
if the output of "perl -V" changes.

Before this change updating e.g. /usr/bin/perl to a new major version
would cause the next "make" command to fail, since perl.mak has
hardcoded paths to perl library paths retrieved from its first run.

Now the logic added in commit ee9be06770 ("perl: detect new files in
MakeMaker builds", 2012-07-27) is extended to regenerate
perl/perl.mak if there's any change to "perl -V".

This will in some cases redundantly trigger perl/perl.mak to be
re-made, e.g. if @INC is modified in ways the build process doesn't
care about through sitecustomize.pl, but the common case is that we
just do the right thing and re-generate perl/perl.mak when needed.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-29 09:48:13 -07:00
Jeff King
f5c2bc2b96 Makefile: detect errors in running spatch
The "make coccicheck" target runs spatch against each source
file. But it does so in a for loop, so "make" never sees the
exit code of spatch. Worse, it redirects stderr to a log
file, so the user has no indication of any failure. And then
to top it all off, because we touched the patch file's
mtime, make will refuse to repeat the command because it
think the target is up-to-date.

So for example:

  $ make coccicheck SPATCH=does-not-exist
      SPATCH contrib/coccinelle/free.cocci
      SPATCH contrib/coccinelle/qsort.cocci
      SPATCH contrib/coccinelle/xstrdup_or_null.cocci
      SPATCH contrib/coccinelle/swap.cocci
      SPATCH contrib/coccinelle/strbuf.cocci
      SPATCH contrib/coccinelle/object_id.cocci
      SPATCH contrib/coccinelle/array.cocci
  $ make coccicheck SPATCH=does-not-exist
  make: Nothing to be done for 'coccicheck'.

With this patch, you get:

  $ make coccicheck SPATCH=does-not-exist
       SPATCH contrib/coccinelle/free.cocci
  /bin/sh: 4: does-not-exist: not found
  Makefile:2338: recipe for target 'contrib/coccinelle/free.cocci.patch' failed
  make: *** [contrib/coccinelle/free.cocci.patch] Error 1

It also dumps the log on failure, so any errors from spatch
itself (like syntax errors in our .cocci files) will be seen
by the user.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-29 09:07:12 -07:00
Junio C Hamano
0330344e0f Merge branch 'jh/memihash-opt'
The name-hash used for detecting paths that are different only in
cases (which matter on case insensitive filesystems) has been
optimized to take advantage of multi-threading when it makes sense.

* jh/memihash-opt:
  name-hash: add test-lazy-init-name-hash to .gitignore
  name-hash: add perf test for lazy_init_name_hash
  name-hash: add test-lazy-init-name-hash
  name-hash: perf improvement for lazy_init_name_hash
  hashmap: document memihash_cont, hashmap_disallow_rehash api
  hashmap: add disallow_rehash setting
  hashmap: allow memihash computation to be continued
  name-hash: specify initial size for istate.dir_hash table
2017-03-28 14:06:00 -07:00
Junio C Hamano
48b3693d3c Merge branch 'jk/sha1dc'
The "detect attempt to create collisions" variant of SHA-1
implementation by Marc Stevens (CWI) and Dan Shumow (Microsoft)
has been integrated and made the default.

* jk/sha1dc:
  Makefile: make DC_SHA1 the default
  t0013: add a basic sha1 collision detection test
  Makefile: add DC_SHA1 knob
  sha1dc: disable safe_hash feature
  sha1dc: adjust header includes for git
  sha1dc: add collision-detecting sha1 implementation
2017-03-24 13:07:38 -07:00
Jeff Hostetler
ea19489532 name-hash: add test-lazy-init-name-hash
Add t/helper/test-lazy-init-name-hash.c test code
to demonstrate performance times for lazy_init_name_hash()
using the original single-threaded and the new multi-threaded
code paths.

Includes a --dump option to dump the created hashmaps to
stdout.  You can use this to run both code paths and
confirm that they generate the same hashmaps.

Includes a --analyze option to analyze performance of both
code paths over a range of index sizes to help you find a
lower bound for the LAZY_THREAD_COST in name-hash.c.
For example, passing "-a 4000" will set "istate.cache_nr"
to 4000 and then try the multi-threaded code -- probably
giving 2 threads with 2000 entries each.  It will then
run both the single-threaded (1x4000) and the multi-threaded
(2x2000) and compare the times.  It will then repeat the
test with 8000, 12000, and etc. so that you can see the
cross over.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-24 11:00:03 -07:00
Junio C Hamano
81944e9b54 Merge branch 'bc/sha1-header-selection-with-cpp-macros'
Our source code has used the SHA1_HEADER cpp macro after "#include"
in the C code to switch among the SHA-1 implementations. Instead,
list the exact header file names and switch among implementations
using "#ifdef BLK_SHA1/#include "block-sha1/sha1.h"/.../#endif";
this helps some IDE tools.

* bc/sha1-header-selection-with-cpp-macros:
  hash.h: move SHA-1 implementation selection into a header file
2017-03-17 13:50:27 -07:00
Junio C Hamano
0bb80ab090 Merge branch 'jk/interop-test'
Picking two versions of Git and running tests to make sure the
older one and the newer one interoperate happily has now become
possible.

* jk/interop-test:
  t/interop: add test of old clients against modern git-daemon
  t: add an interoperability test harness
2017-03-17 13:50:24 -07:00
Junio C Hamano
e6b07da278 Makefile: make DC_SHA1 the default
We used to use the SHA1 implementation from the OpenSSL library by
default.  As we are trying to be careful against collision attacks
after the recent "shattered" announcement, switch the default to
encourage people to use DC_SHA1 implementation instead.  Those who
want to use the implementation from OpenSSL can explicitly ask for
it by OPENSSL_SHA1=YesPlease when running "make".

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-17 10:40:25 -07:00
Jeff King
f5f5e7f06c t0013: add a basic sha1 collision detection test
We don't actually have a Git-object collision, so the best
we can do is to run one of the shattered PDFs through
test-sha1. This should trigger the collision check and die.

In a sense this isn't really checking anything that the
upstream sha1collisiondetection project doesn't cover
already. But it at least makes sure that our build correctly
uses the library.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-17 10:40:25 -07:00
Jeff King
8325e43b82 Makefile: add DC_SHA1 knob
This knob lets you use the sha1dc implementation from:

      https://github.com/cr-marcstevens/sha1collisiondetection

which can detect certain types of collision attacks (even
when we only see half of the colliding pair). So it
mitigates any attack which consists of getting the "good"
half of a collision into a trusted repository, and then
later replacing it with the "bad" half. The "good" half is
rejected by the victim's version of Git (and even if they
run an old version of Git, any sha1dc-enabled git will
complain loudly if it ever has to interact with the object).

The big downside is that it's slower than either the openssl
or block-sha1 implementations.

Here are some timings based off of linux.git:

  - compute sha1 over whole packfile
      sha1dc: 3.580s
    blk-sha1: 2.046s (-43%)
     openssl: 1.335s (-62%)

  - rev-list --all --objects
      sha1dc: 33.512s
    blk-sha1: 33.514s (+0.0%)
     openssl: 33.650s (+0.4%)

  - git log --no-merges -10000 -p
      sha1dc: 8.124s
    blk-sha1: 7.986s (-1.6%)
     openssl: 8.203s (+0.9%)

  - index-pack --verify
      sha1dc: 4m19s
    blk-sha1: 2m57s (-32%)
     openssl: 2m19s (-42%)

So overall the sha1 computation with collision detection is
about 1.75x slower than block-sha1, and 2.7x slower than
sha1. But of course most operations do more than just sha1.
Normal object access isn't really slowed at all (both the
+/- changes there are well within the run-to-run noise); any
changes are drowned out by the other work Git is doing.

The most-affected operation is `index-pack --verify`, which
is essentially just computing the sha1 on every object. This
is similar to the `index-pack` invocation that the receiver
of a push or fetch would perform. So clearly there's some
extra CPU load here.

There will also be some latency for the user, though keep in
mind that such an operation will generally be network bound
(this is about a 1.2GB packfile). Some of that extra CPU is
"free" in the sense that we use it while the pack is
streaming in anyway. But most of it comes during the
delta-resolution phase, after the whole pack has been
received. So we can imagine that for this (quite large)
push, the user might have to wait an extra 100 seconds over
openssl (which is what we use now). If we assume they can
push to us at 20Mbit/s, that's 480s for a 1.2GB pack, which
is only 20% slower.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-17 10:40:25 -07:00
brian m. carlson
f18f816cb1 hash.h: move SHA-1 implementation selection into a header file
Many developers use functionality in their editors that allows for quick
syntax checks, including warning about questionable constructs.  This
functionality allows rapid development with fewer errors.  However, such
functionality generally does not allow the specification of
project-specific defines or command-line options.

Since the SHA1_HEADER include is not defined in such a case,
developers see spurious errors when using these tools.  Furthermore,
there are known implementations of "cc" whose '#include' is unhappy
with this construct.

Instead of using SHA1_HEADER, create a hash.h header and use #if
and #elif to select the desired header.  Have the Makefile pass an
appropriate option to help the header select the right implementation to
use.

[jc: make BLK_SHA1 the fallback default as discussed on list,
e.g. <20170314201424.vccij5z2ortq4a4o@sigill.intra.peff.net>; also
remove SHA1_HEADER and SHA1_HEADER_SQ that are no longer used].

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Reviewed-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-15 11:00:09 -07:00
Jeff King
3d8936153d t: add an interoperability test harness
The current test suite is good at letting you test a
particular version of Git. But it's not very good at letting
you test _two_ versions and seeing how they interact (e.g.,
one cloning from the other).

This commit adds a test harness that will build two
arbitrary versions of git and make it easy to call them from
inside your tests. See the README and the example script for
details.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-03-10 14:30:25 -08:00
Junio C Hamano
fb907176de Merge branch 'rj/remove-unused-mktemp'
Code cleanup.

* rj/remove-unused-mktemp:
  wrapper.c: remove unused gitmkstemps() function
  wrapper.c: remove unused git_mkstemp() function
2017-03-10 13:24:24 -08:00
Ramsay Jones
b2d593a779 wrapper.c: remove unused gitmkstemps() function
The last call to the mkstemps() function was removed in commit 659488326
("wrapper.c: delete dead function git_mkstemps()", 22-04-2016). In order
to support platforms without mkstemps(), this functionality was provided,
along with a Makefile build variable (NO_MKSTEMPS), by the gitmkstemps()
function. Remove the dead code, along with the defunct build machinery.

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-28 11:54:21 -08:00
Junio C Hamano
098ed50e8a Merge branch 'js/rebase-helper'
"git rebase -i" starts using the recently updated "sequencer" code.

* js/rebase-helper:
  rebase -i: use the rebase--helper builtin
  rebase--helper: add a builtin helper for interactive rebases
2017-02-27 13:57:14 -08:00
Johannes Schindelin
4557f1add2 rebase--helper: add a builtin helper for interactive rebases
Git's interactive rebase is still implemented as a shell script, despite
its complexity. This implies that it suffers from the portability point
of view, from lack of expressibility, and of course also from
performance. The latter issue is particularly serious on Windows, where
we pay a hefty price for relying so much on POSIX.

Unfortunately, being such a huge shell script also means that we missed
the train when it would have been relatively easy to port it to C, and
instead piled feature upon feature onto that poor script that originally
never intended to be more than a slightly pimped cherry-pick in a loop.

To open the road toward better performance (in addition to all the other
benefits of C over shell scripts), let's just start *somewhere*.

The approach taken here is to add a builtin helper that at first intends
to take care of the parts of the interactive rebase that are most
affected by the performance penalties mentioned above.

In particular, after we spent all those efforts on preparing the sequencer
to process rebase -i's git-rebase-todo scripts, we implement the `git
rebase -i --continue` functionality as a new builtin, git-rebase--helper.

Once that is in place, we can work gradually on tackling the rest of the
technical debt.

Note that the rebase--helper needs to learn about the transient
--ff/--no-ff options of git-rebase, as the corresponding flag is not
persisted to, and re-read from, the state directory.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-09 14:55:26 -08:00
Jeff King
29c2bd5fa8 add oidset API
This is similar to many of our uses of sha1-array, but it
overcomes one limitation of a sha1-array: when you are
de-duplicating a large input with relatively few unique
entries, sha1-array uses 20 bytes per non-unique entry.
Whereas this set will use memory linear in the number of
unique entries (albeit a few more than 20 bytes due to
hashmap overhead).

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-08 15:39:55 -08:00
Junio C Hamano
a77fe4a976 Merge branch 'bc/use-asciidoctor-opt'
Asciidoctor, an alternative reimplementation of AsciiDoc, still
needs some changes to work with documents meant to be formatted
with AsciiDoc.  "make USE_ASCIIDOCTOR=YesPlease" to use it out of
the box to document our pages is getting closer to reality.

* bc/use-asciidoctor-opt:
  Documentation: implement linkgit macro for Asciidoctor
  Makefile: add a knob to enable the use of Asciidoctor
  Documentation: move dblatex arguments into variable
  Documentation: add XSLT to fix DocBook for Texinfo
  Documentation: sort sources for gitman.texi
  Documentation: remove unneeded argument in cat-texi.perl
  Documentation: modernize cat-texi.perl
  Documentation: fix warning in cat-texi.perl
2017-02-02 13:36:57 -08:00
Junio C Hamano
9dec2c652f Merge branch 'js/retire-relink'
Cruft removal.

* js/retire-relink:
  relink: really remove the command
  relink: retire the command
2017-02-02 13:36:54 -08:00
Junio C Hamano
b7786bb4b0 Merge branch 'js/difftool-builtin'
Rewrite a scripted porcelain "git difftool" in C.

* js/difftool-builtin:
  difftool: hack around -Wzero-length-format warning
  difftool: retire the scripted version
  difftool: implement the functionality in the builtin
  difftool: add a skeleton for the upcoming builtin
2017-01-31 13:15:00 -08:00
Junio C Hamano
6ad8b8e98f Merge branch 'rs/qsort-s'
A few codepaths had to rely on a global variable when sorting
elements of an array because sort(3) API does not allow extra data
to be passed to the comparison function.  Use qsort_s() when
natively available, and a fallback implementation of it when not,
to eliminate the need, which is a prerequisite for making the
codepath reentrant.

* rs/qsort-s:
  ref-filter: use QSORT_S in ref_array_sort()
  string-list: use QSORT_S in string_list_sort()
  perf: add basic sort performance test
  add QSORT_S
  compat: add qsort_s()
2017-01-31 13:15:00 -08:00
Johannes Schindelin
ed21e30fef relink: retire the command
Back in the olden days, when all objects were loose and rubber boots were
made out of wood, it made sense to try to share (immutable) objects
between repositories.

Ever since the arrival of pack files, it is but an anachronism.

Let's move the script to the contrib/examples/ directory and no longer
offer it.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-25 14:42:37 -08:00
René Scharfe
04ee8b875b compat: add qsort_s()
The function qsort_s() was introduced with C11 Annex K; it provides the
ability to pass a context pointer to the comparison function, supports
the convention of using a NULL pointer for an empty array and performs a
few safety checks.

Add an implementation based on compat/qsort.c for platforms that lack a
native standards-compliant qsort_s() (i.e. basically everyone).  It
doesn't perform the full range of possible checks: It uses size_t
instead of rsize_t and doesn't check nmemb and size against RSIZE_MAX
because we probably don't have the restricted size type defined.  For
the same reason it returns int instead of errno_t.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-23 11:02:34 -08:00
brian m. carlson
ec3366eb52 Makefile: add a knob to enable the use of Asciidoctor
While Git has traditionally built its documentation using AsciiDoc, some
people wish to use Asciidoctor for speed or other reasons.  Add a
Makefile knob, USE_ASCIIDOCTOR, that sets various options in order to
produce acceptable output.  For HTML output, XHTML5 was chosen, since
the AsciiDoc options also produce XHTML, albeit XHTML 1.1.

Asciidoctor does not have built-in support for the linkgit macro, but it
is available using the Asciidoctor Extensions Lab.  Add a macro to
enable the use of this extension if it is available.  Without it, the
linkgit macros are emitted into the output.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-23 10:56:57 -08:00
Johannes Schindelin
019678d6b1 difftool: retire the scripted version
It served its purpose, but now we have a builtin difftool. Time for the
Perl script to enjoy Florida.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-19 13:23:43 -08:00
Johannes Schindelin
be8a90e59c difftool: add a skeleton for the upcoming builtin
This adds a builtin difftool that still falls back to the legacy Perl
version, which has been renamed to `legacy-difftool`.

The idea is that the new, experimental, builtin difftool immediately hands
off to the legacy difftool for now, unless the config variable
difftool.useBuiltin is set to true.

This feature flag will be used in the upcoming Git for Windows v2.11.0
release, to allow early testers to opt-in to use the builtin difftool and
flesh out any bugs.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-17 13:32:47 -08:00
Steven Penny
aa38ad2b24 Makefile: put LIBS after LDFLAGS for imap-send
This matches up with the targets git-%, git-http-fetch, git-http-push
and git-remote-testsvn. It must be done this way in Cygwin else lcrypto
cannot find lgdi32 and lws2_32.

Signed-off-by: Steven Penny <svnpenn@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-09 06:31:52 -08:00
Steven Penny
7c44b33f8b Makefile: POSIX windres
When environment variable POSIXLY_CORRECT is set, the
"input -o output" syntax is not supported.

  http://cygwin.com/ml/cygwin/2017-01/msg00036.html

Use "-i input -o output" syntax instead.

Signed-off-by: Steven Penny <svnpenn@gmail.com>
Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-09 01:56:22 -08:00
Junio C Hamano
1d73f8e86d Merge branch 'va/i18n-perl-scripts'
Porcelain scripts written in Perl are getting internationalized.

* va/i18n-perl-scripts:
  i18n: difftool: mark warnings for translation
  i18n: send-email: mark composing message for translation
  i18n: send-email: mark string with interpolation for translation
  i18n: send-email: mark warnings and errors for translation
  i18n: send-email: mark strings for translation
  i18n: add--interactive: mark status words for translation
  i18n: add--interactive: remove %patch_modes entries
  i18n: add--interactive: mark edit_hunk_manually message for translation
  i18n: add--interactive: i18n of help_patch_cmd
  i18n: add--interactive: mark patch prompt for translation
  i18n: add--interactive: mark plural strings
  i18n: clean.c: match string with git-add--interactive.perl
  i18n: add--interactive: mark strings with interpolation for translation
  i18n: add--interactive: mark simple here-documents for translation
  i18n: add--interactive: mark strings for translation
  Git.pm: add subroutines for commenting lines
2016-12-27 00:11:40 -08:00
Junio C Hamano
09b4fdb5f3 Merge branch 'jk/make-tags-find-sources-tweak'
Update the procedure to generate "tags" for developer support.

* jk/make-tags-find-sources-tweak:
  Makefile: exclude contrib from FIND_SOURCE_FILES
  Makefile: match shell scripts in FIND_SOURCE_FILES
  Makefile: exclude test cruft from FIND_SOURCE_FILES
  Makefile: reformat FIND_SOURCE_FILES
2016-12-19 14:45:37 -08:00
Vasco Almeida
0539d5e6d5 i18n: add--interactive: mark patch prompt for translation
Mark prompt message assembled in place for translation, unfolding each
use case for each entry in the %patch_modes hash table.

Previously, this script relied on whether $patch_mode was set to run the
command patch_update_cmd() or show status and loop the main loop. Now,
it uses $cmd to indicate we must run patch_update_cmd() and $patch_mode
is used to tell which flavor of the %patch_modes are we on.  This is
introduced in order to be able to mark and unfold the message prompt
knowing in which context we are.

The tracking of context was done previously by point %patch_mode_flavour
hash table to the correct entry of %patch_modes, focusing only on value
of %patch_modes. Now, we are also interested in the key ('staged',
'stash', 'checkout_head', ...).

Signed-off-by: Vasco Almeida <vascomalmeida@sapo.pt>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-14 11:00:05 -08:00
Vasco Almeida
c4a85c3b8e i18n: add--interactive: mark plural strings
Mark plural strings for translation.  Unfold each action case in one
entire sentence.

Pass new keyword for xgettext to extract.

Update test to include new subroutine __n() for plural strings handling.

Update documentation to include a description of the new __n()
subroutine.

Signed-off-by: Vasco Almeida <vascomalmeida@sapo.pt>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-14 11:00:05 -08:00
Jeff King
046e4c1c09 Makefile: exclude contrib from FIND_SOURCE_FILES
When you're working on the git project, you're unlikely to
care about random bits in contrib/ (e.g., you would not want
to jump to the copy of xmalloc in the wincred credential
helper). Nobody has really complained because there are
relatively few C files in contrib.

Now that we're matching shell scripts, too, we get quite a
few more hits, especially in the obsolete contrib/examples
directory. Looking for usage() should turn up the one in
git-sh-setup, not in some long-dead version of git-clone.

Let's just exclude all of contrib. Any specific projects
there which are big enough to want tags can generate them
separately.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-14 09:54:49 -08:00
Jeff King
8fa2043293 Makefile: match shell scripts in FIND_SOURCE_FILES
We feed FIND_SOURCE_FILES to ctags to help developers
navigate to particular functions, but we only feed C source
code. The same feature can be helpful when working with
shell scripts (especially the test suite). Modern versions
of ctags know how to parse shell scripts; we just need to
feed the filenames to it.

This patch specifically avoids including the individual test
scripts themselves. Those are unlikely to be of interest,
and there are a lot of them to process. It does pick up
test-lib.sh and test-lib-functions.sh.

Note that our negative pathspec already excludes the
individual scripts for the ls-files case, but we need to
loosen the `find` rule to match it.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-14 09:54:49 -08:00
Jeff King
e6fc85b11f Makefile: exclude test cruft from FIND_SOURCE_FILES
The test directory may contain three types of files that
match our patterns:

  1. Helper programs in t/helper.

  2. Sample data files (e.g., t/t4051/hello.c).

  3. Untracked cruft in trash directories and t/perf/build.

We want to match (1), but not the other two, as they just
clutter up the list.

For the ls-files method, we can drop (2) with a negative
pathspec. We do not have to care about (3), since ls-files
will not list untracked files.

For `find`, we can match both cases with `-prune` patterns.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-14 09:54:49 -08:00
Jeff King
e951ebca91 Makefile: reformat FIND_SOURCE_FILES
As we add to this in future commits, the formatting is going
to make it harder and harder to read. Let's write it more as
we would in a shell script, putting each logical block on
its own line.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-14 09:54:49 -08:00
Jeff King
1f7c926132 xdiff: drop XDL_FAST_HASH
The xdiff code hashes every line of both sides of a diff,
and then compares those hashes to find duplicates. The
overall performance depends both on how fast we can compute
the hashes, but also on how many hash collisions we see.

The idea of XDL_FAST_HASH is to speed up the hash
computation. But the generated hashes have worse collision
behavior. This means that in some cases it speeds diffs up
(running "git log -p" on git.git improves by ~8% with it),
but in others it can slow things down. One pathological case
saw over a 100x slowdown[1].

There may be a better hash function that covers both
properties, but in the meantime we are better off with the
original hash. It's slightly slower in the common case, but
it has fewer surprising pathological cases.

[1] http://public-inbox.org/git/20141222041944.GA441@peff.net/

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-06 13:27:11 -08:00
Junio C Hamano
729fb9ad34 Merge branch 'ls/macos-update' into maint
Portability update and workaround for builds on recent Mac OS X.

* ls/macos-update:
  travis-ci: disable GIT_TEST_HTTPD for macOS
  Makefile: set NO_OPENSSL on macOS by default
2016-11-29 13:27:56 -08:00
Junio C Hamano
332fd5655a Merge branch 'ls/macos-update'
Portability update and workaround for builds on recent Mac OS X.

* ls/macos-update:
  travis-ci: disable GIT_TEST_HTTPD for macOS
  Makefile: set NO_OPENSSL on macOS by default
2016-11-11 13:56:30 -08:00
Lars Schneider
f01fe92b82 Makefile: set NO_OPENSSL on macOS by default
Apple removed the OpenSSL header files in macOS 10.11 and above. OpenSSL
was deprecated since macOS 10.7.

Set `NO_OPENSSL` and `APPLE_COMMON_CRYPTO` to `YesPlease` as default for
macOS. It is possible to override this and use OpenSSL by defining
`NO_APPLE_COMMON_CRYPTO`.

Original-patch-by: Torsten Bögershausen <tboegi@web.de>
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-11-10 11:10:36 -08:00
Junio C Hamano
c8fd220175 Merge branch 'rs/cocci' into maint
Code cleanup.

* rs/cocci:
  use strbuf_add_unique_abbrev() for adding short hashes, part 3
  remove unnecessary NULL check before free(3)
  coccicheck: make transformation for strbuf_addf(sb, "...") more precise
  use strbuf_add_unique_abbrev() for adding short hashes, part 2
  use strbuf_addstr() instead of strbuf_addf() with "%s", part 2
  gitignore: ignore output files of coccicheck make target
  use strbuf_addstr() for adding constant strings to a strbuf, part 2
  add coccicheck make target
  contrib/coccinelle: fix semantic patch for oid_to_hex_r()
2016-10-28 09:01:23 -07:00
Junio C Hamano
25ab004c53 Merge branch 'jk/quarantine-received-objects'
In order for the receiving end of "git push" to inspect the
received history and decide to reject the push, the objects sent
from the sending end need to be made available to the hook and
the mechanism for the connectivity check, and this was done
traditionally by storing the objects in the receiving repository
and letting "git gc" to expire it.  Instead, store the newly
received objects in a temporary area, and make them available by
reusing the alternate object store mechanism to them only while we
decide if we accept the check, and once we decide, either migrate
them to the repository or purge them immediately.

* jk/quarantine-received-objects:
  tmp-objdir: do not migrate files starting with '.'
  tmp-objdir: put quarantine information in the environment
  receive-pack: quarantine objects until pre-receive accepts
  tmp-objdir: introduce API for temporary object directories
  check_connected: accept an env argument
2016-10-17 13:25:20 -07:00
Jeff King
2564d994c9 tmp-objdir: introduce API for temporary object directories
Once objects are added to the object database by a process,
they cannot easily be deleted, as we don't know what other
processes may have started referencing them. We have to
clean them up with git-gc, which will apply the usual
reachability and grace-period checks.

This patch provides an alternative: it helps callers create
a temporary directory inside the object directory, and a
temporary environment which can be passed to sub-programs to
ask them to write there (the original object directory
remains accessible as an alternate of the temporary one).

See tmp-objdir.h for details on the API.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-10 13:54:02 -07:00
René Scharfe
a9a884aea5 coccicheck: use --all-includes by default
Add a make variable, SPATCH_FLAGS, for specifying flags for spatch, and
set it to --all-includes by default.  This option lets it consider
header files which would otherwise be ignored.  That's important for
some rules that rely on type information.  It doubles the duration of
coccicheck, however.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-29 20:40:18 -07:00
Junio C Hamano
300e95f7df Merge branch 'js/regexec-buf' into maint
Some codepaths in "git diff" used regexec(3) on a buffer that was
mmap(2)ed, which may not have a terminating NUL, leading to a read
beyond the end of the mapped region.  This was fixed by introducing
a regexec_buf() helper that takes a <ptr,len> pair with REG_STARTEND
extension.

* js/regexec-buf:
  regex: use regexec_buf()
  regex: add regexec_buf() that can work on a non NUL-terminated string
  regex: -G<pattern> feeds a non NUL-terminated string to regexec() and fails
2016-09-29 16:49:45 -07:00
Junio C Hamano
6a67695268 Merge branch 'js/regexec-buf'
Some codepaths in "git diff" used regexec(3) on a buffer that was
mmap(2)ed, which may not have a terminating NUL, leading to a read
beyond the end of the mapped region.  This was fixed by introducing
a regexec_buf() helper that takes a <ptr,len> pair with REG_STARTEND
extension.

* js/regexec-buf:
  regex: use regexec_buf()
  regex: add regexec_buf() that can work on a non NUL-terminated string
  regex: -G<pattern> feeds a non NUL-terminated string to regexec() and fails
2016-09-26 16:09:19 -07:00
Junio C Hamano
85f34a929d Merge branch 'rs/cocci'
Code cleanup.

* rs/cocci:
  use strbuf_addstr() for adding constant strings to a strbuf, part 2
  add coccicheck make target
  contrib/coccinelle: fix semantic patch for oid_to_hex_r()
2016-09-26 16:09:14 -07:00
Johannes Schindelin
2f8952250a regex: add regexec_buf() that can work on a non NUL-terminated string
We just introduced a test that demonstrates that our sloppy use of
regexec() on a mmap()ed area can result in incorrect results or even
hard crashes.

So what we need to fix this is a function that calls regexec() on a
length-delimited, rather than a NUL-terminated, string.

Happily, there is an extension to regexec() introduced by the NetBSD
project and present in all major regex implementation including
Linux', MacOSX' and the one Git includes in compat/regex/: by using
the (non-POSIX) REG_STARTEND flag, it is possible to tell the
regexec() function that it should only look at the offsets between
pmatch[0].rm_so and pmatch[0].rm_eo.

That is exactly what we need.

Since support for REG_STARTEND is so widespread by now, let's just
introduce a helper function that always uses it, and tell people
on a platform whose regex library does not support it to use the
one from our compat/regex/ directory.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-21 13:56:15 -07:00
Junio C Hamano
815a73f714 Merge branch 'rs/compat-strdup' into maint
Code cleanup.

* rs/compat-strdup:
  compat: move strdup(3) replacement to its own file
2016-09-19 13:51:42 -07:00
Junio C Hamano
81358dc238 Merge branch 'cc/apply-am'
"git am" has been taught to make an internal call to "git apply"'s
innards without spawning the latter as a separate process.

* cc/apply-am: (41 commits)
  builtin/am: use apply API in run_apply()
  apply: learn to use a different index file
  apply: pass apply state to build_fake_ancestor()
  apply: refactor `git apply` option parsing
  apply: change error_routine when silent
  usage: add get_error_routine() and get_warn_routine()
  usage: add set_warn_routine()
  apply: don't print on stdout in verbosity_silent mode
  apply: make it possible to silently apply
  apply: use error_errno() where possible
  apply: make some parsing functions static again
  apply: move libified code from builtin/apply.c to apply.{c,h}
  apply: rename and move opt constants to apply.h
  builtin/apply: rename option parsing functions
  builtin/apply: make create_one_file() return -1 on error
  builtin/apply: make try_create_file() return -1 on error
  builtin/apply: make write_out_results() return -1 on error
  builtin/apply: make write_out_one_result() return -1 on error
  builtin/apply: make create_file() return -1 on error
  builtin/apply: make add_index_file() return -1 on error
  ...
2016-09-19 13:47:18 -07:00
René Scharfe
63f0a758a0 add coccicheck make target
Provide a simple way to run Coccinelle against all source files, in the
form of a Makefile target.  Running "make coccicheck" applies each
.cocci file in contrib/coccinelle/ on all source files.  It generates
a .patch file for each .cocci file, containing the actual changes for
effecting the transformations described by the semantic patches.

Non-empty .patch files are reported.  They can be applied to the work
tree using "patch -p0", but should be checked to e.g. make sure they
don't screw up formatting or create circular references.

Coccinelle's diagnostic output (stderr) is piped into .log files.

Linux has a much more elaborate make target of the same name; let's
start nice and easy.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-15 12:23:37 -07:00
Junio C Hamano
27853a85ed Merge branch 'rs/compat-strdup'
* rs/compat-strdup:
  compat: move strdup(3) replacement to its own file
2016-09-12 15:34:36 -07:00
Junio C Hamano
faacc8efe5 Merge branch 'jk/common-main' into maint
There are certain house-keeping tasks that need to be performed at
the very beginning of any Git program, and programs that are not
built-in commands had to do them exactly the same way as "git"
potty does.  It was easy to make mistakes in one-off standalone
programs (like test helpers).  A common "main()" function that
calls cmd_main() of individual program has been introduced to
make it harder to make mistakes.

* jk/common-main:
  mingw: declare main()'s argv as const
  common-main: call git_setup_gettext()
  common-main: call restore_sigpipe_to_default()
  common-main: call sanitize_stdfds()
  common-main: call git_extract_argv0_path()
  add an extra level of indirection to main()
2016-09-08 21:35:51 -07:00
René Scharfe
ca2baa3f75 compat: move strdup(3) replacement to its own file
Move our implementation of strdup(3) out of compat/nedmalloc/ and
allow it to be used independently from USE_NED_ALLOCATOR.  The
original nedmalloc doesn't come with strdup() and doesn't need it.
Only _users_ of nedmalloc need it, which was added when we imported
it to our compat/ hierarchy.

This reduces the difference of our copy of nedmalloc from the
original, making it easier to update, and allows for easier testing
and reusing of our version of strdup().

Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-07 10:41:45 -07:00
Christian Couder
bb493a5c14 builtin/apply: move init_apply_state() to apply.c
To libify `git apply` functionality we must make init_apply_state()
usable outside "builtin/apply.c".

Let's do that by moving it into a new "apply.c".

Helped-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-11 12:41:47 -07:00
Junio C Hamano
43a42aa403 Merge branch 'ew/build-time-pager-tweaks'
The build procedure learned PAGER_ENV knob that lists what default
environment variable settings to export for popular pagers.  This
mechanism is used to tweak the default settings to MORE on FreeBSD.

* ew/build-time-pager-tweaks:
  pager: move pager-specific setup into the build
2016-08-08 14:48:44 -07:00
Junio C Hamano
78849622ec Merge branch 'jk/pack-objects-optim'
"git pack-objects" has a few options that tell it not to pack
objects found in certain packfiles, which require it to scan .idx
files of all available packs.  The codepaths involved in these
operations have been optimized for a common case of not having any
non-local pack and/or any .kept pack.

* jk/pack-objects-optim:
  pack-objects: compute local/ignore_pack_keep early
  pack-objects: break out of want_object loop early
  find_pack_entry: replace last_found_pack with MRU cache
  add generic most-recently-used list
  sha1_file: drop free_pack_by_name
  t/perf: add tests for many-pack scenarios
2016-08-08 14:48:39 -07:00
Junio C Hamano
970994deb1 Merge branch 'nd/test-helpers' into maint
Build clean-up.

* nd/test-helpers:
  t/test-lib.sh: fix running tests with --valgrind
  Makefile: use VCSSVN_LIB to refer to svn library
  Makefile: drop extra dependencies for test helpers
2016-08-08 14:21:43 -07:00
Eric Wong
995bc22d7f pager: move pager-specific setup into the build
Allowing PAGER_ENV to be set at build-time allows us to move
pager-specific knowledge out of our build.  This allows us to
set a better default for FreeBSD more(1), which pretends not to
understand ANSI color escapes if the MORE environment variable
is left empty, but accepts the same variables as less(1)

Originally-from:
 https://public-inbox.org/git/xmqq61piw4yf.fsf@gitster.dls.corp.google.com/

Helped-by: Junio C Hamano <gitster@pobox.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Eric Wong <e@80x24.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-04 13:51:02 -07:00
Jeff King
002f206faf add generic most-recently-used list
There are a few places in Git that would benefit from a fast
most-recently-used cache (e.g., the list of packs, which we
search linearly but would like to order based on locality).
This patch introduces a generic list that can be used to
store arbitrary pointers in most-recently-used order.

The implementation is just a doubly-linked list, where
"marking" an item as used moves it to the front of the list.
Insertion and marking are O(1), and iteration is O(n).

There's no lookup support provided; if you need fast
lookups, you are better off with a different data structure
in the first place.

There is also no deletion support. This would not be hard to
do, but it's not necessary for handling pack structs, which
are created and never removed.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-29 11:05:07 -07:00
Junio C Hamano
ae9ca20c85 Merge branch 'nd/test-helpers'
Build clean-up.

* nd/test-helpers:
  t/test-lib.sh: fix running tests with --valgrind
  Makefile: use VCSSVN_LIB to refer to svn library
  Makefile: drop extra dependencies for test helpers
2016-07-25 14:13:42 -07:00
Junio C Hamano
937be62993 Merge branch 'rw/make-needs-librt'
Makefile assumed that -lrt is always available on platforms that
want to use clock_gettime() and CLOCK_MONOTONIC, which is not a
case for recent Mac OS X.  The necessary symbols are often found in
libc on many modern systems and having -lrt on the command line, as
long as the library exists, had no effect, but when the platform
removes librt.a that is a different matter--having -lrt will break
the linkage.

This change could be seen as a regression for those who do need to
specify -lrt, as they now specifically ask for NEEDS_LIBRT when
building. Hopefully they are in the minority these days.

* rw/make-needs-librt:
  config.mak.uname: define NEEDS_LIBRT under Linux, for now
  Makefile: add NEEDS_LIBRT to optionally link with librt
2016-07-25 14:13:36 -07:00
Junio C Hamano
87492cb24d Merge branch 'mh/ref-iterators'
The API to iterate over all the refs (i.e. for_each_ref(), etc.)
has been revamped.

* mh/ref-iterators:
  for_each_reflog(): reimplement using iterators
  dir_iterator: new API for iterating over a directory tree
  for_each_reflog(): don't abort for bad references
  do_for_each_ref(): reimplement using reference iteration
  refs: introduce an iterator interface
  ref_resolves_to_object(): new function
  entry_resolves_to_object(): rename function from ref_resolves_to_object()
  get_ref_cache(): only create an instance if there is a submodule
  remote rm: handle symbolic refs correctly
  delete_refs(): add a flags argument
  refs: use name "prefix" consistently
  do_for_each_ref(): move docstring to the header file
  refs: remove unnecessary "extern" keywords
2016-07-25 14:13:33 -07:00
Junio C Hamano
d4c6375fd8 Merge branch 'jk/common-main'
There are certain house-keeping tasks that need to be performed at
the very beginning of any Git program, and programs that are not
built-in commands had to do them exactly the same way as "git"
potty does.  It was easy to make mistakes in one-off standalone
programs (like test helpers).  A common "main()" function that
calls cmd_main() of individual program has been introduced to
make it harder to make mistakes.

* jk/common-main:
  mingw: declare main()'s argv as const
  common-main: call git_setup_gettext()
  common-main: call restore_sigpipe_to_default()
  common-main: call sanitize_stdfds()
  common-main: call git_extract_argv0_path()
  add an extra level of indirection to main()
2016-07-19 13:22:19 -07:00
Junio C Hamano
2703572b3a Merge branch 'va/i18n-even-more'
More markings of messages for i18n, with updates to various tests
to pass GETTEXT_POISON tests.

One patch from the original submission dropped due to conflicts
with jk/upload-pack-hook, which is still in flux.

* va/i18n-even-more: (38 commits)
  t5541: become resilient to GETTEXT_POISON
  i18n: branch: mark comment when editing branch description for translation
  i18n: unmark die messages for translation
  i18n: submodule: escape shell variables inside eval_gettext
  i18n: submodule: join strings marked for translation
  i18n: init-db: join message pieces
  i18n: remote: allow translations to reorder message
  i18n: remote: mark URL fallback text for translation
  i18n: standardise messages
  i18n: sequencer: add period to error message
  i18n: merge: change command option help to lowercase
  i18n: merge: mark messages for translation
  i18n: notes: mark options for translation
  i18n: notes: mark strings for translation
  i18n: transport-helper.c: change N_() call to _()
  i18n: bisect: mark strings for translation
  t5523: use test_i18ngrep for negation
  t4153: fix negated test_i18ngrep call
  t9003: become resilient to GETTEXT_POISON
  tests: unpack-trees: update to use test_i18n* functions
  ...
2016-07-13 11:24:10 -07:00
Ronald Wampler
d19e3a5b21 Makefile: add NEEDS_LIBRT to optionally link with librt
We unconditionally link with librt, when HAVE_CLOCK_GETTIME is defined.
But clock_gettime() has been available in most libc implementations for
some time now (e.g., for glibc since version 2.17) and no longer
requires linking with librt. Furthermore, commit a6c3c63 (configure.ac:
check for clock_gettime() and CLOCK_MONOTONIC) will automatically
determined which library (libc or librt) is required for linking when
checking for clock_gettime().

The assumption to unconditionally link with librt was OK, since either
almost every Unix-like system provides a version of librt for backwards
compatibility or other systems, namely Windows or OS X, never provided
clock_gettime(). However, in the latest release of OS X (macOS Sierra),
this function has been added to OS X libc version. As a result, when
running the configuration script, HAVE_CLOCK_GETTIME is set and since
librt is not present, it causes a linker error.

This patches requires those not building via the configuration scripts
to define NEEDS_LIBRT in addition to HAVE_CLOCK_GETTIME, if needed.

Signed-off-by: Ronald Wampler <rdwampler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-07 14:15:08 -07:00
Jeff King
4df7c8a037 Makefile: use VCSSVN_LIB to refer to svn library
We have an abstracted variable; let's use it consistently.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-06 11:51:03 -07:00
Jeff King
1be36b60f1 Makefile: drop extra dependencies for test helpers
A few test-helpers have Makefile dependencies on specific
object files. But since these files are part of libgit.a
(which all of the helpers link against), the inclusion is
simply redundant.

These were once necessary, but became redundant due to
5c5ba73 (Makefile: Use generic rule to build test programs,
2007-05-31), which added the $(GITLIBS) dependency (but
didn't prune the extra dependency lines). Later commits then
cargo-culted the practice (e.g., b4285c7).

Note that we _do_ need to leave the dependencies on the svn
library, as that is not part of the usual link command.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-06 11:50:48 -07:00
Junio C Hamano
de61cebde7 Merge branch 'jk/common-main-2.8' into jk/common-main
* jk/common-main-2.8:
  mingw: declare main()'s argv as const
  common-main: call git_setup_gettext()
  common-main: call restore_sigpipe_to_default()
  common-main: call sanitize_stdfds()
  common-main: call git_extract_argv0_path()
  add an extra level of indirection to main()
2016-07-06 10:02:57 -07:00
Jeff King
3f2e2297b9 add an extra level of indirection to main()
There are certain startup tasks that we expect every git
process to do. In some cases this is just to improve the
quality of the program (e.g., setting up gettext()). In
others it is a requirement for using certain functions in
libgit.a (e.g., system_path() expects that you have called
git_extract_argv0_path()).

Most commands are builtins and are covered by the git.c
version of main(). However, there are still a few external
commands that use their own main(). Each of these has to
remember to include the correct startup sequence, and we are
not always consistent.

Rather than just fix the inconsistencies, let's make this
harder to get wrong by providing a common main() that can
run this standard startup.

We basically have two options to do this:

 - the compat/mingw.h file already does something like this by
   adding a #define that replaces the definition of main with a
   wrapper that calls mingw_startup().

   The upside is that the code in each program doesn't need
   to be changed at all; it's rewritten on the fly by the
   preprocessor.

   The downside is that it may make debugging of the startup
   sequence a bit more confusing, as the preprocessor is
   quietly inserting new code.

 - the builtin functions are all of the form cmd_foo(),
   and git.c's main() calls them.

   This is much more explicit, which may make things more
   obvious to somebody reading the code. It's also more
   flexible (because of course we have to figure out _which_
   cmd_foo() to call).

   The downside is that each of the builtins must define
   cmd_foo(), instead of just main().

This patch chooses the latter option, preferring the more
explicit approach, even though it is more invasive. We
introduce a new file common-main.c, with the "real" main. It
expects to call cmd_main() from whatever other objects it is
linked against.

We link common-main.o against anything that links against
libgit.a, since we know that such programs will need to do
this setup. Note that common-main.o can't actually go inside
libgit.a, as the linker would not pick up its main()
function automatically (it has no callers).

The rest of the patch is just adjusting all of the various
external programs (mostly in t/helper) to use cmd_main().
I've provided a global declaration for cmd_main(), which
means that all of the programs also need to match its
signature. In particular, many functions need to switch to
"const char **" instead of "char **" for argv. This effect
ripples out to a few other variables and functions, as well.

This makes the patch even more invasive, but the end result
is much better. We should be treating argv strings as const
anyway, and now all programs conform to the same signature
(which also matches the way builtins are defined).

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-01 15:09:10 -07:00
Michael Haggerty
0fe5043dad dir_iterator: new API for iterating over a directory tree
The iterator interface is modeled on that for references, though no
vtable is necessary because there is (so far?) only one type of
dir_iterator.

There are obviously a lot of features that could easily be added to this
class:

* Skip/include directory paths in the iteration
* Shallow/deep iteration
* Letting the caller decide which subdirectories to recurse into (e.g.,
  via a dir_iterator_advance_into() function)
* Option to iterate in sorted order
* Option to iterate over directory paths before vs. after their contents

But these are not needed for the current patch series, so I refrain.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-20 11:38:21 -07:00
Michael Haggerty
3bc581b940 refs: introduce an iterator interface
Currently, the API for iterating over references is via a family of
for_each_ref()-type functions that invoke a callback function for each
selected reference. All of these eventually call do_for_each_ref(),
which knows how to do one thing: iterate in parallel through two
ref_caches, one for loose and one for packed refs, giving loose
references precedence over packed refs. This is rather complicated code,
and is quite specialized to the files backend. It also requires callers
to encapsulate their work into a callback function, which often means
that they have to define and use a "cb_data" struct to manage their
context.

The current design is already bursting at the seams, and will become
even more awkward in the upcoming world of multiple reference storage
backends:

* Per-worktree vs. shared references are currently handled via a kludge
  in git_path() rather than iterating over each part of the reference
  namespace separately and merging the results. This kludge will cease
  to work when we have multiple reference storage backends.

* The current scheme is inflexible. What if we sometimes want to bypass
  the ref_cache, or use it only for packed or only for loose refs? What
  if we want to store symbolic refs in one type of storage backend and
  non-symbolic ones in another?

In the future, each reference backend will need to define its own way of
iterating over references. The crux of the problem with the current
design is that it is impossible to compose for_each_ref()-style
iterations, because the flow of control is owned by the for_each_ref()
function. There is nothing that a caller can do but iterate through all
references in a single burst, so there is no way for it to interleave
references from multiple backends and present the result to the rest of
the world as a single compound backend.

This commit introduces a new iteration primitive for references: a
ref_iterator. A ref_iterator is a polymorphic object that a reference
storage backend can be asked to instantiate. There are three functions
that can be applied to a ref_iterator:

* ref_iterator_advance(): move to the next reference in the iteration
* ref_iterator_abort(): end the iteration before it is exhausted
* ref_iterator_peel(): peel the reference currently being looked at

Iterating using a ref_iterator leaves the flow of control in the hands
of the caller, which means that ref_iterators from multiple
sources (e.g., loose and packed refs) can be composed and presented to
the world as a single compound ref_iterator.

It also means that the backend code for implementing reference iteration
will sometimes be more complicated. For example, the
cache_ref_iterator (which iterates over a ref_cache) can't use the C
stack to recurse; instead, it must manage its own stack internally as
explicit data structures. There is also a lot of boilerplate connected
with object-oriented programming in C.

Eventually, end-user callers will be able to be written in a more
natural way—managing their own flow of control rather than having to
work via callbacks. Since there will only be a few reference backends
but there are many consumers of this API, this is a good tradeoff.

More importantly, we gain composability, and especially the possibility
of writing interchangeable parts that can work with any ref_iterator.

For example, merge_ref_iterator implements a generic way of merging the
contents of any two ref_iterators. It is used to merge loose + packed
refs as part of the implementation of the files_ref_iterator. But it
will also be possible to use it to merge other pairs of reference
sources (e.g., per-worktree vs. shared refs).

Another example is prefix_ref_iterator, which can be used to trim a
prefix off the front of reference names before presenting them to the
caller (e.g., "refs/heads/master" -> "master").

In this patch, we introduce the iterator abstraction and many utilities,
and implement a reference iterator for the files ref storage backend.
(I've written several other obvious utilities, for example a generic way
to filter references being iterated over. These will probably be useful
in the future. But they are not needed for this patch series, so I am
not including them at this time.)

In a moment we will rewrite do_for_each_ref() to work via reference
iterators (allowing some special-purpose code to be discarded), and do
something similar for reflogs. In future patch series, we will expose
the ref_iterator abstraction in the public refs API so that callers can
use it directly.

Implementation note: I tried abstracting this a layer further to allow
generic iterators (over arbitrary types of objects) and generic
utilities like a generic merge_iterator. But the implementation in C was
very cumbersome, involving (in my opinion) too much boilerplate and too
much unsafe casting, some of which would have had to be done on the
caller side. However, I did put a few iterator-related constants in a
top-level header file, iterator.h, as they will be useful in a moment to
implement iteration over directory trees and possibly other types of
iterators in the future.

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-20 11:38:20 -07:00
Vasco Almeida
9588c52b75 i18n: rebase-interactive: mark strings for translation
Mark strings in git-rebase--interactive.sh for translation. There is no
need to source git-sh-i18n since git-rebase.sh already does so.

Add git-rebase--interactive.sh to LOCALIZED_SH in Makefile in order to
enable extracting strings marked for translation by xgettext.

Signed-off-by: Vasco Almeida <vascomalmeida@sapo.pt>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-17 15:45:48 -07:00
Vasco Almeida
d323c6b641 i18n: git-sh-setup.sh: mark strings for translation
Positional arguments, such as $0, $1, etc, need to be stored on shell
variables for use in translatable strings, according to gettext manual
[1].

Add git-sh-setup.sh to LOCALIZED_SH variable in Makefile to enable
extraction of string marked for translation by xgettext.

Source git-sh-i18n in git-sh-setup.sh for gettext support.
git-sh-setup.sh is a shell library to be sourced by other shell scripts.
In order to avoid other scripts from sourcing git-sh-i18n twice, remove
line that sources it from them.  Not sourcing git-sh-i18n in any script
that uses gettext would lead to failure due to, for instance, gettextln
not being found.

[1] http://www.gnu.org/software/gettext/manual/html_node/Preparing-Shell-Scripts.html

Signed-off-by: Vasco Almeida <vascomalmeida@sapo.pt>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-17 15:45:48 -07:00
Junio C Hamano
ec5ad66ee2 Merge branch 'mm/makefile-developer-can-be-in-config-mak'
"make DEVELOPER=1" worked as expected; setting DEVELOPER=1 in
config.mak didn't.

* mm/makefile-developer-can-be-in-config-mak:
  Makefile: add $(DEVELOPER_CFLAGS) variable
  Makefile: move 'ifdef DEVELOPER' after config.mak* inclusion
2016-06-03 14:38:02 -07:00
Matthieu Moy
51dd3e81d4 Makefile: add $(DEVELOPER_CFLAGS) variable
This does not change the behavior, but allows the user to tweak
DEVELOPER_CFLAGS on the command-line or in a config.mak* file if
needed.

This also makes the code somewhat cleaner as it follows the pattern

<initialisation of variables>
<include statements>
<actual build logic>

by specifying which flags to activate in the first part, and actually
activating them in the last one.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-01 08:17:15 -07:00
Matthieu Moy
d615628c35 Makefile: move 'ifdef DEVELOPER' after config.mak* inclusion
The DEVELOPER knob was introduced in 658df95 (add DEVELOPER makefile
knob to check for acknowledged warnings, 2016-02-25), and works well
when used as "make DEVELOPER=1", and when the configure script was not
used.

However, the advice given in CodingGuidelines to add DEVELOPER=1 to
config.mak does not: config.mak is included after testing for
DEVELOPER in the Makefile, and at least GNU Make's manual specifies
"Conditional directives are parsed immediately", hence the config.mak
declaration is not visible at the time the conditional is evaluated.

Also, when using the configure script to generate a
config.mak.autogen, the later file contained a "CFLAGS = <flags>"
initialization, which overrode the "CFLAGS += -W..." triggered by
DEVELOPER.

This patch fixes both issues.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-05-31 10:01:51 -07:00
Junio C Hamano
b262b8f889 Merge branch 'va/i18n-misc-updates' into maint
Mark several messages for translation.

* va/i18n-misc-updates:
  i18n: unpack-trees: avoid substituting only a verb in sentences
  i18n: builtin/pull.c: split strings marked for translation
  i18n: builtin/pull.c: mark placeholders for translation
  i18n: git-parse-remote.sh: mark strings for translation
  i18n: branch: move comment for translators
  i18n: branch: unmark string for translation
  i18n: builtin/rm.c: remove a comma ',' from string
  i18n: unpack-trees: mark strings for translation
  i18n: builtin/branch.c: mark option for translation
  i18n: index-pack: use plural string instead of normal one
2016-05-26 13:17:20 -07:00
Junio C Hamano
fa4f29b8a8 Merge branch 'jc/doc-lint'
Find common mistakes when writing gitlink: in our documentation and
drive the check from "make check-docs".

I am not entirely happy with the way the script chooses what input
file to validate, but it is not worse than not having anything, so
let's move it forward and have the logic improved later when people
care about it deeply.

* jc/doc-lint:
  ci: validate "linkgit:" in documentation
2016-05-23 14:54:34 -07:00
Junio C Hamano
e5e7a9115d Merge branch 'va/i18n-misc-updates'
Mark several messages for translation.

* va/i18n-misc-updates:
  i18n: unpack-trees: avoid substituting only a verb in sentences
  i18n: builtin/pull.c: split strings marked for translation
  i18n: builtin/pull.c: mark placeholders for translation
  i18n: git-parse-remote.sh: mark strings for translation
  i18n: branch: move comment for translators
  i18n: branch: unmark string for translation
  i18n: builtin/rm.c: remove a comma ',' from string
  i18n: unpack-trees: mark strings for translation
  i18n: builtin/branch.c: mark option for translation
  i18n: index-pack: use plural string instead of normal one
2016-05-17 14:38:23 -07:00
Junio C Hamano
ab81411ced ci: validate "linkgit:" in documentation
It is easy to add incorrect "linkgit:<page>[<section>]" references
to our documentation suite.  Catch these common classes of errors:

 * Referring to Documentation/<page>.txt that does not exist.

 * Referring to a <page> outside the Git suite.  In general, <page>
   must begin with "git".

 * Listing the manual <section> incorrectly.  The first line of the
   Documentation/<page>.txt must end with "(<section>)".

with a new script "ci/lint-gitlink", and drive it from "make check-docs".

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-05-10 11:15:04 -07:00
Junio C Hamano
a0c9cf51c0 Merge branch 'ky/imap-send-openssl-1.1.0' into maint
Upcoming OpenSSL 1.1.0 will break compilation b updating a few APIs
we use in imap-send, which has been adjusted for the change.

* ky/imap-send-openssl-1.1.0:
  configure: remove checking for HMAC_CTX_cleanup
  imap-send: avoid deprecated TLSv1_method()
  imap-send: check NULL return of SSL_CTX_new()
  imap-send: use HMAC() function provided by OpenSSL
2016-05-06 14:53:24 -07:00
Junio C Hamano
54b0ac57ab Merge branch 'jc/drop-git-spec-in'
As nobody maintains our in-tree git.spec.in and distros use their
own spec file, we stopped pretending that we support "make rpm".

* jc/drop-git-spec-in:
  Makefile: remove dependency on git.spec
  Makefile: stop pretending to support rpmbuild
2016-05-06 14:45:44 -07:00
Junio C Hamano
98eef48257 Merge branch 'jc/makefile-redirection-stderr' into maint
A minor fix in the Makefile.

* jc/makefile-redirection-stderr:
  Makefile: fix misdirected redirections
2016-04-29 14:15:59 -07:00
Junio C Hamano
e0b5851907 Merge branch 'nd/test-helpers'
Sources to many test helper binaries (and the generated helpers)
have been moved to t/helper/ subdirectory to reduce clutter at the
top level of the tree.

* nd/test-helpers:
  test helpers: move test-* to t/helper/ subdirectory
  Makefile: clean *.o files we create
2016-04-29 12:59:06 -07:00
Dennis Kaarsemaker
ef642ff07c Makefile: remove dependency on git.spec
ab214331 (Makefile: stop pretending to support rpmbuild, 2016-04-04)
dropped support for rpmbuild using our own specfile by removing
git.spec.in, but forgot to remove the dependency of the dist target
on git.spec.

Signed-off-by: Dennis Kaarsemaker <dennis@kaarsemaker.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-04-27 13:25:44 -07:00
Junio C Hamano
33e4ec89d9 Merge branch 'ky/imap-send-openssl-1.1.0'
Upcoming OpenSSL 1.1.0 will break compilation b updating a few APIs
we use in imap-send, which has been adjusted for the change.

* ky/imap-send-openssl-1.1.0:
  configure: remove checking for HMAC_CTX_cleanup
  imap-send: avoid deprecated TLSv1_method()
  imap-send: check NULL return of SSL_CTX_new()
  imap-send: use HMAC() function provided by OpenSSL
2016-04-22 15:45:08 -07:00
Vasco Almeida
045fac5845 i18n: git-parse-remote.sh: mark strings for translation
Change Makefile to include git-parse-remote.sh in LOCALIZED_SH.

TODO: remove 3rd argument of error_on_missing_default_upstream function
that is no longer required.

Signed-off-by: Vasco Almeida <vascomalmeida@sapo.pt>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-04-19 12:07:49 -07:00
Nguyễn Thái Ngọc Duy
e6e7530d10 test helpers: move test-* to t/helper/ subdirectory
This keeps top dir a bit less crowded. And because these programs are
for testing purposes, it makes sense that they stay somewhere in t/

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-04-15 10:12:19 -07:00
Junio C Hamano
7897d84b82 Makefile: clean *.o files we create
The part that removes object files in the 'clean' target predates
various Makefile macros that list object files we create, and
instead removes the objects with shell glob, perpetually requiring
updates whenever a new location that builds object files is added.

Simplify the target by removing $(OBJECTS), which is supposed to
have all the objects we create during the build.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-04-15 10:07:35 -07:00
Junio C Hamano
8c9dec985e Merge branch 'jc/makefile-redirection-stderr'
A minor fix in the Makefile.

* jc/makefile-redirection-stderr:
  Makefile: fix misdirected redirections
2016-04-13 14:12:38 -07:00
Kazuki Yamaguchi
1245c74936 configure: remove checking for HMAC_CTX_cleanup
We don't need it, as we no longer use HMAC_CTX_cleanup() directly.

Signed-off-by: Kazuki Yamaguchi <k@rhe.jp>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-04-08 11:46:36 -07:00