Implement a gate between git and mediawiki, allowing git users to push
and pull objects from mediawiki just as one would do with a classic git
repository thanks to remote-helpers.
The following packages need to be installed (available on common
repositories):
libmediawiki-api-perl
libdatetime-format-iso8601-perl
Use remote helpers in order to be as transparent as possible to the git
user.
Download Mediawiki revisions through the Mediawiki API and then
fast-import into git.
Mediawiki revision number and git commits are linked thanks to notes
bound to commits.
The import part is done on a refs/mediawiki/<remote> branch before
coming to refs/remote/origin/master (Huge thanks to Jonathan Nieder
for his help)
We use UTF-8 everywhere: use encoding 'utf8'; does most of the job, but
we also read the output of Git commands in UTF-8 with the small helper
run_git, and write to the console (STDERR) in UTF-8. This allows a
seamless use of non-ascii characters in page titles, but hasn't been
tested on non-UTF-8 systems. In particular, UTF-8 encoding for filenames
could raise problems if different file systems handle UTF-8 filenames
differently. A uri_escape of mediawiki filenames could be imaginable, and
is still to be discussed further.
Partial cloning is supported using one of:
git clone -c remote.origin.pages='A_Page Another_Page' mediawiki::http://wikiurl
git clone -c remote.origin.categories='Some_Category' mediawiki::http://wikiurl
git clone -c remote.origin.shallow='True' mediawiki::http://wikiurl
Thanks to notes metadata, it is possible to compare remote and local last
mediawiki revision to warn non-fast forward pushes and "everything
up-to-date" case.
When allowed, push looks for each commit between remotes/origin/master
and HEAD, catches every blob related to these commit and push them in
chronological order. To do so, it uses git rev-list --children HEAD and
travels the tree from remotes/origin/master to HEAD through children. In
other words:
* Shortest path from remotes/origin/master to HEAD
* For each commit encountered, push blobs related to this commit
Signed-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
Signed-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
Signed-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>
Signed-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>
Signed-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
Signed-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
While it's not a command meant to be used by actual users (hence, not
mentionned in git(1)), this command is a very precious help for
remote-helpers authors.
The best place for such technical doc is the source code, but users may
not find it without a link in a manpage.
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Acked-by: Sverre Rabbelier <srabbelier@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This is important for two reasons:
* when two "import" lines follow each other, only one "done" command
should be issued in the fast-import stream, not one per "import".
* The blank line terminating an import command should not be confused
with the one terminating the sequence of commands.
While we're there, illustrate the corresponding explanation for push
batches with an example.
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Acked-by: Sverre Rabbelier <srabbelier@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The "git fetch" command works in two phases. The remote side tells us what
objects are at the tip of the refs we are fetching from, and transfers the
objects missing from our side. After storing the objects in our repository,
we update our remote tracking branches to point at the updated tips of the
refs.
A broken or malicious remote side could send a perfectly well-formed pack
data during the object transfer phase, but there is no guarantee that the
given data actually fill the gap between the objects we originally had and
the refs we are updating to.
Although this kind of breakage can be caught by running fsck after a
fetch, it is much cheaper to verify that everything that is reachable from
the tips of the refs we fetched are indeed fully connected to the tips of
our current set of refs before we update them.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Often we want to verify everything reachable from a given set of commits
are present in our repository and connected without a gap to the tips of
our refs. We used to do this for this purpose:
$ rev-list --objects $commits_to_be_tested --not --all
Even though this is good enough for catching missing commits and trees,
we show the object name but do not verify their existence, let alone their
well-formedness, for the blob objects at the leaf level.
Add a new "--verify-object" option so that we can catch missing and broken
blobs as well.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The traverse_commit_list() API takes two callback functions, one to show
commit objects, and the other to show other kinds of objects. Even though
the former has a callback data parameter, so that the callback does not
have to rely on global state, the latter does not.
Give the show_objects() callback the same callback data parameter.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"svn dcommit --mergeinfo" replaces the svn:mergeinfo property in an
upstream SVN repository with the given text. The svn:mergeinfo
property may contain commits originating on multiple branches,
separated by newlines.
Cause space characters in the mergeinfo to be replaced by newlines,
allowing a user to create history representing multiple branches being
merged into one.
Update the corresponding documentation and add a test for the new
functionality.
Signed-off-by: Bryan Jacobs <bjacobs@woti.com>
Acked-by: Sam Vilain <sam@vilain.net>
Acked-by: Eric Wong <normalperson@yhbt.net>
The matching step in commit 3235b7053c
did not properly account for users of the "rewriteRoot"
configuration parameter.
ref: <CANWsHyfHtr0EaJtNsDK9UTcmb_AbLg-1jUA-0uWJ-nEeNosb7w@mail.gmail.com>
Suggested-by: H Krishnan <hetchkay@gmail.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Adds a --preserve-empty-dirs flag to the clone operation that will detect
empty directories in the remote Subversion repository and create placeholder
files in the corresponding local Git directories. This allows "empty"
directories to exist in the history of a Git repository.
Also adds the --placeholder-file flag to control the name of any placeholder
files created. Default value is ".gitignore".
Signed-off-by: Ray Chen <rchen@cs.umd.edu>
Acked-by: Eric Wong <normalperson@yhbt.net>
Ensure that the xdl_free_classifier() call on xdlclassifier_t cf is safe
even if xdl_init_classifier() isn't called. This may occur in the case
where diff is run with --histogram and a call to, say, xdl_prepare_ctx()
fails.
Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Some platforms (IRIX, Solaris) provide an ancient /bin/sh which chokes on
modern shell syntax like $(). SHELL_PATH is provided to allow the user to
specify a working sh, let's use it here.
Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Since bc7a96a (mergetool--lib: Refactor tools into separate files,
2011-08-18) the mergetools and difftools related tests fail under
--valgrind because the mergetools/* scriptlets are not in the exec
path.
For now, symlink the mergetools subdir into the t/valgrind/bin
directory as a whole, since it does not contain anything of interest
to the valgrind wrappers.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Acked-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When running large git grep (ie: git grep regexp $(git rev-list --all)), glibc error sometimes occur:
*** glibc detected *** git: double free or corruption (!prev): 0x00000000010abdf0 ***
According to gdb the problem originate from release_delta_cash (sha1_file.c:1703)
free(ent->data);
>From my analysis it seems that git grep threads do acquire lock before calling read_sha1_file but not before calling
read_object_with_reference who ends up calling read_sha1_file too.
Adding the lock around read_object_with_reference seems to fix the issue for me.
I've ran git grep about a dozen time and seen no more error while
it usually happened half the time before.
Signed-off-by: Nicolas Morey-Chaisemartin <nicolas@morey-chaisemartin.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The Makefile enables CHECK_HEADER_DEPENDENCIES when the
compiler supports generating header dependencies.
Make the check use the same flags as the invocation
to avoid a false positive when user-configured compiler
flags contain incompatible options.
For example, without this patch, trying to build universal
binaries on a Mac using CFLAGS='-arch i386 -arch x86_64'
produces:
gcc-4.2: -E, -S, -save-temps and -M options are
not allowed with multiple -arch flags
While at it, remove "sh -c" in the command passed to $(shell);
at this point in the Makefile, SHELL has already been set to
a sensible shell and it is better not to override that.
Signed-off-by: David Aguilar <davvid@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
!"git ..." hopefully always succeeds because "git ..." is not the name
of any executable. However, that's not what was intended. Unquote
it, and while we're at it, also replace ! with test_must_fail since it
is a call to git.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Acked-by: Jon Seymour <jon.seymour@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
'git log -- <path>' does not "show commits that affect the specified
paths" in a literal sense unless --full-history is given (for example,
a file that only existed on a side branch will turn up no commits at
all!).
Reword it to specify the actual intent of the filtering, and point to
the "History Simplification" section.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The current remote helper documentation is from the perspective of
git, so to speak: it presents a full menu of commands for a person
invoking a remote helper to choose from. In practice, that's less
useful than it could be, since the daunted novice remote-helper author
probably just wanted a list of commands needs to implement to get
started. So preface the command list with an overview of each
capability, its purpose, and what commands it requires.
As a side effect, this makes it a little clearer that git doesn't
choose arbitrary commands to run, even if the remote helper advertises
all capabilities --- instead, there are well defined command sequences
for various tasks.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the case where sb is initialized to the slopbuf (through
strbuf_init(sb,0) or STRBUF_INIT), strbuf_grow() loses the terminating
nul: it grows the buffer, but gives ALLOC_GROW a NULL source to avoid
it being freed. So ALLOC_GROW does not copy anything to the new
memory area.
This subtly broke the call to strbuf_getline in read_next_command()
[fast-import.c:1855], which goes
strbuf_detach(&command_buf, NULL); # command_buf is now = STRBUF_INIT
stdin_eof = strbuf_getline(&command_buf, stdin, '\n');
if (stdin_eof)
return EOF;
In strbuf_getwholeline, this did
strbuf_grow(sb, 0); # loses nul-termination
if (feof(fp))
return EOF;
strbuf_reset(sb); # this would have nul-terminated!
Valgrind found this because fast-import subsequently uses prefixcmp()
on command_buf.buf, which after the EOF exit contains only
uninitialized memory.
Arguably strbuf_getwholeline is also broken, in that it touches the
buffer before deciding whether to do any work. However, it seems more
futureproof to not let the strbuf API lose the nul-termination by its
own fault.
So make sure that strbuf_grow() puts in a nul even if it has nowhere
to copy it from. This makes strbuf_grow(sb, 0) a semantic no-op as
far as readers of the buffer are concerned.
Also remove the nul-termination added by strbuf_init, which is made
redudant.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The negated forms introduced in c426003 (format-patch: add --no-cc,
--no-to, and --no-add-headers, 2010-03-07) were not documented
anywhere. Add them to the descriptions of the positive forms.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Since c426003 (format-patch: add --no-cc, --no-to, and
--no-add-headers, 2010-03-07) the tests have checked for an option
called --no-add-headers introduced by letting the user negate
--add-header.
However, the parseopt machinery does not automatically pluralize
anything, so it is in fact called --no-add-header.
Since the option never worked, is not documented anywhere, and
implementing an actual --no-add-headers would lead to silly code
complications, we just adapt the test to the code.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The test wrote something along the lines of 0001-foo.patch to output,
which of course never contained a signature. Luckily the tested
behaviour is actually present.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Most kinds of failure in 'git format-patch --stdout >output' will
result in an empty 'output'. This slips past checks that only verify
absence of output, such as the '! grep ...' that are quite prevalent
in t4014.
Introduce a helper check_patch() that checks that at least From, Date
and Subject are present, thus making sure it looks vaguely like a
patch (or cover letter) email. Then insert calls to it in all tests
that do have positive checks for content.
This makes two of the tests fail. Mark them as such; they'll be
fixed in a moment.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
convert_to_git sets src=dst->buf if any of the preceding conversions
actually did any work. Thus in ident_to_git we have to use memmove
instead of memcpy as far as src->dst copying is concerned.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
And finally, pass the pathspec down through unpack_trees() to traverse_trees()
callchain.
Before and after applying this series, looking for changes in the kernel
repository with a fairly narrow pathspec becomes somewhat faster.
(without patch)
$ /usr/bin/time git diff --raw v2.6.27 -- net/ipv6 >/dev/null
0.48user 0.05system 0:00.53elapsed 100%CPU (0avgtext+0avgdata 163296maxresident)k
0inputs+952outputs (0major+11163minor)pagefaults 0swaps
(with patch)
$ /usr/bin/time git diff --raw v2.6.27 -- net/ipv6 >/dev/null
0.01user 0.00system 0:00.02elapsed 104%CPU (0avgtext+0avgdata 43856maxresident)k
0inputs+24outputs (0major+3688minor)pagefaults 0swaps
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Use the pathspec pruning of traverse_trees() from unpack_trees(). Again,
the unpack_trees() machinery is primarily meant for merging two (or more)
trees, and because a merge is a full tree operation, it didn't support any
pruning with pathspec, and this codepath probably should not be enabled
while running a merge, but the caller in diff-lib.c::diff_cache() should
be able to take advantage of it.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The traverse_trees() machinery is primarily meant for merging two (or
more) trees, and because a merge is a full tree operation, it doesn't
support any pruning with pathspec.
Since d1f2d7e (Make run_diff_index() use unpack_trees(), not read_tree(),
2008-01-19), however, we use unpack_trees() to traverse_trees() callchain
to perform "diff-index", which could waste a lot of work traversing trees
outside the user-supplied pathspec, only to discard at the blob comparison
level in diff-lib.c::oneway_diff() which is way too late.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
More review comments.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Vitor Antunes <vitor.hda@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The error message given when the patch format was not recognized was
wrong, since the variable checked was $parse_patch rather than
$patch_format. Fix by checking the non-emptyness of the correct
variable.
Signed-off-by: Giuseppe Bilotta <giuseppe.bilotta@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
On OS X, the grep pattern
"\"OP .*/objects/$x2/X38_X40 HTTP/[.0-9]*\" 20[0-9] "
is too long ($x38 and $x40 represent 38 and 40 copies of [0-9a-f]) for
grep to handle. In order to still be able to match this, use the sed
invocation to replace what we're looking for with a token.
Improved-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Brian Gernhardt <brian@gernhardtsoftware.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The current explanation of -e can be misread as allowing the user to say
I know 'git clean -XYZ' (substitute -XYZ with any option and/or
parameter) will remove paths A, B, and C, and I want them all removed
except for paths matching this pattern by adding '-e C' to the same
command line, i.e. 'git clean -e C -XYZ'.
But that is not what this option does. It augments the set of ignore rules
from the command line, just like the same "-e <pattern>" argument does
with the "ls-files" command (the user could probably pass "-e \!C" to tell
the command to clean everything the command would normally remove, except
for C). Also error out when both -x and -e are given with an explanation of
what -e means---it is a symptom of misunderstanding what -e does.
It also fixes small style nit in the parameter to add_exclude() call. The
current code only works because EXC_CMDL happens to be defined as 0.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Allow pattern arguments for the list mode just like for git tag -l.
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Currently, there is no way to invoke the list mode explicitly, without
giving -v to force verbose output.
Introduce a --list option which invokes the list mode. This will be
beneficial for invoking list mode with pattern matching, which otherwise
would be interpreted as branch creation.
Along with --list, test also combinations of existing options.
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Long forms are better to memorize and more reliably uniform across
commands.
Names follow precedents, e.g. "git log --remotes".
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Long forms are better to memorize and more reliably uniform across
commands.
Design notes:
-u,--local-user is named following the analogous gnupg option.
-l,--list is not an argument taking option but a mode switch.
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
There is no guarantee that stderr is flushed before stdout when both
channels are redirected to a file. Check the channels using independent
files.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1, SunOS 5.10, and possibly
others do not have exit.h and exitfail.h. Remove the use of these in
obstack.c.
The __block variable was renamed to block to avoid a gcc error:
compat/obstack.h:190: error: __block attribute can be specified on variables only
Initial-patch-by: David Aguilar <davvid@gmail.com>
Reported-by: Brian Gernhardt <brian@gernhardtsoftware.com>
Signed-off-by: Fredrik Kuivinen <frekui@gmail.com>
Acked-by: David Aguilar <davvid@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* nd/decorate-grafts:
log: Do not decorate replacements with --no-replace-objects
log: decorate "replaced" on to replaced commits
log: decorate grafted commits with "grafted"
Move write_shallow_commits to fetch-pack.c
Add for_each_commit_graft() to iterate all grafts
decoration: do not mis-decorate refs with same prefix
* jk/color-and-pager:
want_color: automatically fallback to color.ui
diff: don't load color config in plumbing
config: refactor get_colorbool function
color: delay auto-color decision until point of use
git_config_colorbool: refactor stdout_is_tty handling
diff: refactor COLOR_DIFF from a flag into an int
setup_pager: set GIT_PAGER_IN_USE
t7006: use test_config helpers
test-lib: add helper functions for config
t7006: modernize calls to unset
Conflicts:
builtin/commit.c
parse-options.c
* mh/attr:
Unroll the loop over passes
Change while loop into for loop
Determine the start of the states outside of the pass loop
Change parse_attr() to take a pointer to struct attr_state
Increment num_attr in parse_attr_line(), not parse_attr()
Document struct match_attr
Add a file comment
* di/fast-import-blob-tweak:
fast-import: treat cat-blob as a delta base hint for next blob
fast-import: count and report # of calls to diff_delta in stats
