Blindly checking a path component for falsiness is unwise, as
"0" is false to Perl, but a valid pathname component for SVN
(or any filesystem).
Found via random code reading.
Signed-off-by: Eric Wong <e@80x24.org>
Reusing cached data speeds up git-svn by quite a fair bit. However, if
the YAML module is unavailable, the caches are written to disk in an
architecture-dependent manner. That leads to problems when upgrading,
say, from 32-bit to 64-bit Git for Windows.
Let's just try to read those caches back if we detect the absence of the
YAML module and the presence of the file, and delete the file if it
could not be read back correctly.
Note that the only way to catch the error when the memoized cache could
not be read back is to put the call inside an `eval { ... }` block
because it would die otherwise; the `eval` block should also return `1`
in case of success explicitly since the function reading back the cached
data does not return an appropriate value to test for success.
This fixes https://github.com/git-for-windows/git/issues/233.
[ew: import "retrieve" explictly, check unlink result]
Signed-off-by: Gavin Lambert <github@mirality.co.nz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Eric Wong <e@80x24.org>
"git send-email" attempts to pick up valid e-mails from the
trailers, but people in real world write non-addresses there, like
"Cc: Stable <add@re.ss> # 4.8+", which broke the output depending
on the availability and vintage of Mail::Address perl module.
* mm/send-email-cc-cruft-after-address:
Git.pm: add comment pointing to t9000
t9000-addresses: update expected results after fix
parse_mailboxes: accept extra text after <...> address
parse_mailboxes should probably eventually be completely equivalent to
Mail::Address, and if this happens we can drop the Mail::Address
dependency. Add a comment in the code reminding the current state of the
code, and point to the corresponding failing test to help future
contributors to get it right.
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The test introduced in this commit succeeds without the patch to Git.pm
if Mail::Address is installed, but fails otherwise because our in-house
parser does not accept any text after the email address. They succeed
both with and without Mail::Address after this commit.
Mail::Address accepts extra text and considers it as part of the name,
iff the address is surrounded with <...>. The implementation mimics
this behavior as closely as possible.
This mostly restores the behavior we had before b1c8a11 (send-email:
allow multiple emails using --cc, --to and --bcc, 2015-06-30), but we
keep the possibility to handle comma-separated lists.
Reported-by: Larry Finger <Larry.Finger@lwfinger.net>
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-svn internals were previously not aware of repository
layout differences for users of the "git worktree" command.
Introduce this awareness by using "git rev-parse --git-path"
instead of relying on outdated uses of GIT_DIR and friends.
Thanks-to: Duy Nguyen <pclouds@gmail.com>
Reported-by: Mathieu Arnold <mat@freebsd.org>
Signed-off-by: Eric Wong <e@80x24.org>
Reducing the scope of where we change the record separator ($/)
avoids bugs in calls which rely on the input record separator
further down, such as the 'chomp' usage in command_oneline.
This is necessary for a future change to git-svn, but exists in
Git.pm since it seems useful for gitweb and our other Perl
scripts, too.
Signed-off-by: Eric Wong <e@80x24.org>
It is possible to have refs globbed by git-svn which stores data
purely in git; gently skip those instead of dying and assuming
user error.
ref: http://mid.gmane.org/CALi1mtdtNF_GtzyPTbfb7N51wwxsFY7zm8hsgwxr3tHcZZboyg@mail.gmail.com
Suggested-by: Jacob Godserv <jacobgodserv@gmail.com>
Cc: Christian Couder <chriscool@tuxfamily.org>
Signed-off-by: Eric Wong <e@80x24.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
For repositories without parent following enabled, finding
git parents through svn:mergeinfo or svk::parents can be
expensive and pointless.
Reported-by: Александр Овчинников <proff@proff.email>
http://mid.gmane.org/4094761466408188@web24o.yandex.ru
Signed-off-by: Eric Wong <e@80x24.org>
Many instances of duplicate words (e.g. "the the path") and
a few typoes are fixed, originally in multiple patches.
wildmatch: fix duplicate words of "the"
t: fix duplicate words of "output"
transport-helper: fix duplicate words of "read"
Git.pm: fix duplicate words of "return"
path: fix duplicate words of "look"
pack-protocol.txt: fix duplicate words of "the"
precompose-utf8: fix typo of "sequences"
split-index: fix typo
worktree.c: fix typo
remote-ext: fix typo
utf8: fix duplicate words of "the"
git-cvsserver: fix duplicate words
Signed-off-by: Li Peng <lip@dtdream.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Error messages should attempt to fit within the confines of
an 80-column terminal to avoid compatibility and accessibility
problems. Furthermore the word "directories" can be misleading
when used in the context of git refnames.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Expand the area of globs applicability for branches and tags
in git-svn. It is now possible to use globs like 'a*e', or 'release_*'.
This allows users to avoid long lines in config like:
branches = branches/{release_20,release_21,release_22,...}
In favor of:
branches = branches/release_*
[ew: amended commit message, minor formatting and style fixes]
Signed-off-by: Victor Leschuk <vleschuk@accesssoftek.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
The conversion from "svn.pathnameencoding" to UTF-8 should be applied
first, and then URL encoding should be applied on the resulting UTF-8
path. The reversed order of these transforms (used before this fix)
makes non-UTF-8 URL which causes error from Subversion such as
"Filesystem has no item: '...' path not found" when sending a rename (or
a copy) from non-ASCII path.
[ew: t9115 test case added (requires SVN_HTTPD_PORT set to test),
squash LC_ALL=$a_utf8_locale export from Kazutoshi for Cygwin]
Signed-off-by: Kazutoshi SATODA <k_satoda@f2.dion.ne.jp>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Without the initialization of $self->{pathnameencoding}, conversion in
repo_path() is always skipped as $self->{pathnameencoding} is undefined
even if "svn.pathnameencoding" is configured.
The lack of conversion results in mysterious failure of dcommit (e.g.
"Malformed XML") which happen only when a commit involves a change on
non-ASCII path.
[ew: add test case to t9115,
squash LC_ALL=$a_utf8_locale export from Kazutoshi for Cygwin]
Signed-off-by: Kazutoshi SATODA <k_satoda@f2.dion.ne.jp>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Test scripts have been updated to remove assumptions that are not
portable between Git for POSIX and Git for Windows, or to skip ones
with expectations that are not satisfiable on Git for Windows.
* js/mingw-tests: (21 commits)
gitignore: ignore generated test-fake-ssh executable
mingw: do not bother to test funny file names
mingw: skip a test in t9130 that cannot pass on Windows
mingw: handle the missing POSIXPERM prereq in t9124
mingw: avoid illegal filename in t9118
mingw: mark t9100's test cases with appropriate prereqs
t0008: avoid absolute path
mingw: work around pwd issues in the tests
mingw: fix t9700's assumption about directory separators
mingw: skip test in t1508 that fails due to path conversion
tests: turn off git-daemon tests if FIFOs are not available
mingw: disable mkfifo-based tests
mingw: accomodate t0060-path-utils for MSYS2
mingw: fix t5601-clone.sh
mingw: let lstat() fail with errno == ENOTDIR when appropriate
mingw: try to delete target directory before renaming
mingw: prepare the TMPDIR environment variable for shell scripts
mingw: factor out Windows specific environment setup
Git.pm: stop assuming that absolute paths start with a slash
mingw: do not trust MSYS2's MinGW gettext.sh
...
For users with "store-passwords = no" set in the "[auth]" section of
their ~/.subversion/config, SVN 1.9.0+ would fail with the
following message when attempting to call svn_auth_set_parameter:
Value is not a string (or undef) at Git/SVN/Ra.pm
Ironically, this breakage was caused by r1553823 in subversion:
"Make svn_auth_set_parameter() usable from Perl bindings."
Since 2007 (602015e0e6), git-svn has used a workaround to make
svn_auth_set_parameter usable internally. However this workaround
breaks under SVN 1.9+, which deals properly with the type mapping
and fails to recognize our workaround.
For pre-1.9.0 SVN, we continue to use the existing workaround for
the lack of proper type mapping in the bindings.
Tested under subversion 1.6.17 and 1.9.3.
I've also verified r1553823 was not backported to SVN 1.8.x:
BRANCH=http://svn.apache.org/repos/asf/subversion/branches/1.8.x
svn log -v $BRANCH/subversion/bindings/swig/core.i
ref: https://bugs.debian.org/797705
Cc: 797705@bugs.debian.org
Reported-by: Thierry Vignaud <thierry.vignaud@gmail.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Tested-by: Thierry Vignaud <thierry.vignaud@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
On Windows, absolute paths never start with a slash, unless a POSIX
emulation layer is used. The latter is the case for MSYS2's Perl that
Git for Windows leverages. However, in the tests we also go through
plain `git.exe`, which does *not* leverage the POSIX emulation layer,
and therefore the paths we pass to Perl may actually be DOS-style paths
such as C:/Program Files/Git.
So let's just use Perl's own way to test whether a given path is
absolute or not instead of home-brewing our own.
This patch partially fixes t7800 and t9700 when running in Git for
Windows' SDK.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Processing empty_dir directives becomes extremely slow for svn
repositories with a large enough history.
This is due to using a single hash to store the list of empty
directories, with the expensive step being purging items from
that hash using grep+delete.
Storing directories in a hash of hashes improves the performance
of this purge step and removes a potentially lengthy delay after
every rebase/mkdirs command.
The svn repository with this behaviour has 110K commits with
unhandled.log containing 170K empty_dir directives.
This takes 10 minutes to process when using a single hash, vs
3 seconds with a hash of hashes.
Signed-off-by: Dair Grant <dair@feralinteractive.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
parse_address_line had not the same behavior whether the user had
Mail::Address or not. Teach parse_address_line to behave like
Mail::Address.
When the user input is correct, this implementation behaves
exactly like Mail::Address except when there are quotes
inside the name:
"Jane Do"e <jdoe@example.com>
In this case the result of parse_address_line is:
With M::A : "Jane Do" e <jdoe@example.com>
Without : "Jane Do e" <jdoe@example.com>
When the user input is not correct, the behavior is also mostly
the same.
Unlike Mail::Address, this doesn't parse groups and recursive
commentaries.
Signed-off-by: Remi Lespinet <remi.lespinet@ensimag.grenoble-inp.fr>
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Since b19138b (git-svn: Make it incrementally faster by minimizing temp
files, v1.6.0), git-svn has been using the Git.pm temp_acquire and
temp_release mechanism to avoid unnecessary temp file churn and provide
a speed boost.
However, that change introduced a call to temp_acquire inside the
Git::SVN::Fetcher::close_file function for an 'svn_hash' temp file.
Because an SVN::Pool is active at the time this function is called, if
the Git::temp_acquire function ends up actually creating a new
FileHandle for the temp file (which it will the first time it's called
with the name 'svn_hash') that FileHandle will end up in the SVN::Pool
and should that pool have SVN::Pool::clear called on it that FileHandle
will be closed out from under Git::temp_acquire.
Since the only call site to Git::temp_acquire with the name 'svn_hash'
is inside the close_file function, if an 'svn_hash' temp file is ever
created its FileHandle is guaranteed to be created in the active
SVN::Pool.
This has not been a problem in the past because the SVN::Pool was not
being cleared. However, since dfa72fdb (git-svn: reload RA every
log-window-size, v2.2.0) the pool has been getting cleared periodically
at which point the FileHandle for the 'svn_hash' temp file gets closed.
Any subsequent calls to Git::temp_acquire for 'svn_hash', however,
succeed without creating/opening a new temporary file since it still has
the now invalid FileHandle in its cache. Callers that then attempt to
use that FileHandle fail with an error.
We avoid this problem by making sure the 'svn_hash' temp file is created
in the same place the 'svn_delta_...' and 'git_blob_...' temp files are
(and then temp_release'd) so that it can be safely used inside the
close_file function without having its FileHandle end up in an SVN::Pool
that gets cleared.
Additionally the Git.pm cat_blob function creates a bidirectional pipe
FileHandle using the IPC::Open2::open2 function. If that handle is
created too late, it also gets caught up in the SVN::Pool and incorrectly
closed by the SVN::Pool::clear call. But this only seems to happen with
more recent versions of Perl and svn.
To avoid this problem we add an explicit call to _open_cat_blob_if_needed
before the first call to SVN::Pool->new_default to make sure the open2
handle does not end up in the SVN::Pool.
Signed-off-by: Kyle J. McKay <mackyle@gmail.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git svn uses POSIX::strftime('%s', $sec, $min, ...) to make unix epoch time.
But lowercase %s formatting character is a GNU extention. This causes problem
in git svn fetch --localtime on non-glibc systems, such as msys or cygwin.
Using Time::Local::timelocal($sec, $min, ...) fixes it.
Signed-off-by: Ryuichi Kokubo <ryu1kkb@gmail.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Notes:
lowercase %s format character in strftime is a GNU extension and not widely supported.
POSIX::strftime affected by underlying crt's strftime because POSIX::strftime just calls crt's one.
Time::Local is good function to replace POSIX::strftime because it's a perl core module function.
Document about Time::Local.
http://perldoc.perl.org/Time/Local.html
These are specifications of strftime.
The GNU C Library Reference Manual.
http://www.gnu.org/software/libc/manual/html_node/Formatting-Calendar-Time.html
perl POSIX module's strftime document. It does not have '%s'.
http://perldoc.perl.org/POSIX.html
strftime document of Microsort Windows C Run-Time library.
https://msdn.microsoft.com/en-us/library/fe06s4ak.aspx
The Open Group's old specification does not have '%s' too.
http://pubs.opengroup.org/onlinepubs/007908799/xsh/strftime.html
On my environment, following problems happened.
- msys : git svn fetch does not progress at all with perl.exe consuming CPU.
- cygwin : git svn fetch progresses but time stamp information is dropped.
Every commits have unix epoch timestamp.
I would like to thank git developer and contibutors.
git helps me so much everyday.
Thank you.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We can delay loading some modules until we need them for uncommon
code paths. For example, persistent memoization is not often
needed, so we can avoid loading the modules for it until we
encounter svn::mergeinfo during fetch.
This gives a tiny reduction in syscalls (from 15641 to 15305) when
running "git svn info" and counting via "strace -fc". Further,
more invasive work will be needed to noticeably improve performance.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
ref_id should not match "refs/remotes/".
[ew: dropped initial hunk for GIT_SVN_ID at Ramkumar's request]
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This change allows git-svn to support setting subversion properties.
It is useful for manually setting properties when committing to a
subversion repo that *requires* properties to be set without requiring
moving your changeset to separate subversion checkout in order to
set props.
This change is initially from David Fraser, appearing at:
http://mid.gmane.org/1927112650.1281253084529659.JavaMail.root@klofta.sjsoft.com>
They are now forward-ported to most recent git along with fixes to
deal with files in subdirectories.
Style and functional changes from Eric Wong have been taken
in their entirety from:
http://mid.gmane.org/20141201094911.GA13931@dcvr.yhbt.net
There is a nit to point out: the code does not support
adding props unless there are also content changes to the files as
well. This is demonstrated in the testcase.
[ew - simplify Git.pm usage for check-attr
- improve shell portability for tests
- minor phrasing changes in commit message]
Signed-off-by: David Fraser <davidf@sjsoft.com>
Signed-off-by: Alfred Perlstein <alfred@freebsd.org>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This avoids the following failure with normal "get_dir" on newer
versions of SVN (tested with SVN 1.8.8-1ubuntu3.1):
Incorrect parameters given: Could not convert '%ld' into a number
get_dir2 also has the potential to be more efficient by requesting
less data.
ref: <1414636504.45506.YahooMailBasic@web172304.mail.ir2.yahoo.com>
ref: <1414722617.89476.YahooMailBasic@web172305.mail.ir2.yahoo.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
Memoizing these initialization functions saves some memory for
long fetches which require scanning many unwanted revisions
before any wanted revisions happen.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This should help me track down errors in git-svn more easily:
write .git/Git_XXXXXX: Bad file descriptor
at /usr/lib/perl5/SVN/Ra.pm line 623
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This memoization appears unneeded as the check_cherry_pick2 cache is
in front of it does enough.
With this change applied, importing from local svn+ssh and http copies
of the R repo[1] takes only 2:00 (2 hours) on my system and the git-svn
process never uses more than 60MB RSS on my x86-64 GNU/Linux system[2].
This 60M measurement is only for the git-svn Perl process itself and
does not include memory used by git subprocesses accessing large packs
(subprocess memory usage _is_ measured by my time(1) tool).
Before this change, an import took longer (2:20) on svn+ssh:// but
git-svn used around 240MB during the imports. Worse yet, git-svn
ballooned to over 400M when writing out the cache to the filesystem.
I also tried removing memoization for `has_no_changes', too, but a
local copy of the R repository(*) was not close to finishing within
10 hours on my system.
[1] http://svn.r-project.org/R
[2] file:// repos causes libsvn to use more memory internally
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
There is no reason to keep entries in the %revs hash after we're
done processing a revision, so allow entries become freed as
processing continues.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This override was probably never necessary, but most likely a no-op
as it does not appear to do anything in SVN::Ra itself.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Despite attempting to use local memory pools everywhere we can,
(including our call to SVN::Ra::do_update and all subsequent reporter
calls), there does not appear to be a way to force the Git::SVN::Fetcher
callbacks to use a pool other than the per-SVN::Ra pool.
Git::SVN::Fetcher ends up using the main RA pool which grows
monotonically in size for the lifetime of the RA object.
Thus the only way to free that memory appears to be to destroy and
recreate the RA connection for at every --log-window-size interval.
This reduces memory usage over the course of fetching 10K revisions
using a test repository created with the script at the end of this
commit message.
As reported by time(1) on my x86-64 system:
before: 54024k
after: 28680k
Unfortunately, there remains some yet-to-be-tracked-down slow memory
growth which would be evident as the `nr' parameter increases in
the repository generation script:
-----------------------------8<------------------------------
set -e
tmp=$(mktemp -d svntestrepo-XXXXXXXX)
svnadmin create "$tmp"
repo=file://"$(cd $tmp && pwd)"
svn co "$repo" "$tmp/wd"
cd "$tmp/wd"
if ! test -f a
then
> a
svn add a
svn commit -m 'A'
fi
nr=10000
while test $nr -gt 0
do
echo $nr > a
svn commit -q -m A
nr=$((nr - 1))
done
echo "repository created in $repo"
-----------------------------8<------------------------------
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This should further reduce memory usage from the new mergeinfo
speedups without hurting performance too much, assuming
reasonable latency to the SVN server.
Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
This should reduce excessive memory usage from the new mergeinfo
caches without hurting performance too much, assuming reasonable
latency to the SVN server.
Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
We do not need to store entire lists of commits, only the
number of incomplete and the first commit for reference.
This reduces the amount of data we need to store in memory
and on disk stores.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Subversion can put mergeinfo on any sub-directory to track cherry-picks.
Since cherry-picks are not represented explicitly in git, git-svn should
just ignore it.
Signed-off-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
In a Subversion repository where many feature branches are merged into a
trunk, the svn:mergeinfo property can grow very large. This severely
slows down git-svn's make_log_entry() because it is checking all
mergeinfo entries every time the property changes.
In most cases, the additions to svn:mergeinfo since the last commit are
pretty small, and there is nothing to gain by checking merges that were
already checked for the last commit in the branch.
Add a mergeinfo_changes() function which computes the set of interesting
changes to svn:mergeinfo since the last commit. Filter out merged
branches whose ranges haven't changed, and remove a common prefix of
ranges from other merged branches.
This speeds up "git svn fetch" by several orders of magnitude on a large
repository where thousands of feature branches have been merged.
Signed-off-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Since the very beginning of Git, we gave the LESS environment a
default value "FRSX" when we spawn "less" as the pager. "S" (chop
long lines instead of wrapping) has been removed from this default
set of options, because it is more or less a personal taste thing,
as opposed to others that have good justifications (i.e. "R" is very
much justified because many kinds of output we produce are colored
and "FX" is justified because output we produce is often shorter
than a page).
Existing users who prefer not to see line-wrapped output may want to
set
$ git config core.pager "less -S"
to restore the traditional behaviour. It is expected that people
find output from the most subcommands easier to read with the new
default, except for "blame" which tends to produce really long
lines. To override the new default only for "git blame", you can do
this:
$ git config pager.blame "less -S"
* mm/pager-less-sans-S:
pager: remove 'S' from $LESS by default
By default, Git used to set $LESS to -FRSX if $LESS was not set by
the user. The FRX flags actually make sense for Git (F and X because
sometimes the output Git pipes to less is short, and R because Git
pipes colored output). The S flag (chop long lines), on the other
hand, is not related to Git and is a matter of user preference. Git
should not decide for the user to change LESS's default.
More specifically, the S flag harms users who review untrusted code
within a pager, since a patch looking like:
-old code;
+new good code; [... lots of tabs ...] malicious code;
would appear identical to:
-old code;
+new good code;
Users who prefer the old behavior can still set the $LESS environment
variable to -FRSX explicitly, or set core.pager to 'less -S'.
The documentation in config.txt is made a bit longer to keep both an
example setting the 'S' flag (needed to recover the old behavior)
and an example showing how to unset a flag set by Git.
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Some broken subversion server gives timestamps with only one digit
in the hour part, like this:
2014-01-07T5:01:02.048176Z
Loosen the regexp that expected to see two-digit hour, minute and
second parts to accept a single-digit hour (but not minute or
second).
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
