git-p4 is designed to run correctly under python2.7 and python3, but
its functional behavior wrt importing user-entered text differs across
these environments:
Under python2, git-p4 "naively" writes the Perforce bytestream into git
metadata (and does not set an "encoding" header on the commits); this
means that any non-utf-8 byte sequences end up creating invalidly-encoded
commit metadata in git.
Under python3, git-p4 attempts to decode the Perforce bytestream as utf-8
data, and fails badly (with an unhelpful error) when non-utf-8 data is
encountered.
Perforce clients (especially p4v) encourage user entry of changelist
descriptions (and user full names) in OS-local encoding, and store the
resulting bytestream to the server unmodified - such that different
clients can end up creating mutually-unintelligible messages. The most
common inconsistency, in many Perforce environments, is likely to be utf-8
(typical in linux) vs cp-1252 (typical in windows).
Make the changelist-description- and user-fullname-handling code
python-runtime-agnostic, introducing three "strategies" selectable via
config:
- 'passthrough', behaving as previously under python2,
- 'strict', behaving as previously under python3, and
- 'fallback', favoring utf-8 but supporting a secondary encoding when
utf-8 decoding fails, and finally escaping high-range bytes if the
decoding with the secondary encoding also fails.
Keep the python2 default behavior as-is ('legacy' strategy), but switch
the python3 default strategy to 'fallback' with default fallback encoding
'cp1252'.
Also include tests exercising these encoding strategies, documentation for
the new config, and improve the user-facing error messages when decoding
does fail.
Signed-off-by: Tao Klerks <tao@klerks.biz>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Previously, cleanup_git() would use `test_must_fail test -d` to ensure
that the directory is removed. However, test_must_fail should only be
used for git commands. Use test_path_is_missing() instead to check that
the directory has been removed.
Signed-off-by: Denton Liu <liu.denton@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
According to the comments in 't/lib-git-p4.sh', sometimes 'p4d' seems
to hang, and to deal with that 'start_p4d' starts a watchdog process
to kill it after a long-enough timeout ($P4D_TIMEOUT, defaults to
300s). This watchdog process is implemented as a background subshell
loop iterating once every second until the timeout expires, producing
a few lines of trace output on each iteration when the test script is
run with '-x' tracing enabled. The watchdog loop's trace gets
intermixed with the real test output and trace, and makes that harder
to read.
Send the trace output of this loop to /dev/null to avoid polluting the
real test output.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
'lib-git-p4.sh' uses timeouts in a watchdog process to kill a
potentially stuck 'p4d' process and for certain cleanup operation
between tests. It does so by first computing when the timeout should
expire, and then repeatedly asking for the current time in seconds
until it exceeds the expiration time, and for portability reasons it
uses a one-liner Python script to ask for the current time.
Replace these timeouts with downcounters, which, though not
necessarily shorter, are much simpler, at least in the sense that they
don't execute the Python interpreter every second.
After this change the helper function with that Python one-liner has
no callers left, remove it.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Confusingly, the 'git p4' tests used two cleanup functions:
- 'kill_p4d' was run in the last test before 'test_done', and it not
only killed 'p4d', but it killed the watchdog process, and cleaned
up after 'p4d' as well by removing all directories used by the P4
daemon and client.
This cleanup is not necessary right before 'test_done', because
the whole trash directory is about to get removed anyway, but it
is necessary in 't9801-git-p4-branch.sh', which uses 'kill_p4d' to
stop 'p4d' before re-starting it in the middle of the test script.
- 'cleanup' was run in the trap on EXIT, and it killed 'p4d', but,
it didn't kill the watchdog process, and, contrarily to its name,
didn't perform any cleanup whatsoever.
Make it clearer what's going on by renaming and simplifying the
cleanup functions, so in the end we'll have:
- 'stop_p4d_and_watchdog' replaces 'cleanup' as it will try to live
up to its name and stop both the 'p4d' and the watchdog processes,
and as the sole function registered with 'test_atexit' it will be
responsible for no leaving any stray processes behind after 'git p4'
tests were finished or interrupted.
- 'stop_and_cleanup_p4d' replaces 'kill_p4d' as it will stop 'p4d'
(and the watchdog) and remove all directories used by the P4
daemon and cliean, so it can be used mid-script to stop and then
re-start 'p4d'.
Note that while 'cleanup' sent a single SIGKILL to 'p4d', 'kill_p4d'
was quite brutal, as it first sent SIGTERM to the daemon repeatedly,
either until its pid disappeared or until a given timeout was up, and
then it sent SIGKILL repeatedly, for good measure. This is overkill
(pardon the pun): a single SIGKILL should be able to take down any
process in a sensible state, and if a process were to somehow end up
stuck in the dreaded uninterruptible sleep state then even repeated
SIGKILLs won't bring immediate help. So ditch all the repeated
SIGTERM/SIGKILL parts, and use a single SIGKILL to stop 'p4d', and
make sure that there are no races between asynchron signal delivery
and subsequent restart of 'p4d' by waiting for it to die.
With this change the 'retry_until_fail' helper has no callers left,
remove it.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Use 'test_atexit' to run cleanup commands to stop 'p4d' at the end of
the test script or upon interrupt or failure, as it is shorter,
simpler, and more robust than registering such cleanup commands in the
trap on EXIT in the test scripts.
Note that one of the test scripts, 't9801-git-p4-branch.sh', stops and
then re-starts 'p4d' twice in the middle of the script; take care that
the cleanup functions to stop 'p4d' are only registered once.
Note also that 'git p4' tests invoke different functions in the trap
on EXIT ('cleanup') and in the last test before 'test_done'
('kill_p4d'). Register both of these functions with 'test_atexit' for
now, and a a later patch in this series will then clean up the
redundancy.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Several test scripts run daemons like 'git-daemon' or Apache, and
communicate with them through TCP sockets. To have unique ports where
these daemons are accessible, the ports are usually the number of the
corresponding test scripts, unless the user overrides them via
environment variables, and thus all those tests and test libs contain
more or less the same bit of one-liner boilerplate code to find out
the port. The last patch in this series will make this a bit more
complicated.
Factor out finding the port for a daemon into the common helper
function 'test_set_port' to avoid repeating ourselves.
Take special care of test scripts with "low" numbers:
- Test numbers below 1024 would result in a port that's only usable
as root, so set their port to '10000 + test-nr' to make sure it
doesn't interfere with other tests in the test suite. This makes
the hardcoded port number in 't0410-partial-clone.sh' unnecessary,
remove it.
- The shell's arithmetic evaluation interprets numbers with leading
zeros as octal values, which means that test number below 1000 and
containing the digits 8 or 9 will trigger an error. Remove all
leading zeros from the test numbers to prevent this.
Note that the 'git p4' tests are unlike the other tests involving
daemons in that:
- 'lib-git-p4.sh' doesn't use the test's number for unique port as
is, but does a bit of additional arithmetic on top [1].
- The port is not overridable via an environment variable.
With this patch even 'git p4' tests will use the test's number as
default port, and it will be overridable via the P4DPORT environment
variable.
[1] Commit fc00233071 (git-p4 tests: refactor and cleanup, 2011-08-22)
introduced that "unusual" unique port computation without
explaining why it was necessary (as opposed to simply using the
test number as is). It seems to be just unnecessary complication,
and in any case that commit came way before the "test nr as unique
port" got "standardized" for other daemons in commits c44132fcf3
(tests: auto-set git-daemon port, 2014-02-10), 3bb486e439 (tests:
auto-set LIB_HTTPD_PORT from test name, 2014-02-10), and
bf9d7df950 (t/lib-git-svn.sh: improve svnserve tests with parallel
make test, 2017-12-01).
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The test scripts for "git p4" (but not "git p4" implementation
itself) has been updated so that they would work even on a system
where the installed version of Python is python 3.
* ld/p4-test-py3:
git-p4 tests: time_in_seconds should use $PYTHON_PATH
git-p4 tests: work with python3 as well as python2
git-p4 tests: cd to / before running python
The time_in_seconds script should use $PYTHON_PATH, rather than
just hard-coded python, so that users can override which version
gets used, as is done for other python invocations.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Update the git-p4 tests so that they work with both
Python2 and Python3.
We have to be explicit about the difference between
Unicode text strings (Python3 default) and raw binary
strings which will be exchanged with Perforce.
Additionally, print always takes parentheses in Python3.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The python one-liner for getting the current time prints out
error messages if the current directory is deleted while it is
running if using python3.
Avoid these messages by switching to "/" before running it.
This problem does not arise if using python2.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When migrating from Perforce to git the information about P4 jobs
associated with P4 changelists is lost.
Having these jobs listed on messages of related git commits enables smooth
migration for projects that take advantage of e.g. JIRA integration
(which uses jobs on Perforce side and parses commit messages on git side).
The jobs are added to the message in the same format as is expected when
migrating in the reverse direction.
Signed-off-by: Jan Durovec <jan.durovec@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Preliminary clean-up of testing libraries for git-p4.
* spaces added to both sides of () in function definitions in lib-git-p4
* tab indentation added to git-p4 tests when <<- redirection is used
Signed-off-by: Jan Durovec <jan.durovec@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Sometimes the "prove" test runner hangs on test exit because p4d is
still running. Add a trap to always kill "p4d" on test exit.
You can reproduce the problem by commenting "P4D_TIMEOUT" in
"lib-git-p4.sh" and running "prove ./t9800-git-p4-basic.sh".
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
In rare cases p4d seems to hang. This watchdog will kill the p4d
process after 300s in any case. That means each individual git p4 test
needs to finish before 300s or it will fail.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Jeff King <peff@peff.net>
In rare cases kill/cleanup operations in tests fail. Retry these
operations with a timeout to make the test less flaky.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
When p4d runs on a case-folding OS, git-p4 can end up getting
very confused. This adds failing tests to demonstrate the problem.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Generating the submit template for p4 uses tempfile.mkstemp(),
which by default puts files in /tmp. For a test that fails,
possibly on purpose, this is not cleaned up. Run with TMPDIR
pointing into the trash directory so the temp files go away
with the test results.
To do this required some other minor changes. First, the editor
is launched using system(editor + " " + template_file), using
shell expansion to build the command string. This doesn't work
if editor has a space in it. And is generally unwise as it's
easy to fool the shell into doing extra work. Exec the args
directly, without shell expansion.
Second, without shell expansion, the trick of "P4EDITOR=:" used
in the tests doesn't work. Use a real command, true, as the
non-interactive editor for testing.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The tests use author@example.com as the canonical submitter,
but he does not have an entry in the p4 users database.
This causes the generated change description to complain
that the git and p4 users disagree. The complaint message
is still valid, but isn't useful in tests. It was introduced
in 848de9c (git-p4: warn if git authorship won't be retained,
2011-05-13).
Fix t9813 to use @example.com instead of @localhost due to
change in p4_add_user(). Move the function into the git p4
test library so author can be added at initialization time.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the tests, p4d is started without using "internationalized
mode". Make sure this environment variable is unset, otherwise
a mis-matched user setting would break the tests. The error
message would be "Unicode clients require a unicode enabled server."
[pw: use unset, add commit text]
Signed-off-by: Kazuki Saitoh <ksaitoh560@gmail.com>
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
There are some old versions of p4, compiled for cygwin, that
treat read-only files differently.
Normally, a file that is not open is read-only, meaning that
"test -w" on the file is false. This works on unix, and it works
on windows using the NT version of p4. The cygwin version
of p4, though, changes the permissions, but does not set the
windows read-only attribute, so "test -w" returns false.
Notice this oddity and make the tests work, even on cygiwn.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In all clients, even those created on windows, use unix line
endings. This makes it possible to verify file contents without
doing OS-specific comparisons in all the tests.
Tests in t9802-git-p4-filetype.sh are used to make sure that
the other LineEnd options continue to work.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Native windows binaries do not understand posix-like
path mapping offered by cygwin. Convert paths to native
using "cygpath --windows" before presenting them to p4d.
This is done using the AltRoots mechanism of p4. Both the
posix and windows forms are put in the client specification,
allowing p4 to find its location by native path even though
the environment reports a different PWD.
Shell operations in tests will use the normal form of $cli,
which will look like a posix path in cygwin, while p4 will
use AltRoots to match against the windows form of the working
directory.
This mechanism also handles the symlink issue that was fixed in
23bd0c9 (git p4 test: use real_path to resolve p4 client
symlinks, 2012-06-27). Now that every p4 client view has
an AltRoots with the real_path in it, explicitly calculating
the real_path elsewhere is not necessary.
Thanks-to: Sebastian Schuberth <sschuberth@gmail.com>
Thanks-to: Johannes Sixt <j6t@kdbg.org>
fixup! git p4 test: translate windows paths for cygwin
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This will avoid having to do native path conversion for
windows. Also may be a bit cleaner always to know that p4d
has that working directory, instead of wherever the function
was called from.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Use the standard client_view function from lib-git-p4.sh
instead of building one by hand. This requires a bit of
rework, using the current value of $P4CLIENT for the client
name. It also reorganizes the test to isolate changes to
$P4CLIENT and $cli in a subshell.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The printf command re-interprets the format string as
long as there are arguments to consume. Use this to
simplify a for loop in the client_view() library function.
This requires a fix to one of the client_view callers.
An errant \n in the string was converted into a harmless
newline in the input to "p4 client -i", but now shows up
as a literal \n as passed through by "%s". Remove the \n.
Based-on-patch-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add '--conflict' option to git-p4 subcommand to specify what action
to take when conflicts are found during 'p4 submit'.
* pw/p4-submit-conflicts:
git-p4: add submit --conflict option and config varaiable
git p4: add submit --prepare-p4-only option
git p4: add submit --dry-run option
git p4: accept -v for --verbose
git p4: revert deleted files after submit cancel
git p4: rearrange submit template construction
git p4: test clean-up after failed submit, fix added files
git p4: standardize submit cancel due to unchanged template
git p4: move conflict prompt into run, add [q]uit input
git p4: remove submit failure options [a]pply and [w]rite
git p4: gracefully fail if some commits could not be applied
git p4 test: remove bash-ism of combined export/assignment
This code will be useful in --detect-branches --use-client-spec tests.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This function will be useful in future tests. Move it to
the git-p4 test library. Let it accept an optional argument
to pick a certain marshaled object out of the input stream.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
For convenience, leave one in place at the end of each
test so that it is not necessary to build a new one. This
makes it consistent with $cli.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The p4 program is finicky about making sure the recorded client Root
matches the current working directory. The way it discovers the latter
seems to be to inspect shell variable $PWD. This could involve symlinks,
that while leading to the same place as the client Root, look different,
and cause p4 to fail.
Resolve all client paths using "test-path-utils real_path $path". This
removes ".." and resolves all symlinks.
Discovered while running with --root=/dev/shm, which is a link to
/run/shm.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Running tests at high parallelism on a slow machine, 5 sec is
not enough to wait for p4d to start. Change it to 5 minutes,
adding an environment variable P4D_START_PATIENCE to shrink
that if needed in automated test environments.
Also check if the pid of the p4d that we started is still
around. If not, quit waiting for it immediately.
Remove all the confusing && chaining and simplify the code.
Thanks-to: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If P4EDITOR is set in the environment, test behavior could be
unpredictable. Set it explicitly.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Drop the $GITP4 variable that was used to specify the script in
contrib/fast-import/. The command is called "git p4" now, not
"git-p4".
Note that configuration variables will remain in a section called
"git-p4".
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Move git-p4 out of contrib/fast-import into the main code base,
aside other foreign SCM tools.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a library for functions that are common to
multiple git-p4 test files.
Be a bit more clever about starting and stopping p4d.
Specify a unique port number for each test, so that
tests can run in parallel. Start p4d not in daemon mode,
and save the pid, to be able to kill it cleanly later.
Never kill p4d at startup; always shutdown cleanly.
Handle directory changes better. Always chdir inside
a subshell, and remove any post-test directory changes.
Clean up whitespace, and use test_cmp and test_must_fail
more consistently.
Separate the tests related to detecting p4 branches
into their own file, and add a few more.
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
