Commit Graph

254 Commits

Author SHA1 Message Date
Joel Holdsworth
70c0d55349 git-p4: resolve RCS keywords in bytes not utf-8
RCS keywords are strings that are replaced with information from
Perforce. Examples include $Date$, $Author$, $File$, $Change$ etc.

Perforce resolves these by expanding them with their expanded values
when files are synced, but Git's data model requires these expanded
values to be converted back into their unexpanded form.

Previously, git-p4.py would implement this behaviour through the use of
regular expressions. However, the regular expression substitution was
applied using decoded strings i.e. the content of incoming commit diffs
was first decoded from bytes into UTF-8, processed with regular
expressions, then converted back to bytes.

Not only is this behaviour inefficient, but it is also a cause of a
common issue caused by text files containing invalid UTF-8 data. For
files created in Windows, CP1252 Smart Quote Characters (0x93 and 0x94)
are seen fairly frequently. These codes are invalid in UTF-8, so if the
script encountered any file containing them, on Python 2 the symbols
will be corrupted, and on Python 3 the script will fail with an
exception.

This patch replaces this decoding/encoding with bytes object regular
expressions, so that the substitution is performed directly upon the
source data with no conversions.

A test for smart quote handling has been added to the
t9810-git-p4-rcs.sh test suite.

Signed-off-by: Joel Holdsworth <jholdsworth@nvidia.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-16 14:06:36 -08:00
Joel Holdsworth
4cf67ae1b6 git-p4: open temporary patch file for write only
The patchRCSKeywords method creates a temporary file in which to store
the patched output data. Previously this file was opened in "w+" mode
(write and read), but the code never reads the contents of the file
while open, so it only needs to be opened in "w" mode (write-only).

Signed-off-by: Joel Holdsworth <jholdsworth@nvidia.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-16 14:06:36 -08:00
Joel Holdsworth
9732e2229c git-p4: add raw option to read_pipelines
Previously the read_lines function always decoded the result lines. In
order to improve support for non-decoded binary processing of data in
git-p4.py, this patch adds a raw option to the function that allows
decoding to be disabled.

Signed-off-by: Joel Holdsworth <jholdsworth@nvidia.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-16 14:06:36 -08:00
Joel Holdsworth
e665e98ec1 git-p4: pre-compile RCS keyword regexes
Previously git-p4.py would compile one of two regular expressions for
ever RCS keyword-enabled file. This patch improves simplifies the code
by pre-compiling the two regular expressions when the script first
loads.

Signed-off-by: Joel Holdsworth <jholdsworth@nvidia.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-16 14:06:35 -08:00
Joel Holdsworth
8618d322e0 git-p4: use with statements to close files after use in patchRCSKeywords
Python with statements are used to wrap the execution of a block of code
so that an object can be safely released when execution leaves the
scope.

They are desirable for improving code tidyness, and to ensure that
objects are properly destroyed even when exceptions are thrown.

Signed-off-by: Joel Holdsworth <jholdsworth@nvidia.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-16 14:06:35 -08:00
dorgon.chang
54662d5958 git-p4: fix failed submit by skip non-text data files
If the submit contain binary files, it will throw exception and stop submit when try to append diff line description.

This commit will skip non-text data files when exception UnicodeDecodeError thrown.

The skip will not affect actual submit files in the resulting cl,
the diff line description will only appear in submit template,
so you can review what changed before actully submit to p4.

I don't know if add any message here will be helpful for users,
so I choose to just skip binary content, since it already append filename previously.

Signed-off-by: dorgon.chang <dorgonman@hotmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28 09:49:30 -07:00
Joachim Kuebart
6b79818bfb git-p4: speed up search for branch parent
For every new branch that git-p4 imports, it needs to find the commit
where it branched off its parent branch. While p4 doesn't record this
information explicitly, the first changelist on a branch is usually an
identical copy of the parent branch.

The method searchParent() tries to find a commit in the history of the
given "parent" branch whose tree exactly matches the initial changelist
of the new branch, "target". The code iterates through the parent
commits and compares each of them to this initial changelist using
diff-tree.

Since we already know the tree object name we are looking for, spawning
diff-tree for each commit is wasteful.

Use the "--format" option of "rev-list" to find out the tree object name
of each commit in the history, and find the tree whose name is exactly
the same as the tree of the target commit to optimize this.

This results in a considerable speed-up, at least on Windows. On one
Windows machine with a fairly large repository of about 16000 commits in
the parent branch, the current code takes over 7 minutes, while the new
code only takes just over 10 seconds for the same changelist:

Before:

    $ time git p4 sync
    Importing from/into multiple branches
    Depot paths: //depot
    Importing revision 31274 (100.0%)
    Updated branches: b1

    real    7m41.458s
    user    0m0.000s
    sys     0m0.077s

After:

    $ time git p4 sync
    Importing from/into multiple branches
    Depot paths: //depot
    Importing revision 31274 (100.0%)
    Updated branches: b1

    real    0m10.235s
    user    0m0.000s
    sys     0m0.062s

Signed-off-by: Joachim Kuebart <joachim.kuebart@gmail.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Helped-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-06 12:51:33 +09:00
Junio C Hamano
453e149c8a Merge branch 'dl/p4-encode-after-kw-expansion'
Text encoding fix for "git p4".

* dl/p4-encode-after-kw-expansion:
  git-p4: fix syncing file types with pattern
2021-01-15 21:48:47 -08:00
Daniel Levin
52fc4f195c git-p4: fix syncing file types with pattern
Example of pattern file type: text+k

Text filtered through the p4 pattern regexp must be converted from
string back to bytes, otherwise 'data' command for the fast-import
will receive extra invalid characters, followed by the fast-import
process error.

CC: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Daniel Levin <dendy.ua@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-12-23 13:49:40 -08:00
Junio C Hamano
ca8870d7c1 Merge branch 'js/p4-default-branch'
"git p4" now honors init.defaultBranch configuration.

* js/p4-default-branch:
  p4: respect init.defaultBranch
2020-11-11 13:18:38 -08:00
Johannes Schindelin
1b09d1917f p4: respect init.defaultBranch
In `git p4 clone`, we hard-code the branch name `master` instead of
looking what the _actual_ initial branch name is. Let's fix that.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-09 13:01:47 -08:00
Marlon Rac Cambasis
b7e20b4373 doc: fixing two trivial typos in Documentation/
Fix misspelled "specified" and "occurred" in documentation and
comments.

Signed-off-by: Marlon Rac Cambasis <marlonrc08@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-05 12:52:50 -08:00
Luke Diamand
0acbf5997f git-p4: use HEAD~$n to find parent commit for unshelve
Found-by: Liu Xuhui (Jackson) <Xuhui.Liu@amd.com>
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-09-19 13:44:55 -07:00
Junio C Hamano
7a8fec908a Merge branch 'bk/p4-prepare-p4-only-fix'
The "--prepare-p4-only" option is supposed to stop after replaying
one changeset, but kept going (by mistake?)

* bk/p4-prepare-p4-only-fix:
  git-p4.py: fix --prepare-p4-only error with multiple commits
2020-06-02 13:35:01 -07:00
Ben Keene
2dfdd705ff git-p4.py: fix --prepare-p4-only error with multiple commits
When using git p4 submit with the --prepare-p4-only option, the program
should prepare a single p4 changelist and notify the user that more
commits are pending and then stop processing.

A bug has been introduced by the p4-changelist hook feature that
causes the program to continue to try and process all pending
changelists at the same time.

The function applyCommit returns True when applying the commit
was successful and the program should continue. However, when the
optional flag --prepare-p4-only is set, the program should stop
after the first application.

Change the logic in the run method for P4Submit to check for the
flag --prepare-p4-only after successfully completing the applyCommit
method.

Be aware - this change will fix the existing test error in t9807.23
for --prepare-p4-only. However there is insufficent coverage for
this flag.  If more than 1 commit is pending submission to P4, the
method will properly prepare the P4 changelist, however it will
still exit the application with an exitcode of 1.

The current documentation does not define what the exit code should be
in this condition.
(See: https://git-scm.com/docs/git-p4#Documentation/git-p4.txt---prepare-p4-only)

Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-12 12:42:32 -07:00
Andrew Oakley
82e46d6b83 git-p4: recover from inconsistent perforce history
Perforce allows you commit files and directories with the same name,
so you could have files //depot/foo and //depot/foo/bar both checked
in.  A p4 sync of a repository in this state fails.  Deleting one of
the files recovers the repository.

When this happens we want git-p4 to recover in the same way as
perforce.

Note that Perforce has this change in their 2017.1 version:

     Bugs fixed in 2017.1
     #1489051 (Job #2170) **
        Submitting a file with the same name as an existing depot
        directory path (or vice versa) will now be rejected.

so people hopefully will not creating damaged Perforce repos
anymore, but "git p4" needs to be able to interact with already
corrupt ones.

Signed-off-by: Andrew Oakley <andrew@adoakley.name>
Reviewed-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-10 09:58:50 -07:00
Junio C Hamano
5f2ec211f6 Merge branch 'bk/p4-pre-edit-changelist'
"git p4" learned four new hooks and also "--no-verify" option to
bypass them (and the existing "p4-pre-submit" hook).

* bk/p4-pre-edit-changelist:
  git-p4: add RCS keyword status message
  git-p4: add p4 submit hooks
  git-p4: restructure code in submit
  git-p4: add --no-verify option
  git-p4: add p4-pre-submit exit text
  git-p4: create new function run_git_hook
  git-p4: rewrite prompt to be Windows compatible
2020-04-22 13:42:43 -07:00
Junio C Hamano
9a0fa1709c Merge branch 'yz/p4-py3'
Update "git p4" to work with Python 3.

* yz/p4-py3:
  ci: use python3 in linux-gcc and osx-gcc and python2 elsewhere
  git-p4: use python3's input() everywhere
  git-p4: simplify regex pattern generation for parsing diff-tree
  git-p4: use dict.items() iteration for python3 compatibility
  git-p4: use functools.reduce instead of reduce
  git-p4: fix freezing while waiting for fast-import progress
  git-p4: use marshal format version 2 when sending to p4
  git-p4: open .gitp4-usercache.txt in text mode
  git-p4: convert path to unicode before processing them
  git-p4: encode/decode communication with git for python3
  git-p4: encode/decode communication with p4 for python3
  git-p4: remove string type aliasing
  git-p4: change the expansion test from basestring to list
  git-p4: make python2.7 the oldest supported version
2020-03-25 13:57:43 -07:00
Ben Keene
1ec4a0a356 git-p4: add RCS keyword status message
During the p4 submit process, git-p4 will attempt to apply a patch
to the files found in the p4 workspace. However, if P4 uses RCS
keyword expansion, this patch may fail.

When the patch fails, the user is alerted to the failure and that git-p4
will attempt to clear the expanded text from the files and re-apply
the patch. The current version of git-p4 does not tell the user the
result of the re-apply attempt after the RCS expansion has been removed
which can be confusing.

Add a new print statement after the git patch has been successfully
applied when the RCS keywords have been cleansed.

Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-14 08:58:53 -08:00
Ben Keene
38ecf75244 git-p4: add p4 submit hooks
The git command "commit" supports a number of hooks that support
changing the behavior of the commit command.  The git-p4.py program only
has one existing hook, "p4-pre-submit".  This command occurs early in
the process.  There are no hooks in the process flow for modifying
the P4 changelist text programmatically.

Adds 3 new hooks to git-p4.py to the submit option.

The new hooks are:
* p4-prepare-changelist - Execute this hook after the changelist file
  has been created. The hook will be executed even if the
  --prepare-p4-only option is set.  This hook ignores the --no-verify
  option in keeping with the existing behavior of git commit.

* p4-changelist - Execute this hook after the user has edited the
  changelist. Do not execute this hook if the user has selected the
  --prepare-p4-only option. This hook will honor the --no-verify,
  following the conventions of git commit.

* p4-post-changelist - Execute this hook after the P4 submission process
  has completed successfully. This hook takes no parameters and is
  executed regardless of the --no-verify option.  It's return value will
  not be checked.

The calls to the new hooks: p4-prepare-changelist, p4-changelist,
and p4-post-changelist should all be called inside the try-finally
block.

Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-14 08:58:53 -08:00
Ben Keene
cd1e0dc47a git-p4: restructure code in submit
In preparation for adding new hooks to the submit method of git-p4,
restructure the applyCommit function in the P4Submit class.  Make the
following changes:

* Move all the code after the definition of submitted = False into the
  Try-Finally block. This ensures that any error that occurs will
  properly recover.  This is not necessary with the current code because
  none of it should throw an exception, however the next set of changes
  will introduce exceptional code.

  Existing flow control can remain as defined - the if-block for
  prepare-p4-only sets the local variable "submitted" to True and exits
  the function. New early exits, leave submitted set to False so the
  Finally block will undo changes to the P4 workspace.

* Make the small usability change of adding an empty string to the
  print statements displayed to the user when the prepare-p4-only option
  is selected.  On Windows, the command print() may display a set of
  parentheses "()" to the user when the print() function is called with
  no parameters. By supplying an empty string, the intended blank line
  will print as expected.

* Fix a small bug when the submittedTemplate is edited by the user
  and all content in the file is removed. The existing code will throw
  an exception if the separateLine is not found in the file. Change this
  code to test for the separator line using a find() test first and only
  split on the separator if it is found.

* Additionally, add the new behavior that if the changelist file has
  been completely emptied that the Submit action for this changelist
  will be aborted.

Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-14 08:58:53 -08:00
Ben Keene
4935c458c2 git-p4: add --no-verify option
Add new command line option --no-verify:

Add a new command line option "--no-verify" to the Submit command of
git-p4.py.  This option will function in the spirit of the existing
--no-verify command line option found in git commit. It will cause the
P4 Submit function to ignore the existing p4-pre-submit.

Change the execution of the existing trigger p4-pre-submit to honor the
--no-verify option. Before exiting on failure of this hook, display
text to the user explaining which hook has failed and the impact
of using the --no-verify option.

Change the call of the p4-pre-submit hook to use the new run_git_hook
function. This is in preparation of additional hooks to be added.

Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-11 12:04:05 -08:00
Ben Keene
aa8b766a13 git-p4: add p4-pre-submit exit text
When the p4-pre-submit exits with a non-zero exit code, the application
will abort the process with no additional information presented to the
user. This can be confusing for new users as it may not be clear that
the p4-pre-submit action caused the error.

Add text to explain why the program aborted the submit action.

Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-11 12:04:05 -08:00
Ben Keene
9f59ca4d6a git-p4: create new function run_git_hook
This commit is in preparation of introducing new p4 submit hooks.

The current code in the python script git-p4.py makes the assumption
that the git hooks can be executed by subprocess.call() function.
However, when git is run on Windows, this may not work as expected.

The subprocess.call() does not cover all the use cases for properly
executing the various types of executable files on Windows.

Prepare for remediation by adding a new function, run_git_hook, that
takes 2 parameters:
* the short filename of an optionally registered git hook
* an optional list of parameters

The run_git_hook function will honor the existing behavior seen in the
current code for executing the p4-pre-submit hook:

* Hooks are looked for in core.hooksPath directory.
* If core.hooksPath is not set, then the current .git/hooks directory
  is checked.
* If the hook does not exist, the function returns True.
* If the hook file is not accessible, the function returns True.
* If the hook returns a zero exit code when executed, the function
  return True.
* If the hook returns a non-zero exit code, the function returns False.

Add the following additional functionality if git-p4.py is run on
Windows.
* If hook file is not located without an extension, search for
  any file in the associated hook directory (from the list above) that
  has the same name but with an extension.
* If the file is still not found, return True (the hook is missing)

Add a new function run_hook_command() that wraps the OS dependent
functionality for actually running the subprocess.call() with OS
dependent behavior:

If a hook file exists on Windows:
* If there is no extension, set the launch executable to be SH.EXE
  - Look for SH.EXE under the environmental variable EXEPATH in the
    bin/ directory.
  - If %EXEPATH%/bin/sh.exe exists, use this as the actual executable.
  - If %EXEPATH%/bin/sh.exe does not exist, use sh.exe
  - Execute subprocess.call() without the shell (shell=False)
* If there is an extension, execute subprocess.call() with teh shell
  (shell=True) and consider the file to be the executable.

The return value from run_hook_command() is the subprocess.call()
return value.

These functions are added in this commit, but are only staged and not
yet used.

Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-11 12:04:05 -08:00
Ben Keene
6b602a2f97 git-p4: rewrite prompt to be Windows compatible
The existing function prompt(prompt_text) does not work correctly when
run on Windows 10 bash terminal when launched from the sourcetree
GUI application. The stdout is not flushed properly so the prompt text
is not displayed to the user until the next flush of stdout, which is
quite confusing.

Change this method by:
* Adding flush to stderr, stdout, and stdin
* Use readline from sys.stdin instead of raw_input.

The existing strip().lower() are retained.

Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-11 12:04:05 -08:00
Luke Diamand
43f33e492a git-p4: avoid leak of file handle when cloning
Spotted by Eric Sunshine:

    https://public-inbox.org/git/CAPig+cRx3hG64nuDie69o_gdX39F=sR6D8LyA7J1rCErgu0aMA@mail.gmail.com/

Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-30 12:21:13 -08:00
Luke Diamand
19fa5ac333 git-p4: check for access to remote host earlier
Check we can talk to the remote host before starting the git-fastimport
subchild.

Otherwise we fail to connect, and then exit, leaving git-fastimport
still running since we did not wait() for it.

Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-30 12:20:58 -08:00
Luke Diamand
6026aff5bb git-p4: cleanup better on error exit
After an error, git-p4 calls die(). This just exits, and leaves child
processes still running.

Instead of calling die(), raise an exception and catch it where the
child process(es) (git-fastimport) are created.

This was analyzed in detail here:

    https://public-inbox.org/git/20190227094926.GE19739@szeder.dev/

This change does not address the particular issue of p4CmdList()
invoking a subchild and not waiting for it on error.

Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-30 12:20:58 -08:00
Luke Diamand
ca5b5cce62 git-p4: create helper function importRevisions()
This makes it easier to try/catch around this block of code to ensure
cleanup following p4 failures is handled properly.

Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-30 12:20:58 -08:00
Luke Diamand
4c1d58675d git-p4: disable some pylint warnings, to get pylint output to something manageable
pylint is incredibly useful for finding bugs, but git-p4 has never used
it, so there are a lot of warnings that while important, don't actually
result in bugs.

Let's turn those off for now, so we can get some useful output.

Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-30 12:20:58 -08:00
Luke Diamand
5c3d5020e6 git-p4: add P4CommandException to report errors talking to Perforce
Currently when there is a P4 error, git-p4 calls die() which just exits.

This then leaves the git-fast-import process still running, and can even
leave p4 itself still running.

As a result, git-p4 fails to exit cleanly. This is a particular problem
for people running the unit tests in regression.

Use this exception to report errors upwards, cleaning up as the error
propagates.

Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-30 12:20:57 -08:00
Luke Diamand
837b3a6376 git-p4: make closeStreams() idempotent
Ensure that we can safely call self.closeStreams() multiple times, and
can also call it even if there is no git fast-import stream at all.

Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-30 12:20:57 -08:00
Yang Zhao
7575f4fdec git-p4: use python3's input() everywhere
Python3 deprecates raw_input() from 2.7 and replaced it with input().
Since we do not need 2.7's input() semantics, `raw_input()` is aliased
to `input()` for easy forward compatability.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:40 -08:00
Yang Zhao
ce425eb4e1 git-p4: simplify regex pattern generation for parsing diff-tree
It is not clear why a generator was used to create the regex used to
parse git-diff-tree output; I assume an early implementation required
it, but is not part of the mainline change.

Simply use a lazily initialized global instead.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:40 -08:00
Yang Zhao
2e2aa8d903 git-p4: use dict.items() iteration for python3 compatibility
Python3 uses dict.items() instead of .iteritems() to provide
iteratoration over dict.  Although items() is technically less efficient
for python2.7 (allocates a new list instead of simply iterating), the
amount of data involved is very small and the penalty negligible.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:40 -08:00
Yang Zhao
a6b1306735 git-p4: use functools.reduce instead of reduce
For python3, reduce() has been moved to functools.reduce().  This is
also available in python2.7.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:40 -08:00
Yang Zhao
4294d741cc git-p4: fix freezing while waiting for fast-import progress
As part of its importing process, git-p4 sends a `checkpoint` followed
immediately by `progress` to fast-import to force synchronization.
Due to buffering, it is possible for the `progress` command to not be
flushed before git-p4 begins to wait for the corresponding response.
This causes the script to freeze completely, and is consistently
observable at least on python-3.6.9.

Make sure this command sequence is completely flushed before waiting.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:40 -08:00
Yang Zhao
50da1e7393 git-p4: use marshal format version 2 when sending to p4
p4 does not appear to understand marshal format version 3 and above.
Version 2 was the latest supported by python-2.7.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:40 -08:00
Yang Zhao
5a5577d808 git-p4: open .gitp4-usercache.txt in text mode
Opening .gitp4-usercache.txt in text mode makes python 3 happy without
explicitly adding encoding and decoding.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:40 -08:00
Yang Zhao
d38208a297 git-p4: convert path to unicode before processing them
P4 allows essentially arbitrary encoding for path data while we would
perfer to be dealing only with unicode strings.  Since path data need to
survive round-trip back to p4, this patch implements the general policy
that we store path data as-is, but decode them to unicode before doing
any non-trivial processing.

A new `decode_path()` method is provided that generally does the correct
conversion, taking into account `git-p4.pathEncoding` configuration.

For python2.7, path strings will be left as-is if it only contains ASCII
characters.

For python3, decoding is always done so that we have str objects.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:40 -08:00
Yang Zhao
86dca24b7b git-p4: encode/decode communication with git for python3
Under python3, calls to write() on the stream to `git fast-import` must
be encoded.  This patch wraps the IO object such that this encoding is
done transparently.

Conversely, any text data read from subprocesses must also be decoded
before running through the rest of the pipeline.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:40 -08:00
Yang Zhao
6cec21a82f git-p4: encode/decode communication with p4 for python3
The marshalled dict in the response given on STDOUT by p4 uses `str` for
keys and string values. When run using python3, these values are
deserialized as `bytes`, leading to a whole host of problems as the rest
of the code assumes `str` is used throughout.

This patch changes the deserialization behaviour such that, as much as
possible, text output from p4 is decoded to native unicode strings.
Exceptions are made for the field `data` as it is usually arbitrary
binary data. `depotFile[0-9]*`, `path`, and `clientFile` are also exempt
as they contain path strings not encoded with UTF-8, and must survive
round-trip back to p4.

Conversely, text data being piped to p4 must always be encoded when
running under python3.

encode_text_stream() and decode_text_stream() were added to make these
transformations more convenient.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:39 -08:00
Yang Zhao
1f8b46d0a4 git-p4: remove string type aliasing
Now that python2.7 is the minimum required version and we no longer use
the basestring type, it is not necessary to use type aliasing to ensure
python3 compatibility.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:39 -08:00
Ben Keene
484d09c303 git-p4: change the expansion test from basestring to list
Python 3 handles strings differently than Python 2.7. Since Python 2
is reaching it's end of life, a series of changes are being submitted to
enable python 3.5 and following support. The current code fails basic
tests under python 3.5.

Some codepaths can represent a command line the program
internally prepares to execute either as a single string
(i.e. each token properly quoted, concatenated with $IFS) or
as a list of argv[] elements, and there are 9 places where
we say "if X is isinstance(_, basestring), then do this
thing to handle X as a command line in a single string; if
not, X is a command line in a list form".

This does not work well with Python 3, as there is no
basestring (everything is Unicode now), and even with Python
2, it was not an ideal way to tell the two cases apart,
because an internally formed command line could have been in
a single Unicode string.

Flip the check to say "if X is not a list, then handle X as
a command line in a single string; otherwise treat it as a
command line in a list form".

This will get rid of references to 'basestring', to migrate
the code ready for Python 3.

Thanks-to: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:39 -08:00
Yang Zhao
0b4396f068 git-p4: make python2.7 the oldest supported version
Python2.6 and earlier have been end-of-life'd for many years now, and
we actually already use 2.7-only features in the code. Make the version
check reflect current realities.

This also removes the need to explicitly define CalledProcessError if
it's not available.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-15 12:53:39 -08:00
Junio C Hamano
bc855232bc Merge branch 'bk/p4-misc-usability'
Miscellaneous small UX improvements on "git-p4".

* bk/p4-misc-usability:
  git-p4: show detailed help when parsing options fail
  git-p4: yes/no prompts should sanitize user text
2020-01-02 12:38:29 -08:00
Ben Keene
608e380502 git-p4: show detailed help when parsing options fail
When a user provides invalid parameters to git-p4, the program
reports the failure but does not provide the correct command syntax.

Add an exception handler to the command-line argument parser to display
the command's specific command line parameter syntax when an exception
is thrown. Rethrow the exception so the current behavior is retained.

Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-16 12:32:13 -08:00
Ben Keene
e2aed5fd5b git-p4: yes/no prompts should sanitize user text
When prompting the user interactively for direction, the tests are
not forgiving of user input format.

For example, the first query asks for a yes/no response. If the user
enters the full word "yes" or "no" or enters a capital "Y" the test
will fail.

Create a new function, prompt(prompt_text) where
  * prompt_text is the text prompt for the user
  * returns a single character where valid return values are
      found by inspecting prompt_text for single characters
      surrounded by square brackets

This new function must  prompt the user for input and sanitize it by
converting the response to a lower case string, trimming leading and
trailing spaces, and checking if the first character is in the list
of choices. If it is, return the first letter.

Change the current references to raw_input() to use this new function.

Since the method requires the returned text to be one of the available
choices, remove the loop from the calling code that handles response
verification.

Thanks-to: Denton Liu <liu.denton@gmail.com>
Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-16 12:31:47 -08:00
r.burenkov
ea94b16fb8 git-p4: honor lfs.storage configuration variable
"git lfs" allows users to specify the custom storage location with
the configuration variable `lfs.storage`, but when interacting with
GitLFS pointers, "git p4" always uses the hardcoded default that is
the `.git/lfs/` directory, without paying attention to the
configuration.

Use the value configured in `lfs.storage`, if exists, as all the
"git" operations do, for consistency.

Signed-off-by: r.burenkov <panzercheg@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-11 09:59:17 -08:00
Philip.McGraw
de5abb5f7a git-p4: auto-delete named temporary file
Avoid double-open exceptions on Windows platform when
calculating for lfs compressed size threshold
(git-p4.largeFileCompressedThreshold) comparisons.

Take new approach using the NamedTemporaryFile()
file-like object as input to the ZipFile() which
auto-deletes after implicit close leaving with scope.

Original code had double-open exception on Windows
platform because file still open from NamedTemporaryFile()
using generated filename instead of object.

Thanks to Andrey for patiently suggesting several
iterations on this change for avoiding exceptions!

Also print error details after resulting IOError to make
debugging cause of exception less mysterious when it has
nothing to do with "git version recent enough."

Signed-off-by: Philip.McGraw <Philip.McGraw@bentley.com>
Reviewed-by: Andrey Mazo <ahippo@yandex.com>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-06 20:43:37 +09:00