Merge branch 'mm/credential-plumbing'

Expose the credential API to scripted Porcelain writers.

* mm/credential-plumbing:
  git-remote-mediawiki: update comments to reflect credential support
  git-remote-mediawiki: add credential support
  git credential fill: output the whole 'struct credential'
  add 'git credential' plumbing command

.gitignore

@@ -31,6 +31,7 @@
 /git-commit-tree
 /git-config
 /git-count-objects
+/git-credential
 /git-credential-cache
 /git-credential-cache--daemon
 /git-credential-store
@@ -172,7 +173,6 @@
 /gitweb/static/gitweb.js
 /gitweb/static/gitweb.min.*
 /test-chmtime
-/test-credential
 /test-ctype
 /test-date
 /test-delta

Documentation/git-credential.txt

@@ -0,0 +1,144 @@
+git-credential(1)
+=================
+
+NAME
+----
+git-credential - retrieve and store user credentials
+
+SYNOPSIS
+--------
+------------------
+git credential <fill|approve|reject>
+------------------
+
+DESCRIPTION
+-----------
+
+Git has an internal interface for storing and retrieving credentials
+from system-specific helpers, as well as prompting the user for
+usernames and passwords. The git-credential command exposes this
+interface to scripts which may want to retrieve, store, or prompt for
+credentials in the same manner as git. The design of this scriptable
+interface models the internal C API; see
+link:technical/api-credentials.txt[the git credential API] for more
+background on the concepts.
+
+git-credential takes an "action" option on the command-line (one of
+`fill`, `approve`, or `reject`) and reads a credential description
+on stdin (see <<IOFMT,INPUT/OUTPUT FORMAT>>).
+
+If the action is `fill`, git-credential will attempt to add "username"
+and "password" attributes to the description by reading config files,
+by contacting any configured credential helpers, or by prompting the
+user. The username and password attributes of the credential
+description are then printed to stdout together with the attributes
+already provided.
+
+If the action is `approve`, git-credential will send the description
+to any configured credential helpers, which may store the credential
+for later use.
+
+If the action is `reject`, git-credential will send the description to
+any configured credential helpers, which may erase any stored
+credential matching the description.
+
+If the action is `approve` or `reject`, no output should be emitted.
+
+TYPICAL USE OF GIT CREDENTIAL
+-----------------------------
+
+An application using git-credential will typically use `git
+credential` following these steps:
+
+  1. Generate a credential description based on the context.
++
+For example, if we want a password for
+`https://example.com/foo.git`, we might generate the following
+credential description (don't forget the blank line at the end; it
+tells `git credential` that the application finished feeding all the
+infomation it has):
+
+	 protocol=https
+	 host=example.com
+	 path=foo.git
+
+  2. Ask git-credential to give us a username and password for this
+     description. This is done by running `git credential fill`,
+     feeding the description from step (1) to its standard input. The complete
+     credential description (including the credential per se, i.e. the
+     login and password) will be produced on standard output, like:
+
+	protocol=https
+	host=example.com
+	username=bob
+	password=secr3t
++
+In most cases, this means the attributes given in the input will be
+repeated in the output, but git may also modify the credential
+description, for example by removing the `path` attribute when the
+protocol is HTTP(s) and `credential.useHttpPath` is false.
++
+If the `git credential` knew about the password, this step may
+not have involved the user actually typing this password (the
+user may have typed a password to unlock the keychain instead,
+or no user interaction was done if the keychain was already
+unlocked) before it returned `password=secr3t`.
+
+  3. Use the credential (e.g., access the URL with the username and
+     password from step (2)), and see if it's accepted.
+
+  4. Report on the success or failure of the password. If the
+     credential allowed the operation to complete successfully, then
+     it can be marked with an "approve" action to tell `git
+     credential` to reuse it in its next invocation. If the credential
+     was rejected during the operation, use the "reject" action so
+     that `git credential` will ask for a new password in its next
+     invocation. In either case, `git credential` should be fed with
+     the credential description obtained from step (2) (which also
+     contain the ones provided in step (1)).
+
+[[IOFMT]]
+INPUT/OUTPUT FORMAT
+-------------------
+
+`git credential` reads and/or writes (depending on the action used)
+credential information in its standard input/output. These information
+can correspond either to keys for which `git credential` will obtain
+the login/password information (e.g. host, protocol, path), or to the
+actual credential data to be obtained (login/password).
+
+The credential is split into a set of named attributes.
+Attributes are provided to the helper, one per line. Each attribute is
+specified by a key-value pair, separated by an `=` (equals) sign,
+followed by a newline. The key may contain any bytes except `=`,
+newline, or NUL. The value may contain any bytes except newline or NUL.
+In both cases, all bytes are treated as-is (i.e., there is no quoting,
+and one cannot transmit a value with newline or NUL in it). The list of
+attributes is terminated by a blank line or end-of-file.
+Git will send the following attributes (but may not send all of
+them for a given credential; for example, a `host` attribute makes no
+sense when dealing with a non-network protocol):
+
+`protocol`::
+
+	The protocol over which the credential will be used (e.g.,
+	`https`).
+
+`host`::
+
+	The remote hostname for a network credential.
+
+`path`::
+
+	The path with which the credential will be used. E.g., for
+	accessing a remote https repository, this will be the
+	repository's path on the server.
+
+`username`::
+
+	The credential's username, if we already have one (e.g., from a
+	URL, from the user, or from a previously run helper).
+
+`password`::
+
+	The credential's password, if we are asking it to be stored.

Documentation/technical/api-credentials.txt

@@ -241,42 +241,9 @@ appended to its command line, which is one of:
 	Remove a matching credential, if any, from the helper's storage.
 
 The details of the credential will be provided on the helper's stdin
-stream. The credential is split into a set of named attributes.
-Attributes are provided to the helper, one per line. Each attribute is
-specified by a key-value pair, separated by an `=` (equals) sign,
-followed by a newline. The key may contain any bytes except `=`,
-newline, or NUL. The value may contain any bytes except newline or NUL.
-In both cases, all bytes are treated as-is (i.e., there is no quoting,
-and one cannot transmit a value with newline or NUL in it). The list of
-attributes is terminated by a blank line or end-of-file.
-
-Git will send the following attributes (but may not send all of
-them for a given credential; for example, a `host` attribute makes no
-sense when dealing with a non-network protocol):
-
-`protocol`::
-
-	The protocol over which the credential will be used (e.g.,
-	`https`).
-
-`host`::
-
-	The remote hostname for a network credential.
-
-`path`::
-
-	The path with which the credential will be used. E.g., for
-	accessing a remote https repository, this will be the
-	repository's path on the server.
-
-`username`::
-
-	The credential's username, if we already have one (e.g., from a
-	URL, from the user, or from a previously run helper).
-
-`password`::
-
-	The credential's password, if we are asking it to be stored.
+stream. The exact format is the same as the input/output format of the
+`git credential` plumbing command (see the section `INPUT/OUTPUT
+FORMAT` in linkgit:git-credential[7] for a detailed specification).
 
 For a `get` operation, the helper should produce a list of attributes
 on stdout in the same format. A helper is free to produce a subset, or

Makefile

@@ -488,7 +488,6 @@ X =
 PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS))
 
 TEST_PROGRAMS_NEED_X += test-chmtime
-TEST_PROGRAMS_NEED_X += test-credential
 TEST_PROGRAMS_NEED_X += test-ctype
 TEST_PROGRAMS_NEED_X += test-date
 TEST_PROGRAMS_NEED_X += test-delta
@@ -836,6 +835,7 @@ BUILTIN_OBJS += builtin/commit-tree.o
 BUILTIN_OBJS += builtin/commit.o
 BUILTIN_OBJS += builtin/config.o
 BUILTIN_OBJS += builtin/count-objects.o
+BUILTIN_OBJS += builtin/credential.o
 BUILTIN_OBJS += builtin/describe.o
 BUILTIN_OBJS += builtin/diff-files.o
 BUILTIN_OBJS += builtin/diff-index.o

builtin.h

@@ -67,6 +67,7 @@ extern int cmd_commit(int argc, const char **argv, const char *prefix);
 extern int cmd_commit_tree(int argc, const char **argv, const char *prefix);
 extern int cmd_config(int argc, const char **argv, const char *prefix);
 extern int cmd_count_objects(int argc, const char **argv, const char *prefix);
+extern int cmd_credential(int argc, const char **argv, const char *prefix);
 extern int cmd_describe(int argc, const char **argv, const char *prefix);
 extern int cmd_diff_files(int argc, const char **argv, const char *prefix);
 extern int cmd_diff_index(int argc, const char **argv, const char *prefix);

builtin/credential.c

@@ -0,0 +1,31 @@
+#include "git-compat-util.h"
+#include "credential.h"
+#include "builtin.h"
+
+static const char usage_msg[] =
+	"git credential [fill|approve|reject]";
+
+int cmd_credential(int argc, const char **argv, const char *prefix)
+{
+	const char *op;
+	struct credential c = CREDENTIAL_INIT;
+
+	op = argv[1];
+	if (!op)
+		usage(usage_msg);
+
+	if (credential_read(&c, stdin) < 0)
+		die("unable to read credential from stdin");
+
+	if (!strcmp(op, "fill")) {
+		credential_fill(&c);
+		credential_write(&c, stdout);
+	} else if (!strcmp(op, "approve")) {
+		credential_approve(&c);
+	} else if (!strcmp(op, "reject")) {
+		credential_reject(&c);
+	} else {
+		usage(usage_msg);
+	}
+	return 0;
+}

contrib/mw-to-git/git-remote-mediawiki

@@ -26,9 +26,6 @@
 # - Git renames could be turned into MediaWiki renames (see TODO
 #   below)
 #
-# - login/password support requires the user to write the password
-#   cleartext in a file (see TODO below).
-#
 # - No way to import "one page, and all pages included in it"
 #
 # - Multiple remote MediaWikis have not been very well tested.
@@ -43,6 +40,8 @@ use encoding 'utf8';
 binmode STDERR, ":utf8";
 
 use URI::Escape;
+use IPC::Open2;
+
 use warnings;
 
 # Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced
@@ -72,9 +71,7 @@ my @tracked_categories = split(/[ \n]/, run_git("config --get-all remote.". $rem
 chomp(@tracked_categories);
 
 my $wiki_login = run_git("config --get remote.". $remotename .".mwLogin");
-# TODO: ideally, this should be able to read from keyboard, but we're
-# inside a remote helper, so our stdin is connect to git, not to a
-# terminal.
+# Note: mwPassword is discourraged. Use the credential system instead.
 my $wiki_passwd = run_git("config --get remote.". $remotename .".mwPassword");
 my $wiki_domain = run_git("config --get remote.". $remotename .".mwDomain");
 chomp($wiki_login);
@@ -151,28 +148,108 @@ while (<STDIN>) {
 
 ########################## Functions ##############################
 
+## credential API management (generic functions)
+
+sub credential_from_url {
+	my $url = shift;
+	my $parsed = URI->new($url);
+	my %credential;
+
+	if ($parsed->scheme) {
+		$credential{protocol} = $parsed->scheme;
+	}
+	if ($parsed->host) {
+		$credential{host} = $parsed->host;
+	}
+	if ($parsed->path) {
+		$credential{path} = $parsed->path;
+	}
+	if ($parsed->userinfo) {
+		if ($parsed->userinfo =~ /([^:]*):(.*)/) {
+			$credential{username} = $1;
+			$credential{password} = $2;
+		} else {
+			$credential{username} = $parsed->userinfo;
+		}
+	}
+
+	return %credential;
+}
+
+sub credential_read {
+	my %credential;
+	my $reader = shift;
+	my $op = shift;
+	while (<$reader>) {
+		my ($key, $value) = /([^=]*)=(.*)/;
+		if (not defined $key) {
+			die "ERROR receiving response from git credential $op:\n$_\n";
+		}
+		$credential{$key} = $value;
+	}
+	return %credential;
+}
+
+sub credential_write {
+	my $credential = shift;
+	my $writer = shift;
+	while (my ($key, $value) = each(%$credential) ) {
+		if ($value) {
+			print $writer "$key=$value\n";
+		}
+	}
+}
+
+sub credential_run {
+	my $op = shift;
+	my $credential = shift;
+	my $pid = open2(my $reader, my $writer, "git credential $op");
+	credential_write($credential, $writer);
+	print $writer "\n";
+	close($writer);
+
+	if ($op eq "fill") {
+		%$credential = credential_read($reader, $op);
+	} else {
+		if (<$reader>) {
+			die "ERROR while running git credential $op:\n$_";
+		}
+	}
+	close($reader);
+	waitpid($pid, 0);
+	my $child_exit_status = $? >> 8;
+	if ($child_exit_status != 0) {
+		die "'git credential $op' failed with code $child_exit_status.";
+	}
+}
+
 # MediaWiki API instance, created lazily.
 my $mediawiki;
 
 sub mw_connect_maybe {
 	if ($mediawiki) {
-	    return;
+		return;
 	}
 	$mediawiki = MediaWiki::API->new;
 	$mediawiki->{config}->{api_url} = "$url/api.php";
 	if ($wiki_login) {
-		if (!$mediawiki->login({
-			lgname => $wiki_login,
-			lgpassword => $wiki_passwd,
-			lgdomain => $wiki_domain,
-		})) {
-			print STDERR "Failed to log in mediawiki user \"$wiki_login\" on $url\n";
-			print STDERR "(error " .
-			    $mediawiki->{error}->{code} . ': ' .
-			    $mediawiki->{error}->{details} . ")\n";
-			exit 1;
+		my %credential = credential_from_url($url);
+		$credential{username} = $wiki_login;
+		$credential{password} = $wiki_passwd;
+		credential_run("fill", \%credential);
+		my $request = {lgname => $credential{username},
+			       lgpassword => $credential{password},
+			       lgdomain => $wiki_domain};
+		if ($mediawiki->login($request)) {
+			credential_run("approve", \%credential);
+			print STDERR "Logged in mediawiki user \"$credential{username}\".\n";
 		} else {
-			print STDERR "Logged in with user \"$wiki_login\".\n";
+			print STDERR "Failed to log in mediawiki user \"$credential{username}\" on $url\n";
+			print STDERR "  (error " .
+				$mediawiki->{error}->{code} . ': ' .
+				$mediawiki->{error}->{details} . ")\n";
+			credential_run("reject", \%credential);
+			exit 1;
 		}
 	}
 }

credential.c

@@ -191,7 +191,7 @@ static void credential_write_item(FILE *fp, const char *key, const char *value)
 	fprintf(fp, "%s=%s\n", key, value);
 }
 
-static void credential_write(const struct credential *c, FILE *fp)
+void credential_write(const struct credential *c, FILE *fp)
 {
 	credential_write_item(fp, "protocol", c->protocol);
 	credential_write_item(fp, "host", c->host);

credential.h

@@ -26,6 +26,7 @@ void credential_approve(struct credential *);
 void credential_reject(struct credential *);
 
 int credential_read(struct credential *, FILE *);
+void credential_write(const struct credential *, FILE *);
 void credential_from_url(struct credential *, const char *url);
 int credential_match(const struct credential *have,
 		     const struct credential *want);

git.c

@@ -351,6 +351,7 @@ static void handle_internal_command(int argc, const char **argv)
 		{ "commit-tree", cmd_commit_tree, RUN_SETUP },
 		{ "config", cmd_config, RUN_SETUP_GENTLY },
 		{ "count-objects", cmd_count_objects, RUN_SETUP },
+		{ "credential", cmd_credential, RUN_SETUP_GENTLY },
 		{ "describe", cmd_describe, RUN_SETUP },
 		{ "diff", cmd_diff },
 		{ "diff-files", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE },

t/lib-credential.sh

@@ -4,10 +4,20 @@
 # stdout and stderr should be provided on stdin,
 # separated by "--".
 check() {
+	credential_opts=
+	credential_cmd=$1
+	shift
+	for arg in "$@"; do
+		credential_opts="$credential_opts -c credential.helper='$arg'"
+	done
 	read_chunk >stdin &&
 	read_chunk >expect-stdout &&
 	read_chunk >expect-stderr &&
-	test-credential "$@" <stdin >stdout 2>stderr &&
+	if ! eval "git $credential_opts credential $credential_cmd <stdin >stdout 2>stderr"; then
+		echo "git credential failed with code $?" &&
+		cat stderr &&
+		false
+	fi &&
 	test_cmp expect-stdout stdout &&
 	test_cmp expect-stderr stderr
 }
@@ -41,7 +51,7 @@ reject() {
 		echo protocol=$2
 		echo host=$3
 		echo username=$4
-	) | test-credential reject $1
+	) | git -c credential.helper=$1 credential reject
 }
 
 helper_test() {
@@ -52,6 +62,8 @@ helper_test() {
 		protocol=https
 		host=example.com
 		--
+		protocol=https
+		host=example.com
 		username=askpass-username
 		password=askpass-password
 		--
@@ -74,6 +86,8 @@ helper_test() {
 		protocol=https
 		host=example.com
 		--
+		protocol=https
+		host=example.com
 		username=store-user
 		password=store-pass
 		--
@@ -85,6 +99,8 @@ helper_test() {
 		protocol=http
 		host=example.com
 		--
+		protocol=http
+		host=example.com
 		username=askpass-username
 		password=askpass-password
 		--
@@ -98,6 +114,8 @@ helper_test() {
 		protocol=https
 		host=other.tld
 		--
+		protocol=https
+		host=other.tld
 		username=askpass-username
 		password=askpass-password
 		--
@@ -112,6 +130,8 @@ helper_test() {
 		host=example.com
 		username=other
 		--
+		protocol=https
+		host=example.com
 		username=other
 		password=askpass-password
 		--
@@ -133,6 +153,9 @@ helper_test() {
 		host=path.tld
 		path=bar.git
 		--
+		protocol=http
+		host=path.tld
+		path=bar.git
 		username=askpass-username
 		password=askpass-password
 		--
@@ -150,6 +173,8 @@ helper_test() {
 		protocol=https
 		host=example.com
 		--
+		protocol=https
+		host=example.com
 		username=askpass-username
 		password=askpass-password
 		--
@@ -176,6 +201,8 @@ helper_test() {
 		host=example.com
 		username=user1
 		--
+		protocol=https
+		host=example.com
 		username=user1
 		password=pass1
 		EOF
@@ -184,6 +211,8 @@ helper_test() {
 		host=example.com
 		username=user2
 		--
+		protocol=https
+		host=example.com
 		username=user2
 		password=pass2
 		EOF
@@ -200,6 +229,8 @@ helper_test() {
 		host=example.com
 		username=user1
 		--
+		protocol=https
+		host=example.com
 		username=user1
 		password=askpass-password
 		--
@@ -213,6 +244,8 @@ helper_test() {
 		host=example.com
 		username=user2
 		--
+		protocol=https
+		host=example.com
 		username=user2
 		password=pass2
 		EOF
@@ -234,6 +267,8 @@ helper_test_timeout() {
 		protocol=https
 		host=timeout.tld
 		--
+		protocol=https
+		host=timeout.tld
 		username=askpass-username
 		password=askpass-password
 		--

t/t0300-credentials.sh

@@ -82,6 +82,9 @@ test_expect_success 'credential_fill passes along metadata' '
 	host=example.com
 	path=foo.git
 	--
+	protocol=ftp
+	host=example.com
+	path=foo.git
 	username=one
 	password=two
 	--
@@ -213,6 +216,8 @@ test_expect_success 'match configured credential' '
 	host=example.com
 	path=repo.git
 	--
+	protocol=https
+	host=example.com
 	username=foo
 	password=bar
 	--
@@ -225,6 +230,8 @@ test_expect_success 'do not match configured credential' '
 	protocol=https
 	host=bar
 	--
+	protocol=https
+	host=bar
 	username=askpass-username
 	password=askpass-password
 	--
@@ -239,6 +246,8 @@ test_expect_success 'pull username from config' '
 	protocol=https
 	host=example.com
 	--
+	protocol=https
+	host=example.com
 	username=foo
 	password=askpass-password
 	--
@@ -252,6 +261,8 @@ test_expect_success 'http paths can be part of context' '
 	host=example.com
 	path=foo.git
 	--
+	protocol=https
+	host=example.com
 	username=foo
 	password=bar
 	--
@@ -265,6 +276,9 @@ test_expect_success 'http paths can be part of context' '
 	host=example.com
 	path=foo.git
 	--
+	protocol=https
+	host=example.com
+	path=foo.git
 	username=foo
 	password=bar
 	--

test-credential.c

@@ -1,38 +0,0 @@
-#include "cache.h"
-#include "credential.h"
-#include "string-list.h"
-
-static const char usage_msg[] =
-"test-credential <fill|approve|reject> [helper...]";
-
-int main(int argc, const char **argv)
-{
-	const char *op;
-	struct credential c = CREDENTIAL_INIT;
-	int i;
-
-	op = argv[1];
-	if (!op)
-		usage(usage_msg);
-	for (i = 2; i < argc; i++)
-		string_list_append(&c.helpers, argv[i]);
-
-	if (credential_read(&c, stdin) < 0)
-		die("unable to read credential from stdin");
-
-	if (!strcmp(op, "fill")) {
-		credential_fill(&c);
-		if (c.username)
-			printf("username=%s\n", c.username);
-		if (c.password)
-			printf("password=%s\n", c.password);
-	}
-	else if (!strcmp(op, "approve"))
-		credential_approve(&c);
-	else if (!strcmp(op, "reject"))
-		credential_reject(&c);
-	else
-		usage(usage_msg);
-
-	return 0;
-}