Commit Graph

95 Commits

Author SHA1 Message Date
Vincent van Ravesteijn
a3428205e6 t: Replace 'perl' by $PERL_PATH
GIT-BUILD-OPTIONS defines PERL_PATH to be used in the test suite. Only a
few tests already actually use this variable when perl is needed. The
other test just call 'perl' and it might happen that the wrong perl
interpreter is used.

This becomes problematic on Windows, when the perl interpreter that is
compiled and installed on the Windows system is used, because this perl
interpreter might introduce some unexpected LF->CRLF conversions.

This patch makes sure that $PERL_PATH is used everywhere in the test suite
and that the correct perl interpreter is used.

Signed-off-by: Vincent van Ravesteijn <vfr@lyx.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-06-12 09:30:41 -07:00
Jens Lehmann
cb8ad289c6 Consistently use "superproject" instead of "supermodule"
We fairly consistently say "superproject" and never "supermodule" these
days. But there are seven occurrences of "supermodule" left in the current
work tree. Three appear in Release Notes for 1.5.3 and 1.7.7, three in
test names and one in a C-code comment.

Replace all occurrences of "supermodule" outside of the Release Notes
(which shouldn't be changed after the fact) with "superproject" for
consistency.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-05-20 14:58:38 -07:00
Junio C Hamano
9a10117560 Merge branch 'js/fast-import-test-9300' into maint
By Johannes Sixt
* js/fast-import-test-9300:
  t9300-fast-import: avoid 'exit' in test_expect_success snippets
2012-05-11 11:17:49 -07:00
Johannes Sixt
05880b0222 t9300-fast-import: avoid 'exit' in test_expect_success snippets
Exiting from a for-loop early using '|| break' does not propagate the
failure code, and for this reason, the tests used just 'exit'. But this
ends the test script with 'FATAL: Unexpected exit code 1' in the case of
a failed test.

Fix this by moving the loop into a shell function, from which we can
simply return early.

While at it, modernize the style of the affected test cases.

Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-04-20 13:38:51 -07:00
Pete Wyckoff
06454cb9a3 fast-import: tighten parsing of datarefs
The syntax for the use of mark references in fast-import
demands either a SP (space) or LF (end-of-line) after
a mark reference.  Fast-import does not complain when garbage
appears after a mark reference in some cases.

Factor out parsing of mark references and complain if
errant characters are found.  Also be a little more careful
when parsing "inline" and SHA1s, complaining if extra
characters appear or if the form of the dataref is unrecognized.

Buggy input can cause fast-import to produce the wrong output,
silently, without error.  This makes it difficult to track
down buggy generators of fast-import streams.  An example is
seen in the last line of this commit command:

    commit refs/heads/S2
    committer Name <name@example.com> 1112912893 -0400
    data <<COMMIT
    commit message
    COMMIT
    from :1M 100644 :103 hello.c

It is missing a newline and should be:

    [...]
    from :1
    M 100644 :103 hello.c

What fast-import does is to produce a commit with the same
contents for hello.c as in refs/heads/S2^.  What the buggy
program was expecting was the contents of blob :103.  While
the resulting commit graph looked correct, the contents in
some commits were wrong.

Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-04-10 14:34:02 -07:00
Junio C Hamano
79efeae69d Merge branch 'jn/maint-fast-import-empty-ls' into maint
* jn/maint-fast-import-empty-ls:
  fast-import: don't allow 'ls' of path with empty components
  fast-import: leakfix for 'ls' of dirty trees
2012-03-26 12:10:25 -07:00
Junio C Hamano
5087aace2d two fixes for fast-import's "ls" command
Andrew Sayers noticed that the svn-fe | git fast-import pipeline
 mishandles a subversion history that copies the root directory to a
 sub-directory (e.g. doing `svn cp . trunk` to standardise your
 layout).  As David Barr explained, the bug arises when the following
 command is sent to git fast-import:
 
   'ls' SP ':1' SP LF
 
 Instead of reading back what is at the root of r1, it unconditionally
 reports the path as missing.
 
 After sleeping on it, here are two patches for 'maint'.  One plugs a
 memory leak.  The other ensures that trying to pass an empty path to
 the 'ls' command results in an error message that can help the
 frontend author instead of the silently broken conversion Andrew
 found.
 
 Then we can carefully add 'ls ""' support in 1.7.11.
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.12 (GNU/Linux)
 
 iQIbBAABCAAGBQJPWtgPAAoJEN/Gce6zM/olpU4P92DnurNCuUOsCIOVvEHT7YC+
 JdBou7wDKug2U0XH4yvTCglJZQ+zWpoobPoJlGu4c+RPWYGoUnOvc2jUmhFGD2r4
 jLlNhq3Uob4CSdEA8JLz8IOvgW41kc2LuUsZrr19dC3eS/19U72NQpD5nA958wZw
 xmvcu1DjhrsYHSdh4azg0Ccsp3PzesssCGJSckKoYpOZSlbznK3Dyn2KnLPZvAmx
 fF6uyPAZXKsKDOyVUPczIQ+mitb5YhOa3eFDcZk/EBM1LUCqbE0qmFyPhFqtE2pP
 f0BcG2MwXc6S8FIRTSkzxV3WP8Cl5+3Y3LX+aB9Y5Zl7Vx1LBdDEhEZS0DkJFW1u
 c/64Ge89FD/mSKpJfu8iZdga+qalyF1u5fPPFvOKV3p6UO1ou1RuIVEDMUhMzPhi
 244i7VnaPq232aMr8Jn4eAnpg+mZpOdDuqsJn4Q/BOpJ2D5UmKzja4LUZk2tkmgh
 EzGYLbs//TZXKjeNGH1rBdbW9lO0fU8oGRaTKiRwWTQVAJ48hUlOYfMBsWvUnvI6
 A5ar/iT8UthEc54OZIkBefjpFrwIXl3wG1iafDY8Z1rcnGsbH9MpNFLFxcoGQTBJ
 9A6ZyqL7dEKg2SWfESXf93xkF/fFZYl+0jFm+VNuJUzkrsvoi4tkxs/3qxaE74Kb
 zQtYbp/z+KWFh4eg830=
 =wrup
 -----END PGP SIGNATURE-----

Merge "two fixes for fast-import's 'ls' command" from Jonathan

Andrew Sayers noticed that the svn-fe | git fast-import pipeline
mishandles a subversion history that copies the root directory to a
sub-directory (e.g. doing `svn cp . trunk` to standardise your
layout).  As David Barr explained, the bug arises when the following
command is sent to git fast-import:

  'ls' SP ':1' SP LF

Instead of reading back what is at the root of r1, it unconditionally
reports the path as missing.

After sleeping on it, here are two patches for 'maint'.  One plugs a
memory leak.  The other ensures that trying to pass an empty path to
the 'ls' command results in an error message that can help the
frontend author instead of the silently broken conversion Andrew
found.

Then we can carefully add 'ls ""' support in 1.7.11.

* commit 'refs/pull-request-tags/jn/maint-fast-import-empty-ls':
  fast-import: don't allow 'ls' of path with empty components
  fast-import: leakfix for 'ls' of dirty trees
2012-03-16 08:19:18 -07:00
Jonathan Nieder
178e1deaae fast-import: don't allow 'ls' of path with empty components
As the fast-import manual explains:

	The value of <path> must be in canonical form. That is it must
	not:
	. contain an empty directory component (e.g. foo//bar is invalid),
	. end with a directory separator (e.g. foo/ is invalid),
	. start with a directory separator (e.g. /foo is invalid),

Unfortunately the "ls" command accepts these invalid syntaxes and
responds by declaring that the indicated path is missing.  This is too
subtle and causes importers to silently misbehave; better to error out
so the operator knows what's happening.

The C, R, and M commands already error out for such paths.

Reported-by: Andrew Sayers <andrew-git@pileofstuff.org>
Analysis-by: David Barr <davidbarr@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
2012-03-09 22:07:22 -06:00
Johannes Schindelin
b52612ed4f t9300: do not run --cat-blob-fd related tests on MinGW
As diagnosed by Johannes Sixt, msys.dll does not hand through file
descriptors > 2 to child processes, so these test cases cannot passes when
run through an MSys bash.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-10-14 22:46:45 -07:00
Dmitry Ivankov
0bc69881a6 fast-import: don't allow to note on empty branch
'reset' command makes fast-import start a branch from scratch. It's name
is kept in lookup table but it's sha1 is null_sha1 (special value).
'notemodify' command can be used to add a note on branch head given it's
name. lookup_branch() is used it that case and it doesn't check for
null_sha1. So fast-import writes a note for null_sha1 object instead of
giving a error.

Add a check to deny adding a note on empty branch and add a corresponding
test.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-09-22 13:30:59 -07:00
Dmitry Ivankov
2c9c8ee2de fast-import: don't allow to tag empty branch
'reset' command makes fast-import start a branch from scratch. It's name
is kept in lookup table but it's sha1 is null_sha1 (special value).
'tag' command can be used to tag a branch by it's name. lookup_branch()
is used it that case and it doesn't check for null_sha1. So fast-import
writes a tag for null_sha1 object instead of giving a error.

Add a check to deny tagging an empty branch and add a corresponding test.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-09-22 13:30:57 -07:00
Junio C Hamano
0dc691a4f3 Merge branch 'di/fast-import-tagging'
* di/fast-import-tagging:
  fast-import: allow to tag newly created objects
  fast-import: add tests for tagging blobs
2011-08-28 21:18:48 -07:00
Junio C Hamano
45792b64c1 Merge branch 'di/fast-import-deltified-tree'
* di/fast-import-deltified-tree:
  fast-import: prevent producing bad delta
  fast-import: add a test for tree delta base corruption
2011-08-28 21:18:47 -07:00
Junio C Hamano
0b98954975 Merge branch 'di/fast-import-ident'
* di/fast-import-ident:
  fsck: improve committer/author check
  fsck: add a few committer name tests
  fast-import: check committer name more strictly
  fast-import: don't fail on omitted committer name
  fast-import: add input format tests
2011-08-28 21:18:47 -07:00
Junio C Hamano
1d21112a94 Merge branch 'di/fast-import-doc'
* di/fast-import-doc:
  doc/fast-import: document feature import-marks-if-exists
2011-08-25 16:00:32 -07:00
Dmitry Ivankov
6c447f633c fast-import: allow to tag newly created objects
fast-import allows to tag objects by sha1 and to query sha1 of objects
being imported. So it should allow to tag these objects, make it do so.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-08-23 11:25:59 -07:00
Dmitry Ivankov
2efe38e7da fast-import: add tests for tagging blobs
fast-import allows to create an annotated tag that annotates a blob,
via mark or direct sha1 specification.

For mark it works, for sha1 it tries to read the object. It tries to
do so via read_sha1_file, and then checks the size to be at least 46.

That's weird, let's just allow to (annotated) tag any object referenced
by sha1. If the object originates from our packfile, we still fail though.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-08-23 11:25:56 -07:00
Dmitry Ivankov
3beb4fc461 doc/fast-import: document feature import-marks-if-exists
fast-import command-line option --import-marks-if-exists was introduced
in commit dded4f1 (fast-import: Introduce --import-marks-if-exists, 2011-01-15)

--import-marks option can be set via a "feature" command in a fast-import
stream and --import-marks-if-exists had support for such specification
from the very beginning too due to some shared codebase. Though the
documentation for this feature wasn't written in dded4f1.

Add the documentation for "feature import-marks-if-exists=<file>". Also add
a minimalistic test for it.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-08-17 16:51:49 -07:00
Dmitry Ivankov
8fb3ad76b1 fast-import: prevent producing bad delta
To produce deltas for tree objects fast-import tracks two versions
of tree's entries - base and current one. Base version stands both
for a delta base of this tree, and for a entry inside a delta base
of a parent tree. So care should be taken to keep it in sync.

tree_content_set cuts away a whole subtree and replaces it with a
new one (or NULL for lazy load of a tree with known sha1). It
keeps a base sha1 for this subtree (needed for parent tree). And
here is the problem, 'subtree' tree root doesn't have the implied
base version entries.

Adjusting the subtree to include them would mean a deep rewrite of
subtree. Invalidating the subtree base version would mean recursive
invalidation of parents' base versions. So just mark this tree as
do-not-delta me. Abuse setuid bit for this purpose.

tree_content_replace is the same as tree_content_set except that is
is used to replace the root, so just clearing base sha1 here (instead
of setting the bit) is fine.

[di: log message]

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-08-14 14:40:01 -07:00
Dmitry Ivankov
9a0edb79f2 fast-import: add a test for tree delta base corruption
fast-import is able to write imported tree objects in delta format.
It holds a tree structure in memory where each tree entry may have
a delta base sha1 assigned. When delta base data is needed it is
reconstructed from this in-memory structure. Though sometimes the
delta base data doesn't match the delta base sha1 so wrong or even
corrupt pack is produced.

Add a small test that produces a corrupt pack. It uses just tree
copy and file modification commands aside from the very basic commit
and blob commands.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-08-14 14:40:00 -07:00
Dmitry Ivankov
4b4963c0e1 fast-import: check committer name more strictly
The documentation declares following identity format:
(<name> SP)? LT <email> GT
where name is any string without LF and LT characters.
But fast-import just accepts any string up to first GT
instead of checking the whole format, and moreover just
writes it as is to the commit object.

git-fsck checks for [^<\n]* <[^<>\n]*> format. Note that the
space is mandatory. And the space quirk is already handled via
extending the string to the left when needed.

Modify fast-import input identity format to a slightly stricter
one - deny LF, LT and GT in both <name> and <email>. And check
for it.

This is stricter then git-fsck as fsck accepts "Name> <email>"
currently, but soon fsck check will be adjusted likewise.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-08-11 12:21:03 -07:00
Dmitry Ivankov
17fb00721b fast-import: don't fail on omitted committer name
fast-import format declares 'committer_name SP' to be optional in
'committer_name SP LT email GT'. But for a (commit) object SP is
obligatory while zero length committer_name is ok. git-fsck checks
that SP is present, so fast-import must prepend it if the name SP
part is omitted. It doesn't do so and thus for "LT email GT" ident
it writes a bad object.

Name cannot contain LT or GT, ident always comes after SP in fast-import.
So if ident starts with LT reuse the SP as if a valid 'SP LT email GT'
ident was passed.

This fixes a ident parsing bug for a well-formed fast-import input.
Though the parsing is still loose and can accept a ill-formed input.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-08-11 12:20:56 -07:00
Dmitry Ivankov
4cedb78cb5 fast-import: add input format tests
Documentation/git-fast-import.txt says that git-fast-import is strict
about it's input format. But committer/author field parsing is a bit
loose. Invalid values can be unnoticed and written out to the commit,
either with format-conforming input or with non-format-conforming one.

Add one passing and one failing test for empty/absent committer name
with well-formed input. And a failed test with unnoticed ill-formed
input.

Reported-by: SASAKI Suguru <sss.sonik@gmail.com>
Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-08-11 12:20:56 -07:00
Junio C Hamano
59d9ba869e Merge branch 'sr/transport-helper-fix'
* sr/transport-helper-fix: (21 commits)
  transport-helper: die early on encountering deleted refs
  transport-helper: implement marks location as capability
  transport-helper: Use capname for refspec capability too
  transport-helper: change import semantics
  transport-helper: update ref status after push with export
  transport-helper: use the new done feature where possible
  transport-helper: check status code of finish_command
  transport-helper: factor out push_update_refs_status
  fast-export: support done feature
  fast-import: introduce 'done' command
  git-remote-testgit: fix error handling
  git-remote-testgit: only push for non-local repositories
  remote-curl: accept empty line as terminator
  remote-helpers: export GIT_DIR variable to helpers
  git_remote_helpers: push all refs during a non-local export
  transport-helper: don't feed bogus refs to export push
  git-remote-testgit: import non-HEAD refs
  t5800: document some non-functional parts of remote helpers
  t5800: use skip_all instead of prereq
  t5800: factor out some ref tests
  ...
2011-08-01 15:00:14 -07:00
Sverre Rabbelier
be56862f19 fast-import: introduce 'done' command
Add a 'done' command that causes fast-import to stop reading from the
stream and exit.

If the new --done command line flag was passed on the command line
(or a "feature done" declaration included at the start of the stream),
make the 'done' command mandatory.  So "git fast-import --done"'s
input format will be prefix-free, making errors easier to detect when
they show up as early termination at some convenient time of the
upstream of a pipe writing to fast-import.

Another possible application of the 'done' command would to be allow a
fast-import stream that is only a small part of a larger encapsulating
stream to be easily parsed, leaving the file offset after the "done\n"
so the other application can pick up from there.  This patch does not
teach fast-import to do that --- fast-import still uses buffered input
(stdio).

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
Acked-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-07-19 11:17:47 -07:00
Dmitry Ivankov
7be8b3baba Fix typo: existant->existent
refs.c had a error message "Trying to write ref with nonexistant object".
And no tests relied on the wrong spelling.
Also typo was present in some test scripts internals, these tests still pass.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-06-16 10:33:50 -07:00
David Barr
8dc6a373d2 fast-import: add 'ls' command
Lazy fast-import frontend authors that want to rely on the backend to
keep track of the content of the imported trees _almost_ have what
they need in the 'cat-blob' command (v1.7.4-rc0~30^2~3, 2010-11-28).
But it is not quite enough, since

 (1) cat-blob can be used to retrieve the content of files, but
     not their mode, and

 (2) using cat-blob requires the frontend to keep track of a name
     (mark number or object id) for each blob to be retrieved

Introduce an 'ls' command to complement cat-blob and take care of the
remaining needs.  The 'ls' command finds what is at a given path
within a given tree-ish (tag, commit, or tree):

	'ls' SP <dataref> SP <path> LF

or in fast-import's active commit:

	'ls' SP <path> LF

The response is a single line sent through the cat-blob channel,
imitating ls-tree output.  So for example:

	FE> ls :1 Documentation
	gfi> 040000 tree 9e6c2b599341d28a2a375f8207507e0a2a627fe9	Documentation
	FE> ls 9e6c2b599341d28a2a375f8207507e0a2a627fe9 git-fast-import.txt
	gfi> 100644 blob 4f92954396e3f0f97e75b6838a5635b583708870	git-fast-import.txt
	FE> ls :1 RelNotes
	gfi> 120000 blob b942e49944	RelNotes
	FE> cat-blob b942e49944
	gfi> b942e49944 blob 32
	gfi> Documentation/RelNotes/1.7.4.txt

The most interesting parts of the reply are the first word, which is
a 6-digit octal mode (regular file, executable, symlink, directory,
or submodule), and the part from the second space to the tab, which is
a <dataref> that can be used in later cat-blob, ls, and filemodify (M)
commands to refer to the content (blob, tree, or commit) at that path.

If there is nothing there, the response is "missing some/path".

The intent is for this command to be used to read files from the
active commit, so a frontend can apply patches to them, and to copy
files and directories from previous revisions.

For example, proposed updates to svn-fe use this command in place of
its internal representation of the repository directory structure.
This simplifies the frontend a great deal and means support for
resuming an import in a separate fast-import run (i.e., incremental
import) is basically free.

Signed-off-by: David Barr <david.barr@cordelta.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Improved-by: Junio C Hamano <gitster@pobox.com>
Improved-by: Sverre Rabbelier <srabbelier@gmail.com>
2011-02-26 04:57:58 -06:00
Junio C Hamano
fc180d98a2 Merge branch 'rr/fi-import-marks-if-exists'
* rr/fi-import-marks-if-exists:
  fast-import: Introduce --import-marks-if-exists
2011-02-09 16:41:16 -08:00
Junio C Hamano
99e63ef24e Merge branch 'maint'
* maint:
  rebase -i: clarify in-editor documentation of "exec"
  tests: sanitize more git environment variables
  fast-import: treat filemodify with empty tree as delete
  rebase: give a better error message for bogus branch
  rebase: use explicit "--" with checkout

Conflicts:
	t/t9300-fast-import.sh
2011-01-27 10:27:49 -08:00
Junio C Hamano
5ce3258122 Merge branch 'jn/fast-import-empty-tree-removal' into maint
* jn/fast-import-empty-tree-removal:
  fast-import: treat filemodify with empty tree as delete
2011-01-27 10:23:53 -08:00
Jonathan Nieder
8fe533f686 fast-import: treat filemodify with empty tree as delete
Normal git processes do not allow one to build a tree with an empty
subtree entry without trying hard at it.  This is in keeping with the
general UI philosophy: git tracks content, not empty directories.

v1.7.3-rc0~75^2 (2010-06-30) changed that by making it easy to include
an empty subtree in fast-import's active commit:

	M 040000 4b825dc642 subdir

One can trigger this by reading an empty tree (for example, the tree
corresponding to an empty root commit) and trying to move it to a
subtree.  It is better and more closely analogous to 'git read-tree
--prefix' to treat such commands as requests to remove the subtree.

Noticed-by: David Barr <david.barr@cordelta.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-01-27 10:22:37 -08:00
Ramkumar Ramachandra
dded4f12a4 fast-import: Introduce --import-marks-if-exists
When a frontend uses a marks file to ensure its state persists between
runs, it may represent "clean slate" when bootstrapping with "no marks
yet". In such a case, feeding the last state with --import-marks and
saving the state after the current run with --export-marks would be a
natural thing to do.

The --import-marks option however errors out when the specified marks file
doesn't exist; this makes bootstrapping a bit difficult.  The location of
the marks file becomes backend-dependent when --relative-marks is in
effect, and the frontend cannot check for the existence of the file in
such a case.

The --import-marks-if-exists option does the same thing as --import-marks
but does not flag an error if the named file does not exist yet to help
these frontends.

Helped-by: Junio C Hamano <gitster@pobox.com>
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-01-18 07:07:01 -08:00
Jonathan Nieder
4de0bbd898 t9300: use perl "head -c" clone in place of "dd bs=1 count=16000" kluge
It is unfortunate to have to issue thousands of one-byte read calls to
work around dd's refusal to buffer input that would fill a block after
a short read (a3a6f4, 2010-12-13).  We could do better by using
"head -c", if it were available on all platforms we cared about.
Replace it with some simple perl.

While doing so, restructure 9300.114 to use a subshell instead of a
script.  Subshells can inherit functions (like the new head_c) from
the parent shell while external scripts cannot.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-19 13:51:09 -08:00
Junio C Hamano
914584266c Merge branch 'jn/fast-import-blob-access'
* jn/fast-import-blob-access:
  t9300: avoid short reads from dd
  t9300: remove unnecessary use of /dev/stdin
  fast-import: Allow cat-blob requests at arbitrary points in stream
  fast-import: let importers retrieve blobs
  fast-import: clarify documentation of "feature" command
  fast-import: stricter parsing of integer options

Conflicts:
	fast-import.c
2010-12-16 12:58:38 -08:00
Jonathan Nieder
a3a6f4c4cd t9300: avoid short reads from dd
dd is a thin wrapper around read(2).  As open group Issue 7 explains:

	It shall read the input one block at a time, using the specified
	input block size; it shall then process the block of data
	actually returned, which could be smaller than the requested
	block size.

Any short read --- for example from a pipe whose capacity cannot fill
a block --- results in that block being truncated.  As a result, the
first cat-blob test (9300.114) fails on Mac OS X, where the pipe
capacity is around 8 KiB.

Fix the test by using a block size of 1.  Each read will block until
the next byte of input is available.

It would be even nicer to use head -c which expresses the intention
more clearly.  Alas, IRIX "head" does not support the -c option.

Reported-by: Brian Gernhardt <brian@gernhardtsoftware.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-12 23:17:44 -08:00
Junio C Hamano
491e359c94 t9300: remove unnecessary use of /dev/stdin
We really shouldn't be using these funny /dev/* files that did not exist
in V7 UNIX in our tests when we do not have to.

Output from

    $ git grep -n -e /dev/ --and --not -e /dev/null t/

tells us that, aside from use of /dev/urandom in apache.conf used in http
tests, "dd if=/dev/stdin" added recently to t/t9300-fast-import.sh are the
only offenders, and "dd" reads from the standard input by default, so
removing them should be straightforward.

Reported-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-03 12:28:00 -08:00
Jonathan Nieder
777f80d742 fast-import: Allow cat-blob requests at arbitrary points in stream
The new rule: a "cat-blob" can be inserted wherever a comment is
allowed, which means at the start of any line except in the middle of
a "data" command.

This saves frontends from having to loop over everything they want to
commit in the next commit and cat-ing the necessary objects in
advance.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: David Barr <david.barr@cordelta.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-01 13:28:04 -08:00
David Barr
85c62395b1 fast-import: let importers retrieve blobs
New objects written by fast-import are not available immediately.
Until a checkpoint has been started and finishes writing the pack
index, any new blobs will not be accessible using standard git tools.

So introduce a new way to access them: a "cat-blob" command in the
command stream requests for fast-import to print a blob to stdout or a
file descriptor specified by the argument to --cat-blob-fd.  The value
for cat-blob-fd cannot be specified in the stream because that would
be a layering violation: the decision of where to direct a stream has
to be made when fast-import is started anyway, so we might as well
make the stream format is independent of that detail.

Output uses the same format as "git cat-file --batch".

Thanks to Sverre Rabbelier and Sam Vilain for guidance in designing
the protocol.

Based-on-patch-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: David Barr <david.barr@cordelta.com>
Acked-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-01 13:27:37 -08:00
Jonathan Nieder
a9ff277e58 fast-import: stricter parsing of integer options
Check the result from strtoul to avoid accepting arguments like
--depth=-1 and --active-branches=foo,bar,baz.

Requested-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-01 13:26:52 -08:00
Junio C Hamano
ed8298dc34 Merge branch 'jn/fast-import-fix'
* jn/fast-import-fix:
  fast-import: do not clear notes in do_change_note_fanout()
  t9300 (fast-import): another test for the "replace root" feature
  fast-import: tighten M 040000 syntax
  fast-import: filemodify after M 040000 <tree> "" crashes
2010-11-29 17:52:32 -08:00
Junio C Hamano
b3ff808b71 Merge branch 'en/and-cascade-tests'
* en/and-cascade-tests: (25 commits)
  t4124 (apply --whitespace): use test_might_fail
  t3404: do not use 'describe' to implement test_cmp_rev
  t3404 (rebase -i): introduce helper to check position of HEAD
  t3404 (rebase -i): move comment to description
  t3404 (rebase -i): unroll test_commit loops
  t3301 (notes): use test_expect_code for clarity
  t1400 (update-ref): use test_must_fail
  t1502 (rev-parse --parseopt): test exit code from "-h"
  t6022 (renaming merge): chain test commands with &&
  test-lib: introduce test_line_count to measure files
  tests: add missing &&, batch 2
  tests: add missing &&
  Introduce sane_unset and use it to ensure proper && chaining
  t7800 (difftool): add missing &&
  t7601 (merge-pull-config): add missing &&
  t7001 (mv): add missing &&
  t6016 (rev-list-graph-simplify-history): add missing &&
  t5602 (clone-remote-exec): add missing &&
  t4026 (color): remove unneeded and unchained command
  t4019 (diff-wserror): add lots of missing &&
  ...

Conflicts:
	t/t7006-pager.sh
2010-11-24 15:51:49 -08:00
Jonathan Nieder
a48fcd8369 tests: add missing &&
Breaks in a test assertion's && chain can potentially hide
failures from earlier commands in the chain.

Commands intended to fail should be marked with !, test_must_fail, or
test_might_fail.  The examples in this patch do not require that.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-11-09 11:59:49 -08:00
Jonathan Nieder
971728c853 t9300 (fast-import): another test for the "replace root" feature
Another test for the replace root feature.  One can imagine an
implementation for which R "some/subdir" "" would free some state
associated to the subdir and leave fast-import confused.

Luckily, git's is not such an implementation.

While at it, change the previous test to use C "some/subdir" ""
instead of R (i.e., test both syntaxes).

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-19 22:55:15 -07:00
Jonathan Nieder
3421578393 fast-import: tighten M 040000 syntax
When tree_content_set() is asked to modify the path "foo/bar/",
it first recurses like so:

	tree_content_set(root, "foo/bar/", sha1, S_IFDIR) ->
	 tree_content_set(root:foo, "bar/", ...) ->
	  tree_content_set(root:foo/bar, "", ...)

And as a side-effect of 2794ad5 (fast-import: Allow filemodify to set
the root, 2010-10-10), this last call is accepted and changes
the tree entry for root:foo/bar to refer to the specified tree.

That seems safe enough but let's reject the new syntax (we never meant
to support it) and make it harder for frontends to introduce pointless
incompatibilities with git fast-import 1.7.3.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-18 16:42:26 -07:00
Jonathan Nieder
5edde51018 fast-import: filemodify after M 040000 <tree> "" crashes
Until M 040000 <tree> "" syntax was introduced in commit 2794ad5
(fast-import: Allow filemodify to set the root, 2010-10-10), it
was impossible for the root entry to refer to an unloaded tree.
Update various functions to take that possibility into account.
Otherwise

	M 040000 <tree> ""
	M 100644 :1 "foo"

and similar commands (using D, C, or R after resetting the root
tree) segfault.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-18 16:41:27 -07:00
David Barr
2794ad5244 fast-import: Allow filemodify to set the root
v1.7.3-rc0~75^2 (Teach fast-import to import subtrees named by tree id,
2010-06-30) has a shortcoming - it doesn't allow the root to be set.
Extend this behaviour by allowing the root to be referenced as the
empty path, "".

For a command (like filter-branch --subdirectory-filter) that wants
to commit a lot of trees that already exist in the object db, writing
undeltified objects as loose files only to repack them later can
involve a significant amount of overhead.
(23% slow-down observed on Linux 2.6.35, worse on Mac OS X 10.6)

Fortunately we have fast-import (which is one of the only git commands
that will write to a pack directly) but there is not an advertised way
to tell fast-import to commit a given tree without unpacking it.

This patch changes that, by allowing

	M 040000 <tree id> ""

as a filemodify line in a commit to reset to a particular tree without
any need to parse it.  For example,

	M 040000 4b825dc642 ""

is a synonym for the deleteall command and the fast-import equivalent of

	git read-tree 4b825dc642

Signed-off-by: David Barr <david.barr@cordelta.com>
Commit-message-by: Jonathan Nieder <jrnieder@gmail.com>
Acked-by: Sverre Rabbelier <srabbelier@gmail.com>
Tested-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-13 15:10:31 -07:00
Junio C Hamano
347c47e61e Merge branch 'jl/maint-fix-test'
* jl/maint-fix-test:
  Several tests: cd inside subshell instead of around

Conflicts:
	t/t9600-cvsimport.sh
2010-09-06 16:46:36 -07:00
Jens Lehmann
fd4ec4f2bb Several tests: cd inside subshell instead of around
Fixed all places where it was a straightforward change from cd'ing into a
directory and back via "cd .." to a cd inside a subshell.

Found these places with "git grep -w "cd \.\.".

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-09-06 14:30:53 -07:00
Junio C Hamano
ebb561bcfc Merge branch 'jn/fast-import-subtree'
* jn/fast-import-subtree:
  Teach fast-import to import subtrees named by tree id
2010-08-18 12:14:41 -07:00
Raja R Harinath
7e7db5e452 fast-import: export correctly marks larger than 2^20-1
dump_marks_helper() has a bug when dumping marks larger than 2^20-1,
i.e., when the sparse array has more than two levels.  The bug was
that the 'base' counter was being shifted by 20 bits at level 3, and
then again by 10 bits at level 2, rather than a total shift of 20 bits
in this argument to the recursive call:

  (base + k) << m->shift

There are two ways to fix this correctly, the elegant:

  (base + k) << 10

and the one I chose due to edit distance:

  base + (k << m->shift)

Signed-off-by: Raja R Harinath <harinath@hurrynot.org>
Acked-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-08-11 10:45:15 -07:00