Historically, the diff machinery for rename detection had a
hardcoded limit of 32k paths; this is being lifted to allow users
trade cycles with a (possibly) easier to read result.
* en/rename-progress:
diffcore-rename: make diff-tree -l0 mean -l<large>
sequencer: show rename progress during cherry picks
diff: remove silent clamp of renameLimit
progress: fix progress meters when dealing with lots of work
sequencer: warn when internal merge may be suboptimal due to renameLimit
Since 180a9f2268 (provide a facility for "delayed" progress
reporting, 2007-04-20), the progress code has allowed
callers to skip showing progress if they have reached a
percentage-threshold of the total work before the delay
period passes.
But since 8aade107dd (progress: simplify "delayed" progress
API, 2017-08-19), that parameter is not available to outside
callers (we always passed zero after that commit, though
that was corrected in the previous commit to "100%").
Let's drop the threshold code, which never triggers in
any meaningful way.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Commit 8aade107dd (progress: simplify "delayed" progress
API, 2017-08-19) dropped the parameter by which callers
could say "show my progress only if I haven't passed M%
progress after N seconds". The intent was to just show
nothing for 2 seconds, and then always progress after that.
But we flipped the logic in the wrapper: it sets M=0,
meaning that we'd almost _never_ show progress after 2
seconds, since we'd generally have made some progress. This
should have been 100%, not 0%.
We were fooled by existing calls like:
start_progress_delay("foo", 0, 0, 2);
which behaved this way. The trick is that the first "0"
there is "how many items total", and there zero means "we
don't know". And without knowing that, we cannot compute a
completed percent at all, and we ignored the threshold
parameter entirely! Modeling our wrapper after that broke
callers which pass a non-zero value for "total".
We can switch to the intended behavior by using "100" in the
wrapper call.
Reported-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The possibility of setting merge.renameLimit beyond 2^16 raises the
possibility that the values passed to progress can exceed 2^32.
Use uint64_t, because it "ought to be enough for anybody". :-)
Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We used to expose the full power of the delayed progress API to the
callers, so that they can specify, not just the message to show and
expected total amount of work that is used to compute the percentage
of work performed so far, the percent-threshold parameter P and the
delay-seconds parameter N. The progress meter starts to show at N
seconds into the operation only if we have not yet completed P per-cent
of the total work.
Most callers used either (0%, 2s) or (50%, 1s) as (P, N), but there
are oddballs that chose more random-looking values like 95%.
For a smoother workload, (50%, 1s) would allow us to start showing
the progress meter earlier than (0%, 2s), while keeping the chance
of not showing progress meter for long running operation the same as
the latter. For a task that would take 2s or more to complete, it
is likely that less than half of it would complete within the first
second, if the workload is smooth. But for a spiky workload whose
earlier part is easier, such a setting is likely to fail to show the
progress meter entirely and (0%, 2s) is more appropriate.
But that is merely a theory. Realistically, it is of dubious value
to ask each codepath to carefully consider smoothness of their
workload and specify their own setting by passing two extra
parameters. Let's simplify the API by dropping both parameters and
have everybody use (0%, 2s).
Oh, by the way, the percent-threshold parameter and the structure
member were consistently misspelled, which also is now fixed ;-)
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The values in struct throughput are only updated every 0.5 seconds. If
we're all done before that time span then the final update will show a
rate of 0 bytes/s, which is misleading if some bytes had been handled.
Remember the start time and show the total throughput instead.
And avoid division by zero by enforcing a minimum time span value of 1
(unit: 1/1024th of a second). That makes the resulting rate an
underestimation, but it's closer to the actual value than the currently
shown 0 bytes/s.
Reported-by: 積丹尼 Dan Jacobson <jidanni@jidanni.org>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Simplify code by replacing buffer allocation with a call to xstrfmt().
Signed-off-by: Maxim Moseychuk <franchesko.salias.hudro.pedros@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We frequently allocate strings as xmalloc(len + 1), where
the extra 1 is for the NUL terminator. This can be done more
simply with xmallocz, which also checks for integer
overflow.
There's no case where switching xmalloc(n+1) to xmallocz(n)
is wrong; the result is the same length, and malloc made no
guarantees about what was in the buffer anyway. But in some
cases, we can stop manually placing NUL at the end of the
allocated buffer. But that's only safe if it's clear that
the contents will always fill the buffer.
In each case where this patch does so, I manually examined
the control flow, and I tried to err on the side of caution.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The usual arguments for using xsnprintf over sprintf apply,
but this case is a little tricky. We print to a fixed-size
buffer if we have room, and otherwise to an allocated
buffer. So there should be no overflow here, but it is still
good to communicate our intention, as well as to check our
earlier math for how much space the string will need.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Coverity noticed that we strncpy() into a fixed-size buffer
without making sure that it actually ended up
NUL-terminated. This is unlikely to be a bug in practice,
since throughput strings rarely hit 32 characters, but it
would be nice to clean it up.
The most obvious way to do so is to add a NUL-terminator.
But instead, this patch switches the fixed-size buffer out
for a strbuf. At first glance this seems much less
efficient, until we realize that filling in the fixed-size
buffer is done by writing into a strbuf and copying the
result!
By writing straight to the buffer, we actually end up more
efficient:
1. We avoid an extra copy of the bytes.
2. Rather than malloc/free each time progress is shown, we
can strbuf_reset and use the same buffer each time.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
progress: treat "no terminal" as being in the foreground
Commit 85cb890 (progress: no progress in background,
2015-04-13) avoids sending progress from background
processes by checking that the process group id of the
current process is the same as that of the controlling
terminal.
If we don't have a terminal, however, this check never
succeeds, and we print no progress at all (until the final
"done" message). This can be seen when cloning a large
repository; instead of getting progress updates for
"counting objects", it will appear to hang then print the
final count.
We can fix this by treating an error return from tcgetpgrp()
as a signal to show the progress.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Disable the display of the progress if stderr is not the
current foreground process.
Still display the final result when done.
Signed-off-by: Luke Mewburn <luke@mewburn.net>
Acked-by: Nicolas Pitre <nico@fluxnic.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Calculating duration from a single uint64_t is simpler than from a struct
timeval. Change throughput measurement from gettimeofday() to
getnanotime().
Also calculate misec only if needed, and change integer division to integer
multiplication + shift, which should be slightly faster.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Humanization of downloaded size is done in the same function as text
formatting in 'process.c'. The code cannot be reused easily elsewhere.
Separate text formatting from size simplification and make the
function public in strbuf so that it can easily be used by other
callers.
We now can use strbuf_humanise_bytes() for both downloaded size and
download speed calculation. One of the drawbacks is that speed will
now look like this when download is stalled: "0 bytes/s" instead of
"0 KiB/s".
Signed-off-by: Antoine Pelisse <apelisse@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Switch to MiB/s when the connection is fast enough (i.e. on a LAN).
Signed-off-by: Nicolas Pitre <nico@fluxnic.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Due to problems at cam.org, my nico@cam.org email address is no longer
valid. From now on, nico@fluxnic.net should be used instead.
Signed-off-by: Nicolas Pitre <nico@fluxnic.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Often the throughput output is requested when the data read so far is
one smaller than multiple of 1024; because 1023/1024 is ~0.999, it often
shows up as 0.99 because the code currently truncates.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Dynamically sized arrays are gcc and C99 construct. Using them hurts
portability to older compilers, although using them is nice in this case
it is not desirable. This patch removes the only use of the construct
in stop_progress_msg(); the function is about writing out a single line
of a message, and the existing callers of this function feed messages
of only bounded size anyway, so use of dynamic array is simply overkill.
Signed-off-by: Boyd Lynn Gerber <gerberb@zenez.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This will make progress display from pack-objects (invoked via
upload-pack) more responsive on platforms with an implementation
of stdio whose stderr is line buffered.
The standard error stream is defined to be merely "not fully
buffered"; this is different from "unbuffered". If the
implementation of the stdio library chooses to make it line
buffered, progress reports that end with CR but do not contain
LF will accumulate in the stdio buffer before written out. A
fflush() after each progress display gives a nice continuous
progress.
Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the same spirit of prettifying Git's output display for mere mortals,
here's a simple extension to the progress API allowing for a final
message to be provided when terminating a progress line, and use it for
the display of the number of objects needed to complete a thin pack,
saving yet one more line of screen display.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The minimum delay of 1/2 sec between successive throughput updates might
not have been elapsed when display_throughput() is called for the last
time, potentially making the display of total transferred bytes not
right when progress is said to be done.
Let's force an update of the throughput display as well when the
progress is complete. As a side effect, the total transferred will
always be displayed even if the actual transfer rate doesn't have time
to kickin.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The throughput display needs a delay period before accounting and
displaying anything. Yet it might be called after some amount of data
has already been transferred. The display of total data is therefore
accounted late and therefore smaller than the reality.
Let's call display_throughput() with an absolute amount of transferred
data instead of a relative number, and let the throughput code find the
relative amount of data by itself as needed. This way the displayed
total is always exact.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Right now it is infeasible to offer to the user a reasonable concept
of when a clone will be complete as we aren't able to come up with
the final pack size until after we have actually transferred the
entire thing to the client. However in many cases users can work
with a rough rule-of-thumb; for example it is somewhat well known
that git.git is about 16 MiB today and that linux-2.6.git is over
120 MiB.
We now show the total amount of data we have transferred over
the network as part of the throughput meter, organizing it in
"human friendly" terms like `ls -h` would do. Users can glance at
this, see that the total transferred size is about 3 MiB, see the
throughput of X KiB/sec, and determine a reasonable figure of about
when the clone will be complete, assuming they know the rough size
of the source repository or are able to obtain it.
This is also a helpful indicator that there is progress being made
even if we stall on a very large object. The thoughput meter may
remain relatively constant and the percentage complete and object
count won't be changing, but the total transferred will be increasing
as additional data is received for this object.
[from an initial proposal from Shawn O. Pearce]
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Currently the progress/throughput display update happens only through
display_progress(). If the progress based on object count remains
unchanged because a large object is being received, the latest throughput
won't be displayed. The display update should occur through
display_throughput() as well.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This adds the ability for the progress code to also display transfer
throughput when that makes sense.
The math was inspired by commit c548cf4ee0
from Linus.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This allows for better management of progress "object" existence,
as well as making the progress display implementation more independent
from its callers.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Each progress can be on a single line instead of two.
[sp: Changed "Checking files out" to "Checking out files" at
Johannes Sixt's suggestion as it better explains the
action that is taking place]
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This allows for progress to be displayed only if the progress has not
reached a specified percentage treshold within a given delay in seconds.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
If the progress bar ends up in a box, better provide a title for it too.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Instead of having this code duplicated in multiple places, let's have
a common interface for progress display. If someday someone wishes to
display a cheezy progress bar instead then only one file will have to
be changed.
Note: I left merge-recursive.c out since it has a strange notion of
progress as it apparently increase the expected total number as it goes.
Someone with more intimate knowledge of what that is supposed to mean
might look at converting it to the common progress interface.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
