Reliable ETA and progress percentage. #1963

LebedevRI · 2021-04-26T17:55:15Z

Reliable ETA and progress percentage.

This has been bugging me for years. :)
Count of finished edges isn't a great statistic, it isn't
really obvious if LLVM will take 8 minues to build, or 10 minutes.

But, it's actually pretty straight-forward to get some
more useful information. We already know how much time each edge
has taken, so we could just do the dumb thing, and assume that
every edge in the plan takes the same amount of time.

Or, we can do better. .ninja_log already contains
the historical data on how long each edge took to produce it's outs,
so we simply need to ensure that we populate edges with that info,
and then we can greatly improve our predictions.
The math is pretty simple i think.

This is largely a port of a similar change i did to LLVM LIT:
https://reviews.llvm.org/D99073

With this, i get something quite lovely:

llvm-project/build-Clang12$ NINJA_STATUS="[%f/%t %p %P][%w + %W] " /repositories/ninja/build-Clang-debug/ninja opt
[288/2527  11%   4%][00:27 + 08:52] Building CXX object lib/DebugInfo/CodeView/CMakeFiles/LLVMDebugInfoCodeView.dir/AppendingTypeTableBuilder.cpp.o

I hope people will find this useful, and it could be merged.
CC @nico

Please let me know which kinds of test coverage this needs?

Fix #115.

LebedevRI · 2021-04-27T17:47:27Z

So far, i'm aware of two caveats:

All times are in milliseconds. That may or may not affect the quality of prediction. Honestly, i'm not sure why they are in milliseconds. Even if they were in nanoseconds, int64_t would be enough for ~3 centuries. .ninja_log file size conservation?
The prediction becomes bogus if the last rebuild was recovered from ccache.
I guess we need to stop using previous times if the actual rate is very different from the predicted one.

jhasse · 2021-04-29T07:12:56Z

Can you add "Fix #115." to the commit message?

LebedevRI · 2021-04-29T10:47:20Z

FWIW i've fixed second point (by hardcoding some magic numbers).
First one i'm not so sure.

Will look into failing build.

mathstuf

Overall the code looks OK to me. One question I have is how this interacts with restat = 1 removing edges and dyndeps adding and removing edges from the plan during the build. Can tests be added?

doc/manual.asciidoc

LebedevRI · 2021-04-29T11:16:19Z

I have no idea what restat = 1 is, but this does gracefully handles adding/removing edges
(that haven't started yet i presume?) during the build. That was indeed important to handle.
See builder_->status_->EdgeAddedToPlan(edge); in Plan::EdgeWanted() e.g.

I haven't looked into that yet.

LebedevRI · 2021-05-01T13:52:26Z

Did some analysis. With clean build:

With rebuild from ccache:

For ccache the largest avg dt is 0.2s at 100% completion,
while for clean build the minimal avg dt is always larger than ~2.0s after ~5% completion.
So looks like we can say that we can reuse the previous timings if the curr/prev averages differ by less than ~10x.

ninja_log.clean-sheet.csv
ninja_log.ccache.csv

LebedevRI · 2021-06-16T11:57:20Z

(please don't merge. this is still broken when e.g. ccache is used)

LebedevRI · 2022-08-19T08:43:17Z

Hm, and i just realized this is still here.

Essentially, this functionality requires that the last execution time, and the time it will take this time,
are similar. If they are significantly different (e.g. if on the second run it's being restored from ccache,
or it was restored from ccache last time, but this time it didn't hit), prediction will just be wrong.

As far as i can tell, the best thing we can do, is to heuristically stop using last execution time
when the new time is significantly different. It kinda works, but not really.

Can codeowners here please comment if the functionality this PR provides is acceptable given that drawback?

LebedevRI · 2022-08-24T19:10:17Z

So i take, in other words this would be fine.
Let me rebase this...

LebedevRI · 2022-08-24T19:49:57Z

Okay, rebased.
I've tested it somewhat, and it does appear to be still working after the rebase.
What is needed to get this through the finish line this time? :)

LebedevRI · 2022-08-24T20:59:58Z

src/graph.h

+
+  // Historical info: how long did this edge take last time,
+  // as per .ninja_log, if known? Defaults to -1 if unknown.
+  int64_t prev_elapsed_time;


Yep, naked unitless data is a problem, i agree. I would think it would make sense to change
all relevant occurrences at once, so i'm not sure an one-off change is a win?

My main problem with that is that it (and some other changes) goes directly against the current code style,
and having uniform code is generally better, and can be uplifted all at once,
as opposed to having a weird mismatch of different coding styles.

But if the maintainers (@jhasse ? @nico ?) can chime in and review the change as a whole,
i can sure make those changes if they are deemed better by maintainers.

src/ninja.cc

src/status.cc

LebedevRI · 2022-10-19T00:41:20Z

Re build failure:
i wonder if instead of curl -L -O https://github.com/Kitware/CMake/releases/download/v3.16.4/cmake-3.16.4-Linux-x86_64.sh
it would be more stable to use https://apt.kitware.com/

doc/manual.asciidoc

src/status.cc

doc/manual.asciidoc

src/status.cc

LebedevRI · 2023-01-07T17:50:31Z

ping

LebedevRI · 2023-02-05T17:20:38Z

LebedevRI · 2023-04-04T01:33:59Z

LebedevRI · 2023-05-16T23:10:20Z

LebedevRI · 2023-06-09T21:13:20Z

@jhasse

LebedevRI · 2023-07-09T20:01:15Z

LebedevRI · 2023-08-05T19:07:32Z

LebedevRI · 2023-09-11T19:20:28Z

LebedevRI · 2023-10-09T21:00:06Z

mathstuf · 2023-10-10T00:18:50Z

Please do not repeatedly "poke" issues/PRs with content-less comments such as this. Note that there has been activity lately (mostly merging bug fixes and some CI updates). Hopefully there will be more time in the future for such feature PRs to get the attention the need and deserve, but "pings" really do not help.

However, given the CI changes lately, a rebase of this PR to adapt to the new configuration is likely useful.

This has been bugging me for *years*. :) Count of finished edges isn't a great statistic, it isn't really obvious if LLVM will take 8 minues to build, or 10 minutes. But, it's actually pretty straight-forward to get some more useful information. We already know how much time each edge has taken, so we could just do the dumb thing, and assume that every edge in the plan takes the same amount of time. Or, we can do better. `.ninja_log` already contains the historical data on how long each edge took to produce it's outs, so we simply need to ensure that we populate edges with that info, and then we can greatly improve our predictions. The math is pretty simple i think. This is largely a port of a similar change i did to LLVM LIT: https://reviews.llvm.org/D99073 With this, i get something quite lovely: ``` llvm-project/build-Clang12$ NINJA_STATUS="[%f/%t %p %P][%w + %W] " /repositories/ninja/build-Clang-debug/ninja opt [288/2527 11% 4%][00:27 + 08:52] Building CXX object lib/DebugInfo/CodeView/CMakeFiles/LLVMDebugInfoCodeView.dir/AppendingTypeTableBuilder.cpp.o ``` I hope people will find this useful, and it could be merged.

LebedevRI · 2024-03-14T15:34:32Z

ping.

jhasse · 2024-03-16T12:10:31Z

Thank you for your hard work and your patience! A great feature :)

The pings resulted in me unsubscribing and not being interested in having a look at this PR. Please refrain from doing that next time.

LebedevRI · 2024-03-16T12:45:33Z

Wow thank you @jhasse, i had almost given up on this thing :)
If there are issues with it, just let me know.

Yeah, believe me, i very VERY VERY much know the feeling :(
Sorry about that, i didn't quite know what the preferred etiquette was here, and thought i was helping.

LebedevRI force-pushed the reliable-eta branch 2 times, most recently from 3b578c8 to 556bc11 Compare April 26, 2021 22:41

LebedevRI force-pushed the reliable-eta branch from 556bc11 to 49cb784 Compare April 28, 2021 12:48

jhasse mentioned this pull request Apr 29, 2021

Add remaining time estimate to Ninja status. #1602

Closed

jhasse added the feature label Apr 29, 2021

This was referenced Apr 29, 2021

ENH: show remaining time #1360

Closed

Add a new NINJA_STATUS format: %E for ETA time. #1428

Closed

mathstuf reviewed Apr 29, 2021

View reviewed changes

doc/manual.asciidoc Outdated Show resolved Hide resolved

LebedevRI force-pushed the reliable-eta branch 2 times, most recently from f34e70d to 5c3ebdf Compare May 1, 2021 13:42

ilyapopov mentioned this pull request Aug 25, 2021

status: print roughly estimated remaining time #2017

Closed

This comment was marked as abuse.

Sign in to view

LebedevRI force-pushed the reliable-eta branch from 5c3ebdf to 14b88f4 Compare August 24, 2022 19:48

LebedevRI requested a review from mathstuf August 24, 2022 19:48

LebedevRI force-pushed the reliable-eta branch 3 times, most recently from 5f04500 to c0af2fc Compare August 24, 2022 20:33

This comment was marked as abuse.

Sign in to view

jonesmz reviewed Aug 24, 2022

View reviewed changes

src/ninja.cc Outdated Show resolved Hide resolved

jonesmz reviewed Aug 24, 2022

View reviewed changes

src/status.cc Show resolved Hide resolved

mathstuf reviewed Oct 21, 2022

View reviewed changes

doc/manual.asciidoc Outdated Show resolved Hide resolved

doc/manual.asciidoc Outdated Show resolved Hide resolved

LebedevRI force-pushed the reliable-eta branch 2 times, most recently from 044dc50 to 966de6a Compare October 24, 2022 20:49

LebedevRI commented Oct 24, 2022

View reviewed changes

src/status.cc Outdated Show resolved Hide resolved

LebedevRI force-pushed the reliable-eta branch from 966de6a to 4a0e197 Compare October 24, 2022 22:40

LebedevRI requested a review from mathstuf October 25, 2022 22:55

mathstuf reviewed Oct 25, 2022

View reviewed changes

doc/manual.asciidoc Outdated Show resolved Hide resolved

src/status.cc Outdated Show resolved Hide resolved

LebedevRI force-pushed the reliable-eta branch from 4a0e197 to e0d4ada Compare October 27, 2022 19:09

LebedevRI requested a review from mathstuf October 27, 2022 19:11

LebedevRI force-pushed the reliable-eta branch from e0d4ada to e3734cb Compare October 27, 2022 19:16

mathstuf approved these changes Oct 28, 2022

View reviewed changes

LebedevRI force-pushed the reliable-eta branch from e3734cb to 44a402f Compare October 10, 2023 21:50

jhasse added this to the 1.12.0 milestone Mar 16, 2024

jhasse merged commit 8d47b88 into ninja-build:master Mar 16, 2024
10 checks passed

LebedevRI deleted the reliable-eta branch March 16, 2024 12:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reliable ETA and progress percentage. #1963

Reliable ETA and progress percentage. #1963

LebedevRI commented Apr 26, 2021 •

edited

Loading

LebedevRI commented Apr 27, 2021

jhasse commented Apr 29, 2021

LebedevRI commented Apr 29, 2021

mathstuf left a comment

LebedevRI commented Apr 29, 2021

LebedevRI commented May 1, 2021 •

edited

Loading

LebedevRI commented Jun 16, 2021

LebedevRI commented Aug 19, 2022

This comment was marked as abuse.

LebedevRI commented Aug 24, 2022

LebedevRI commented Aug 24, 2022

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

LebedevRI Aug 24, 2022

This comment was marked as abuse.

LebedevRI Aug 24, 2022

This comment was marked as abuse.

LebedevRI commented Oct 19, 2022

LebedevRI commented Jan 7, 2023

LebedevRI commented Feb 5, 2023

LebedevRI commented Apr 4, 2023

LebedevRI commented May 16, 2023

LebedevRI commented Jun 9, 2023

LebedevRI commented Jul 9, 2023

LebedevRI commented Aug 5, 2023

LebedevRI commented Sep 11, 2023

LebedevRI commented Oct 9, 2023

mathstuf commented Oct 10, 2023

LebedevRI commented Mar 14, 2024

jhasse commented Mar 16, 2024 •

edited

Loading

LebedevRI commented Mar 16, 2024 •

edited

Loading

Reliable ETA and progress percentage. #1963

Reliable ETA and progress percentage. #1963

Conversation

LebedevRI commented Apr 26, 2021 • edited Loading

LebedevRI commented Apr 27, 2021

jhasse commented Apr 29, 2021

LebedevRI commented Apr 29, 2021

mathstuf left a comment

Choose a reason for hiding this comment

LebedevRI commented Apr 29, 2021

LebedevRI commented May 1, 2021 • edited Loading

LebedevRI commented Jun 16, 2021

LebedevRI commented Aug 19, 2022

This comment was marked as abuse.

LebedevRI commented Aug 24, 2022

LebedevRI commented Aug 24, 2022

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

LebedevRI Aug 24, 2022

Choose a reason for hiding this comment

This comment was marked as abuse.

LebedevRI Aug 24, 2022

Choose a reason for hiding this comment

This comment was marked as abuse.

LebedevRI commented Oct 19, 2022

LebedevRI commented Jan 7, 2023

LebedevRI commented Feb 5, 2023

LebedevRI commented Apr 4, 2023

LebedevRI commented May 16, 2023

LebedevRI commented Jun 9, 2023

LebedevRI commented Jul 9, 2023

LebedevRI commented Aug 5, 2023

LebedevRI commented Sep 11, 2023

LebedevRI commented Oct 9, 2023

mathstuf commented Oct 10, 2023

LebedevRI commented Mar 14, 2024

jhasse commented Mar 16, 2024 • edited Loading

LebedevRI commented Mar 16, 2024 • edited Loading

LebedevRI commented Apr 26, 2021 •

edited

Loading

LebedevRI commented May 1, 2021 •

edited

Loading

jhasse commented Mar 16, 2024 •

edited

Loading

LebedevRI commented Mar 16, 2024 •

edited

Loading