[Feature Request] Parallel downloads on upgrade

[chrisidefix]

brew processes the upgrade command sequentially. I propose to parallelise the download of formulas to reduce wait times when trying to upgrade multiple formulas at the same time.

Option 1:

• initiate all downloads at the same time and start installing packages as soon as the download completed (considering dependencies, etc and continue to avoid installing packages in parallel)

• this especially improves the update experience for people with solid internet access, but would probably delay the whole process for people on limited bandwidths (something I am more concerned about)

Option 2 (preferred):

• initiate the first download as done currently, but immediately start the download of the next package, while the first one is being installed

• this obviously could significantly reduce the upgrade wait time for people, who compile packages

OLD upgrade time line

    |-download-|-install-|download-|-install-|-download-|-install-|...

PROPOSED upgrade time line

    |-download-|-install-|
               |-download-|-install-|
                          |-download-|-install-|
                                     ...

[MikeMcQuaid]

This will cause some issues with how to produce output and, as you mentioned, people without solid internet access. I'd be open to having all the downloads happen up front but in that case it'll need to be a PR rather than a feature request, I'm afraid. This document should help and we're happy to walk you through anything else.

Thanks!

[chrisidefix]

Thanks, just in case that point didn't come across very well, this was actually supposed to help people (like me) without very fast internet access, because the current design actually drags out the upgrade process.

I understand the concerns about having consistent output and preferring serial execution, but in the end, I will still compile with make -j8, because it's about 8 times faster, even if the output is harder to read.

[chrisidefix]

PS: also agree that a PR is more suitable, but I didn't want to invest a bunch of time to eventually be told it's not going to be merged.

[MikeMcQuaid]

@chrisidefix Ok, I see that, thanks. Yeh, I think if the output situation could be handled this could be a good idea. I think we'd still probably prefer "download everything first and then install it".

[ELLIOTTCABLE]

Hi!

1. Is this definitely an “on upgrade only” thing, i.e. is there a reason parallel-downloading is only relevant to upgrade, as opposed to brew install a list_of various_packages, or even a set of dependencies for brew install a_big_package?

2. What, a little more precisely, are the concerns about output? I'd assume the major effort of such a branch would be adding any sort of curses-and-or-other-rawmode output to Homebrew, given that it appears to be entirely static / synchronous line-by-line output, right now (unless Homebrew is entirely missing any sort of background-task-queue and asynchronous functionality — I haven't had time to look at the codebase yet, but with a project of this age and size, I'm just assuming … :P)

   Am I assuming too much to think something like pnpm's output would be ideal for a verbose format? Given that Homebrew has no rawmode output right now (at least, that I've noticed), is there an explicit “static output” directive for the project that would prevent work like this being merged?

   

   (The work would also obviously contribute to a future ‘short’ format, à la yarn's current install output; or even something like yarn update-interactive.)

3. Is this still explicitly wanted / open for contributions? I'm an old hand at Ruby and curses (even maintained the ncurses-ruby codebase at one point, way back in the day — although I haven't written much since 2.0 … yeah, operative word “old’), and kinda on a FOSS-contributions kick at the moment; I might take a stab at it.

[reitermarkus]

1. Is this definitely an “on upgrade only” thing

I'd say this should ideally be done for every command that can download multiple things, i.e. fetch, upgrade, install.

2. What, a little more precisely, are the concerns about output?

The main problem is probably the output when building from source, or when there are caveats that need to be shown.

Am I assuming too much to think something like pnpm's output would be ideal for a verbose format?

For option 1, this would work perfectly. For option 2, the caveats could be either show in the listing or afterwards, but the problem when building from source remains.

3. Is this still explicitly wanted / open for contributions?

This could potentially speed up downloads quite a bit, so I'd say give it a go!

[ELLIOTTCABLE]

Oh, I personally don't at all think the behaviour for installation should be changed — there's so many complications, there (dependencies changing how other packages install, etc). A package-manager would literally have to be designed around doing so.

I only suggest doing the download stage, initially, in parallel, for for the fully-resolved tree of dependencies — and then proceeding exactly as Homebrew does now, for installation / output / debugging / caveats … does this sound like a good plan?

[MikeMcQuaid]

I only suggest doing the download stage, initially, in parallel, for for the fully-resolved tree of dependencies — and then proceeding exactly as Homebrew does now, for installation / output / debugging / caveats … does this sound like a good plan?

👍

[tolgraven]

Yeah, since download stage-only would be infinitely less complex, and anything else very much would approach diminishing returns territory even without all the possible gotchas, the former is clearly the way to go.

As far as output, I would propose to simply limit shown progress/output to the currently installing package. Can't see any reason why that wouldn't solve any and all possible complications.
Basically, don't bother indicating that the next package is downloading, just do it. If it's down by the time the previous installation finishes, simply move on to installing the next and no download output is needed anyways. If not, simply show the meter just how it's displayed now. If something went wrong, just barf errors as before.

Though I don't know how things are setup currently, seems to me this shouldn't have to involve further changes than a simple switch to mass-spawning the curl processes, redirecting their output and watching it, even maintaining the current parser. Ought to be simply since no actual processing is parallelised...

@ELLIOTTCABLE have you done any work on this yet? If not I think I'll have a go.

[MikeMcQuaid]

As far as output, I would propose to simply limit shown progress/output to the currently installing package. Can't see any reason why that wouldn't solve any and all possible complications.
Basically, don't bother indicating that the next package is downloading, just do it. If it's down by the time the previous installation finishes, simply move on to installing the next and no download output is needed anyways. If not, simply show the meter just how it's displayed now. If something went wrong, just barf errors as before.

This seems like a sensible approach. Alternatively, you could at least print a message saying the download is starting in the background and otherwise not display it.

If not I think I'll have a go.

There's no open PR so: please do.