Implement concurrent downloads in brew fetch.

[reitermarkus]

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same change?
• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your changes? Here's an example.
• Have you successfully run brew style with your changes locally?
• Have you successfully run brew typecheck with your changes locally?
• Have you successfully run brew tests with your changes locally?

---

$ hyperfine 'brew fetch --force --build-from-source azure-cli --concurrency 1'
Benchmark 1: brew fetch --force --build-from-source azure-cli --concurrency 1
  Time (mean ± σ):     63.889 s ±  2.812 s    [User: 16.251 s, System: 14.726 s]
  Range (min … max):   61.079 s … 70.527 s    10 runs

$ hyperfine 'brew fetch --force --build-from-source azure-cli --concurrency 2'
Benchmark 1: brew fetch --force --build-from-source azure-cli --concurrency 2
  Time (mean ± σ):     35.500 s ±  3.112 s    [User: 15.205 s, System: 14.819 s]
  Range (min … max):   30.537 s … 40.762 s    10 runs

concurrent-downloads.mov

[MikeMcQuaid]

Makes sense to me so far! Some questions:

• what made you decide on concurrent-ruby and whirly gems?
• does this use threads or ractors?
• does this slow down/add additional requires to a brew help command at all?

Thanks!

[reitermarkus]

what made you decide on concurrent-ruby and whirly gems?

concurrent-ruby because I have been using it for parallel brew fetch in my dotfiles for years, whirly to get at least some sort of progress indicator. The end goal is to have a progress bar, but for that all download strategies need to support progress updates first.

does this use threads or ractors?

Threads. From reading ruby-concurrency/concurrent-ruby#899, it seems ractors are quite limited in what they can do. In any case, network is the bottleneck here.

does this slow down/add additional requires to a brew help command at all?

No, the additional requires are only called when the thread pool is created, i.e. only if the downloads are actually started.

[reitermarkus]

Improved output which shows all concurrent downloads at once.

concurrent-downloads.mov

[MikeMcQuaid]

Looks great! A few tiny nits here and then can self-merge. Nice work @reitermarkus!

[MikeMcQuaid]

Looks good when comments addressed.

[MikeMcQuaid]

@reitermarkus converted to draft until you're ready to merge

[MikeMcQuaid]

@reitermarkus do you have any ETA on this being mergeable?

[Bo98]

Looks ok though I have a few questions that might have been from me reading things in the wrong order as there's a few different things going on.

[MikeMcQuaid]

Probable makes sense to merge this and open an issue for the things still missing.

@reitermarkus Sounds good. Could you address @Bo98's comments, consider adding an opt-in undocumented environment variable (HOMEBREW_FETCH_PARALLEL or something, could do in a follow-up PR instead) and then self-merge?

[MikeMcQuaid]

consider adding an opt-in undocumented environment variable (HOMEBREW_FETCH_PARALLEL or something, could do in a follow-up PR instead)

I forgot: you can do this for switch but not flag so: let's hold off for now.

---

@reitermarkus good to merge when @Bo98 is ✅
@Bo98 given this is "feature flagged": please mainly just check for if you have concerns about breaking existing functionality. Any issues with the new functionality we can fix forward.

[reitermarkus]

I forgot: you can do this for switch but not flag so: let's hold off for now.

It's hidden for now anyways, i.e. undocumented and unsupported.

[carlocab]

Tested brew fetch locally with and without --concurrency. Seems to work well.

[MikeMcQuaid]

We've just made a new tag so let's try to land this as it's now blocking several other PRs.

Please feel free to make follow-up PRs to clean this up a bit, anyone/everyone.