feat: parallel docker-compose pull, improve ddev debug download-images, fixes #7163

[stasadev]

The Issue

• Enable parallel download via docker compose during pre-pull of images #7163

How This PR Solves The Issue

• Removes docker pull completely
• Uses docker-compose pull everywhere
• Refactors ddev debug download-images:
  • Make it always pull, even if the image exists
  • Add optional project arg
  • Add --all flag to pull everything for all projects

Manual Testing Instructions

Speed comparison for pulling basic images (Linux AMD64):

ddev delete images --all --yes && docker builder prune -f
start=$(date +%s%N)
ddev debug download-images
echo "Elapsed time: $((($(date +%s%N) - start) / 1000000)) ms"

Before (three retries):

Elapsed time: 46599 ms
Elapsed time: 47083 ms
Elapsed time: 53672 ms

After (three retries):

Elapsed time: 25162 ms
Elapsed time: 24964 ms
Elapsed time: 22870 ms

---

Check out:

ddev debug download-images

cd d11 && ddev debug download-images

cd ~ && ddev debug download-images d11

ddev debug download-images --all

Automated Testing Overview

Release/Deployment Notes

[github-actions]

Download the artifacts for this pull request:

• all-ddev-executables.zip
• ddev-macos-amd64.zip
• ddev-macos-arm64.zip
• ddev-linux-arm64.zip
• ddev-linux-amd64.zip
• ddev-windows-amd64.zip
• ddev-windows-amd64-installer.zip
• ddev-windows-arm64.zip
• ddev-windows-arm64-installer.zip

See Testing a PR.

[rfay]

Congratulations! That's significant speed improvement!

[stasadev]

Regarding implementing pull on ddev start for existing images:

• feat: use docker-compose to parallel download images, fixes #7163 #7217 (comment)

After thinking about it, I would prefer pulling once per day on ddev start rather than on every ddev start.

But I think this is out of scope for this PR.

---

Key change here:

• ddev debug download-images always pulls
• ddev start works as before, no pull for existing images

[tyler36]

Great work.

I see solid speed improvements

Results

v1.24.7
Elapsed time: 72134 ms
Elapsed time: 72152 ms
Elapsed time: 67262 ms

v1.24.7-9-g249c4e63e
Elapsed time: 34329 ms
Elapsed time: 49075 ms
Elapsed time: 45244 ms

OS: Ubuntu (WSL2 on Win11)
Docker: Docker Desktop (4.42.0)

[tyler36]

I really appreciate the display tweak too. It adds a great "polished" feel.

[rfay]

I have terrible internet today so bad testing environment, but will get to this before long.

[rfay]

I got stymied by bad internet testing this. I like the beautiful new presentation, but a couple of comments:

• Instead of saying "pulled" with a checkmark, could it say the image that was pulled?
• There seems to be a leftover "lling" from the pull in the very nice presentation. (This was on a ddev config --auto in a project that hadn't been started)

[stasadev]

• Instead of saying "pulled" with a checkmark, could it say the image that was pulled?
• There seems to be a leftover "lling" from the pull in the very nice presentation. (This was on a ddev config --auto in a project that hadn't been started)

Thank you! Fixed both issues.

[rfay]

It's looking good!

More nits... The name given isn't really the name of the pulled image. For example, we didn't pull ddev-hobobiker-xhgui or ddev-d11simple-web

Downloading images for all projects
[+] Pulling 45/75
[+] Pulling 73/75                                                   1.3s
 ✔ busybox Pulled                                                   1.3s
 ✔ ddev-router Pulled                                               1.1s  ✔ ddev-t3v13-db Pulled                                            23.6s
 ✔ ddev-utilities Pulled                                            1.0s
 ✔ ddev-hobobiker-xhgui Pulled                                      1.0s
 ⠋ ddev-d11simple-web [⣤] 202.6MB / 395MB   Pulling                61.1s
 ✔ ddev-d11-db Pulled                                               1.1s
 ✔ ddev-d11-redis Pulled                                           31.9s
 ✔ ddev-hobobiker-db Pulled                                        48.7s
 ✔ ddev-ssh-agent Pulled                                            1.1s

[rfay]

Did you already check this out for offline usage? I'll try to give it a shot.

[rfay]

Comprehensive PR Analysis (Claude Code Review)

Overview

This PR implements parallel Docker image pulling to significantly improve DDEV's image download performance, addressing issue #7163. The implementation shows impressive performance gains - roughly 50% faster image downloads (from ~47s to ~24s).

Issue Context

Issue #7163 requested enabling parallel downloads via Docker Compose during pre-pull of images. The original approach downloaded images sequentially, causing slow startup times on first boot or when Docker images were pruned.

Key Changes

1. Enhanced ddev debug download-images Command

• Added optional project argument: ddev debug download-images [project]
• Added --all flag to download images for all projects
• Now always pulls images even if they exist locally when specified

2. Refactored Image Pulling Architecture

• Replaced individual docker pull commands with docker-compose pull for parallel execution
• Created new pkg/dockerutil/composeutils.go with PullImages() function
• Updated PullBaseContainerImages() and PullContainerImages() to use new parallel approach

3. Improved Docker Compose Integration

• Enhanced ComposeCmd and ComposeWithStreams to accept YAML directly via stdin
• Added CreateComposeProject() helper function
• Moved utility functions like EscapeDollarSign() to pkg/util/yamltools.go

4. Data Structure Changes

• Changed FindAllImages() return type from []string to map[string]string for service-to-image mapping
• Updated FindNotOmittedImages() similarly for consistency

Function Moves and Refactoring Analysis

The PR includes several strategic function moves to improve code organization:

1. escapeDollarSign → util.EscapeDollarSign

• Location: pkg/ddevapp/compose_yaml.go → pkg/util/yamltools.go
• Changes: ✅ Identical implementation - only renamed from private to public (capitalized)
• Rationale: Better placement in utility package for broader reuse

2. createComposeProject → dockerutil.CreateComposeProject

• Location: pkg/ddevapp/compose_yaml.go → pkg/dockerutil/composeutils.go
• Changes: ✅ Identical implementation - only renamed from private to public (capitalized)
• Rationale: More logical placement in Docker utilities package

3. Pull Function - Completely Refactored

• Location: pkg/dockerutil/dockerutils.go → pkg/dockerutil/composeutils.go
• Original: Used docker pull command directly with existence check
• New: Uses PullImages() function which leverages docker-compose pull for consistency
• Rationale: Aligns with new parallel pulling architecture

Technical Implementation

The core innovation is creating temporary Docker Compose projects with only image definitions and using docker-compose pull for parallel downloads. This leverages Docker Compose's built-in parallelization rather than sequential pulls.

Before: Sequential docker pull image1 && docker pull image2 && ...
After: Parallel docker-compose pull with dynamically generated compose file

Performance Impact

• Speed improvement: ~50% faster (47s → 24s for basic images)
• User experience: Better progress indication with compose pull output
• Resource efficiency: More efficient network utilization through parallelization

Code Quality Assessment

Refactoring Excellence:

• Functions moved to more appropriate packages (util for general utilities, dockerutil for Docker-specific operations)
• Previously private functions made public following Go naming conventions
• Byte-for-byte identical utility functions with no functional changes
• Improved code organization and reusability

Implementation Quality:

• Well-structured refactoring with clear separation of concerns
• Maintains backward compatibility
• Good error handling and fallback behavior
• Comprehensive test updates for new ComposeWithStreams signature

Documentation

Updated command documentation in docs/content/users/usage/commands.md with new usage examples for the enhanced command options.

Final Assessment

This is a well-implemented enhancement that delivers significant performance improvements through both:

1. Performance optimization - Parallel pulling reduces download times by ~50%
2. Code architecture improvement - Strategic function moves create better package organization

The function moves strengthen the overall architecture by:

• Improving code organization and reusability without changing functionality
• Unifying the image pulling strategy across the codebase
• Following Go best practices for package structure
• Supporting the primary performance enhancement goals

The refactoring improves code organization while maintaining backward compatibility, and the performance gains justify the complexity of the changes.

Recommendation: ✅ Strongly Approve - This PR successfully addresses the performance issue with a clean, well-tested implementation that also improves the overall codebase architecture through thoughtful refactoring.

---

Analysis performed using Claude Code

[stasadev]

Did you already check this out for offline usage? I'll try to give it a shot.

Yes, works as before.

More nits... The name given isn't really the name of the pulled image. For example, we didn't pull ddev-hobobiker-xhgui or ddev-d11simple-web

That's not an image name, it's a service name. I avoided using the image name because I wasn't sure what pattern was valid there.

I found the allowed pattern here https://github.com/compose-spec/compose-go/blob/0bd910723fa2d79a3da0334cbec6f958f632936d/schema/compose-spec.json#L31

So now, each image name used as a service name during pull will look like this:

-ddev/ddev-webserver:v1.24.7
+ddev-ddev-webserver-v1.24.7

[rfay]

This looks great, works great. Thanks for all the finesse and for getting the actual docker image names (or at least pseudo-names) in there.