Fix repair_metadata OOM on large repositories#1189
Merged
Conversation
github-actions
bot
added
multi-commit
jobselko
requested changes
Apr 10, 2026
jobselko
reviewed
Apr 13, 2026
jobselko
reviewed
Apr 13, 2026
Contributor
|
Could you please squash the commits into one? Otherwise, it looks good! |
decko
force-pushed
the
fix/repair-metadata-memory
branch
from
025a2f3 to
9cda1db
Compare
github-actions
bot
removed
multi-commit
Member
Author
Done @jobselko. Just one commit. |
gerrod3
requested changes
Apr 13, 2026
Contributor
gerrod3
left a comment
gerrod3
left a comment
There was a problem hiding this comment.
We can accept this change and backport it. In the future we will need to take a look at our util methods and refactor them, they've grown a bit absurd.
Comment on lines
+1
to
+3
| Reduced peak memory consumption of repair_metadata by lowering batch size from 1000 to 250, | ||
| eliminating double S3 reads for wheel files, and closing artifact file handles after each | ||
| iteration. This fixes "Worker has gone missing" errors on repositories with 1000+ packages. |
Contributor
There was a problem hiding this comment.
Simplify this, too verbose. Keep it on one line.
Large repositories (1000+ packages) cause workers to OOM during repair_metadata. Three changes reduce peak memory: - Reduce BULK_SIZE from 1000 to 250, flushing batches 4x more often - Copy artifact to temp file once via helper, reuse for both content data extraction and metadata extraction (eliminates double S3 read) - Extract metadata bytes while temp file exists, pass bytes through the metadata batch instead of file paths Closes pulp#1188 JIRA: PULP-1573 Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
decko
force-pushed
the
fix/repair-metadata-memory
branch
from
9cda1db to
208ab04
Compare
gerrod3
approved these changes
Apr 14, 2026
Backport to 3.27: 💚 backport PR created✅ Backport PR branch: Backported as #1196 🤖 @patchback |
patchback bot
pushed a commit
that referenced
this pull request
Apr 14, 2026
Fix repair_metadata OOM on large repositories (cherry picked from commit 7aaf53f)
Backport to 3.28: 💚 backport PR created✅ Backport PR branch: Backported as #1197 🤖 @patchback |
3 tasks
patchback bot
pushed a commit
that referenced
this pull request
Apr 14, 2026
Fix repair_metadata OOM on large repositories (cherry picked from commit 7aaf53f)
3 tasks
jobselko
added a commit
that referenced
this pull request
Apr 14, 2026
…f2a46f21be5363e677f503d6ea86fc9/pr-1189 [PR #1189/7aaf53fc backport][3.27] Fix repair_metadata OOM on large repositories
jobselko
added a commit
that referenced
this pull request
Apr 14, 2026
…f2a46f21be5363e677f503d6ea86fc9/pr-1189 [PR #1189/7aaf53fc backport][3.28] Fix repair_metadata OOM on large repositories
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BULK_SIZEfrom 1000 to 250, flushing batches 4x more often to cap peak memoryFixes #1188
Test plan
test_repair.pytests pass (metadata repair command, endpoint, artifact repair)test_metadata_repair_batch_boundarypasses with reduced BULK_SIZErepair-python-metadata.py --env stage --domain <large-domain>to verify no OOMJIRA: PULP-1573
🤖 Generated with Claude Code