fix(Gmail Node): Add V2 maxResults limit to prevent OOM on large inboxes#28470
Open
BerniWittmann wants to merge 1 commit intomasterfrom
Open
fix(Gmail Node): Add V2 maxResults limit to prevent OOM on large inboxes#28470BerniWittmann wants to merge 1 commit intomasterfrom
BerniWittmann wants to merge 1 commit intomasterfrom
Conversation
- Add node version 2 with configurable "Max Emails per Poll" (default 10, max 50) - Store unfetched message IDs as pendingMessageIds for next poll - Use lightweight MessageBookkeeping type instead of full Message for state tracking - Fix possibleDuplicates loss during pending drain by merging when lastTimeChecked doesn't advance - Floor timestamps to integers to prevent float comparison issues - Add comprehensive V2 test coverage (12 new tests)
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Contributor
There was a problem hiding this comment.
No issues found across 3 files
Architecture diagram
sequenceDiagram
participant SD as Static Data (Storage)
participant Node as Gmail Trigger Node
participant GAPI as Gmail API (External)
Note over Node: NEW: Version 2 Polling Logic
Node->>SD: Retrieve lastTimeChecked & pendingMessageIds
SD-->>Node: lastTimeChecked, pendingMessageIds[]
alt NEW: Pending messages from previous poll exist
Node->>Node: Calculate budget (maxResults)
Node->>Node: Slice first N IDs from pendingMessageIds
loop For each ID in slice
Node->>GAPI: GET /messages/{id} (fetch full body/metadata)
GAPI-->>Node: Message data
Node->>Node: NEW: Extract MessageBookkeeping (lightweight metadata)
end
Node->>SD: NEW: Update pendingMessageIds (remove fetched)
opt NEW: pendingMessageIds still not empty
Node-->>Node: Early Exit: Return items to workflow
end
end
Note over Node: List New Messages (if budget remains)
Node->>GAPI: GET /messages?q=after:{lastTimeChecked}
GAPI-->>Node: List of {id, threadId}
Node->>Node: CHANGED: Filter IDs against possibleDuplicates
alt NEW: Listed count > remaining budget
Node->>Node: Slice IDs to fit budget
Node->>SD: NEW: Store overflow in pendingMessageIds
end
loop For each message in budget
Node->>GAPI: GET /messages/{id}
GAPI-->>Node: Full Message
Note right of Node: OOM Prevention: Only keep <br/>Bookkeeping objects in memory <br/>for date calculation
Node->>Node: NEW: push to allFetchedMessages (id, internalDate)
end
Note over Node, SD: State Update & Deduplication logic
Node->>Node: CHANGED: Calculate lastEmailDate (Math.floor)
alt NEW: lastTimeChecked did not advance
Node->>SD: CHANGED: Merge current possibleDuplicates with new IDs
else lastTimeChecked advanced
Node->>SD: Replace possibleDuplicates with new IDs
end
Node->>SD: Update lastTimeChecked
Node-->>Node: Return processed responseData
Contributor
Performance ComparisonComparing current → latest master → 14-day baseline Memory consumption baseline with starter plan resources
docker-stats
Idle baseline with Instance AI module loaded
How to read this table
|
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Gmail Trigger causes OOM on Cloud when polling many or large emails. Root causes:
format=rawcan be 10MB+ per email)Messageobjects were kept in memory just for timestamp/duplicate detectionThis PR adds node version 2 with a configurable "Max Emails per Poll" field (default 10, max 50). The approach: list all IDs (cheap), limit full fetches (expensive), store unfetched IDs as
pendingMessageIdsfor the next poll cycle.Key changes:
maxResultsproperty gated behind@version >= 2, disabled in manual modeMessageBookkeepingtype — lightweight{id, internalDate, date, headers.date}for timestamp/dedup computation without retaining full message bodiespendingMessageIdsstate — unfetched IDs stored for next poll (when # of emails > maxResults), drained before listing new messagespossibleDuplicatesare now merged (not replaced) whenlastTimeCheckeddoesn't advance during pending drain, preventing boundary duplicates from reappearingMath.floor()onlastTimeCheckedto prevent float precision issues in equality comparisonsHow to test:
maxResults=2Related Linear tickets, Github issues, and Community forum posts
https://linear.app/n8n/issue/NODE-4595
Review / Merge checklist
Backport to Beta,Backport to Stable, orBackport to v1(if the PR is an urgent fix that needs to be backported)🤖 PR Summary generated by AI