VLE cache: replace snapshot invalidation with per-graph

[jrgemignani]

Replace snapshot-based VLE cache invalidation with per-graph version
counters and add performance optimizations for VLE traversal.

Cache invalidation:

• Add DSM (PG 17+) and shmem (PG <17) per-graph monotonic version
  counters for cross-backend cache invalidation
• Replace snapshot xmin/xmax/curcid comparison with version counter
  check in is_ggctx_invalid(), with snapshot fallback for safety
• Add executor hooks in CREATE/DELETE/SET/MERGE to increment the
  graph version counter on mutations
• Add SQL trigger function (age_invalidate_graph_cache) for catching
  SQL-level mutations (INSERT/UPDATE/DELETE/TRUNCATE)
• Auto-install trigger on new label tables via label_commands.c
  with LookupFuncName check for backward compatibility
• Add TRUNCATE interception in ProcessUtility hook (ag_catalog.c)
• Add shmem_request/startup hooks for PG <17 (age.c)

Thin entries (lazy property fetch):

• Replace Datum edge_properties/vertex_properties with 6-byte
  ItemPointerData TID in vertex_entry and edge_entry
• Add get_vertex_entry_properties() and get_edge_entry_properties()
  that do heap_fetch via stored TID on demand
• Add is_an_edge_match() fast path: skip property access entirely
  for label-only VLE patterns (the common case)

Edge cleanup list removal:

• Remove ggctx->edges linked list from GRAPH_global_context. With
  thin entries, edge_entry has no palloc'd sub-structures to free
  (TIDs are inline). The cleanup list existed to pfree datumCopy'd
  edge_properties Datums, which no longer exist. hash_destroy()
  handles all edge hash table memory. Saves ~5.6 GB at SF10.

Performance optimizations:

• Reduce VERTEX/EDGE_HTAB_INITIAL_SIZE from 1,000,000 to 10,000.
  PG dynahash grows automatically; large initial size wastes memory.
• Eliminate extract_variadic_args in age_match_vle_terminal_edge:
  direct PG_GETARG_DATUM + cached arg types via fn_extra
• Eliminate extract_variadic_args in age_match_vle_edge_to_id_qual:
  same pattern
• Add agtype_access_operator 2-arg fast path: bypasses
  extract_variadic_args_min for the common property access case
• Add age_tointeger 1-arg fast path: bypasses extract_variadic_args
  with cached arg type
• Add GraphIdStack (flat array-based) DFS stacks replacing
  ListGraphId linked-list stacks for push/pop without palloc/pfree

Regression tests:

• Add VLE cache invalidation tests (CREATE, DELETE, SET mutations)
• Add thin entry edge property fetch tests (RETURN p, UNWIND)

All 32 regression tests pass.

SF3 Benchmarks (9.3M vertices, 52.7M edges, warm cache, median):

Total IC1-IC12: 1,530s -> 1,159s (-24.3%, 371s saved)

VLE-heavy queries:
IC3 (KNOWS1..2 + 2 countries): 34.9s -> 20.7s (-40.6%)
IC5 (forum members, KNOWS1..2): 46.2s -> 29.9s (-35.4%)
IC6 (tag co-occurrence, KNOWS1..2): 28.2s -> 19.2s (-31.8%)
IC9 (recent messages, KNOWS1..2): 86.0s -> 60.4s (-29.7%)
IC11 (KNOWS1..2 + WORK_AT): 18.7s -> 11.0s (-41.5%)
IC1 (KNOWS1..3 + profile): 1269.4s -> 974.9s (-23.2%)

Short Reads (IS1-IS7): No meaningful change — non-VLE queries,
sub-millisecond to sub-second. Within run-to-run variance.

Updates (IU1-IU8, SF3, median, 50 ops each):

IU2 (Like Post): 99.5ms -> 6.3ms (-93.6%)
IU3 (Like Comment): 318.3ms -> 5.7ms (-98.2%)
IU7 (Add Comment): 344.4ms -> 11.3ms (-96.7%)
IU5 (Forum Member): 12.3ms -> 5.9ms (-52.0%)

Version counter eliminates redundant VLE cache rebuilds on mutations.
Previously, every INSERT/UPDATE/DELETE invalidated the cache via
snapshot comparison, forcing a full rebuild on the next VLE query.
Now, mutations just bump a counter; rebuild only occurs when VLE
actually runs and finds the counter changed.

Graph cache memory (SF3, calculated):

Total: ~15.7 GB -> ~8.7 GB (-45%, 7.0 GB saved)

Thin entries (TID replaces datumCopy'd properties): -5.3 GB
Edge cleanup list removal (no longer needed): -1.7 GB

Co-authored-by: Claude Opus noreply@anthropic.com

Files changed (17):
src/backend/age.c — shmem hook registration (PG <17)
src/backend/catalog/ag_catalog.c — TRUNCATE interception
src/backend/commands/label_commands.c — conditional trigger auto-install on label creation
src/backend/executor/cypher_create.c — increment_graph_version after CREATE
src/backend/executor/cypher_delete.c — increment_graph_version after DELETE
src/backend/executor/cypher_merge.c — increment_graph_version after MERGE
src/backend/executor/cypher_set.c — increment_graph_version after SET
src/backend/utils/adt/age_global_graph.c — version counter, thin entries, trigger fn, lazy fetch
src/backend/utils/adt/age_vle.c — is_an_edge_match fast path
src/include/utils/age_global_graph.h — new declarations
sql/age_main.sql — trigger function registration for next-version SQL
regress/sql/age_global_graph.sql — VLE cache regression tests
regress/expected/age_global_graph.out — expected output for new tests
age--1.7.0--y.y.y.sql — upgrade template: trigger function for existing installs
src/backend/utils/adt/age_graphid_ds.c
src/backend/utils/adt/agtype.c
src/include/utils/age_graphid_ds.h

[Copilot]

Pull request overview

This PR updates Apache AGE’s VLE (variable-length edge) cache invalidation to be graph-specific, avoiding server-wide false invalidations, and reduces VLE cache memory by switching vertex/edge cache entries to “thin” TID-based storage with lazy property fetch.

Changes:

• Replace snapshot-based invalidation with per-graph monotonic version counters backed by DSM (PG17+), SHMEM hooks (PG<17 + shared_preload_libraries), or snapshot fallback.
• Reduce VLE cache memory by storing tuple TIDs in the cache and fetching properties lazily when constructing results.
• Add a VLE edge-match fast path and new regression tests for invalidation + thin-entry property fetching.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/include/utils/age_global_graph.h	Exposes graph version counter APIs and SHMEM init hooks.
src/backend/utils/adt/age_vle.c	Adds label-only fast path in edge matching to avoid property access.
src/backend/utils/adt/age_global_graph.c	Implements version counters + thin entries + lazy property fetch + trigger function.
src/backend/executor/cypher_set.c	Increments per-graph version on SET mutations.
src/backend/executor/cypher_merge.c	Increments per-graph version when MERGE creates new paths.
src/backend/executor/cypher_delete.c	Increments per-graph version on DELETE mutations.
src/backend/executor/cypher_create.c	Increments per-graph version on CREATE mutations.
src/backend/commands/label_commands.c	Conditionally installs SQL mutation invalidation triggers on new label tables.
src/backend/catalog/ag_catalog.c	Intercepts TRUNCATE to invalidate affected graph caches.
src/backend/age.c	Registers SHMEM hooks for PG<17 to enable shared invalidation state.
sql/age_main.sql	Registers the trigger function in the extension SQL.
regress/sql/age_global_graph.sql	Adds regression coverage for invalidation + thin-entry behavior.
regress/expected/age_global_graph.out	Adds expected output for the new regression cases.
age--1.7.0--y.y.y.sql	Upgrade template adds the trigger function for existing installs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jrgemignani]

@MuhammadTahaNaveed rebased with master.