<fix>[ha]: defer skip-trace list cleanup on MN departure to prevent split-brain

[MatheMatrix]

When a management node departs, its VM skip-trace entries were
immediately removed. If VMs were still being started by kvmagent,
the next VM sync would falsely detect them as Stopped and trigger
HA, causing split-brain.

Fix: transfer departed MN skip-trace entries to an orphaned set with
10-minute TTL instead of immediate deletion. VMs in the orphaned set
remain skip-traced until the TTL expires or they are explicitly
continued, preventing false HA triggers during MN restart scenarios.

Resolves: ZSTAC-80821

Change-Id: I3222e260b2d7b33dc43aba0431ce59a788566b34

Conflicts:
plugin/kvm/src/main/java/org/zstack/kvm/KvmVmSyncPingTask.java

sync from gitlab !9626

[coderabbitai]

工作流程

在KVM插件中新增了孤立虚拟机跳过条目的全局配置超时参数，并在虚拟机同步任务中实现了基于TTL的孤立跳过条目跟踪机制，用于管理已离线管理节点的遗留VM跳过标记。

变更

队列/文件	摘要
KVM全局配置 plugin/kvm/src/main/java/org/zstack/kvm/KVMGlobalConfig.java	新增 ORPHANED_VM_SKIP_TIMEOUT 全局配置项，类型为 Long，默认值 600 秒，用于设置孤立VM跳过条目的超时时间，包含大于0的验证规则。
虚拟机同步任务 plugin/kvm/src/main/java/org/zstack/kvm/KvmVmSyncPingTask.java	实现孤立跳过条目的TTL跟踪机制，在节点离线时将VM UUID移入带时间戳的 orphanedSkipVms 容器，新增 cleanupExpiredOrphanedSkipVms() 清理过期条目，新增公有方法 isVmDoNotNeedToTrace() 检查VM是否需要跳过跟踪。

代码审查工作量估计

🎯 3 (Moderate) | ⏱️ ~20 minutes

诗

🐰 孤立的虚机啊，何处是归程，
TTL计时器里，岁月悄悄行，
节点离去留残影，清理机制来帮忙，
跳过标记有新家，管理更加妥贴当！✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	标题清晰准确地总结了主要变更：推迟MN离线时跳过追踪列表清理以防止脑裂问题。
Description check	✅ Passed	描述充分说明了问题背景、解决方案和具体实现细节，与变更集内容高度相关。

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch sync/yaohua.wu/84234@@2

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@plugin/kvm/src/main/java/org/zstack/kvm/KvmVmSyncPingTask.java`:
- Around line 494-504: The explicit "continue" path currently only removes
entries from vmsToSkip but not from orphanedSkipVms, so VMs explicitly continued
still remain skipped until TTL expires; update the VM_CONTINUE_TRACE_PATH
handler (the callback that clears vmsToSkip) to also remove the VM UUID from
orphanedSkipVms (use orphanedSkipVms.remove(vmUuid)) and ensure any code paths
that mark a VM as resumed/continued clear both vmsToSkip and orphanedSkipVms so
explicit continue immediately resumes tracing.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 6cae9136-1be4-4162-b06a-b5a4f2a5dd9b

📥 Commits

Reviewing files that changed from the base of the PR and between 97a38fa and 2939f82.

📒 Files selected for processing (2)

• plugin/kvm/src/main/java/org/zstack/kvm/KVMGlobalConfig.java
• plugin/kvm/src/main/java/org/zstack/kvm/KvmVmSyncPingTask.java

[coderabbitai]

⚠️ Potential issue | 🟠 Major

显式 continue 还不会结束 orphaned skip。

这里把 orphaned 条目的退出条件实现成了“仅 TTL 过期”，但 PR 目标还有“explicitly continued”。当前 VM_CONTINUE_TRACE_PATH 回调只清理 vmsToSkip，不会清理 orphanedSkipVms，所以 VM 实际已经完成启动/恢复后，仍然会继续被 skip 到 TTL 结束，HA 会被额外延后。

🔧 建议修复

         if (data1.getApiId() != null && vmApis.containsKey(data1.getManagementNodeId()) && vmApis.get(data1.getManagementNodeId()).contains(data1.getApiId())) {
             String vmUuid = vmApis.get(data1.getManagementNodeId()).remove(data1.getApiId());
             logger.info("Continuing tracing VM: " + vmUuid);
             vmsToSkip.get(data1.getManagementNodeId()).remove(vmUuid);
+            orphanedSkipVms.remove(vmUuid);
             return;
         }

         if (data1.getVmUuid() != null) {
             logger.info("Continuing tracing VM: " + data1.getVmUuid());
             vmsToSkip.get(data1.getManagementNodeId()).remove(data1.getVmUuid());
+            orphanedSkipVms.remove(data1.getVmUuid());
         }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@plugin/kvm/src/main/java/org/zstack/kvm/KvmVmSyncPingTask.java` around lines
494 - 504, The explicit "continue" path currently only removes entries from
vmsToSkip but not from orphanedSkipVms, so VMs explicitly continued still remain
skipped until TTL expires; update the VM_CONTINUE_TRACE_PATH handler (the
callback that clears vmsToSkip) to also remove the VM UUID from orphanedSkipVms
(use orphanedSkipVms.remove(vmUuid)) and ensure any code paths that mark a VM as
resumed/continued clear both vmsToSkip and orphanedSkipVms so explicit continue
immediately resumes tracing.