Fix TestNG session status reported with EFD by daniel-mohedano · Pull Request #11087 · DataDog/dd-trace-java

daniel-mohedano · 2026-04-13T10:33:15Z

What Does This Do

Fixes a bug that would make the test session status dependent of the result orders when using EFD with TestNG:
- If the result order was pass and fail, the test session was correctly failed.
- If the result order was fail and pass, the test session would incorrectly pass, while EFD should fail the session when a flaky is found.
This was caused by TestNG's retry analyzer suppressing failures on retried tests automatically, which would only trigger if the first test execution was a fail.
This bug only affects EFD as it depends on tests being retried after a failure and the failures not being suppressed:
- ATR suppresses failures by default
- Attempt to Fix will not retry a test if the first execution is a failure

Motivation

Aligns EFD behavior for TestNG with the rest of the testing frameworks.

Additional Information

Two unit tests have been introduced to verify the behavior.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors

Jira ticket: SDTEST-3720

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

cit-pr-commenter-54b7da · 2026-04-13T10:40:22Z

Test Environment - sbt-scalatest

Job Status: success

Scenario	Overhead (%)
agent	55.20
agentEvpProxy	55.99

cit-pr-commenter-54b7da · 2026-04-13T10:43:06Z

Test Environment - nebula-release-plugin

Job Status: success

Scenario	Overhead (%)
agent	34.55
agentless	33.70
agentlessCodeCoverage	42.53
agentlessLineCoverage	74.30

cit-pr-commenter-54b7da · 2026-04-13T10:44:40Z

Test Environment - netflix-zuul

Job Status: success

Scenario	Overhead (%)
agent	88.24
agentless	82.24
agentlessCodeCoverage	97.79
agentlessLineCoverage	114.13

cit-pr-commenter-54b7da · 2026-04-13T10:45:50Z

Test Environment - reactive-streams-jvm

Job Status: success

Scenario	Overhead (%)
agent	21.64
agentless	18.75
agentlessCodeCoverage	20.15
agentlessLineCoverage	30.07

cit-pr-commenter-54b7da · 2026-04-13T10:48:45Z

Test Environment - pass4s

Job Status: success

Scenario	Overhead (%)
agent	10.74
agentless	13.95
agentlessCodeCoverage	21.23

cit-pr-commenter-54b7da · 2026-04-13T10:51:40Z

Test Environment - sonar-kotlin

Job Status: success

Scenario	Overhead (%)
agent	13.23
agentless	11.95
agentlessCodeCoverage	16.46
agentlessLineCoverage	19.17

cit-pr-commenter-54b7da · 2026-04-13T10:56:48Z

Test Environment - jolokia

Job Status: success

Scenario	Overhead (%)
agent	94.23
agentless	90.61
agentlessCodeCoverage	99.92
agentlessLineCoverage	100.99

cit-pr-commenter-54b7da · 2026-04-13T11:00:28Z

Test Environment - okhttp

Job Status: success

Scenario	Overhead (%)
agent	19.52
agentless	18.18
agentlessCodeCoverage	20.70
agentlessLineCoverage	43.75

cit-pr-commenter-54b7da · 2026-04-13T11:07:37Z

Test Environment - spring_boot

Job Status: success

Scenario	Overhead (%)
agent	16.59
agentless	10.11
agentlessCodeCoverage	14.23
agentlessLineCoverage	33.45

pr-commenter · 2026-04-13T11:15:18Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/testng-failure-propagation
git_commit_date	1776255945	1776255197
git_commit_sha	93449b6f0f	`4fd99ee`
release_version	1.62.0-SNAPSHOT~693449b6f0f	1.62.0-SNAPSHOT~4fd99ee3ec

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1776257235	1776257235
ci_job_id	1597134230	1597134230
ci_pipeline_id	107796383	107796383
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-6v0jxluk 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-6v0jxluk 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 61 metrics, 10 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~4fd99ee3ec, baseline=1.62.0-SNAPSHOT~693449b6f0f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.06 s) : 0, 1059732
Total [baseline] (8.867 s) : 0, 8866719
Agent [candidate] (1.062 s) : 0, 1062006
Total [candidate] (8.814 s) : 0, 8814323
section iast
Agent [baseline] (1.22 s) : 0, 1220364
Total [baseline] (9.505 s) : 0, 9504653
Agent [candidate] (1.218 s) : 0, 1217934
Total [candidate] (9.525 s) : 0, 9524547

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.06 s	-
Agent	iast	1.22 s	160.632 ms (15.2%)
Total	tracing	8.867 s	-
Total	iast	9.505 s	637.934 ms (7.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.062 s	-
Agent	iast	1.218 s	155.928 ms (14.7%)
Total	tracing	8.814 s	-
Total	iast	9.525 s	710.225 ms (8.1%)

gantt
    title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~4fd99ee3ec, baseline=1.62.0-SNAPSHOT~693449b6f0f

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.228 ms) : 0, 1228
crashtracking [candidate] (1.238 ms) : 0, 1238
BytebuddyAgent [baseline] (633.99 ms) : 0, 633990
BytebuddyAgent [candidate] (636.664 ms) : 0, 636664
AgentMeter [baseline] (29.512 ms) : 0, 29512
AgentMeter [candidate] (29.514 ms) : 0, 29514
GlobalTracer [baseline] (248.565 ms) : 0, 248565
GlobalTracer [candidate] (249.546 ms) : 0, 249546
AppSec [baseline] (32.425 ms) : 0, 32425
AppSec [candidate] (32.487 ms) : 0, 32487
Debugger [baseline] (59.281 ms) : 0, 59281
Debugger [candidate] (59.413 ms) : 0, 59413
Remote Config [baseline] (610.1 µs) : 0, 610
Remote Config [candidate] (596.495 µs) : 0, 596
Telemetry [baseline] (8.047 ms) : 0, 8047
Telemetry [candidate] (8.079 ms) : 0, 8079
Flare Poller [baseline] (9.798 ms) : 0, 9798
Flare Poller [candidate] (8.242 ms) : 0, 8242
section iast
crashtracking [baseline] (1.237 ms) : 0, 1237
crashtracking [candidate] (1.22 ms) : 0, 1220
BytebuddyAgent [baseline] (798.668 ms) : 0, 798668
BytebuddyAgent [candidate] (797.596 ms) : 0, 797596
AgentMeter [baseline] (11.305 ms) : 0, 11305
AgentMeter [candidate] (11.32 ms) : 0, 11320
GlobalTracer [baseline] (238.247 ms) : 0, 238247
GlobalTracer [candidate] (237.937 ms) : 0, 237937
AppSec [baseline] (31.199 ms) : 0, 31199
AppSec [candidate] (31.188 ms) : 0, 31188
Debugger [baseline] (60.02 ms) : 0, 60020
Debugger [candidate] (60.661 ms) : 0, 60661
Remote Config [baseline] (1.155 ms) : 0, 1155
Remote Config [candidate] (1.724 ms) : 0, 1724
Telemetry [baseline] (12.738 ms) : 0, 12738
Telemetry [candidate] (11.177 ms) : 0, 11177
Flare Poller [baseline] (3.477 ms) : 0, 3477
Flare Poller [candidate] (3.398 ms) : 0, 3398
IAST [baseline] (25.827 ms) : 0, 25827
IAST [candidate] (25.708 ms) : 0, 25708

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~4fd99ee3ec, baseline=1.62.0-SNAPSHOT~693449b6f0f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1057571
Total [baseline] (11.007 s) : 0, 11007226
Agent [candidate] (1.053 s) : 0, 1053091
Total [candidate] (10.965 s) : 0, 10964946
section appsec
Agent [baseline] (1.244 s) : 0, 1244292
Total [baseline] (11.209 s) : 0, 11208844
Agent [candidate] (1.263 s) : 0, 1263025
Total [candidate] (11.116 s) : 0, 11116381
section iast
Agent [baseline] (1.226 s) : 0, 1225875
Total [baseline] (11.236 s) : 0, 11236165
Agent [candidate] (1.221 s) : 0, 1221082
Total [candidate] (11.128 s) : 0, 11128310
section profiling
Agent [baseline] (1.184 s) : 0, 1183654
Total [baseline] (11.081 s) : 0, 11080881
Agent [candidate] (1.18 s) : 0, 1180402
Total [candidate] (11.149 s) : 0, 11149138

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	appsec	1.244 s	186.721 ms (17.7%)
Agent	iast	1.226 s	168.304 ms (15.9%)
Agent	profiling	1.184 s	126.084 ms (11.9%)
Total	tracing	11.007 s	-
Total	appsec	11.209 s	201.619 ms (1.8%)
Total	iast	11.236 s	228.94 ms (2.1%)
Total	profiling	11.081 s	73.656 ms (0.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.053 s	-
Agent	appsec	1.263 s	209.934 ms (19.9%)
Agent	iast	1.221 s	167.991 ms (16.0%)
Agent	profiling	1.18 s	127.311 ms (12.1%)
Total	tracing	10.965 s	-
Total	appsec	11.116 s	151.435 ms (1.4%)
Total	iast	11.128 s	163.365 ms (1.5%)
Total	profiling	11.149 s	184.192 ms (1.7%)

gantt
    title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~4fd99ee3ec, baseline=1.62.0-SNAPSHOT~693449b6f0f

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.232 ms) : 0, 1232
crashtracking [candidate] (1.213 ms) : 0, 1213
BytebuddyAgent [baseline] (631.903 ms) : 0, 631903
BytebuddyAgent [candidate] (630.407 ms) : 0, 630407
AgentMeter [baseline] (29.549 ms) : 0, 29549
AgentMeter [candidate] (29.389 ms) : 0, 29389
GlobalTracer [baseline] (249.698 ms) : 0, 249698
GlobalTracer [candidate] (247.95 ms) : 0, 247950
AppSec [baseline] (32.539 ms) : 0, 32539
AppSec [candidate] (32.206 ms) : 0, 32206
Debugger [baseline] (60.301 ms) : 0, 60301
Debugger [candidate] (59.772 ms) : 0, 59772
Remote Config [baseline] (608.179 µs) : 0, 608
Remote Config [candidate] (586.109 µs) : 0, 586
Telemetry [baseline] (8.135 ms) : 0, 8135
Telemetry [candidate] (8.071 ms) : 0, 8071
Flare Poller [baseline] (7.571 ms) : 0, 7571
Flare Poller [candidate] (7.492 ms) : 0, 7492
section appsec
crashtracking [baseline] (1.219 ms) : 0, 1219
crashtracking [candidate] (1.27 ms) : 0, 1270
BytebuddyAgent [baseline] (659.274 ms) : 0, 659274
BytebuddyAgent [candidate] (672.857 ms) : 0, 672857
AgentMeter [baseline] (12.054 ms) : 0, 12054
AgentMeter [candidate] (12.176 ms) : 0, 12176
GlobalTracer [baseline] (247.836 ms) : 0, 247836
GlobalTracer [candidate] (250.589 ms) : 0, 250589
AppSec [baseline] (184.54 ms) : 0, 184540
AppSec [candidate] (185.594 ms) : 0, 185594
Debugger [baseline] (65.868 ms) : 0, 65868
Debugger [candidate] (66.345 ms) : 0, 66345
Remote Config [baseline] (601.241 µs) : 0, 601
Remote Config [candidate] (610.359 µs) : 0, 610
Telemetry [baseline] (8.495 ms) : 0, 8495
Telemetry [candidate] (8.609 ms) : 0, 8609
Flare Poller [baseline] (3.54 ms) : 0, 3540
Flare Poller [candidate] (3.56 ms) : 0, 3560
IAST [baseline] (24.536 ms) : 0, 24536
IAST [candidate] (24.803 ms) : 0, 24803
section iast
crashtracking [baseline] (1.225 ms) : 0, 1225
crashtracking [candidate] (1.227 ms) : 0, 1227
BytebuddyAgent [baseline] (802.652 ms) : 0, 802652
BytebuddyAgent [candidate] (798.789 ms) : 0, 798789
AgentMeter [baseline] (11.391 ms) : 0, 11391
AgentMeter [candidate] (11.372 ms) : 0, 11372
GlobalTracer [baseline] (239.317 ms) : 0, 239317
GlobalTracer [candidate] (238.327 ms) : 0, 238327
AppSec [baseline] (31.198 ms) : 0, 31198
AppSec [candidate] (31.935 ms) : 0, 31935
Debugger [baseline] (64.236 ms) : 0, 64236
Debugger [candidate] (59.417 ms) : 0, 59417
Remote Config [baseline] (548.741 µs) : 0, 549
Remote Config [candidate] (1.108 ms) : 0, 1108
Telemetry [baseline] (9.858 ms) : 0, 9858
Telemetry [candidate] (13.434 ms) : 0, 13434
Flare Poller [baseline] (3.464 ms) : 0, 3464
Flare Poller [candidate] (3.504 ms) : 0, 3504
IAST [baseline] (25.73 ms) : 0, 25730
IAST [candidate] (25.73 ms) : 0, 25730
section profiling
ProfilingAgent [baseline] (94.614 ms) : 0, 94614
ProfilingAgent [candidate] (93.737 ms) : 0, 93737
crashtracking [baseline] (1.168 ms) : 0, 1168
crashtracking [candidate] (1.166 ms) : 0, 1166
BytebuddyAgent [baseline] (689.584 ms) : 0, 689584
BytebuddyAgent [candidate] (689.131 ms) : 0, 689131
AgentMeter [baseline] (9.11 ms) : 0, 9110
AgentMeter [candidate] (9.039 ms) : 0, 9039
GlobalTracer [baseline] (207.159 ms) : 0, 207159
GlobalTracer [candidate] (206.374 ms) : 0, 206374
AppSec [baseline] (32.662 ms) : 0, 32662
AppSec [candidate] (32.747 ms) : 0, 32747
Debugger [baseline] (65.944 ms) : 0, 65944
Debugger [candidate] (65.281 ms) : 0, 65281
Remote Config [baseline] (575.585 µs) : 0, 576
Remote Config [candidate] (572.171 µs) : 0, 572
Telemetry [baseline] (7.877 ms) : 0, 7877
Telemetry [candidate] (7.849 ms) : 0, 7849
Flare Poller [baseline] (3.548 ms) : 0, 3548
Flare Poller [candidate] (3.551 ms) : 0, 3551
Profiling [baseline] (95.189 ms) : 0, 95189
Profiling [candidate] (94.3 ms) : 0, 94300

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/testng-failure-propagation
git_commit_date	1776255788	1776255197
git_commit_sha	93449b6f0f	`4fd99ee`
release_version	1.62.0-SNAPSHOT~693449b6f0f	1.62.0-SNAPSHOT~4fd99ee3ec

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1776257557	1776257557
ci_job_id	1597134231	1597134231
ci_pipeline_id	107796383	107796383
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-wmsqn2u6 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-wmsqn2u6 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 1 performance regressions! Performance is the same for 18 metrics, 15 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:petclinic:tracing:high_load	worse [+365.354µs; +1140.588µs] or [+2.079%; +6.489%]	unsure [+340.509µs; +1579.891µs] or [+1.185%; +5.499%]	unstable [-35.064op/s; +16.751op/s] or [-13.504%; +6.451%]	18.330ms	29.691ms	250.500op/s	17.577ms	28.731ms	259.656op/s
scenario:load:petclinic:profiling:high_load	better [-1.757ms; -0.661ms] or [-9.144%; -3.440%]	unsure [-1691.656µs; -112.512µs] or [-5.523%; -0.367%]	unstable [-12.367op/s; +38.367op/s] or [-5.147%; +15.968%]	18.007ms	29.725ms	253.281op/s	19.216ms	30.627ms	240.281op/s
scenario:load:petclinic:no_agent:high_load	better [-1.733ms; -0.499ms] or [-9.289%; -2.675%]	unsure [-2.999ms; -0.152ms] or [-9.658%; -0.489%]	unstable [-10.451op/s; +42.889op/s] or [-4.307%; +17.672%]	17.542ms	29.477ms	258.906op/s	18.658ms	31.053ms	242.688op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~4fd99ee3ec, baseline=1.62.0-SNAPSHOT~693449b6f0f
    dateFormat X
    axisFormat %s
section baseline
no_agent (19.232 ms) : 19038, 19427
.   : milestone, 19232,
appsec (18.745 ms) : 18548, 18942
.   : milestone, 18745,
code_origins (17.757 ms) : 17584, 17930
.   : milestone, 17757,
iast (18.091 ms) : 17912, 18270
.   : milestone, 18091,
profiling (19.43 ms) : 19236, 19624
.   : milestone, 19430,
tracing (17.966 ms) : 17789, 18143
.   : milestone, 17966,
section candidate
no_agent (18.023 ms) : 17841, 18204
.   : milestone, 18023,
appsec (18.779 ms) : 18588, 18970
.   : milestone, 18779,
code_origins (17.823 ms) : 17648, 17997
.   : milestone, 17823,
iast (17.757 ms) : 17581, 17932
.   : milestone, 17757,
profiling (18.423 ms) : 18237, 18608
.   : milestone, 18423,
tracing (18.629 ms) : 18442, 18815
.   : milestone, 18629,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	19.232 ms [19.038 ms, 19.427 ms]	-
appsec	18.745 ms [18.548 ms, 18.942 ms]	-487.369 µs (-2.5%)
code_origins	17.757 ms [17.584 ms, 17.93 ms]	-1.475 ms (-7.7%)
iast	18.091 ms [17.912 ms, 18.27 ms]	-1.141 ms (-5.9%)
profiling	19.43 ms [19.236 ms, 19.624 ms]	197.677 µs (1.0%)
tracing	17.966 ms [17.789 ms, 18.143 ms]	-1.267 ms (-6.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.023 ms [17.841 ms, 18.204 ms]	-
appsec	18.779 ms [18.588 ms, 18.97 ms]	756.308 µs (4.2%)
code_origins	17.823 ms [17.648 ms, 17.997 ms]	-200.131 µs (-1.1%)
iast	17.757 ms [17.581 ms, 17.932 ms]	-265.968 µs (-1.5%)
profiling	18.423 ms [18.237 ms, 18.608 ms]	399.881 µs (2.2%)
tracing	18.629 ms [18.442 ms, 18.815 ms]	606.154 µs (3.4%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~4fd99ee3ec, baseline=1.62.0-SNAPSHOT~693449b6f0f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.258 ms) : 1247, 1270
.   : milestone, 1258,
iast (3.288 ms) : 3237, 3339
.   : milestone, 3288,
iast_FULL (5.758 ms) : 5700, 5816
.   : milestone, 5758,
iast_GLOBAL (3.688 ms) : 3628, 3749
.   : milestone, 3688,
profiling (2.272 ms) : 2249, 2296
.   : milestone, 2272,
tracing (1.825 ms) : 1810, 1840
.   : milestone, 1825,
section candidate
no_agent (1.24 ms) : 1227, 1252
.   : milestone, 1240,
iast (3.195 ms) : 3154, 3236
.   : milestone, 3195,
iast_FULL (5.91 ms) : 5850, 5970
.   : milestone, 5910,
iast_GLOBAL (3.594 ms) : 3534, 3654
.   : milestone, 3594,
profiling (2.331 ms) : 2311, 2352
.   : milestone, 2331,
tracing (1.905 ms) : 1889, 1921
.   : milestone, 1905,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.258 ms [1.247 ms, 1.27 ms]	-
iast	3.288 ms [3.237 ms, 3.339 ms]	2.03 ms (161.3%)
iast_FULL	5.758 ms [5.7 ms, 5.816 ms]	4.5 ms (357.6%)
iast_GLOBAL	3.688 ms [3.628 ms, 3.749 ms]	2.43 ms (193.2%)
profiling	2.272 ms [2.249 ms, 2.296 ms]	1.014 ms (80.6%)
tracing	1.825 ms [1.81 ms, 1.84 ms]	566.879 µs (45.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.24 ms [1.227 ms, 1.252 ms]	-
iast	3.195 ms [3.154 ms, 3.236 ms]	1.955 ms (157.7%)
iast_FULL	5.91 ms [5.85 ms, 5.97 ms]	4.67 ms (376.7%)
iast_GLOBAL	3.594 ms [3.534 ms, 3.654 ms]	2.354 ms (189.8%)
profiling	2.331 ms [2.311 ms, 2.352 ms]	1.091 ms (88.0%)
tracing	1.905 ms [1.889 ms, 1.921 ms]	664.787 µs (53.6%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/testng-failure-propagation
git_commit_date	1776255709	1776255197
git_commit_sha	93449b6f0f	`4fd99ee`
release_version	1.62.0-SNAPSHOT~693449b6f0f	1.62.0-SNAPSHOT~4fd99ee3ec

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1776257221	1776257221
ci_job_id	1597134232	1597134232
ci_pipeline_id	107796383	107796383
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-pjgoh61r 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-pjgoh61r 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~4fd99ee3ec, baseline=1.62.0-SNAPSHOT~693449b6f0f
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.942 s) : 14942000, 14942000
.   : milestone, 14942000,
appsec (14.836 s) : 14836000, 14836000
.   : milestone, 14836000,
iast (18.306 s) : 18306000, 18306000
.   : milestone, 18306000,
iast_GLOBAL (18.189 s) : 18189000, 18189000
.   : milestone, 18189000,
profiling (14.721 s) : 14721000, 14721000
.   : milestone, 14721000,
tracing (14.875 s) : 14875000, 14875000
.   : milestone, 14875000,
section candidate
no_agent (15.573 s) : 15573000, 15573000
.   : milestone, 15573000,
appsec (14.867 s) : 14867000, 14867000
.   : milestone, 14867000,
iast (18.745 s) : 18745000, 18745000
.   : milestone, 18745000,
iast_GLOBAL (18.264 s) : 18264000, 18264000
.   : milestone, 18264000,
profiling (14.872 s) : 14872000, 14872000
.   : milestone, 14872000,
tracing (15.146 s) : 15146000, 15146000
.   : milestone, 15146000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.942 s [14.942 s, 14.942 s]	-
appsec	14.836 s [14.836 s, 14.836 s]	-106.0 ms (-0.7%)
iast	18.306 s [18.306 s, 18.306 s]	3.364 s (22.5%)
iast_GLOBAL	18.189 s [18.189 s, 18.189 s]	3.247 s (21.7%)
profiling	14.721 s [14.721 s, 14.721 s]	-221.0 ms (-1.5%)
tracing	14.875 s [14.875 s, 14.875 s]	-67.0 ms (-0.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.573 s [15.573 s, 15.573 s]	-
appsec	14.867 s [14.867 s, 14.867 s]	-706.0 ms (-4.5%)
iast	18.745 s [18.745 s, 18.745 s]	3.172 s (20.4%)
iast_GLOBAL	18.264 s [18.264 s, 18.264 s]	2.691 s (17.3%)
profiling	14.872 s [14.872 s, 14.872 s]	-701.0 ms (-4.5%)
tracing	15.146 s [15.146 s, 15.146 s]	-427.0 ms (-2.7%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~4fd99ee3ec, baseline=1.62.0-SNAPSHOT~693449b6f0f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.483 ms) : 1471, 1494
.   : milestone, 1483,
appsec (3.832 ms) : 3609, 4054
.   : milestone, 3832,
iast (2.271 ms) : 2201, 2341
.   : milestone, 2271,
iast_GLOBAL (2.319 ms) : 2249, 2390
.   : milestone, 2319,
profiling (2.106 ms) : 2051, 2162
.   : milestone, 2106,
tracing (2.091 ms) : 2036, 2145
.   : milestone, 2091,
section candidate
no_agent (1.488 ms) : 1476, 1500
.   : milestone, 1488,
appsec (3.82 ms) : 3596, 4043
.   : milestone, 3820,
iast (2.266 ms) : 2196, 2336
.   : milestone, 2266,
iast_GLOBAL (2.316 ms) : 2246, 2386
.   : milestone, 2316,
profiling (2.091 ms) : 2036, 2147
.   : milestone, 2091,
tracing (2.078 ms) : 2023, 2132
.   : milestone, 2078,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.483 ms [1.471 ms, 1.494 ms]	-
appsec	3.832 ms [3.609 ms, 4.054 ms]	2.349 ms (158.4%)
iast	2.271 ms [2.201 ms, 2.341 ms]	788.15 µs (53.1%)
iast_GLOBAL	2.319 ms [2.249 ms, 2.39 ms]	836.471 µs (56.4%)
profiling	2.106 ms [2.051 ms, 2.162 ms]	623.322 µs (42.0%)
tracing	2.091 ms [2.036 ms, 2.145 ms]	607.606 µs (41.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.488 ms [1.476 ms, 1.5 ms]	-
appsec	3.82 ms [3.596 ms, 4.043 ms]	2.331 ms (156.7%)
iast	2.266 ms [2.196 ms, 2.336 ms]	778.023 µs (52.3%)
iast_GLOBAL	2.316 ms [2.246 ms, 2.386 ms]	827.841 µs (55.6%)
profiling	2.091 ms [2.036 ms, 2.147 ms]	603.136 µs (40.5%)
tracing	2.078 ms [2.023 ms, 2.132 ms]	589.642 µs (39.6%)

cit-pr-commenter-54b7da · 2026-04-13T11:23:55Z

Test Environment - sonar-java

Job Status: success

Scenario	Overhead (%)
agent	-16.60
agentless	-16.32
agentlessCodeCoverage	80.27
agentlessLineCoverage	92.46

nikita-tkachenko-datadog · 2026-04-15T09:59:18Z

...-7.0/src/main/java/datadog/trace/instrumentation/testng7/TestNGExecutionInstrumentation.java

+        result.setStatus(ITestResult.FAILURE);
+        result.setThrowable(
+            new AssertionError(
+                "Datadog Early Flake Detection: test has flaky results (mixed pass/fail)"));


Nit: IMO explicitly mentioning "Datadog Early Flake Detection" here adds implicit coupling/breaks encapsulation. It is true that EFD is currently the only policy that can return true from shouldPropagateFailure, but this isn't enforced in any way in the code and may change in the future. I'd choose a more generic error message here

Good point, updated it to "Datadog: propagating test failure based on aggregated execution results" to align it better with the overall functionality 👍

daniel-mohedano · 2026-04-15T14:19:42Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-04-15T14:19:46Z

View all feedbacks in Devflow UI.

2026-04-15 14:19:46 UTC ℹ️ Start processing command /merge
Use /merge -c to cancel this operation!

2026-04-15 14:19:52 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

Use /merge -c to cancel this operation!

⏳ Processing

fix: fail testng session on new flaky (efd) independent of result order

986199e

daniel-mohedano added type: bug Bug report and fix comp: ci visibility Continuous Integration Visibility labels Apr 13, 2026

daniel-mohedano added 3 commits April 13, 2026 14:31

refactor: move logic to failure suppression instrumentation

a38d861

chore: rename

bc787c0

Merge branch 'master' into daniel.mohedano/testng-failure-propagation

e7ff3d7

daniel-mohedano marked this pull request as ready for review April 15, 2026 08:15

daniel-mohedano requested a review from a team as a code owner April 15, 2026 08:15

nikita-tkachenko-datadog approved these changes Apr 15, 2026

View reviewed changes

fix: update propagated error message

4fd99ee

Conversation

daniel-mohedano commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Information

Contributor Checklist

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sbt-scalatest

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - nebula-release-plugin

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - netflix-zuul

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - reactive-streams-jvm

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - pass4s

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sonar-kotlin

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - jolokia

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - okhttp

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - spring_boot

Uh oh!

pr-commenter bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sonar-java

Uh oh!

nikita-tkachenko-datadog Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

daniel-mohedano Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

daniel-mohedano commented Apr 15, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

daniel-mohedano commented Apr 13, 2026 •

edited

Loading

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •

edited

Loading

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •

edited

Loading

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •

edited

Loading

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •

edited

Loading

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •

edited

Loading

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •

edited

Loading

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •

edited

Loading

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •

edited

Loading

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •

edited

Loading

pr-commenter bot commented Apr 13, 2026 •

edited

Loading

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Apr 15, 2026 •

edited

Loading