Add custom attribute for payment address in checkout service

[bpschmitt]

Changes

Summary

Adds an OpenTelemetry span attribute app.payment.service.address to the chargeCard function in the checkout service to improve observability of payment service routing.

Changes in src/checkout/main.go:

• Added app.payment.service.address span attribute to every call through chargeCard, recording which gRPC endpoint was targeted for the payment charge request
• When the paymentUnreachable feature flag is disabled, the attribute reflects the configured payment service address (from PAYMENT_ADDR)
• When the paymentUnreachable feature flag is enabled, the attribute reflects the injected bad address (badAddress:50051), making fault injection clearly visible in traces

Attribute naming follows OpenTelemetry semantic conventions, using the app.payment.* namespace consistent with other checkout service span attributes (e.g., app.payment.transaction.id).

Motivation

Previously, the paymentUnreachable feature flag would silently redirect traffic to a bad address with no trace-level indication of which endpoint was called. This change makes payment service routing observable in every request, which is especially valuable when diagnosing failures caused by fault injection.

Merge Requirements

For new features contributions, please make sure you have completed the following
essential items:

• CHANGELOG.md updated to document new feature additions

[cyrille-leclerc]

Why would we add a custom attribute to the checkout service's oteldemo.CheckoutService/PlaceOrder span for the payment service hostname? That information is already available on the RPC client span oteldemo.PaymentService/Charge via the server.address attribute (trace screenshot below).

In practice, invocation details such as server.address or status_code are captured on RPC/HTTP/DB client spans.

That said, it could be worth verifying whether we’re making proper use of the Feature Flags attributes (semconv). We should also consider whether relying on them might make troubleshooting too straightforward, which could undermine one of the goals of the OTel Demo: providing a realistic environment to evaluate observability troubleshooting workflows.

[bpschmitt]

@cyrille-leclerc the paymentUnreachable feature flag uses badAddress:50051 as the payment service URL. When this feature flag is enabled, the server.address field does not contain a value. My thinking is that, considering this is meant to be a demo application, wouldn't you want to be able to explain why this feature flag is generating errors?

[cyrille-leclerc]

Here is below the error RPC client span, as you highlight, it lacks the server.address and server.port that are required for efficient troubleshooting.

Should we discuss with the instrumentation library go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc and see if they could enrich the error spans with server.address & server.port even when invalid. My reading of the semconv for RPC client spans (here) is that those attributes should be set even when invalid.

@julianocosta89 what do you think of this next step to engage with the go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc maintainers to improve the context attributes when gRPC calls fail?

I think that the code relates to:
https://github.com/open-telemetry/opentelemetry-go-contrib/blob/468d883fac706bf86431642429dd96bd9700172c/instrumentation/google.golang.org/grpc/otelgrpc/stats_handler.go#L355-L357

[julianocosta89]

• ery call through chargeCard, recording which gRPC endpoint was targeted for the payment charge request
• When the paymentUnreachable feature flag is disabled, the attribute reflects the configured payment service address (from PAYMENT_ADDR)

I believe this is gold!
That's one of the key points of the Demo: Provide actual data to OTel maintainers and contributors.

Yes! We should definitely engage with otelgrpc instrumentation folks.

[cyrille-leclerc]

See:

• [Feature]: otelgrpc — capture server.address and server.port for failed RPC client spans opentelemetry-go-contrib#8781

[puckpuck]

Given this is a bug upstream in grpc go, should we create an issue our side to track and close this PR?