Investigate Lambda AI integration in the CI.

[ameukam]

We should look at how to integrate Lambda AI witch our current CI so we can offer access for more recent GPUs for testing.

/milestone v1.36
/priority important-longterm

[ameukam]

/kind feature

[elieserr]

Would the proper way to implement this be to build a custom kubetest2 deployer for this particular provider?

Im also curious if we know wich kind of jobs will be candidates for using this? I mean, based on the credits im not sure if usage will be on demand by some jobs on reuquest from certain privileged users or if we want ot have this integrated into periodic/presubmits jobs.

[ameukam]

Would the proper way to implement this be to build a custom kubetest2 deployer for this particular provider?

Completely agree on this. I vibe-coded one https://github.com/ameukam/kubetest2-lambdalabs for experimentation.

Im also curious if we know wich kind of jobs will be candidates for using this? I mean, based on the credits im not sure if usage will be on demand by some jobs on reuquest from certain privileged users or if we want ot have this integrated into periodic/presubmits jobs.

I think what's matter is the CI signal we want to get by using real hardware. IMO, subprojects toward opiniated scheduling like Kueue are the first audience for this.

[ameukam]

Talked to SIG testing about this. We don't necessarily want to aim the k/k for testing.
We should start with sub-projects in need of special accelerators.
Probably build some karpenter provider with kOps integration to start.

[BenTheElder]

Some subproject that runs on K8s and has accelerator integration ... perhaps DRANet? cc @gauravkghildiyal

DRANet would need higher end GPUs for integrating with the NICs IIRC, though it should be able to run pretty quickly and then release them ...

[dims]

see https://gist.github.com/dims/b033e62eab37b0a13807d1ef05304bee on how to run the GPU device plugin tests in lambda

lambdactl is here - https://github.com/dims/lambdactl