Nvidia's Driver Stopped Building on Proxmox. The Fix Lives in a Branch You Cannot Download.

The upgrade looked routine. A Proxmox host in a homelab, a Friday apt full-upgrade, the usual stream of packages. Then the DKMS hook ran, tried to rebuild Nvidia's open kernel modules against the new kernel, and failed. The GPU that ran the local AI models, the photo library's face recognition, and the media server's hardware transcoding went dark, not because the hardware changed, but because a small guard in Nvidia's driver source was checking for the wrong thing.

The report landed on Nvidia's open GPU kernel module tracker at 05:55 UTC on October 9, as issue 1433. Driver 615.71.09 builds cleanly on Proxmox kernel 7.0.14-19-pve, the reporter wrote, and fails to compile on the newer kernels. Both affected kernels are official Proxmox packages, installed through the normal upgrade path, not locally compiled builds or release candidates. The package at fault is nvidia-kernel-open-dkms 615.71.09-2, distributed from Nvidia's own CUDA repository for Debian 13.

What breaks, and where

DKMS, the Dynamic Kernel Module Support system, exists to make exactly this scenario boring. When the kernel updates, DKMS rebuilds out-of-tree modules like Nvidia's automatically, in a hook during the upgrade. It is the piece that is supposed to keep GPU drivers, ZFS modules, and vendor hardware support alive across routine patching. On Proxmox kernels 7.0.14-20 and later, that quiet automation now fails for Nvidia's open modules, and it fails the same way every time: a compile error deep in the CXL code.

CXL, Compute Express Link, is the interconnect standard for attaching memory and accelerators to servers, mostly a datacenter technology. The irony: a feature most affected users will never touch is standing between their GPU and a working module. The build dies on a field the compiler considers incompletely typed, and the community's fix is small enough to read in a minute.

A guard that tests for the wrong thing

Here is the mechanism, simplified. Nvidia's driver defines a structure, nv_cxl, that embeds the kernel's struct cxl_dev_state by value. Embedding by value means the compiler must see the full definition of that struct, not just a promise that it exists. The driver source guards this code with a check for whether the kernel's CXL header file is present. If the header exists, the code compiles.

Proxmox's 7.0.14-20 kernel and later ship a stub cxl.h that declares a single function and nothing else. The header is present, so the guard passes, but struct cxl_dev_state is never fully defined. Meanwhile Nvidia's own NV_CXL_INIT_SUPPORTED macro, the correct signal, stays undefined. The compiler then hits a field with an incomplete type and stops. The header is there; the definition is not.

One of the community submitters published a test matrix to prove the point. On 7.0.14-19, both the stock and patched driver sources build all five modules. On kernels -20, -22, and -23, the stock source fails with the reported error while the patched source builds all five modules without errors. Small patch, clean matrix, exactly the kind of contribution open-source projects are supposed to absorb quickly.

Two fixes, both closed

Two people wrote fixes independently within a day of the report. Pull request 1435, opened October 9, changes how the nv_cxl fields are generated. Pull request 1438, opened October 10, swaps the header-presence test for the NV_CXL_INIT_SUPPORTED check. Then, on October 12, Andy Ritger, an Nvidia engineer whose GitHub profile carries the company label, closed both. First 1438 at 06:02 UTC, then 1435 at 21:30 UTC, each with almost the same message: "We have an equivalent fix in our tree already, and it should be included in our upcoming 620.xx releases."

The responses do not dispute the diagnosis or the patches. Ritger thanked both authors and noted the second was "nearly identical" to the first. The operative phrase is "our tree": the repository the contributors are writing in is a snapshot published per release, not the tree Nvidia develops in, so the merged fix lives somewhere the community cannot see, and the public code is not the code that will ship. This is how Nvidia has always run the open kernel module repo, a mirror with pull requests attached, but it still stings when the mirror rejects a correct patch.

Two correct patches arrived within a day of the bug report. Nvidia closed both, thanked the authors, and pointed at a branch with no release date.

About that branch: "620.xx" is a family, not a version. Nvidia has offered no date, no number, and no list of contents. The newest published open kernel module release, 615.78.08 from October 7, is a 615 build. There is no release to watch for, only a future to wait through.

Why the homelab cares

If your server has no Nvidia GPU, this is a curiosity. If it does, it joins a long ledger of driver-versus-kernel friction. Homelab GPUs usually do three jobs: local AI inference with tools like Ollama, machine learning for photo libraries (Immich's face recognition is often the heaviest container in the stack), and hardware transcoding for media servers like Jellyfin.

The stakes are higher because the open modules are now mandatory for recent hardware. Blackwell-generation GPUs, the RTX 50 series and professional cards, do not work with the older proprietary modules; install those and the card fails to initialize. For anyone who bought current-generation hardware for local models, the open modules are the only path, which makes this build failure a total GPU outage, not a degradation.

Unattended upgrades make it worse. One homelab operator documented an overnight driver upgrade that left the kernel module and userspace libraries out of sync and took their Immich instance offline until a reboot let DKMS rebuild. The lesson applies here in stronger form: after any dist-upgrade, check dkms status before trusting the GPU. A clean apt run can coexist with no working module at all.

Your options until 620.xx arrives

Three paths, none perfect. The simplest is pinning the kernel: proxmox-boot-tool's kernel pin command holds the host on 7.0.14-19, the last kernel where the stock driver builds. You keep the GPU and skip newer kernel patches, a trade that gets less comfortable the longer Nvidia takes.

The second is to apply one of the closed community patches yourself. Both are public, both have published test matrices, and the change is a handful of lines. The cost is maintenance: you are now carrying a local patch through every future driver update until the 620.xx branch lands and makes it redundant.

Or simply wait: if your GPU is not load-bearing, hold the working kernel and upgrade when 620.xx appears. Whichever path you take, never reboot a GPU host without verifying the modules rebuilt, and treat driver and kernel as a unit, not two independent upgrades.

It is a small, sharp illustration of what self-hosting costs. The software is free, the control is real, and the maintenance is yours. A guard in a driver you did not write can take out your GPU on a Friday night, and the fix can sit in a tree you cannot see. That is the deal, and homelabbers keep choosing it anyway.