Fix the Cannot Find -lcuda Linker Error

Fix /usr/bin/ld: cannot find -lcuda: No such file or directory by making a matching NVIDIA driver library available under the filename and search path your linker requires. The usual causes are a missing libcuda.so link, a driver directory absent from the compiler’s search path, or an incompatible library architecture. Check the failing command before reinstalling CUDA: a working GPU runtime does not guarantee that a compiler can link against the driver.

Use the diagnostic immediately before the failure to choose the repair.

Symptom	Likely Cause	Fix
The driver library exists outside the compiler’s search directories	Missing build-time library path	Add its directory through `LIBRARY_PATH` for GCC, or an explicit `-L` option
Only the versioned driver library exists	Missing unversioned linker filename	Restore the package-provided link or use a private linker-only symlink
The error names an exact versioned filename	The requested file is absent from linker search directories	Locate that exact file and expose its containing directory
The output includes `skipping incompatible`	Library and compilation target have different architectures	Select the matching-architecture driver library
Failure starts after changing container images	Development libraries disappeared, or mounted drivers are not compiler-visible	Check image contents and runtime-mounted driver paths separately
No usable driver library exists	Missing driver components, container injection, or build stubs	Restore the appropriate component for the execution environment

What the CUDA Linker Error Means

/usr/bin/ld is the linker. Its “cannot find” message means it cannot resolve a requested library through the applicable search directories, not necessarily that CUDA is completely missing.

The library argument determines the filename it needs:

-lcuda normally searches for libcuda.so, or a suitable static archive.
-l:libcuda.so.1 requests that exact filename. The colon changes how GNU ld interprets the library name.

Consequently, having libcuda.so.1 on disk does not, by itself, satisfy a request for libcuda.so.

Also distinguish the two CUDA libraries:

libcuda is the CUDA Driver API library, supplied by the NVIDIA driver or, for linking without a driver, a toolkit stub.
libcudart is the CUDA Runtime API library, supplied with CUDA runtime/toolkit components. It is not a replacement for libcuda.

A program can encounter this linker error after it has already started. vLLM, for example, can reach it while Triton compiles a cuda_utils extension during model initialization (vLLM issue tracker). That is still a build-time failure, even though it appears inside a Python traceback.

How to Fix the CUDA Linker Error

The error is resolved by locating the required driver library, adding its verified directory to the build-time search path, and restoring a missing linker filename.

Step 1: Inspect the Failing Command in the Affected Environment

Find the complete compiler command in the log, not just the final collect2 or Python exception.
Identify the requested library filename and the directories following each -L option.
Run the following checks inside the same container, environment, or job that fails, not only on its host.

For a process expected to use a GPU, check driver visibility:

nvidia-smi

Successful output is useful, but the required linker filename must exist. If this also fails, investigate driver installation or GPU exposure before treating the problem as only a library-path issue.

Check the existing environment:

printenv LIBRARY_PATH LD_LIBRARY_PATH

The important distinction is:

Setting	What It Controls
`LIBRARY_PATH`	Additional library directories used by GCC during linking
`-L/path/to/lib`	A library search directory explicitly passed to the linker
`LD_LIBRARY_PATH`	Shared-library lookup by the runtime dynamic loader

For GCC’s direct library lookup, changing only LD_LIBRARY_PATH is not a dependable fix. Also, GNU ld invoked directly does not use GCC’s LIBRARY_PATH mechanism; give it an explicit -L option.

Step 2: Locate and Inspect the Driver Libraries

Search for the available files and symlinks:

sudo find / -name 'libcuda.so*' 2>/dev/null

The quotes prevent your shell from expanding the wildcard before find receives it. In a container running as root, omit sudo; an unprivileged search may miss inaccessible directories.

Use the directory that actually contains the appropriate library. Common locations include /usr/lib/x86_64-linux-gnu on x86-64 Linux and, in some installations, /usr/local/cuda/lib64. Do not assume either contains the driver library merely because the directory exists.

For example, if your search finds the installed driver under the first location, record it:

export CUDA_DRIVER_LIBDIR=/usr/lib/x86_64-linux-gnu

Inspect the filenames and symlink targets:

ls -l "$CUDA_DRIVER_LIBDIR"/libcuda.so*

Inspect the files behind those links:

file -L "$CUDA_DRIVER_LIBDIR"/libcuda.so*

Check for a broken symlink, the wrong architecture, or a path under a toolkit stubs directory. If the linker reports an incompatible candidate, resolve the architecture mismatch before selecting that directory.

Step 3: Add the Correct Build-Time Search Path

For a build or JIT compilation launched through GCC, expose the directory you verified:

export LIBRARY_PATH="$CUDA_DRIVER_LIBDIR${LIBRARY_PATH:+:$LIBRARY_PATH}"

This preserves an existing value without adding an empty path entry when the variable was unset.

If you control the compiler command, an explicit linker directory is another option. For an existing object file named main.o, the command would be:

gcc main.o -L"$CUDA_DRIVER_LIBDIR" -lcuda -o app

Keep the library after the object files that use it. Adapt the filenames to your build rather than running this example against nonexistent files.

Rerun the original failing build or application from this shell. If it succeeds, the directory was the missing piece. If it still cannot resolve the requested library, inspect the exact filename rather than adding more unrelated CUDA directories.

Step 4: Restore a Missing Linker Filename if Necessary

If the command requests the unversioned library but the directory contains only a valid libcuda.so.1, the search path alone cannot fix it.

The preferred permanent repair is to restore the distribution package that supplies the driver’s development/linker files. Package names vary by distribution and NVIDIA driver branch; use the package matching the installed driver and target architecture, not a package name guessed from the CUDA Toolkit version.

When package repair is unavailable, a private linker directory can provide a narrowly scoped workaround:

Confirm that CUDA_DRIVER_LIBDIR contains a readable, real driver library for your build’s architecture.
Create a user-owned directory:

mkdir -p "$HOME/.local/lib/cuda-link"

Create the missing libcuda.so symlink in the private directory, pointing to the real driver’s libcuda.so.1:

ln -s "$CUDA_DRIVER_LIBDIR/libcuda.so.1" "$HOME/.local/lib/cuda-link/libcuda.so"

If that destination already exists, inspect it rather than forcing an overwrite.

Add the private linker directory to GCC’s LIBRARY_PATH while preserving any existing entries:

export LIBRARY_PATH="$HOME/.local/lib/cuda-link${LIBRARY_PATH:+:$LIBRARY_PATH}"

From the shell with the updated LIBRARY_PATH, rerun the failing build or restart the application and trigger its JIT compilation again.

This does not replace the installed driver or alter system-managed libraries. Keep the real driver available to the runtime loader, and do not create a versioned driver filename pointing to a toolkit stub.

If neither the real driver nor a suitable build stub exists, a symlink cannot help. Install the appropriate driver components on the GPU host, restore driver injection into the container, or provide toolkit stubs for a build-only environment.

Docker, Kubernetes, and vLLM Special Cases

Containers require the mounted real driver directory in LIBRARY_PATH for JIT linking, while image builds use development components such as linker stubs.

Runtime-Mounted NVIDIA Drivers

The NVIDIA Container Toolkit can expose driver libraries only when a container starts. A common mounted file is /usr/local/nvidia/lib64/libcuda.so.1, but the location depends on the image and deployment.

Find the exact versioned library inside the affected container:

find /usr/ -name 'libcuda.so.1'

If the mounted driver directory is /usr/local/nvidia/lib64, run this before starting vLLM:

export LIBRARY_PATH=/usr/local/nvidia/lib64:$LIBRARY_PATH

Then start vLLM from that shell and rerun the request or initialization that previously failed. This exposes the mounted directory to GCC rather than only to the runtime loader.

For a command using -l:libcuda.so.1, the mounted versioned file can satisfy the request directly. A command requesting the unversioned library also needs the corresponding unversioned filename.

For a persistent Kubernetes fix, place the verified directory in the application container’s environment within the workload’s Pod template:

env:
  - name: LIBRARY_PATH
    value: /usr/local/nvidia/lib64

Preserve any other build-time directories your image requires. Apply the updated workload configuration, wait for the replacement Pod to become ready, and rerun the failing workload.

Exporting a variable inside an interactive kubectl exec shell does not change the environment of an application process that is already running.

Development Images Versus Runtime Images

NVIDIA CUDA devel images include development components such as linker stubs; their runtime counterparts do not include those development stubs. Switching image types can therefore expose a JIT linking failure even when the driver is mounted successfully at container startup (llm-d issue tracker).

Choose the fix based on when linking happens:

During image build: An ordinary Docker build does not automatically receive the host’s runtime-mounted driver libraries. Use an appropriate development build stage and its linker stubs.
After container startup: Make the mounted real driver directory visible to the compiler. A smaller runtime image may also need a compiler and other development dependencies if the application performs JIT compilation.

After changing the image or environment, rebuild or redeploy it and test the actual JIT-compiling workload. A successful container start alone is not sufficient.

When the Linker Skips an Incompatible Library

A skipped library needs an architecture fix; a compatible library outside the search directories needs a path fix.

This diagnostic identifies a different problem from a simple missing directory:

/usr/bin/ld: skipping incompatible /lib/i386-linux-gnu/libcuda.so when searching for -lcuda

That path commonly identifies a 32-bit library. A 64-bit build cannot use it.

Check the machine architecture:

uname -m

Check the compiler’s configured target:

gcc -dumpmachine

If the build has already produced an object file, inspect it too, replacing main.o with an actual build artifact:

file main.o

The object file matters because compilation flags or a cross-compiler can select a target different from the host.

To finish the repair:

Locate or install the driver library matching the compilation target.
Remove incorrect architecture directories from explicit linker options and build environment settings.
Point the build at the matching library directory.
Rebuild affected objects if their architecture was also wrong, then rerun the application.

Do not copy an incompatible library into a preferred directory or change its symlink name. Neither action changes its architecture.

Using CUDA Stubs and Compatibility Libraries

CUDA compatibility libraries are distinct from toolkit stubs and are intended only for deployments using NVIDIA CUDA forward compatibility with a supported driver and GPU combination.

Toolkit Stubs for Build-Only Environments

A CUDA driver stub lets a program link without a working GPU driver in the build environment. It does not provide a functioning driver for execution.

A common toolkit stub directory is /usr/local/cuda/lib64/stubs; other installations place it beneath the toolkit’s target-specific directory. Locate the toolkit stub named libcuda.so with find before adding its directory to LIBRARY_PATH.

If that directory contains the stub needed by your build, expose it to GCC:

export LIBRARY_PATH="/usr/local/cuda/lib64/stubs${LIBRARY_PATH:+:$LIBRARY_PATH}"

Run the build, then deploy the result into an environment with the real NVIDIA driver. Keep stub directories out of runtime LD_LIBRARY_PATH, and test the resulting application against the real driver.

A stub named libcuda.so does not satisfy an exact versioned-filename request simply because it represents the same API.

CUDA Compatibility Libraries

A search may also find a path such as /usr/local/cuda-12.9/compat/libcuda.so.1. The toolkit version and directory vary by image, so use the actual search output.

Files under compat are CUDA forward-compatibility driver components, not the same thing as toolkit stubs. Finding one does not make it the best library for your deployment.

Use compatibility libraries only when the deployment intentionally uses NVIDIA CUDA forward compatibility with a supported driver and GPU combination. Otherwise, prefer the driver supplied by the host or container runtime rather than adding every discovered CUDA directory to both search paths.

Verify the Fix and Make It Persistent

The decisive test is the operation that originally failed. A basic GPU availability check can succeed while a later Triton or PyTorch compilation still fails.

Confirm all of the following:

The original compiler or JIT command finishes successfully.
The generated extension loads without a missing-driver error.
The application completes the initialization, inference, or training operation that triggered the failure.
A fresh process works using the saved configuration, not just the current interactive shell.

For a separate runtime-loading check in a Python environment, run:

python -c 'import ctypes; ctypes.CDLL("libcuda.so.1"); print("CUDA driver library loaded")'

This checks that the dynamic loader can open the driver library. Loading libcuda.so.1 does not verify that a GPU is usable or that the compiler can resolve the filename requested by the failing command.

Save the working build-time path where the application actually receives its environment:

In the build configuration for a compiled application.
In the launch script or service configuration for a server.
In the container environment or Kubernetes workload template for a deployment.

An interactive shell configuration is not automatically read by services or container entrypoints. Restart from the saved configuration and rerun the workload that originally failed.

Frequently Asked Questions

Can Installing Python CUDA Packages Fix This Error?

Installing Python CUDA runtime packages alone usually does not fix a missing driver-library link. Packages providing components such as the CUDA runtime or NVRTC do not replace the host’s NVIDIA driver installation or automatically expose its library directory to GCC.

A Python package reinstall helps only if the package itself supplies or configures the missing build dependency.

Should I Downgrade vLLM to Fix This Error?

A downgrade may avoid the compilation path that triggers the error, but it does not necessarily repair the linker configuration. A different release can change its Triton dependency, container base, or generated compiler command.

Compare the failing command and available libraries before treating the problem as a CUDA version mismatch.

Does This Approach Work for Other “Cannot Find” Linker Errors?

Yes: identify the requested filename, locate a compatible library, and make its directory available to the linker. Errors involving -lz, -lgfortran, or -llapack need their respective libraries or development packages, not NVIDIA driver changes.

A generic Linux “No such file or directory” message needs context: a missing executable, a broken symlink, and an unresolved linker library require different repairs.