Depends on what you mean by active. The kernel has the code in the binary, but on boot it goes through device discovery and initialization, so it doesn't initialize or run drivers for things that aren't present.
There are also architecture specific builds, so the aarch64 build won't include driver code at all for devices that are specific to x86.
There are also architecture specific builds, so the aarch64 build won't include driver code at all for devices that are specific to x86.