Skip to content

Checking Linux Kernel Modules Offline for Rootkits

How to check a Linux host's kernel modules from an image: signatures, modules.order and modules.dep, package ownership, taint, symbols, live outputs and limits.

Published on 10 min read

TL;DR. From an image or a collection, work through the module tree of each installed kernel, /lib/modules/<release>/:

  1. List every .ko, .ko.zst and .ko.xz file. Compare the list with modules.order (what the kernel build produced) and modules.dep (what depmod indexed).
  2. Check which package owns each file, and whether its digest still matches.
  3. Look for the signature marker at the end of each module.
  4. Compare vermagic with the folder.
  5. Read the undefined symbols and strings of anything left over.
  6. Read the load configuration and the kernel log's taint warnings. If live outputs exist, compare /proc/modules with /sys/module.

None of this proves that a kernel is clean. A rootkit that never touched the disk, or that hid from the live collection, only shows in memory.

What to collect

ItemPathWhy
Module tree/lib/modules/<release>/ (/usr/lib/modules/ on merged-/usr systems), every kernelThe .ko files, modules.order, modules.dep, modules.builtin
Load configuration/etc/modules, /etc/modules-load.d/, /usr/lib/modules-load.d/, /etc/modprobe.d/, /usr/lib/modprobe.d/; RHEL family also /etc/sysconfig/modules/What loads at boot, and install lines that run commands
Package database/var/lib/dpkg/info/*.list and *.md5sums; /var/lib/rpm/Ownership and digests
Kernel logs/var/log/kern.log*, /var/log/messages*, /var/log/journal/Taint and signature warnings
Live state, if the host is running/proc/modules, a listing of /sys/module, /proc/sys/kernel/tainted, uname -a, dmesgWhat the kernel reports now
Boot files/boot/vmlinuz-*, /boot/initrd.img-* or /boot/initramfs-*.imgWhich kernels exist; modules loaded before the root file system

A disk image read on a clean analysis machine is the best source. A collection made on the running host reads files through the suspect kernel: a rootkit that hides files hides them from the collection too.

rootkits.dev's Linux collect script saves the live outputs under _live/ in the archive. The Linux Forensics kernel modules page covers each artifact in detail.

Pick the kernel first. _live/ gives the running release (uname -a). Otherwise look at /boot and at the folder names under /lib/modules. Check every installed kernel: the next reboot may use another one.

1. Files the kernel's own lists do not know

Two index files tell you what belongs:

  • modules.order is written by the kernel build. It lists every module built with that kernel, in build order.
  • modules.dep is written by depmod. It indexes every module present when depmod last ran, including updates/ and extra/, where DKMS and vendor packages install theirs.
PatternUsual meaning
In modules.order and modules.depBuilt with the kernel
In modules.dep only, under updates/ or extra/DKMS or a vendor package; check the owner
In neitherCopied in after depmod last ran. modprobe cannot load it by name; insmod with a path can. Suspicious
R=/mnt/evidence; K=6.8.0-45-generic; M="$R/lib/modules/$K"; W=./work; mkdir -p "$W"
( cd "$M" && find . -type f -name '*.ko*' | sed -E 's#^\./##; s/\.(zst|xz|gz)$//' | sort -u ) > "$W/on-disk.txt"
sed -E 's/\.(zst|xz|gz)$//' "$M/modules.order" | sort -u > "$W/order.txt"
cut -d: -f1 "$M/modules.dep" | sed -E 's/\.(zst|xz|gz)$//' | sort -u > "$W/dep.txt"
comm -23 "$W/on-disk.txt" "$W/dep.txt"      # never indexed by depmod
comm -23 "$W/on-disk.txt" "$W/order.txt"    # not built with this kernel

The sed -E expressions work with GNU and BSD sed. A root intruder can run depmod after planting a module, so presence in modules.dep is not a clean bill. modules.order comes from the kernel package and is rarely rewritten.

2. Package ownership and digests

On Debian and Ubuntu, each package's file list is in /var/lib/dpkg/info/<package>.list:

cat "$R"/var/lib/dpkg/info/*.list | grep -E '^(/usr)?/lib/modules/' \
  | sed -E 's#^/usr##; s#^/lib/modules/[^/]+/##; s/\.(zst|xz|gz)$//' | sort -u > "$W/owned.txt"
comm -23 "$W/on-disk.txt" "$W/owned.txt"    # module files no package installed
debsums --root="$R" -c 2>/dev/null | grep /lib/modules/    # packaged modules whose digest changed

Paths are normalised because merged-/usr systems list /lib/modules or /usr/lib/modules depending on the package. The comparison assumes the image holds the one kernel K, or that you filter owned.txt to it.

On the RHEL family:

find "$M" -type f -name '*.ko*' | while read -r f; do
  rpm --root "$R" -qf "${f#"$R"}" >/dev/null 2>&1 || echo "not owned: $f"
done
rpm --root "$R" -Va 2>/dev/null | grep /lib/modules/

rpm --root uses the database inside the image, so the analysis machine needs an rpm that reads its format. Recent RHEL and Fedora releases use SQLite.

Both databases live on the disk you are examining. An intruder with root can edit *.md5sums or the RPM database. For any module that matters, compare it with the same package version downloaded from the distribution's mirror. As the debsums manual puts it, the tool is "of limited use as a security tool". Package manager forensics on Linux Forensics covers the logs that show when packages came and went.

3. Signatures

A signed module ends with the marker ~Module signature appended~. The kernel's module signing documentation says the string "confirms that a signature is present", not that it is valid.

Compressed modules are signed before compression, so decompress first:

find "$M" -type f -name '*.ko*' | while read -r f; do
  case "$f" in
    *.zst) m=$(zstd -dcq "$f" | tail -c 28) ;;
    *.xz)  m=$(xz -dc "$f" | tail -c 28) ;;
    *.gz)  m=$(gzip -dc "$f" | tail -c 28) ;;
    *)     m=$(tail -c 28 "$f") ;;
  esac
  case "$m" in *"Module signature appended"*) ;; *) echo "unsigned: ${f#"$M"/}" ;; esac
done
modinfo -F signer "$M/kernel/drivers/net/dummy.ko.zst"     # compare signers across modules

How to read the result:

  • Unsigned on a signed host. Distribution kernels for Ubuntu, Debian, Fedora and RHEL sign their modules. One unsigned module among thousands of signed ones stands out.
  • A different signer. Modules with another signer are usually DKMS builds signed with a local machine owner key (MOK). Confirm which package built them.
  • Old tools. modinfo needs kmod 27 or later to print PKCS#7 signer fields. Older versions show empty fields or md4 (Ubuntu, 2020).
  • Validity. Checking that a signature is cryptographically valid needs the keys the kernel trusted: built-in keys, and on Secure Boot systems the platform and MOK keys. Those live in the kernel image and in firmware variables, not in the module tree. A present signature with the expected signer name is a strong hint, not proof.

4. Version magic

modinfo -F vermagic "$M/path/to/suspect.ko"

The first word is the kernel release the module was built for. A module whose vermagic names another release than its folder was built elsewhere and copied in. Syslogk's analysed sample, for example, carries a CentOS 6 release string.

5. Symbols and strings

Undefined symbols are what the module asks the kernel for. Strings show names it looks up at run time. Decompress compressed modules to a work folder first.

nm -u suspect.ko | awk '{print $2}' | sort
modprobe --show-modversions suspect.ko | head      # versioned imports; older kmod: --dump-modversions
strings -a -n 6 suspect.ko | grep -E 'sys_call_table|kallsyms|getdents|filldir|tcp4_seq_show|/proc/|/dev/shm|ld\.so\.preload'
SeenWhat it suggestsLegitimate uses
The string sys_call_tableLooking up the syscall table to patch it (Diamorphine, PUMAKIT)Almost none in modern modules
register_kprobe with the string kallsyms_lookup_nameFinding the unexported symbol lookup function, unexported since 5.7 (LWN)Some tracers, such as LTTng
register_ftrace_function with ftrace_set_filter_ipDiverting kernel functions through ftrace (KoviD, PUMAKIT)Live patching, tracers, some security agents
Strings naming filldir, filldir64, getdents64, tcp4_seq_show, killHook targets for hiding files, processes and connections (Reptile, Adore-ng)Rare outside debugging tools
commit_creds, prepare_credsChanging a process's credentialsoverlayfs, NFS, keyrings and many more: never a finding alone
nf_register_net_hookNetfilter hooksEvery firewall module: context only
call_usermodehelperStarting a user process from the kernelMany drivers: context only

Judge the combination, not one symbol. A module calling itself a sound driver that imports ftrace hooks and names filldir64 needs analysis. A Netfilter module importing nf_register_net_hook does not.

6. Load configuration

grep -rhvE '^\s*([#;]|$)' "$R"/etc/modules "$R"/etc/modules-load.d "$R"/usr/lib/modules-load.d 2>/dev/null
grep -rnE '^\s*(install|remove|softdep)\s' "$R"/etc/modprobe.d "$R"/usr/lib/modprobe.d 2>/dev/null
ls -la "$R"/etc/sysconfig/modules/ 2>/dev/null
grep -rnE 'insmod|modprobe' "$R"/etc/rc.local "$R"/etc/rc.d "$R"/etc/systemd/system "$R"/etc/cron* "$R"/var/spool/cron 2>/dev/null

Check every module name against step 1. Check every install command for a program in a writable folder. Kernel modules on Linux Persistence explains the files and how attackers use them. Look into the initramfs too (lsinitramfs on Debian, lsinitrd on RHEL): modules there load before the root file system.

7. Taint in the logs

The kernel warns when a module taints it:

<module>: loading out-of-tree module taints kernel.
<module>: module verification failed: signature and/or required key missing - tainting kernel
zgrep -hE 'taint|module verification' "$R"/var/log/kern.log* "$R"/var/log/messages* 2>/dev/null
journalctl -D "$R"/var/log/journal _TRANSPORT=kernel -o short-iso | grep -iE 'taint|module verification'

Each warning appears only for the first such module after a boot (kernel source). On a host that loads VirtualBox or ZFS at every boot, that module's name fills the slot and a later rootkit loads silently.

To decode a saved taint value, use the kernel's tainted kernels table. Bit 12 (O) means an out-of-tree module, bit 13 (E) an unsigned one.

KoviD resets the live value to 0. A taint warning in the log for a boot whose saved value is 0 is worth explaining.

8. The live outputs

If the collect script ran on the live host, compare what the kernel listed with what is on disk:

ComparisonMeaning
In /sys/module (with an initstate file) but not in /proc/modulesUnlinked from the module list but not from sysfs: what Diamorphine's and Reptile's public sources leave
In /proc/modules with no file in the module treeLoaded from another path, from memory, or deleted after loading
Configured to load at boot but absent from both listsFailed to load, was removed, or hides from both views
Taint letters in /proc/modulesWhich loaded modules set the taint

What offline checks cannot prove

  • A module that never reached the disk is invisible. Reptile's loader decrypts its module in memory. PUMAKIT's dropper runs from memfd. A module can also be deleted after insmod.
  • eBPF programs are not modules. TripleCross leaves cron and sudoers files, not a .ko.
  • Live outputs come from the suspect kernel. A rootkit that hides from sysfs and the module list, or fakes the taint value, makes them agree with each other.
  • A collection taken on the running host is filtered by the same rootkit.
  • A valid signature is not a verdict. Signed malicious and vulnerable drivers exist on Windows; on Linux a key enrolled as a MOK signs whatever its holder wants.
  • Package databases on the disk can be edited.

When the stakes justify it, capture memory and analyse it. Volatility 3's linux.malware.hidden_modules, linux.malware.check_modules, linux.malware.check_syscall and linux.tracing.ftrace cover module hiding and hooking (memory forensics with LiME and Volatility). What an offline rootkit check can prove goes through every layer.

With the checker

Drop the image's files or the collect script's archive on the rootkits.dev checker. Its linuxkernel module runs most of steps 1 to 8 in the browser. For each module under lib/modules/ it shows its modinfo fields, signer and version magic, whether modules.order and modules.dep list it, which dpkg package owns it, and its notable imports and strings. It flags:

StepFindings
1, 2"Module unknown to the kernel build and to every package" (not in modules.order, owned by no dpkg package); "Module not indexed by depmod", a low finding, since a root intruder can rerun depmod; "Out-of-tree module inside the kernel's own folder"; "Packaged differently from the other modules"
3"Unsigned module among signed modules"; "Signed by another key than the kernel's modules"
4"Module built for another kernel"; "Module name differs from its file name"
5"Looks up the system call table"; "Resolves unexported kernel symbols"; "Redirects kernel functions with ftrace"; "Names kernel functions that rootkits hook". Credential, user-mode helper and netfilter imports are context only
6"Boot configuration loads a flagged module"; "Boot configuration names a module that is not on disk"; "modprobe runs a command"
7The taint and refusal lines of kern.log, messages and a saved dmesg, such as "Kernel log: unsigned module loaded" and "Kernel log: out-of-tree module loaded"
8"Loaded module missing from /proc/modules"; "Loaded module with no file on disk"; "Set to load at boot, missing from the live module list"; "Kernel tainted by an out-of-tree or unsigned module"; "Loaded module carries taint flags"

What it leaves to you: it reads signers without verifying signatures, checks ownership in the dpkg database only (not RPM) and does not compare package digests, does not read the journal, /etc/sysconfig/modules/, rc scripts, cron or the initramfs, and reads no module outside lib/modules/. It also reads /etc/ld.so.preload and the libraries it names. "Try a sample" loads fin-jump-01, a synthetic Ubuntu host with a planted unsigned module, intel_rapl_perf, a preload library and live outputs where the module hides from /proc/modules; the sample walk-through reads it finding by finding. Nothing leaves the browser.

Frequently asked questions

Is an unsigned kernel module always malicious?

No. Out-of-tree drivers built locally (DKMS packages, VirtualBox, ZFS, some GPU and security agents) are often unsigned or signed with a local key, and on kernels that do not enforce signatures they load with only a taint warning. An unsigned module is a lead: check which package owns it, who the vendor is, and whether its symbols and strings fit its stated purpose.

Why not just run lsmod on the suspect host?

lsmod only formats /proc/modules, and most public LKM rootkits unlink themselves from that list as soon as they load. Some, such as Diamorphine and Reptile, leave their /sys/module entry, so comparing the two views helps; others, such as KoviD and Adore-ng, remove both. A memory image is the only independent view of what is loaded.

Why does modinfo show an empty signer or sig_hashalgo md4?

Since Linux 5.2 module signatures use a PKCS#7 layout that only kmod 27 and later can parse. An older modinfo prints incomplete signature fields even for properly signed modules. Use a current kmod on the analysis machine, or check for the signature marker at the end of the file.

What if the rootkit module is not under /lib/modules?

insmod loads a module from any path, and some loaders never write a .ko at all. Search the whole image for ELF relocatable files with a .modinfo section, read the load configuration and scripts for insmod paths, and remember that a module loaded from memory leaves no file to find.

Related articles

How Linux userland rootkits use /etc/ld.so.preload and LD_PRELOAD, why ls, cat and ldd on the host can lie, and how to check a system offline and live.
Layer by layer (firmware, boot, kernel, userland): what files and logs show about a rootkit, what they never show, and what to do when offline checks run out.
Hunt vulnerable (BYOVD) and malicious Windows drivers offline: service keys, driver files and signers, Amcache, LOLDrivers, the blocklist and CodeIntegrity.