Skip to content

Linux LKM rootkit

Skidmap Linux Rootkit: Traces and Offline Checks

Skidmap is a Linux cryptominer that loads kernel modules to hide files and fake CPU and network figures, and backdoors PAM. Its traces, checks and recovery.

Kind
Linux LKM rootkit · Kernel layer
Platforms
Linux
First seen
2019
Status
In the wild
Also known as
SkidMap
Checked by
Linux kernel

Overview

Skidmap is a Linux cryptomining malware family that goes further than most miners. It loads its own kernel modules to hide the miner. Those modules make the CPU look idle and falsify network figures. Skidmap also leaves several ways back in, among them a backdoored PAM module that accepts one password for every account.

Trend Micro described it in September 2019. Trend Micro notes that the kernel modules make cleanup harder than for user-mode miners, and that the extra access routes let it reinfect a host.

In July 2023, Trustwave SpiderLabs reported a new variant on its honeypots. It broke in through Redis servers without authentication. It carried kernel-module packages for many distributions and kernels, Alibaba, openEuler, EulerOS and Anolis among them.

Neither vendor attributes Skidmap to a named actor. MITRE ATT&CK tracks it as S0468.

How it loads and hides

2019 version (Trend Micro).

  • Access and install. A cron job fetches and runs an install script every minute. The script downloads the main binary to /var/lib/pc and runs it. The binary weakens the host:
    • it runs setenforce 0 and edits /etc/selinux/config to disable SELinux;
    • it adds the attackers' public key to authorized_keys;
    • it replaces pam_unix.so with a version that accepts a fixed password for any user.
  • Miner by distribution. On Debian-family systems the miner goes to /tmp/miner2. On CentOS and RHEL, an archive cos7.tar.gz is downloaded, decrypted with OpenSSL (Triple DES) and installed.
  • Components:
    • kaudited (/usr/bin/kaudited) drops and loads the kernel modules, picking the build that matches the running kernel, and installs a watchdog for the miner;
    • a replacement rm sometimes adds a cron job that downloads and runs a file;
    • the iproute module hooks getdents to hide files;
    • the netlink module falsifies network statistics for chosen addresses and ports, hides the pamdicks process, and reports a low CPU load.

2023 variant (Trustwave).

  • Access. The attackers write cron jobs through Redis keys named HA. Every ten minutes a job fetches a script, named b, c or f on different hosts.
  • Dropper. The script installs an ELF dropper at /var/lib/gif (earlier jpeg), marked immutable with chattr +i. The dropper reads /etc/*-release and picks one of several encrypted packages, staged under a new folder in /usr/include/: gold16, gold18, gold20, stream9, euler21, euler22.
  • Kernel modules:
    • mcpuinfo.ko fakes system load through a replaced /proc handler (fake_loading_proc_show);
    • kmeminfo.ko registers netfilter hooks (nf_register_net_hook) to inspect, change or drop packets;
    • mzoneinfo.ko is referenced but was never recovered.
  • Other changes:
    • the miner runs from /tmp/.miner (or .mimer) on Debian and Ubuntu, and from memory on RHEL and CentOS;
    • attacker SSH keys go into files that Trustwave's text spells /root/.ssh/authoried_keys and authoried_keys2. Check both spellings;
    • a reverse shell is appended to /etc/cron.hourly/prelink;
    • SELinux and auditd settings are changed;
    • a clean.sh script and a binary named wtmp wipe logs.

Trustwave adds that the attackers appear to block the unhide tool. ClamAV and rkhunter reported nothing on the infected honeypot, while chkrootkit flagged malicious module indicators.

Traces on disk

LocationWhatSource
Crontabs fetching scripts every minute (2019) or ten minutes (2023)Infection and reinfectionTrend Micro, Trustwave
/var/lib/pc; /usr/bin/kaudited2019 main binary; module loader and watchdogTrend Micro
File names crypto514, iproute514, netlink514, kswaped, systemd_network2019 components listed as IoCsTrend Micro
/tmp/miner2; cos7.tar.gz2019 miner, by distributionTrend Micro
A pam_unix.so that differs from its packageMaster-password backdoorTrend Micro
/var/lib/gif (immutable), earlier /var/lib/jpeg2023 dropperTrustwave
/usr/include/<new folder>/ holding gold*, stream*, euler* packages and a version.txt2023 stagingTrustwave
mcpuinfo.ko, kmeminfo.ko2023 kernel modulesTrustwave
/tmp/.miner, /tmp/.mimer2023 miner on Debian and UbuntuTrustwave
/root/.ssh/authoried_keys, authoried_keys2, or authorized_keys with unknown keysSSH backdoorTrend Micro, Trustwave
/etc/cron.hourly/prelink with a reverse shell appended2023 persistenceTrustwave
/etc/selinux/config set to disabled; altered audit configurationWeakened hostTrend Micro, Trustwave
Gaps in wtmp and system logsLog wipingTrustwave

Both reports list hashes and network indicators. Take them from the reports.

How to check a host

Live, with caution. The modules falsify CPU and network figures, so the usual "high CPU, find the process" approach fails. Look from outside and below:

cat /proc/sys/kernel/tainted
lsattr /var/lib/gif /var/lib/pc /usr/bin/kaudited 2>/dev/null   # immutable files are unusual there
for m in /sys/module/*; do [ -e "$m/initstate" ] && ! grep -q "^${m##*/} " /proc/modules && echo "only in sysfs: ${m##*/}"; done
crontab -l; ls -la /etc/cron.* /var/spool/cron* 2>/dev/null
  • Measure power draw, temperature or the hypervisor's view of the guest's CPU. Trustwave suggests unexplained fan noise and heat as a physical sign.
  • Read network flows on the switch or the cloud provider's flow logs. The host's own counters may be faked.
  • Capture memory before cleaning.

Offline.

R=/mnt/evidence
ls -la "$R"/var/lib/pc "$R"/var/lib/gif "$R"/var/lib/jpeg "$R"/usr/bin/kaudited "$R"/tmp/.miner "$R"/tmp/.mimer "$R"/tmp/miner2 2>/dev/null
find "$R" -xdev \( -name 'mcpuinfo.ko' -o -name 'kmeminfo.ko' -o -name 'mzoneinfo.ko' -o -name '*514' \) 2>/dev/null
ls -la "$R"/usr/include | grep -vE '\.h$'                       # unexpected folders among headers
cat "$R"/etc/cron.hourly/prelink; ls -la "$R"/root/.ssh/
debsums --root="$R" -c 2>/dev/null | grep -i pam               # Debian/Ubuntu
rpm --root "$R" -V pam 2>/dev/null                              # RHEL family
zgrep -hE 'taint|module verification' "$R"/var/log/kern.log* "$R"/var/log/messages* 2>/dev/null

The modules may sit outside /lib/modules: neither report gives their path. Search the whole image by name and by type (file reports ELF 64-bit LSB relocatable for .ko files). For each module you find, see checking kernel modules offline.

What the checker looks for

The linuxkernel module can see:

  • Module files under lib/modules/: "Unsigned module among signed modules" and "Module unknown to the kernel build and to every package". Package ownership is read from the dpkg database only, so on a RHEL-family host the second finding rests on modules.order alone. Hook-target names such as getdents, when the module carries them as strings, raise "Names kernel functions that rootkits hook"; netfilter registration imports appear only as context in that finding.
  • Kernel taint warnings in kern.log or messages. The journal is not read.
  • A module hidden from /proc/modules, if a module unlinks itself from /proc/modules but not from sysfs: "Loaded module missing from /proc/modules" from the collect script's _live/ outputs. Neither report says what Skidmap's modules do there.

It does not read:

  • cron, the Redis-written jobs or authorized_keys;
  • the PAM module, so it cannot spot the replaced pam_unix.so;
  • modules stored outside the module tree, which the reports suggest is likely.

Linux Persistence covers the PAM, cron and SSH key side.

Removal and recovery

  1. Preserve memory and the disk, including the immutable files and the staging folders.
  2. Expect reinfection.
    • Cut the entry point first: Redis exposed without authentication (2023) or whatever let the 2019 install script run.
    • Remove every cron entry, the SSH keys and the prelink reverse shell.
    • The 2019 replacement rm and the watchdog exist to put things back.
  3. Rebuild. A replaced pam_unix.so means any account could be entered with the master password. Kernel modules ran with full control. Reinstalling packages in place is not enough to trust the host. Rebuild it and restore data from backups.
  4. Rotate every password and key that authenticated on the host, since the PAM backdoor saw them. Revoke the attacker keys wherever the same authorized_keys might have been copied.
  5. Harden.
    • Require authentication on Redis and keep it off the internet; Trustwave notes that protected mode alone was not enough.
    • Keep SELinux enforcing, enforce module signatures, and alert on changes to pam_unix.so and /etc/cron.*.

See Diamorphine and libprocesshider for the lighter hiding tools other mining crews use.

Sources

Primary public reports, newest first.

  1. 2023-07-30Honeypot Recon: New Variant of SkidMap Targeting RedisTrustwave SpiderLabs (LevelBlue)
  2. 2019-09-16Skidmap Linux Malware Uses Rootkit Capabilities to Hide Cryptocurrency-Mining PayloadTrend Micro

See also

Related families