Linux user-land rootkit
libprocesshider: ld.so.preload Process Hiding and Checks
libprocesshider is a small open-source preload library that hides one process from ps and top, used by cryptomining crews. Its traces, checks and cleanup.
- Kind
- Linux user-land rootkit · Userland layer
- Platforms
- Linux
- First seen
- 2014
- Status
- In the wild
- Checked by
- Linux kernel
Overview
libprocesshider is a public, open-source shared library that hides a process from tools such as ps, top and lsof. Its repository and the accompanying Sysdig blog post date from August 2014. The post was a demonstration, and it showed that a system-call tracer still sees the hidden process. The code has not changed since 2019 and fits in about a hundred lines.
It is the simplest kind of userland rootkit. That is why it turns up in commodity intrusions:
- TeamTNT, January 2021. AT&T Alien Labs reported TeamTNT adding the tool, "copied from open source repositories", to its cryptominer and IRC bot. A base64-encoded script inside the binary unpacked it to
/usr/local/lib/systemhealt.soand registered it in/etc/ld.so.preload. The same script changed DNS settings, added systemd persistence and deleted bash history. - Hildegard, February 2021. Unit 42 described TeamTNT's Hildegard malware using the same technique. It wrote
/etc/ld.so.preloadand overrodereaddir()andreaddir64()to hide processes namedtmate,xmrigandziggy. Unit 42 names the librariesx86_64.soandxmrig.soand does not say they are libprocesshider.
How it loads and hides
Loading. The library does nothing until the dynamic linker loads it. The project's README shows the system-wide route: the library's path in /etc/ld.so.preload. From then on, every dynamically linked program started on the host loads it before libc (ld.so(8)). No restart is needed, and processes already running are not affected.
Hiding. The library defines its own readdir() and readdir64(). For each entry, it checks whether the directory being read is /proc and whether the entry is a PID. It reads the process name from /proc/<pid>/stat and drops the entry if the name equals one string compiled into the library (evil_script.py in the public source). It then hands the call to the real libc function, found with dlsym(RTLD_NEXT, …).
That is all it does, which leaves a lot visible:
| Not hidden | Why it matters |
|---|---|
/etc/ld.so.preload and the library file | cat /etc/ld.so.preload works on the live host |
The hidden process's /proc/<pid>/ directory | Reachable by PID; only the listing skips it |
| Its network sockets | ss and /proc/net/tcp still list them, with no owning process |
| Its CPU use | Load average and /proc/stat include it |
| Statically linked tools, and anything reading the kernel directly | They never load the library |
Programs that do not use libc's readdir | They read /proc without the filter |
The hidden name is a plain string in the library. Running strings on the .so usually shows what was hidden.
Traces on disk
| Location | What | Source |
|---|---|---|
/etc/ld.so.preload | One line with the library's path; the file is absent on a default Debian, Ubuntu or RHEL install | Repository, Alien Labs |
/usr/local/lib/libprocesshider.so | Default name in the README | Repository |
/usr/local/lib/systemhealt.so | TeamTNT's copy | Alien Labs |
x86_64.so, xmrig.so named in /etc/ld.so.preload | Hildegard's preload libraries | Unit 42 |
A small .so exporting readdir and readdir64 and importing dlsym and fopen | The library itself, whatever its name | Repository |
The hidden name inside it (strings) | The process it hides: a miner, bot or shell | Repository |
Times of /etc/ld.so.preload and the library | When hiding started; compare with service restarts | General |
Alien Labs lists the hash of TeamTNT's systemhealt.so in its report. Copy it from there if needed.
How to check a host
Live. The library does not hide its configuration, so ordinary commands already help. A static tool confirms.
cat /etc/ld.so.preload
ls -la --time-style=full-iso /etc/ld.so.preload "$(cat /etc/ld.so.preload)"
strings -n 4 "$(cat /etc/ld.so.preload)" | head -50 # the hidden process name is usually here
/media/ir/busybox ps # a static ps never loads the library
ss -tanp | grep -v 'users:' # sockets whose owner could not be listed
unhide brute proc # PIDs that exist but are missing from listings
ssfinds a socket's owner by walking/proc. A connection with nousers:field, while other connections show theirs, often belongs to a hidden process. Sockets of kernel threads and of other users' processes, when you are not root, also lack one.- Compare the busybox
pswith the system'sps: the difference is the hidden process. - To restore normal output, remove the line or move the file aside, after preserving them. New programs stop loading the library immediately.
Offline.
R=/mnt/evidence
cat "$R"/etc/ld.so.preload
readelf --dyn-syms -W /path/to/lib.so | grep -wE 'readdir|readdir64'
readelf -d /path/to/lib.so | grep NEEDED
strings -n 4 /path/to/lib.so | grep -vE '^(GLIBC|_|\.)' | head -40
grep -l "$(basename /path/to/lib.so)" "$R"/var/lib/dpkg/info/*.list 2>/dev/null || echo "no package owns it"
Then find what the library hid. The name inside it points to a miner, a bot or a shell. Look for that program, its unit or cron entry, and its configuration. The ld.so.preload guide and Linux Forensics' ld.so.preload artifact page go further.
What the checker looks for
The linuxkernel module reads etc/ld.so.preload, the libraries it names and other shared objects in the collection. For libprocesshider it shows:
- The preload entry. "Library preloaded into every program":
/etc/ld.so.preloadis absent on stock Debian, Ubuntu and RHEL installs. A library outside the packaged library folders, as/usr/local/libis, adds "Preloaded library outside the system library folders", which also says whether a package owns it. - The library's exports.
readdirandreaddir64raise "Library replaces libc directory and file functions", with the function names as evidence. The library is flagged this way even when no preload entry names it, if it is in the collection. - A missing library. When the preload file names a library the collection lacks, "Preloaded library not read" (the checker also skips libraries over 256 KiB in the system folders).
The checker's sample host, fin-jump-01, includes a synthetic library of this shape.
The checker does not see:
LD_PRELOADset in an environment or unit file;- the hidden program, its persistence or its traffic.
Use Linux Persistence for the configuration side, and follow the hidden name by hand.
Removal and recovery
- Preserve
/etc/ld.so.preload, the library and their times. Savepsoutput from a static tool and the list of running processes with their maps before changing anything. - Remove the preload. Delete the line, or the whole file if nothing legitimate uses it, then delete the library. Running processes keep the library mapped until they restart. Restart services, or reboot.
- Deal with what it hid. The hidden process is the real intrusion: a miner, an IRC bot, a reverse shell. Remove it and its persistence: units, cron,
authorized_keys, the DNS changes Alien Labs describes. - Decide on a rebuild. libprocesshider needs root to install system-wide, so the intruder had root. In-place cleaning is only reasonable when you can account for everything that root session did. Otherwise rebuild, and rotate the credentials and cloud keys on the host.
- Prevent it. Alert on writes to
/etc/ld.so.preload. Keep container hosts' Docker and Kubernetes APIs off the internet, which is how TeamTNT got in, per Unit 42 and Alien Labs.
Azazel and Symbiote use the same loading route with far more hooks. Diamorphine is the kernel-module tool TeamTNT used when it could build one.
Sources
Primary public reports, newest first.
- 2021-02-03Hildegard: New TeamTNT Cryptojacking Malware Targeting KubernetesPalo Alto Networks Unit 42
- 2021-01-27TeamTNT delivers malware with new detection evasion toolAT&T Alien Labs (LevelBlue)
- 2014-08-28Hiding Linux processes for fun + profitSysdig
- 2014-08-16libprocesshider source repositoryGitHub
See also
Related families
- AzazelLinux user-land rootkit
- SymbioteLinux user-land rootkit
- DiamorphineLinux LKM rootkit