Skip to content

Linux user-land rootkit

libprocesshider: ld.so.preload Process Hiding and Checks

libprocesshider is a small open-source preload library that hides one process from ps and top, used by cryptomining crews. Its traces, checks and cleanup.

Kind
Linux user-land rootkit · Userland layer
Platforms
Linux
First seen
2014
Status
In the wild
Checked by
Linux kernel

Overview

libprocesshider is a public, open-source shared library that hides a process from tools such as ps, top and lsof. Its repository and the accompanying Sysdig blog post date from August 2014. The post was a demonstration, and it showed that a system-call tracer still sees the hidden process. The code has not changed since 2019 and fits in about a hundred lines.

It is the simplest kind of userland rootkit. That is why it turns up in commodity intrusions:

  • TeamTNT, January 2021. AT&T Alien Labs reported TeamTNT adding the tool, "copied from open source repositories", to its cryptominer and IRC bot. A base64-encoded script inside the binary unpacked it to /usr/local/lib/systemhealt.so and registered it in /etc/ld.so.preload. The same script changed DNS settings, added systemd persistence and deleted bash history.
  • Hildegard, February 2021. Unit 42 described TeamTNT's Hildegard malware using the same technique. It wrote /etc/ld.so.preload and overrode readdir() and readdir64() to hide processes named tmate, xmrig and ziggy. Unit 42 names the libraries x86_64.so and xmrig.so and does not say they are libprocesshider.

How it loads and hides

Loading. The library does nothing until the dynamic linker loads it. The project's README shows the system-wide route: the library's path in /etc/ld.so.preload. From then on, every dynamically linked program started on the host loads it before libc (ld.so(8)). No restart is needed, and processes already running are not affected.

Hiding. The library defines its own readdir() and readdir64(). For each entry, it checks whether the directory being read is /proc and whether the entry is a PID. It reads the process name from /proc/<pid>/stat and drops the entry if the name equals one string compiled into the library (evil_script.py in the public source). It then hands the call to the real libc function, found with dlsym(RTLD_NEXT, …).

That is all it does, which leaves a lot visible:

Not hiddenWhy it matters
/etc/ld.so.preload and the library filecat /etc/ld.so.preload works on the live host
The hidden process's /proc/<pid>/ directoryReachable by PID; only the listing skips it
Its network socketsss and /proc/net/tcp still list them, with no owning process
Its CPU useLoad average and /proc/stat include it
Statically linked tools, and anything reading the kernel directlyThey never load the library
Programs that do not use libc's readdirThey read /proc without the filter

The hidden name is a plain string in the library. Running strings on the .so usually shows what was hidden.

Traces on disk

LocationWhatSource
/etc/ld.so.preloadOne line with the library's path; the file is absent on a default Debian, Ubuntu or RHEL installRepository, Alien Labs
/usr/local/lib/libprocesshider.soDefault name in the READMERepository
/usr/local/lib/systemhealt.soTeamTNT's copyAlien Labs
x86_64.so, xmrig.so named in /etc/ld.so.preloadHildegard's preload librariesUnit 42
A small .so exporting readdir and readdir64 and importing dlsym and fopenThe library itself, whatever its nameRepository
The hidden name inside it (strings)The process it hides: a miner, bot or shellRepository
Times of /etc/ld.so.preload and the libraryWhen hiding started; compare with service restartsGeneral

Alien Labs lists the hash of TeamTNT's systemhealt.so in its report. Copy it from there if needed.

How to check a host

Live. The library does not hide its configuration, so ordinary commands already help. A static tool confirms.

cat /etc/ld.so.preload
ls -la --time-style=full-iso /etc/ld.so.preload "$(cat /etc/ld.so.preload)"
strings -n 4 "$(cat /etc/ld.so.preload)" | head -50     # the hidden process name is usually here
/media/ir/busybox ps                                    # a static ps never loads the library
ss -tanp | grep -v 'users:'                             # sockets whose owner could not be listed
unhide brute proc                                       # PIDs that exist but are missing from listings
  • ss finds a socket's owner by walking /proc. A connection with no users: field, while other connections show theirs, often belongs to a hidden process. Sockets of kernel threads and of other users' processes, when you are not root, also lack one.
  • Compare the busybox ps with the system's ps: the difference is the hidden process.
  • To restore normal output, remove the line or move the file aside, after preserving them. New programs stop loading the library immediately.

Offline.

R=/mnt/evidence
cat "$R"/etc/ld.so.preload
readelf --dyn-syms -W /path/to/lib.so | grep -wE 'readdir|readdir64'
readelf -d /path/to/lib.so | grep NEEDED
strings -n 4 /path/to/lib.so | grep -vE '^(GLIBC|_|\.)' | head -40
grep -l "$(basename /path/to/lib.so)" "$R"/var/lib/dpkg/info/*.list 2>/dev/null || echo "no package owns it"

Then find what the library hid. The name inside it points to a miner, a bot or a shell. Look for that program, its unit or cron entry, and its configuration. The ld.so.preload guide and Linux Forensics' ld.so.preload artifact page go further.

What the checker looks for

The linuxkernel module reads etc/ld.so.preload, the libraries it names and other shared objects in the collection. For libprocesshider it shows:

  • The preload entry. "Library preloaded into every program": /etc/ld.so.preload is absent on stock Debian, Ubuntu and RHEL installs. A library outside the packaged library folders, as /usr/local/lib is, adds "Preloaded library outside the system library folders", which also says whether a package owns it.
  • The library's exports. readdir and readdir64 raise "Library replaces libc directory and file functions", with the function names as evidence. The library is flagged this way even when no preload entry names it, if it is in the collection.
  • A missing library. When the preload file names a library the collection lacks, "Preloaded library not read" (the checker also skips libraries over 256 KiB in the system folders).

The checker's sample host, fin-jump-01, includes a synthetic library of this shape.

The checker does not see:

  • LD_PRELOAD set in an environment or unit file;
  • the hidden program, its persistence or its traffic.

Use Linux Persistence for the configuration side, and follow the hidden name by hand.

Removal and recovery

  1. Preserve /etc/ld.so.preload, the library and their times. Save ps output from a static tool and the list of running processes with their maps before changing anything.
  2. Remove the preload. Delete the line, or the whole file if nothing legitimate uses it, then delete the library. Running processes keep the library mapped until they restart. Restart services, or reboot.
  3. Deal with what it hid. The hidden process is the real intrusion: a miner, an IRC bot, a reverse shell. Remove it and its persistence: units, cron, authorized_keys, the DNS changes Alien Labs describes.
  4. Decide on a rebuild. libprocesshider needs root to install system-wide, so the intruder had root. In-place cleaning is only reasonable when you can account for everything that root session did. Otherwise rebuild, and rotate the credentials and cloud keys on the host.
  5. Prevent it. Alert on writes to /etc/ld.so.preload. Keep container hosts' Docker and Kubernetes APIs off the internet, which is how TeamTNT got in, per Unit 42 and Alien Labs.

Azazel and Symbiote use the same loading route with far more hooks. Diamorphine is the kernel-module tool TeamTNT used when it could build one.

Sources

Primary public reports, newest first.

  1. 2021-02-03Hildegard: New TeamTNT Cryptojacking Malware Targeting KubernetesPalo Alto Networks Unit 42
  2. 2021-01-27TeamTNT delivers malware with new detection evasion toolAT&T Alien Labs (LevelBlue)
  3. 2014-08-28Hiding Linux processes for fun + profitSysdig
  4. 2014-08-16libprocesshider source repositoryGitHub

See also