Linux Malware Detection: The Right Way To Do It
The recent (as of May 2013) publicity of malware found on many Linux web servers, dynamically injecting malicious content into the legitimate content being served, is leading many people responsible for administering or securing Linux systems to re-examine how they should be determining if their Linux systems are compromised by this malware, or any other for that matter.
The malware families known as Cdorked and Darkleech have infected servers running a variety of different distributions (Debian, CentOS, etc.) and a variety of different web servers (Apache, nginx, lighttpd). They have taken a variety of forms, including iframe-injecting Apache modules, replacement daemon binaries, and malicious kernel modules. The attackers are reported to have gained access to the infected servers by a number of different means (Cpanel vulnerabilities, stolen credentials, etc.).
We've seen two different kinds of advice on how to detect Cdorked and Darkleech infections. The first and worst suggests looking for symptoms that only apply to a specific instance of this kind of infection, which can change quite easily over time. The second, and slightly better, suggests using commands on the system to verify the integrity of files on disk and find unexpected files. For example, "rpm -v" on Red Hat-based systems or "debsums" on Debian-based ones. The problem with this approach is that one cannot trust commands executed on a rooted system to tell you the truth. As others have pointed out, sometimes even as they advise this approach to detecting Linux malware, the attacker can modify the rpm database or the rpm command itself (or the Debian equivalents), or even the kernel, to cover up evidence of the infection.
But the idea behind this second approach, trying to verify the software on the system, is the right one. It's just the means suggested that is flawed. The right way to get trustworthy verification is via memory forensics. It enables inspection of the state of a system without relying on or making assumptions about the integrity of the kernel or any programs on the system. Indeed, it enables detection of modifications to the kernel such as made by rootkits. It also enables one to identify and verify the integrity of programs running on a system, by comparing the programs (and shared libraries) in memory with the software one expects to be running on the system. Typically on Linux this means software from vendor-provided packages.
This is what Second Look® does, and it is the most reliable way to detect not only the Linux malware currently in the spotlight, Cdorked and Darkleech, but also the future, increasingly sophisticated infections that will inevitably follow. If you're serious about Linux security, then invest in the leading edge Linux malware detection technology, Second Look®: Linux Memory Forensics.