tlug: Security: Finding evidence of your cracker

["Atlas21 Shawn" <gray_s@example.com>]

This post was taken from the "GUIDE TO (mostly) HARMLESS HACKING," (=>) and
I found it to be quite informative. Some of it may be old hat to you Linux
security experts out there, but hopefully it will still find an audience
with some people like myself who may not be too familiar with Linux
security issues. I wasn't sure about posting it here, but cleared it with
Scott Stone first, so I'll just put on my asbestos suit and prepare for the
worst. :->

GUIDE TO (mostly) HARMLESS HACKING
Vol. 3 No. 11
Through the looking glass: Finding evidence of your cracker
____________________________________________________________
by Chris Kuethe
University of Alberta
Department of Mathematical Sciences
ckuethe@example.com
You've subscribed to Bugtraq and The Happy Hacker list, bought yourself a
copy of _The_Happy_Hacker_, and read _The_Cuckoo's_Egg_ a few times. It's
been a very merry Christmas, with the arrival of a cable modem and a load
of
cash for you, so you run out and go shopping to start your own hacker lab.
A
week or so later, you notice that one of your machines is being an
especially slow slug (Hi Sherwood!) and you've got no disk space. Guess
what
- you got cracked, and now it's time to clean up the mess. The only way to
be sure you get it right is to restore from a clean backup - usually
install
media and canonical source - but let's see what the h4x0r left for us to
study.
In late October of this year, we experienced a rash of attacks on some
workstations here at the University of Alberta's Department of Mathematical
Sciences. Most of our faculty machines run RedHat 5.1 (there's a good
platform  to learn how to try to secure...) since it's cheap and easy to
install.  Workstations are often dual-boot with Windows 95, but we'll be
phasing that out  as we get Citrix WinFrame installed. This paper is an
analysis of the compromise of one professor's machine.
One fine day I was informed that we'd just had another break-in, and it was
time for me to show my bosses some magic. But like a skilled card shark
who's forced to use an unmarked deck, my advantage of being at the console
had been tainted. Our cracker had used a decent rootkit and almost covered
her tracks.
In general, a rootkit is a collection of utilities a cracker will install
in
order to keep her root access. Things like versions of ps, ls, passwd, sh,
and other fairly essential utilities will be replaced with versions
containing back doors. In this way, the cracker can control how much
evidence she leaves behind. Ls gets replaced so that the cracker's files
don't show up, and ps is done so that her processes are not displayed
either. In general, after a crack, you can bet something very important on
a
sniffer having been installed. Packet sniffers - programs that record
network traffic - are not part of a rootkit per se, but they are nearly as
loved by hackers as a buggered copy of ls. Who wouldn't want to try snarf
other user's passwords?
In nearly all cases, you can trust the copy of ls on the cracked box to lie
like a rug. Don't bet on finding any suspicious files with it, and  don't
trust the file sizes or dates it reports; there's a reason why a rootkit
binary is generally bigger than the real one, but we'll get there in a
moment. In order to find anything interesting, you'll have to use find.
Find
is a clever version of 'ls -RalF <w> | grep <x> | grep <y> | ... | grep <z>
'. It has a powerful  matching syntax to allow precise specification of
where to look and what to look for. I wasn't being picky - anything whose
name began with a dot was worth  looking at. The command: find / -name ".*"
-ls
Sandwiched in the middle of a ton of useless temporary files and the usual
'.thingrc' files (settings like MS-DOS's .ini) we found
'/etc/rc.d/init.d/...'. Yes, with 3 dots. One dot by itself isn't
suspicious, nor are two. Play around with DOS for about two seconds and
you'll see why: '.' means "this directory" and '..' means "one directory
up." They exist in every directory and are necessary for the proper
operation of the file system. But '...' ? That has no special reason to
exist.
Well, it was getting late, and I was fried after a day of class and my
contacts were drying up, so I listed /etc/rc.d/init.d/ to check for this
object. Nada. Just the usual System V / RedHat 5.1 init files. To see who
was lying, changed my directory into /tmp/foo, the echoed the current date
into a file called '...' and tried ls on it. '...' was not found. I'd found
the first rootkit binary: a copy of ls written to not show the name '...' .
Now that we knew that '...' was not part of a canonical distribution, I
moved into to it and had a look. There were only two files; linsniffer and
tcp.log. I viewed tcp.log with more and made a list of the staff who would
get some unhappy news. Ps didn't show the sniffer running, but ps should
not
be trusted in this case, so I had to check another way.
We were running in tcsh, an enhanced C-syntax shell which supports
asynchronous (background) job execution. I typed './linsniffer &' which
told
tcsh to run the program called linsniffer in this directory, and background
it. Tcsh said that was job #1, with process ID 2640. Time for another ps -
and no linsniffer. Well, that wasn't too shocking. Either ps was hacked or
linsniffer changed its name to something else. The kicker: 'ps 2640'
reported that there were no processes available. Good enough. Ps got
cracked. This was the second rootkit binary. Kill the sniffer.
Now we check the obvious: /etc/passwd. There were no strange entries and
all
the logins worked. That is, the passwords were unchanged. In fact the only
weird thing was that the file had been modified earlier in the day. An
invocation of last showed us that 'bomb' had logged in for a short time
around 2:35 am. That time would prove to be significant. Ain't nobody here
but us chickens, and none of us is called bomb...
I went and got my crack-detection disk - a locked floppy with binaries I
trust - and mounted the RedHat CD. I used my clean ls and found that the
real ls was about 28K, while the rootkit one was over 130K! Would anyone
like to explain to me what all those extra bytes are supposed to be? File
has the answer: ELF 32-bit LSB executable, Intel 80386, version 1,
dynamically linked, not stripped. Aha! So when she compiled it, our script
kiddie forgot to strip the file. That means that gcc left all its debugging
info in the file. Indeed, stripping the program brings it down to 36K,
which
is about reasonable for the extra functionality (hiding certain files) that
was added.
Remember how I mentioned that the increased file size is important? This is
where we find out why. First, new "features" have been added. Second, the
binaries have verbose symbol tables, to aid debugging without having to
include full debug code. And third, many script kiddies like to compile
things with debugging enabled, thinking that it'll give them more
debug-mode
backdoors. Certainly our 'kiddie was naive enough to think so. Her copy of
ls had a full symbol table, and source and was compiled from
/home/users/c/chlorine/fileutils-3.13/ls.c - which is useful info. We can
fetch canonical distributions and compare those against what's installed to
get another clue into what she may have damaged.
I naively headed for the log files, which were, of course, pure as the
driven snow, but I still did find out something useful: this box seemed to
have TCP wrappers installed. OK, those must have failed somehow since she
got in to our system. On RH51, the TCP wrappers live in /usr/sbin/in.* so
what's this in.sockd doing in /sbin? Being Naughty, that's what. I munged
in.sockd through strings, and found some very interesting strings indeed. I
quote: You are being logged , FUCK OFF , /bin/sh , Password: , backon . I
doubt that this is part of an official RedHat release.
I quickly checked the other TCP wrappers, and found that RedHat's in.rshd
is
11K, and the one on the HD was 200K. OK, 2 bogus wrappers. It seems that,
looking at the file dates, this cracked wrapper came out the day after RH51
was released. Spooky, huh?
I noticed that these binaries, though dynamically linked, used libc5, not
libc6 which we have. Sure, libc5 exists, but nothing, and I mean nothing at
all uses it. Pure background compatibility code. After checking the other
suspect binaries, they too used libc5. That's where strings and grep (or a
pager) gets used.
Now I'm getting bored of looking by hand, so lets narrow our search a
little
using find. Try everything in October of this year... I doubt our cracker
was the patient sort - look at her mistakes so far - so she probably didn't
get on before the beginning of the month. I don't claim to be a master of
the find syntax, so I did this:
     find / -xdev -ls | grep "Oct" | grep -v "19[89][0-7]" > octfiles.txt
In English: start from the root, and don't check on other drives, print out
all the file names. Pass this through a grep which filters everything
except
for "Oct" and then another grep to filter out years that I don't care
about.
Sure, the 80's produced some good music (Depeche Mode) and good code (UN*X
/
BSD) but this is not the time to study history.
One of the files reported by the find was /sbin/in.sockd. Interestingly
enough, ps said that there was one unnamed process with a low (76) process
id owned by uid=0, gid=26904. That group is unknown on campus here - whose
is it? And how did this file get run so early so as to get that low a PID?
In.sockd has that uid/gid pair... funky. It has to get called from the init
scripts since this process appears on startup, with a consistently low PID.
Grepping the rc.sysinit file for in.sockd, the last 2 lines of the file are
this:
     #Start Socket Deamon
     exec in.sockd
Yeah, sure... That's not part of the normal install. And Deamon is spelled
wrong. Should a spell checker be included as an crack-detector? Well,
RedHat
isn't famous for poor docs and tons of typos, but it is possible to add
words to a dictionary. So our cracker tried to install a backdoor and tried
to disguise it by stuffing it in with some related programs. This adds
credibility to my theory that our cracker has so far confined her skills to
net searches for premade exploits.
The second daemon that was contaminated was rshd. About 10 times as big as
the standard copy, it can't be up to anything but trouble. What does rsh
mean here? RemoteSHell or RootShell? Your guess is as good as mine.
So far what we've found are compromised versions of ls, ps, rshd, in.sockd,
and the party's just beginning. I suggest that once you're finished reading
this, you do a web search for rootkit and see how many you can scrounge up
and defeat. You have to know what to look for in order to be able to remove
it.
While the log files had been all but wiped clean, the console still had
some
errors printed on it, quite a few after 0235h. One of these was a refusal
to
serve root access to / via nfs at 0246h. That coincided perfectly with the
last access time to the NFS manpage. So our script kiddie found something
neat, and she tried to mount this computer via NFS, but she didn't set it
up
properly. All crackers, I'd say, make mistakes. If they did everything
perfectly we'd never notice them and there would be no problems. But it's
the problems that arise from their flaws that cause us any amount of grief.
So read your manuals. The more thoroughly you know your system, the more
likely you are to notice abnormalities.
One of the useful things (for stopping a cracker) about NFS is that if the
server goes down, all the NFS clients with directories still mounted will
hang. You'll have to 120-cycle the machine to get it back. Hmmm. This
presents an interesting tool opportunity: write a script to detect an NFS
hack, and if a remote machine gets in, ifconfig that interface off.
Granted,
that presents a possible denial-of-service if authorized users get cut off.
But it's useful if you don't want your workstation getting compromised.
At this point I gave up. I learned what I'd set out to do - how to find the
crap left behind by a cracker. Since the owner of this system had all her
files on (removed) removable media there was no danger of them being in any
way compromised. The ~janedoe directory was mounted off a jaz disk which
she
took home at night, so I just dropped the CD into her drive and
reinstalled.
This is why you always keep user files on a separate partition, why you
always keep backups and why it's a good plan to write down where to get the
sources for things you downloaded, if you can't keep the original archives.
Now that we've accumulated enough evidence and we're merely spirited
sluggers pulverizing and equine cadaver, it's time to consider the
appropriate response. Similar to Carolyn's you-can-get-punched and
you-can-go-to-jail warnings, I would suggest that a vicious hack back is
not
appropriate. In Canada, the RCMP does actually have their collective head
out of the sand. I am not a lawyer, so don't do anything based on these
words except find a lawyer of your own. With that out of the way, suffice
it
to say that we're big on property protection here. Aside from finding a
lawyer of your own, my advice here is for you to call the national police,
whoever they are. People like the RCMP, FBI, BKA and KGB probably don't
mind
a friendly phone call, especially if you're calling to see how you can
become a better law-abiding citizen. Chances are, you'll get some really
good tips, or at least some handy references. And of course you'll know
someone who'll help you prosecute.
My communication with RCMP's Commercial Crimes unit (that includes theft of
computing and/or network services) can be summarized as follows: E-mail has
no expectation of privacy. You wish email was a secret, but wake up and
realize that it's riskier than a postcard. As a sysadmin, you can do
_ANYTHING_ you want with your computer - since it's your responsibility
either because you own it or because you are its appointed custodian - so
long as you warn the users first. So I can monitor each and every byte all
of my users send or receive, since they've been warned verbally,
electronically and in writing, of my intent to do so. My browse of the
FBI's
website shows similar things. But that was only browsing. Don't run afoul
of
provincial or state laws regulating the interception of electronic
communication either.
NOTE:     While I have attempted to make this reconstruction of events
     as accurate as possible, there's always a chance I might have
     misread a log entry, or have misinterpreted something. Further,
     this article is solely my opinion, and should not be read as
     the official position of my employer.
Appendix A: Programs you want to have in a crack-detection kit
===========
     find, ps, ls, cp, rm, mv
     gdb, nm, strings, file, strip
     (gnu)tar, gzip, grep
     less / more
     vi / pico
     tcsh / bash / csh / sh
     mount
     These should all be statistically linked. Items separated by commas
     mean "all of these," items separated by forward slashes mean "pick
     your favorite."

Appendix B: References
===========
WinFrame
     www.citrix.com
RedHat 5.1
     www.redhat.com
     www.rootshell.com
     www.netspace.org/lsv-archive/bugtraq.html
About the filesystem
     McKusik, M.K. , Joy, W.N. , Leffler, S.J. , Fabry, R.S.
     "A Fast File System for UNIX" Unix System Manager's Manual
     Computer Systems Reseach Group, Berkeley. SMM-14. April 1986
LEA and Computer Crime
     www.rcmp-grc.gc.ca/html/cpu-cri.htm
     www.fbi.gov/programs/compcrim.htm

--
  Chris Kuethe: System Administrator - U of A Math Dept
      http://www.[math.]ualberta.ca/~ckuethe/
pager: 403.917.6448             office: CAB553, x1704   cell: 903.9475
wargames@example.com               ckuethe@example.com
ckuethe@example.com        ckuethe@example.com


------------------------------------------------------------------
Next Technical Meeting: 12 December, 12:30 at Temple Univ. Japan
*NEW LOCATION* a map is available at http://www.tuj.ac.jp/maps.html
Next Nomikai: 15 January 1999, 19:30 Tengu TokyoEkiMae 03-3275-3691
------------------------------------------------------------------
more info: http://tlug.linux.or.jp                     Sponsor: PHT