Re: [tlug] server installation best practices/ worksheet

[Sigurd Urdahl <sigurdur@example.com>]

Erin D. Hughes wrote:
> My personal ones are pretty short but have always gotten me through.
This probably is a good demonstration on why it can be hard to find good 
BP/worksheets, what is a best practice on place is bad practice 
somewhere else.

The department I work in does system administration for customers, we 
are about fifteen in the team, most customers have to primary sysadmins, 
but we have 24/7 on call rotation (so you might end up having to work on 
"someone else's" system in the middle of the night.

Parts of Erin's list is BP (best) at my work place, but some would be 
"bad practices". Not because they are bad, but because we have 
standardised on other ways of doing things.
> 3.Disallow root log ons.[2]
> 4.Change SSH default port to something else.[2]
Both of these are no-no's in my department. It makes it easier to be on 
call sysadmin if we keep SSH at the standard port, and there have to be 
a very good reason for it if we expose that port to the internet. 
Allowing root logins also makes the on call sysadmin's job easier, we 
just try to change the passwords quite often. (we keep a secure password 
database, and we have different passwords for every security domain)

> 6.Don't install anything you do not need .  I usually do a minimal 
> install.
Indeed a good start, but; Over time the systems will have software you 
don't really need, accept it. Run things like debfoster to locate 
unnecessary packages, uninstall stuff you take out of use. But remember; 
having unecessary stuff running (say GDM on a server) does not impact 
performance much (it will swap out and be gone), it shouldn't affect 
security remotely (keep your firewalls updated, seperate into different 
DMZs). 

> 7.Scan it with Nessus/nmap to make sure you didn't miss anything.
And keep doing it, compare and locate changes.
> [1]Funny thing about hard drive mirroring it never seems to work if 
> you can't convince your boss to buy the second hard disc.
Educate your customer (or boss) that the extra cost in hardware needed 
to give systems some redundancy is nothing compared to the monetary or 
human resource cost of fixing things when they fail because of stupid 
things like a failing disk. Remember to make sure they understand that 
they usually answer for downtime to someone above them in the system. 
Another thing they should want to pay for is a remote management card so 
that you can power cycle the crashed server via VPN instead of commuting 
to the co-loc.

A server installation worksheet should also (IMHO) include

* document the installation properly
* recheck that all users change their passwords before the system goes 
into production use
* Use your distro's packages if possible, package other software to use 
the distro's package manager if  you need a package that is unavailble, 
use "manual" installations (./configure; make, make install) as a last 
resort.
* Make sure applications that log a lot get their own log-partition (we 
often use /var/log/app for things like Java applications servers), and 
applications that store a lot of data (e.g. a database) get it's own 
parition. Your base partitioning can be more or less the same on all 
kinds of servers.
* Make sure rootmail is delivered to an account that is read regularily 
- and that the amount of emails is kept down.
* Test critical stuff;
** Everything starts when you boot the server
** The server boots no matter what disk you remove from the RAID
** Failover actually works (if HA cluster)
* Make sure your monitoring system is correctly configured
* Make sure backup is correctly configured, and working.
* Time synchronisation is configured and working (ntp)
* logrotating must work
* get someone else to have a look at the documentation

kind regards,
-sig



--
Sigurd Urdahl
Linux, goofing, cooking, making fire, computer security, having a
beer. Give me good music.