A watch dog against SSH login attacks

Since some days I'm trying to secure a tiny pc that I am planning to connect to internet full-time to be used by me as development server and for deploying some custom web application to handle some job documentation (i.e. to generate invoices, manage job time, archive documents etc.). After I put it on internet I checked logs and I discovered that the world is full of boys that repeatedly try to guess root password for some obscure purposes, and I decided to configure the firewall(s) and to secure services at best of my knowledge of unix. And this knowledge, is not so much.
After taking some basic measures to prevent unwelcome guests, I checked if something OpenSSH configuration could resolve repeated login attempts in banning after more than X tryes. I was really disappointed to see that OpenSSH didn't offer any feature like this. I found different solutions but nothing that I really liked.

The PAM solution

Initially I found pam_abl that is a little module that can be configured with PAM (Pluggable Authentication Modules). It was not easy to recompile (at least for me) as it lacks an autoconfig makefile, and needed some tricks and patches to the bash scripts.
I found that PAM configuration is a mess. After some basic understanding on what PAM is and reading how pam_abl docs, I was able to compile, configure and put it running. It seemed that, to ban an IP you have to fail a full login attempt, and for "full login attempt" I mean "log-in wrongly 3 time consequently" and being disconnetted by the server. You can, for example try to log-in one time, then disconnect manually, and login again after reconnecting, and pam_abl, seemed to be not disturbed at all by this. But it's not what I liked.
The nice thing is that PAM runs as a module, so you don't have any process running apart: pam responds to the user and it is pam that keeps action based on user iteraction.
But...to use pam_abl, you should use OpenSSH with PAM: this means that the user will be asked for a login and a password. But... what about using public key authentication? PAM does not join the game in this case, and this means that if you use this approach pam_abl cannot do nothing.

Log monitors

A reader of a my recent posts pointed me to sshblack. Based on what is written in sshblack site, it is a Perl script that monitors log files for suspicious activity and react blacklisting source IP using firewalls, etc.
If I understood correctly installation notes, sshblack periodically reads the logs (/var/log/secure or similar) and reacts updating iptables. The problems in this case are two, for me. Number one: I've already tried to read some firewall howto based on iptables, and I only succeeded to ban myself after some attempts, but never ever I got some wanted results. It's a complete mess, even worse than PAM! I am sure that iptables is really flexible and I'm sure that some superman can do anything with it... but, not I. I tried shorewall, and I rapidly understood the basics to make it working. Oh... well, not so rapidly, I must confess. Second problem: how often does sshblack check logs? Every second? Or hour, or day? Maybe not so often to block an intruder after its third attempt. I am supposing here.
My desired behavior would be that the user, after its third attempt failed, gets banned: three attempts, and strike out. If the script reads logs every ten minutes, for example, the user can make a dozen of attempts in the meantime.
I didn't tried sshblack as it seems to have this limitation. Yes, on the site, it says that it's a real-time security tool. I've checked at source code, but I'm a complete incompetent of Perl, and I just bet that it was acting like tail.
I found other log monitors, and monitors means delay. And delay means no real time. This is an example with the common monitor tail. Those two commands should give the same result, but they don't: the second command looses some data (I think because of some buffering)

root@myhost:~# tail -1000f /var/log/auth.log | grep "Illegal user"
root@myhost:~# tail -1000f /var/log/auth.log | grep "Illegal user" | tee

The command tee just copies stdin to stdout (and the copy is unbuffered). But why I get different outputs? I don't know. Also the man page of tail on my debian system, speaks about sleep-interval parameter. This means that tail periodically checks the file for updates and then loads last "n" lines.
I don't know Perl, and the 450-line installation notes seemed too obscure with all those iptables stuff.
I like to think about to know how my server works, and the NIH syndrome was started to tempt me.

FIFO and syslog

I knew that syslogd is much flexible, and checking at the documentation I discovered fifo files. That's wondeful! You just read from a file, and you get messages from the syslogd (or any other process). All I have to do is to make a fifo file that receives failed login attempts and black list related ip addresses. To blacklist ips the easiest way I found is to update the /etc/hosts.deny, that defines a language to block connections for specific, or all, services (short and clear example).

First of all I updated my /etc/syslog.conf:

#  /etc/syslog.conf     Configuration file for syslogd.
#
auth.info                       |/opt/watch-dog/dev/watch-dog.fifo
# other config statements follow...

All authentication fails now are sent to /opt/watch-dog/dev/watch-dog.fifo file. The "|" at the top means that the syslog have to flush its buffer after each line, so that reading from the fifo will result in reading the line without any delay.
To create the fifo file just execute the command mkfifo /opt/watch-dog/dev/watch-dog.fifo, easy!
The next thing is writing a java program that sequentially read this file as text and process each line matching some patterns related to "Illegal user" or "Invalid password" patterns and, when needed rewrites the /etc/hosts.deny. The java program will continue to read the fifo until EOF, and EOF will come when syslog daemon closes the file.

I wrote that java program as quick prototype. It was the first time I tried to use the java.util.logging package and I decided that it's not for me, also I didn't want to include libraries for displaying just some messages and I wrote some plain old System.out... that I think it's more flexible and customizable than java.util.logging ;-)
Also if this is just a prototype I didn't like to rewrite the /etc/hosts.deny file, as I want to mantain the possibility to add other configuration to this file, so I decided to use a template file to wich my program appends its black list. So I can customize the template file if I need.
Here's the source.

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.text.MessageFormat;
import java.util.Date;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.Properties;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

/**
 * @author Luigi R. Viggiano
 * @version $Revision: 23 $, $Date: 2005-09-20 00:34:04 +0200 (mar, 20 set 2005) $
 */
public class WatchDog {
    private static final String IP_PATTERN = "\\D(\\d*(\\.\\d*){3})";
    private static final Pattern ILLEGAL_USER = Pattern
            .compile(".*Illegal user.*from.*" + IP_PATTERN + ".*");
    private static final Pattern INVALID_PASS = Pattern
            .compile(".*Failed password for.*from.*" + IP_PATTERN + ".*");

    private BufferedReader input;
    private File denyFile;
    private int threshold;
    private Map ipMap;

    public WatchDog(String log, String deny, int threshold)
            throws FileNotFoundException {
        ipMap = new HashMap(10);
        input = new BufferedReader(new FileReader(log));
        denyFile = new File(deny);
        this.threshold = threshold;
    }

    // do guard...
    public void guard() throws IOException {
        Util.info("Bobby watching the sheeps...");
        String line = null;
        while ((line = input.readLine()) != null) {
            if ("SHUTDOWN".equals(line))
                break; // command to kill

            boolean exceeded = update(ILLEGAL_USER.matcher(line))
                    || update(INVALID_PASS.matcher(line));

            if (exceeded)
                save(ipMap);
        }
        Util.info("Bobby sleeping peacefully...");
    }

    // returns true if (and only if) threeshold is exceeded by this match
    private boolean update(Matcher matcher) {
        if (!matcher.matches())
            return false;
        String ip = matcher.group(1);
        Integer value = (Integer) ipMap.get(ip);
        int count = (value != null) ? value.intValue() : 0;
        int newCount = count + 1;
        ipMap.put(ip, new Integer(newCount));
        Util.info("ip: " + ip + "\t fails:" + newCount);
        return count < threshold && newCount >= threshold;
    }

    // save hosts.deny file
    private void save(Map ipMap) throws IOException {
        Util.info("Saving " + denyFile);
        BufferedReader template = new BufferedReader(new InputStreamReader(
                Util.getStream("/hosts.deny")));
        BufferedWriter output = new BufferedWriter(new FileWriter(denyFile));
        String line = null;
        while ((line = template.readLine()) != null) {
            output.write(line);
            output.write("\n");
        }
        output.write("ALL:");
        int c = 0;
        for (Iterator iter = ipMap.entrySet().iterator(); iter.hasNext(); c++) {
            Map.Entry element = (Map.Entry) iter.next();
            output.write("\\\n    ");
            output.write(element.getKey().toString());
        }
        output.write("\n\n");
        output.close();
        template.close();
    }

    public static void main(String[] args) throws IOException {
        Properties conf = new Properties();
        conf.load(Util.getStream("/WatchDog.config"));
        WatchDog bobby = new WatchDog(
            conf.getProperty("watchdog.input.file", "dev/watch-dog.fifo"),
            conf.getProperty("watchdog.deny.file", "/etc/hosts.deny"),
            Integer.parseInt(conf.getProperty("watchdog.allowed.fails", "3")));
        bobby.guard();
    }

    private static class Util {
        private static Object[] data = new Object[] { new Date(), "INFO",
                new StringBuffer() };

        private static void info(String message) {
            Date time = (Date) data[0];
            time.setTime(System.currentTimeMillis());
            StringBuffer buffer = (StringBuffer) data[2];
            buffer.setLength(0);
            buffer.append(message);
            System.out.println(MessageFormat.format("{0,date} {0,time} [{1}]: {2}", data));
            System.out.flush();
        }

        private static InputStream getStream(String path) {
            return WatchDog.class.getResourceAsStream(path);
        }
    }
}

As you can see, it reads from a text file (the FIFO) and checks every line against some regular expressions, extracts the IP address of the source and after 3 attempts, rewrites the /etc/hosts.deny denying ALL services to offending sources. The template file is obtained from the classpath, as well as a simple configuration file, that enables configuring max fail attempts allowed, the FIFO path and the hosts.deny path.

To simplify steps I made a little package that contains sources, binaries, shell scripts, the FIFO and a tree structure to make a quick install. Just follow those steps:

  1. login as root
  2. if you haven't yet, set-up your $JAVA_HOME and $PATH to allow java programs to run (i.e. in ~/.bashrc file)
  3. unpack this archive (watch-dog.tar.bz2) in your /opt/ directory: /opt# tar -xvjf /root/watch-dog.tar.bz2
  4. Update syslog configuration /opt# vi /etc/syslog.conf adding the "auth.info" line at the top. ie:
    #  /etc/syslog.conf     Configuration file for syslogd.
    #
    auth.info                       |/opt/watch-dog/dev/watch-dog.fifo
    # ...other config statements follow...
    
  5. backup your /etc/hosts.deny file, and replace it with a customized version of the one provided in the classes directory of the package
  6. restart the syslog daemon with /etc/init.d/sysklogd restart command
  7. start the process with the startup script:
    /opt# cd /opt/watch-dog/bin
    /opt/watch-dog/bin# ./startup.sh
    
  8. check the log file: /opt/watch-dog/bin# tail -f ../logs/watch-dog.out
You find also a shutdown.sh script, that sends to the FIFO a special command that tells the program to shutdown. Both startup.sh and shutdown.sh call watchdog.sh, with "start" or "stop" parameter. Startup simply redirects stdout to the log file, and runs the watchdog.sh in background.

To run the program on System V startup I created a startup script like this in /etc/init.d/watchdog:

#! /bin/sh

set -e

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

DESC="WatchDog SSH Guardian daemon"
NAME=watchdog
DAEMON_START=/opt/watch-dog/bin/startup.sh
DAEMON_STOP=/opt/watch-dog/bin/shutdown.sh
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

export JAVA_HOME=/opt/java

# Gracefully exit if the package has been removed.
test -x $DAEMON || exit 0

d_start() {
        $DAEMON_START
}

d_stop() {
        $DAEMON_STOP
}

d_restart() {
        d_stop
        sleep 2
        d_start
}

case "$1" in

  start)
        echo -n "Starting $DESC: $NAME"
        d_start
        echo "."

        ;;
  stop)
        echo -n "Stopping $DESC: $NAME"
        d_stop
        echo "."

        ;;
  restart|force-reload)
        echo -n "Restarting $DESC: $NAME"
        d_restart
        echo "."

        ;;
  *)
        echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
        exit 1
        ;;
esac

exit 0

Last step is to instal this script with the command /etc/init.d# update-rc.d watchdog defaults 30. That's all.

It is working well on my linux box. There are many improvements that can be done. For example, it could be good to save and restore the black-listed IP addresses between restarts. But this is just a quick prototype (I've adapted the source to be short and contained in a single file, to be easily posted here). I think I'll do more on the java program as I'll need on the next. Now it's just a matter of extending it with some java coding.

Final notes

See also: More ssh ideas

Note: usual disclaimer... don't blame me if this don't work in your linux box, or if something in this post causes any problem to you. I'm just a newbie of linux, and you should know what you do while following anything written in this blog. Anyway, everything here is licensed under the Creative Commons license: non-derivative restriction just apply to prose and comments, derivative works are allowed for any source code listed in this blog.


One Response to “A watch dog against SSH login attacks”  

  1. 1 The Watchdog Engine - NewInstance


Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>



Calendar

September 2005
M T W T F S S
« Aug   Oct »
 1234
567891011
12131415161718
19202122232425
2627282930  

Follow me

twitter flickr LinkedIn feed

Subscribe by email

Enter your email address:

Archives


Categories

Tag Cloud


Listening