http://www.debian-administration.org/articles/573
*Tags*: botnet, compromise, cracklib, denyhosts, firewall, openssh, pwgen, security, ssh
I'm sure if you're responsible for any server connected to the Internet with a "real" IP address, or if you port forward :22 to a box on your LAN, your logs will contain a lot of failed SSH attempts. Here we'll look at some simple solutions.
The logged entries will probably look something like the following:
sshd[22768]: Invalid user carol from 91.194.85.65 sshd[22770]: Invalid user carolr from 91.194.85.65 sshd[22772]: Invalid user sergi from 91.194.85.65 sshd[22774]: Invalid user quala from 91.194.85.65
Now these are an annoyance, but not much of a concern, as none of these usernames exist on my system - its just filling up my logs, nothing more.
There are other more alarming messages hiding in your logs too, like these:
sshd[17810]: Failed password for root from 91.194.85.65 port 44229 ssh2 sshd[17811]: Failed password for root from 91.194.85.65 port 38154 ssh2 sshd[17814]: pam_unix(ssh:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=91.194.85.65 user=root sshd[17815]: pam_unix(ssh:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=91.194.85.65 user=root
Both of these logs come from different physical boxes, with IPs located in totally different /16's and yet it is the same IP address attacking both? Scary! Even more worrying is the fact the attacks are many hours apart, this guy has been hammering hundreds or thousands of servers all day…
What happens if they eventually get lucky? The "best case" scenario is they get an unprivileged account on your system, with which they can:
If they end up with root@yourserver, things get more interesting - they can literally do anything they want with the box. You're screwed. Really.
Every single decision you ever make about these sort of issues will involve balancing usability and accessibility vs. security and safety. There are lots of options available to you, some more "annoying" to real users than others.
1) DenyHosts
Simply put this is "apt-get install denyhosts" and the default configuration is pretty good. It'll send you alert e-mails if you want them and the level at which block compromised IPs with /etc/hosts.deny is tunable.
Pros:
Cons:
2) Firewall
A simple firewall rule can totally drop all traffic to your SSH server if the traffic isn't coming from your IP.
Pros:
Cons:
3) Strong Passwords
This is often overlooked, but a real winner. Ensure that all accounts on your system have strong passwords using cracklib.
Generating these passwords is easy with pwgen, to make really strong passwords invoke like so: "pwgen -syB 23"
Pros:
Cons:
4) Public Key Authentication
Remembering all these strong passwords can be a chore. Often when you're dealing with 2-3+ boxes it's easier to just install the public part of a certificate in ~/.ssh/authorized_keys.
Then you may authenticate to the server with your private key.
Simply put: drop your key in authorized_keys and try to login by doing "ssh -i key.file user@your.hostname.here". Once you have this working you can disable other authentication methods.
Pros:
Cons:
5) sshd_config
Using options such as "AllowGroups" to limit the users which are considered valid for logins, regardless of the shell which may be set in /etc/passwd.
Perhaps run the daemon on a non-standard port so that scans only looking at :22 don't detect you're even running SSH.
Most of the things suggested above are covered in manual pages and other articles on this site, I am planning to write up some more in-depth coverage as part of this series, including coverage of sshd_config "Best Practice", cracklib and PAM, authentication using public key, etc.
For reference you might want to take a look at our earlier collection of advice upon Keeping SSH access secure.
documented on: 10 Jan 2008, ahowells
In addition to the idea of using ~/.ssh/authorized_keys almost exclusively, might I recommend editing /etc/ssh/sshd_config, and setting
# Change to no to disable tunnelled clear text passwords #PasswordAuthentication yes PasswordAuthentication no
followed by `/etc/init.d/ssh reload`
. This means that you can access this
Box ONLY via an entry in ~/.ssh/authorized_keys.
I do this on ALL Boxen which have an open ssh port on an untrusted network. (and I know very few trusted nets ;-)
Also, while we're at it, if you are behind a firewall which you control,
(IPCop at home) then you might want to consider mapping the "visible ports"
to something other than 22. You can do this for several Boxen if you wish; I
have used the convention "22+last-byte-of-IP" at times. On the road I could
call home with `ssh -p22123 user@outsideaddr.ISP.TLD`
and land on port 22 of
the Box x.y.z.123
rhi
The only option from "sshd_config" mentioned in the article is "AllowGroup", which is a really interesting option to lower risks, but I think another very interesting option is "PermitRootLogin". Setting this option to "no" will disable the access to user root. This is very valuable in my (humble) opinion, because this is the only user whose name is always known to exist on your system by the attacker.
So, by combining "PermitRootLogin" and "AllowGroup" it is very easy to restrict access to SSH to one or several real users (meaning not system with standard and predictable logins as 'www-data', 'ftp', etc…), whose logins are very unlikely to be found by an attacker.
So the brute force attack has to found the password AND the user, which significantly reduces chances of success because of course OpenSSH gives no details when refusing authentication: the attacker cannot know if it is the password or the login which is wrong.
I was having the same problems as you mention (lots of login tries to ssh) so I did lots of research into SSH security and how to fix these problems. I read lots of articles by experts and most of the stuff simply seemed very complex.
I am a KISS person by nature so I decided to go my own way. I set up a list of things I would try but I would try one thing at a time to see what effect it would have. The first thing on my list was to simply change the port number (you mentioned this at the end of your article).
FYI, that is as far as I got on my list. Attacks on my machine through SSH have virtually disappeared altogether.
It is interesting to note that none of the articles I read during my research mentioned this step yet it was extremely effective and the simplest solution by far. The reason why it is effective is probably because the scans that are being done are automated, most probably by people who don't really understand what they are doing, or automated scans from machines that have been pwned. Nobody is portscanning me and trying to figure out what I am running behind each port. As a homeuser I am simply not worth the effort.
So my opinion is to try this first and foremost and then monitor your logs. I am certain that you will see these attacks die down. However, if you are looking into these things for something other than home network then make sure you dig deeper into the security solutions out there.
/David
> Attacks on my machine through SSH have virtually disappeared altogether.
I'm not sure changing the port number is a panacea - anyone with a bit of time on their hands and a decent scanner can find it again ;) Not to mention the fact you might want to connect from "restrictive" places like WiFi hotspots, corporate networks and such - they may block anything other than 'standard' ports :(
A large number of British ISPs also adversely QoS access to anything other than 'standard' ports during peak hours, which generates lots of lag. Boooo.
ahowells
relocating ssh to an off-port is not perfect, but it sure is peaceful.
i've never had ssh on a standard port since i got broadband years ago. i've only had one connection ever and that's because i was abusing "http/1.1 connect" to connect to my ssh server through a web proxy. (within an hour an ip address associated with blue coat created a tcp connection on that port, but didn't attempt login because it was probably just seeing what kind of server i had used "http/1.1 connect" for since it wasn't port 443.)
yes, it doesn't block against a determined adversary who will scan all 65535 ports and try to brute force ssh once he finds it, but because there's no noise in the logs, from thousands of bots trying to brute force (or just connect to) my ssh server, you can easily see the real threats.
assuming your adversary gets that far, you can implemented ssh keys (which i only use from my own hardware so as to not allow keylogging or bruteforcing of my key's passphrase) and otp with opie (on "untrusted" hardware, though session hijacking by way of the network socket is possible by a rootkit), eliminating password brute forcing. you can even eliminate brute-forcing root's otp with pam's securetty (brute forcing root's 2048 rsa/dsa ssh key is still a possibility, but one that most people are willing to risk ;-). this requires a two-step process for a valid user to log in as root (ssh to normal user using ssh key or otp, then su to root using password or otp), but requires the attacker to know a user name before even attempting to brute force ssh key or otp.
about ssh not being on port 22 and therefor blocked: most company or public access firewalls i've encountered block everything but http/https, usually through a (transparent) proxy, with the occasional allowance for telnet & ftp, so ssh gets blocked regardless of what port it is on. that's where ajaxterm comes in. ;-)
moving ssh to an off-port is equivalent to moving the safe from outside at the mail box to inside behind the picture on the wall.
by undefined (192.31.xx.xx)
The firewall is able to do some funky stuff also \u2014 I use a setup somewhat like this:
iptables -P INPUT DROP iptables -N ssh iptables -A ssh -m state --state ESTABLISHED -j ACCEPT iptables -A ssh -m recent --update --seconds 300 --hitcount 3 -j REJECT iptables -A ssh -m recent --set -j ACCEPT iptables -A INPUT -i lo -j ACCEPT iptables -A INPUT -i tun0 -j ACCEPT iptables -A INPUT -s $INTERNALNET -j ACCEPT iptables -A INPUT -p tcp --dport 22 -j ssh
The ssh-table will now reject the fourth and later attempts to create an SSH connection within five minutes.
Since it can be hard to guess a matching username and password in three guesses, this is good enough for me (root login is not permitted in my setup).
tweek
That's the way I do too, and it works just great.
iptables -A INPUT -p tcp --dport 22 -i eth2 -m state --state NEW -m recent --set iptables -A INPUT -p tcp --dport 22 -i eth2 -m state --state NEW -m recent --update --seconds 600 --hitcount 5 -j DROP iptables -A INPUT -p tcp --dport 22 -i eth2 -j ACCEPT
Nilshar
> The ssh-table will now reject the fourth and > later attempts to create an SSH connection > within five minutes.
I do this with my machine too but it does have a downside. It doesn't differentiate between valid and invalid connections. If you do genuinely ssh several times in quick succession to the machine it will still ban you from connecting for a while. This might be because you are opening several terminal windows which run ssh on your desktop (either "ssh host xterm" or "xterm -e "ssh host") or if a script runs ssh to the same host several times.
quboid
> but it does have a downside
Hehe … I know. :-)
To get around that problem I use VPN. Incoming connections from tun0 and the local network is then accepted unconditionally.
tweek
You can also use a master/slave ssh setup, so you only connect once and then further logins to the same user@host combination use that already-existing connection.
by Anonymous
This downside doesn't exist IIRC when reusing an existing socket… http://www.debian-administration.org/articles/290
Steve
Do not forgive opiekey/skey to secure password and SSH.
Cons : Hard to master, like public keys Pros : Secure and you can easily compute your one time password with a web page (javascript).
Just remember to add [:skey] to your login and compute them ;)
by Anonymous (86.66.xx.xx)
This is exactly what I did to my system.
I permit either PKI or OPIE/SKEY authentication from remote. I do not permit password authentication at all. With this configuration it is incredibly unlikely that a brute-force attack would ever work. OPIE/SKEY clients exist for many platforms, including my Nokia mobile phone.
If on the console, it will still prompt for OPIE/SKEY but also accept a password as a failsafe.
I also rate-limit SSH connections per second on the firewall, and use fail2ban.
I found that I get some brute-force attacks on postfix as well. This is more difficult to secure and I rely on fail2ban in this case. If the level of brute-force attacks increases, I'll have to think of something else.
by Anonymous (68.15.xx.xx)
> Sounds exactly like what I would like to do, but can you be a bit more specific? > > How do you manage to have a different authentication scheme from console > than via openssh?
I should have been a bit more specific; ssh uses OPIE or PKI and will not permit passwords, everything else user-interactive (which in my case is just the console) will permit OPIE and password. I am not using OPIE for things like pop3, imap, etc.
Authentication methods for various things can be controlled with PAM. I'm using Debian, so this might need significant modification for your system. My /etc/pam.d/ssh looks like (the part that's different than default):
auth sufficient pam_opie.so auth required pam_deny.so
You'll want to comment out "@include common-auth" once you know that OPIE is working.
I've also modified /etc/pam.d/common-auth to permit both OPIE (first) and then password (second):
auth sufficient pam_opie.so auth sufficient pam_unix.so nullok_secure auth required pam_deny.so
In /etc/ssh/sshd:
PubkeyAuthentication yes ChallengeResponseAuthentication yes PasswordAuthentication no UsePAM yes
I realize this is brief but hope it helps, or at least gives you an idea of what to look for.
by Anonymous (68.15.xx.xx)
I want to be able to access my home server via SSH from wherever I happen to be at the moment. I tunnel my music via mt-daap to iTunes at work, OpenNX for dessktop access, and I sometimes tunnel to my privoxy web proxy when I'm on a wireless network I don't trust.
The solution I've settled on is a combination of fail2ban and a restrictive AllowUsers entry in my sshd_config file. I allow only a couple obscurely named accounts to login via SSH (don't totally discount security through obscurity!). I also use fail2ban to block repeated failed login attempts by modifying the server's local firewall. The firewall drops all packets from the attacker, while hosts.deny only affects a few services. Offending IP addresses are only banned for 15 minutes which is long enough to make the script kiddies go away, but prevents all but the most targeted DoS attacks. The 15 minute lockout also prevents me from DoSing myself with my poor typing skills!
While I use PuTTY's Pageant for public key auth all the time at work, I don't like it for accessing my home server. I feel better typing my password every time.
hypatia
OpenSSH (using keys only and with root login disallowed) ` Netfilter (delay
after wrong authentication in rather minutes than seconds [already mentioned
here?] or port-knocking http://vger.cz/faqs/netfilter/portknock_multi.faq
[works very well for me, using this script
http://vger.cz/hacks/shell/bash/iptknock/ to setup the complicated rules) `
SE Linux (just for case somebody really got in) can secure any host quite
well. Can the system wide security be better? But, still, there are many
other ways to compromite your host either… and always consider that NO
security is perfect enough (for instance when you talk too much when drunken
etc.)..
by Anonymous (86.49.xx.xx)
> What about port knocking to open the port on demand ? > http://en.wikipedia.org/wiki/Port_knocking[]
Port Knocking adds very little true security to an SSH service. Abstractly, in a full-connected internet, it's worth considering the port numbers chosen and timing required to open up an SSH service as a relatively short secret string required in addition to the ssh daemon's authentication requirements. Since the short sequence is known by everyone who uses it, it is a shared secret (which is bad). And since there is no way to encrypt a sequence of "port knocks" over the internet, the secret is transmitted in the clear (which is extra bad).
Compounding the problems, port knocking is fragile in the presence of firewalls between you and the remote host. If any firewall or filter decides to block any of the ports you've tried to use, your port knocking will fail, cutting you off from the remote machine.
If your SSH daemon's authentication is weak enough that the gain provided by port knocking seems significant, you should probably instead spend more time thinking about how to improve the choices you've made for SSH authentication (reading this article and the comments here is a good start!).
dkg
Denyhosts in testing/sid (that's works also in stable branch) has a patch for the security issue.
If you want be cutting-edge, lastest debian package (that actually is not yet in sid) is avaible at the maintainer homepage (myself):
http://bertorello.ns0.it/debian/binary/denyhosts/denyhosts_2.6-3_ all.deb
Marco Bertorello
crattis
> If your box is compromised, all they have to do, is go through your history > to get a list of additional boxes to get access too.
You mean the box holding the private key, not the server running sshd, right? Of course, the private key itself would be stored on disk with a strong passphrase as well, no? So yes, it could be a cascaded compromise if your local machine is compromised with the key, but it would require the compromising attacker to sort out your key's passphrase as well.
Another option is to offload the key itself to a removable RSA engine. I do this, and detailed how it works with the Cryptoflex eGate http://lair.fifthhorseman.net/~dkg/egate/ (back under debian sarge — that page is seriously out of date, sorry!). In that case, even if your local machine is compromised, the private key is still inaccessible.
dkg