Configuring SSH

May 15, 2025 · Tyler Yeager · 18 min reading time

Featured Image of blog post

Servers are so easy to issue out these days. There are so many hobby level VPS’s (Virtual Private Servers) that you can rent for as little as $1/month, sometimes even $10/year. Once you have one of these, you can do so many cool things.

You can host:

  • Games
  • Chat servers
  • Your websites
  • Your programs
  • All sorts of other self-hosted services (awesome-selfhosted)

However, once you do have a server out there, you need to connect to it. The ubiquitous way to do this for Linux servers is through SSH (Secure SHell). The thing is, attackers know this, and they put a lot of time and effort into trying to get access to your server using SSH.

So whenever you get a VPS server, it’s important to secure it by locking down SSH. This post is about securing your SSH server so that you don’t have to worry about this part of it. I also include more complex examples to see the flexibility SSH offers.

While you read through this, make sure to never close out your original connection. Always use more than one connection to test if it connects. You might otherwise lock yourself out! SSH won’t kill your connection on a service restart, giving you a window to recover your mistake. You have been warned!

Finally, some of this content is older and written about 10 years ago. It’s still relevant, but my writing style has changed a little bit. See if you can notice and enjoy!


Some things I will cover:

  • Explanation and examples of AllowUsers, AllowGroups, DenyUsers, DenyGroups
  • Explanation and examples of Matching statements
  • Using combinations of Allow/Deny and Match statements to support complex situations
  • Connecting via password from external address

The OpenSSH Server configuration file is located at /etc/ssh/sshd_config
The options inside are obeyed from top to bottom.
So if you put in:

sshd_config
PasswordAuthentication no
PasswordAuthentication yes

Then passwords are allowed. Thus, all default configurations must be above the conditional options in the file.

Best Practice: In general, make sure to make permissions as limited as possible. This is much better than opening the gates and putting up numerous barriers to stop intruders. Say no to everyone, then say yes to a very select choice of users. If you must have password access, ensure the users using password access have exceptionally limited permissions. No matter how small the door is, if your one user permitted passwords has a easy password and root (or sudo) access, you are asking for trouble.

AllowUsers, AllowGroups, DenyUsers, DenyGroups

The names should explain themselves. e.g. if you set a AllowUsers option, that user will be permitted while everyone else is denied. If you set DenyGroups, that group will be denied but all others will be permitted. If a person matches multiple (in allow users but in deny groups) behavior has been a little inconsistent. Deny seems to be more powerful than Allow. I suggest testing to see how it acts on your system. My sshd_config man page says it is processed in this order: DenyUsers, AllowUsers, DenyGroups, then AllowGroups.

If you specify multiple AllowUsers (or any other) it will compound them.

Here’s some examples of setups.

If I want everyone in group admin but user hazcheezburger to have access while everyone else is denied:

sshd_config
AllowGroups admin
DenyUsers hazcheezburger

Multiple options are permitted. Users alice and bob are permitted, everyone else is denied.

sshd_config
AllowUsers alice bob

Wildcards are permitted as well. * means any or all. ? Meaning anything for that character ! negates the option. This will deny all groups starting with user_ and permit user alice but not bob from accessing:

sshd_config
AllowUsers ?????
DenyGroups user_*

You do not want user bob connecting but wish to permit alice and charlie
(yes, denyuser would the better method of doing this)

sshd_config
AllowUsers !bob alice charlie

SSH deals with network connections. Oftentimes, people will connect from one ip address. If they connect from another one, that could indicate a risk. Let’s ensure the user backup_files (a automated user that connects from 72.45.21.664) can only login from one address. However, we want to ensure user sam can login from any lan address

sshd_config
AllowUsers backup_files@72.45.21.664 sam@192.168.1.*

Now let’s make sure sam can login from lan and his work ip, 23.432.36.1. However, the local ip 192.168.1.20 is known to be extremely insecure, so we won’t let sam login from there

sshd_config
AllowUsers sam@192.168.1.*,23.432.36.1,!192.168.1.20

Let’s make a bad example. I’m trying to block user hacker in group evil who uses a lan connection but want to permit everyone else to access via lan.

sshd_config
AllowUsers *@192.168.1.* !hacker

This will permit hacker to login. Do you know why? I’ll post the answer later.

Matching statements

Sometimes we want fine-grained control of what goes on. While the Allow/Deny options are useful, they are sweeping in that the two possible options are to deny or permit access.

This is where using Match comes into play. Remember to ensure these are placed at the bottom of the file, to prevent overruling. Also, not all config options can be changed. Look through man sshd_config for full list & make sure to test your sshd config

There are a few possible options. There are more, but I see these to be the most useful.

Address
User
Group

Address refers to a ip address (e.g. 192.168.1.25) and hosts refer to dns lookup. Someone from linuxquestions.org logging in? Use Host to modify their login.

To write a generic match statement, it goes like this

sshd_config
Match Filter Argument
Option Argument

The best example of using addresses to match is to ensure external connections cannot use passwords, but local connections can use passwords This will set a default of no password auth, but anyone connecting from 192.168.1.* is permitted

sshd_config
PasswordAuthentication no
Match Address 192.168.1.*
PasswordAuthentication yes

alice & bob are permitted to use X11 forwarding, however the default is set to deny.

sshd_config
X11Forwarding no
Match User alice,bob
X11Forwarding yes

Groups work in a similar way and can be combined with other options. Let’s permit groups admin or homework or user alice to use x11 forwarding

sshd_config
X11Forwarding no
Match Group admin,homework User alice
X11Forwarding yes

These options can be combined for specialized purposes.

Generally, the security risks of X11 forwarding are reduced in the lan network. But the only users who need it belong in group gui. I will also give everyone in the lan network permission to use password authentication Notice if I combined the below Matching statements, it would only permit those in lan and group gui to use pass auth. Everyone else would not be permitted.

sshd_config
PasswordAuthentication no
X11Forwarding no
Match Group gui Address 192.168.1.*
X11Forwarding yes
Match Address 192.168.1.*
PasswordAuthentication yes

Here we mandate that banners will be displayed to all users except those in the group nobanner.

sshd_config
Banner none
Match User * Group !nobanner
Banner /etc/banner

Complex examples

Let’s say you are setting up the ssh server for the main connection between external and internal connections for a school. Administrators in the group admin are permitted to login from anywhere but must not use x11 forwarding or passwords if from a external network. Students in the group student must be within the lan to connect, are permitted to use password auth but not permitted to use x11 forwarding. There is a IT support tech that manages schools up and down the district who must have x11 forwarding access from her home address (32.65.435.63) lan and work ip which may vary but will start with 92.45.13., her username is sally and is part of the admin group. Everyone else should be denied

sshd_config
X11Forwarding no
PasswordAuthentication no
AllowGroups admin student@192.168.1.*
Match Address 192.168.1.* Group admin
PasswordAuthentication yes
X11Forwarding yes
Match Group student
PasswordAuthentication yes
Match User sally Address 192.168.1.*,32.65.435.63,92.45.13.*
X11Forwarding yes

An individual uses his home computer to write documents for his business. He wants to be able to access this from his work with a ip address of 24.562.2.526 but sometimes will connect from the local cafe which has the ip of 75.743.1.632. He trusts the security of his work network so password auth is ok but does not trust the cafe, which should be key based. He will be the only user ever connecting to this computer and the server should never permit any other user from accessing, nor should anyone be able to access the server from any other ip address. His username is uplink

sshd_config
PasswordAuthentication no
AllowUser uplink@24.562.2.526,75.743.1.632
Match Address 24.562.2.526
PasswordAuthentication yes

A group of people use ssh in their building to login to each other’s computer for work. One of them will be going to another country for a few months to work on a project. Let’s ensure he can continue to do his job, however we must make sure no one else can access via external network. He will be using key based auth but passwords are acceptable in lan. He doesn’t know what ip he’ll be connecting from but plans to followup with the info. his username is rocker

sshd_config
PasswordAuthentication no
AllowUser *@192.168.1.* rocker
Match Address 192.168.1.*
PasswordAuthentication yes

Later on, you notice in the logs rocker is connecting from 999.23.53.1. He confirms he will be consistently connecting from this ip. He is using key based auth, which is secure, however you decide to update the config to be safe. If someone manages to get his private key, it will be useless unless from the one ip address.

sshd_config
AllowUser *@192.168.1.* rocker@999.23.53.1

Password based Authentication open to external networks.

Pretty much every guide out there says do not permit passwords to external ips. I agree. But people need it sometimes. Before you set it though, think for a moment and answer these questions:

  • Do you need ALL users to be able to login with password from anywhere? Maybe just one user?
  • Would it be ok to permit passwords in the lan, but restrict users using passwords from external ip’s?
  • Will users be using the same network to connect from? Would it be ok to restrict passwords to limited ip’s?

Also, if you have to use passwords for external ips… Make sure to use a strong password and additional protections such as banning programs when too many invalid attempts are made Even a 5-minute ban on an ip after 3 guesses can be the difference between being cracked and making the attack too expensive to pursue. There are an assortment of easy to use and effective programs out there. I use fail2ban. A 5 minute ban will not ruin your day. A hacked computer will.

If you have a username of (don’t use something easy, like admin):

sshd_config
PasswordAuthentication no
Match User samual
PasswordAuthentication yes

or to only permit one user to login:

sshd_config
PasswordAuthentication no
AllowUser samual
Match User samual
PasswordAuthentication yes

Conclusion

Overall, for most situations setting password authentication to no and only allowing public key authentication is all you need. Most VPN servers permit password login initially, use this as your opportunity to copy your key in and then disable password auth.

Then you can consider it good to go and continue setting up your server 😊

Did you find this interesting?

Consider subscribing 😊

No AI-generated content or SEO garbage.

Unsubscribe anytime