David - Musings of an SRE

Secure your Server - Send Slack Alert on SSH Login

In my line of work, besides architecting and building infrastructure, part of the scope includes building safeguards to ensure that systems that were setup are properly secured.

Here’s a quick trick for folks looking for an easy first-line defence that they can implement easily across their servers right now.

In order to quickly respond to a server compromise, it is a good idea to setup some form of IDS to alert you when a potential headache is about to happen. Simplest way to do that is to set up an alerting system to alert you or your team when a server has been accessed.

Prerequisite

  • Prepare your Slack’s incoming webhook tokens

Lets Go

First, login to your server as root and open up /etc/pam.d/sshd

Add the following at the bottom of the file

session optional  pam_exec.so /etc/slack_ssh.sh

Bonus: To find out more about what else pam_exec can do, checkout man pam_exec, man pam.conf

Don’t worry, I’ll explain it afterwards!

Now, still as root, create /etc/slack_ssh.sh

$ vim /etc/slack_ssh.sh

# slack_ssh.sh

#!/bin/bash
#!/bin/sh
[ "$PAM_TYPE" = "open_session" ] || exit 0
{
curl -X POST --data-urlencode "payload={'channel': '#builds', 'text': \"[login-alert] Login attempt by $PAM_RHOST as $PAM_USER @ $(hostname)\"}" \
https://hooks.slack.com/services/xxx/yyy/xxx
}

Remember to chmod the /etc/slack_ssh.sh

$ chmod +x /etc/slack_ssh.sh

Breaking it down

So what just happened?

First,

Defining session optional pam_exec.so /etc/slack_ssh.sh, what we’re telling the server to do is to execute the file /etc/slack_ssh.sh whenever a pam login event happens. If you noticed, we’ve edited the file /etc/pam.d/sshd which describe the ssh login service. If we wanted to, we can also add a similar executable line whenever someone makes sudo for example. How? We just modify /etc/pam.d/sudo of course!

Moving on.

So now we need to create the actual script that will be triggered.

You might notice that I’ve defined a conditional to check for: $PAM_TYPE = "open_session". In this case, we want to ensure that only logins are tracked else, even ssh logouts trigger as an event and will fire off the script.

In our case, our script consists of a quick and dirty incoming slack webhook which will send a message to a channel that we defined. This way your team will be quickly informed at minute 0 of someone connecting to your servers.

If you want to also track for logouts, just remove the conditionals.

Simple enough!

And you’re done!

Slack SSH Alert

References