One of my clients got a Rackspace cloud server which I am to build for one of the applications we developed for him. It was so natural for me to build a Ubuntu server because that is the OS I have been very familiar with for many years now. I built the server, crossed many huddles and finally deployed the application. Yeah, we were good.

Some months later, something ridiculous happened and the server was suspended. I was baffled and after much consultation with the customer personnel person I got to know Rackspace suspended our account because of a DDOS (Distriduted Denial of Service) attack on it. Then my eyes opened and my hears sprung up, I shouted shouted “DDOS?”. It was so annoying because I felt Rackspace is keeping an eyes on this server against all these threats but Alas you gats secure your thing yourself.

This increases my client's billing from $15 per month to over $200+. Rackspace asked us to send the steps we would be taking to stop this attack before the server would be unsuspended. Below is what I sent and you can also use same, heheee...

"We are aware our server has been suspended because of DoS attack.

We have planned to first do an audit trail of our System security, analyse the system log files, scan logs and ban suspicious hosts, Protect from DDOS (Denial of Service) attacks with ModEvasive, Install and configure Apache application firewall - ModSecurity, Restrict Apache Information Leakage, Disable Open DNS Recursion and Remove Version Info - Bind9 DNS, Harden network with sysctl settings, Secure shared memory - fstab, Intrusion Detection - PSAD.

We hope the following steps would further secure the server and stop the attack.

Thanks."

You might need to bear the high bill as you fix the server, don't worry your bill will return to normal.

Okay here are the steps I took to savage this situation:

1. Firewall – UFW

A good place to start is to install a Firewall. Please check if you already have this on the server installed before overwriting the initial settings by installing this one. I did it and I got logged out.

UFW - Uncomplicated Firewall is a basic firewall that works very well and easy to configure with its Firewall configuration tool - gufw, or use Shorewall, fwbuilder, or Firestarter.

Use Firestarter GUI to configure your firewall or refer to the Ubuntu Server Guide, UFW manual pages or the Ubuntu UFW community documentation.

Install UFW and enable, open a terminal window and enter :

sudo apt-get install ufw

Allow SSH and Http services.

sudo ufw allow ssh
sudo ufw allow http

Enable the firewall.

sudo ufw enable

Check the status of the firewall.

sudo ufw status verbose

 

2. Prevent IP Spoofing.

Open a Terminal and enter the following :

sudo vi /etc/host.conf

Add or edit the following lines :

order bind,hosts
nospoof on

3. Harden PHP for security.

Edit the php.ini file :

sudo vi /etc/php5/apache2/php.ini

Add or edit the following lines an save :

disable_functions = exec,system,shell_exec,passthru
register_globals = Off
expose_php = Off
display_errors = Off
track_errors = Off
html_errors = Off
magic_quotes_gpc = Off

Restart Apache server. Open a Terminal and enter the following :

sudo /etc/init.d/apache2 restart

4. Web Application Firewall (ModSecurity) and Protect from DDOS (Denial of Service) attacks (ModEvasive).

Install ModSecurity:

sudo apt-get install libxml2 libxml2-dev libxml2-utils libaprutil1 libaprutil1-dev libapache-mod-security

If your Ubuntu is 64bit, you need to fix a bug:

sudo ln -s /usr/lib/x86_64-linux-gnu/libxml2.so.2 /usr/lib/libxml2.so.2

Configure ModSecurity:

sudo mv /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf; sudo vi /etc/modsecurity/modsecurity.conf

Enable the rule engine:

SecRuleEngine On

Increase the request body size limit to 10Mb(Optional, only if your site accepts uploads):

SecRequestBodyLimit 10000000
SecRequestBodyInMemoryLimit 10000000

Check the ModSecurity version:

dpkg -s libapache-mod-security | grep Version

The installed ModSecurity version is:

Version: 2.6.3-1ubuntu0.2

Install OWASP ModSecurity Core Rule Set:

Download the rule set(version 2.2.5 because the latest version requires ModSecurity 2.7.0+):

wget https://github.com/SpiderLabs/owasp-modsecurity-crs/tarball/v2.2.5 -O /tmp/owasp.tar.gz

Extract the package:

cd /tmp; tar -zxvf owasp.tar.gz; rm owasp.tar.gz

Copy the directory to /etc/modsecurity, and set the permissions:

sudo mv SpiderLabs-owasp-modsecurity-crs-5c28b52/ /etc/modsecurity/owasp-crs
sudo chmod -R 644 /etc/modsecurity/owasp-crs

Link the rules to /etc/modsecruity/owasp-crs/activated_rules directory:

sudo mv /etc/modsecurity/owasp-crs/modsecurity_crs_10_setup.conf.example /etc/modsecurity/owasp-crs/modsecurity_crs_10_setup.conf
cd /etc/modsecurity/owasp-crs/activated_rules/
sudo ln -s ../modsecurity_crs_10_setup.conf
for f in $(ls ../base_rules/); do sudo ln -s ../base_rules/$f; done
for f in $(ls ../optional_rules/); do sudo ln -s ../optional_rules/$f; done

Modify /etc/apache2/mods-available/mod-security.conf to include the rules:

sudo vi /etc/apache2/mods-available/mod-security.conf

Add the following line:

Include "/etc/modsecurity/owasp-crs/activated_rules/*.conf"

Enable headers module:

sudo a2enmod headers

This to fix the following error:

Syntax error on line 29 of /etc/apache2/conf.d/modsecurity/optional_rules/modsecurity_crs_49_header_tagging.conf:

Invalid command 'RequestHeader', perhaps misspelled or defined by a module not included in the server configuration

Action 'configtest' failed.

The Apache error log may have more information.

...fail!

when restarting apache2.

Enable ModSecurity module and restart apache2:

sudo a2enmod mod-security; sudo /etc/init.d/apache2 restart

5. Scan logs and ban suspicious hosts - DenyHosts and Fail2Ban.

DenyHosts is a python program that automatically blocks SSH attacks by adding entries to /etc/hosts.deny. DenyHosts will also inform Linux administrators about offending hosts, attacked users and suspicious logins.

Open a Terminal and enter the following :

sudo apt-get install denyhosts

After installation edit the configuration file /etc/denyhosts.conf and change the email, and other settings as required.

To edit the admin email settings open a terminal window and enter:

sudo vi /etc/denyhosts.conf

Change the following values as required on your server :

ADMIN_EMAIL = root@localhost
SMTP_HOST = localhost
SMTP_PORT = 25
#SMTP_USERNAME=foo
#SMTP_PASSWORD=bar
SMTP_FROM = DenyHosts nobody@localhost
#SYSLOG_REPORT=YES

Fail2ban is more advanced than DenyHosts as it extends the log monitoring to other services including SSH, Apache, Courier, FTP, and more.

Fail2ban scans log files and bans IPs that show the malicious signs -- too many password failures, seeking for exploits, etc.

Generally Fail2Ban then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action could also be configured.

Out of the box Fail2Ban comes with filters for various services (apache, courier, ftp, ssh, etc).

Open a Terminal and enter the following :

sudo apt-get install fail2ban

After installation edit the configuration file /etc/fail2ban/jail.local and create the filter rules as required.

To edit the settings open a terminal window and enter:

sudo vi /etc/fail2ban/jail.conf

Activate all the services you would like fail2ban to monitor by changing enabled = false to enabled = true

For example if you would like to enable the SSH monitoring and banning jail, find the line below and change enabled from false to true. Thats it.

[ssh]

enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3

If you have selected a non-standard SSH port in step 3 then you need to change the port setting in fail2ban from ssh which by default is port 22, to your new port number, for example if you have chosen 1234 then port = 1234

[ssh]

enabled = true
port = <ENTER YOUR SSH PORT NUMBER HERE>
filter = sshd
logpath = /var/log/auth.log
maxretry = 3

If you would like to receive emails from Fail2Ban if hosts are banned change the following line to your email address.

destemail = root@localhost

and change the following line from :

action = %(action_)s

to:

action = %(action_mwl)s

You can also create rule filters for the various services that you would like fail2ban to monitor that is not supplied by default.

sudo vi /etc/fail2ban/jail.local

Good instructions on how to configure fail2ban and create the various filters can be found on HowtoForge - click here for an example

When done with the configuration of Fail2Ban restart the service with :

sudo /etc/init.d/fail2ban restart

You can also check the status with.

sudo fail2ban-client status

6. Intrusion Detection - PSAD.

Cipherdyne PSAD is a collection of three lightweight system daemons that run on Linux machines and analyze iptables log messages to detect port scans and other suspicious traffic.

Currently version 2.1 causes errors during install on Ubuntu 12.04, but apparently does work. Version 2.2 resolves these issues but is not yet available on the Ubuntu software repositories. It is recommended to manually compile and install version 2.2 from the source files available on the Ciperdyne website.

To install the latest version from the source files follow these instruction : How to install PSAD Intrusion Detection on Ubuntu 12.04 LTS server

OR install the older version from the Ubuntu software repositories, open a Terminal and enter the following :

sudo apt-get install psad

Then for basic configuration see How to install PSAD Intrusion Detection on Ubuntu 12.04 LTS server and follow from step 2:

7. Check for rootkits - RKHunter and CHKRootKit.

Both RKHunter and CHKRootkit basically do the same thing - check your system for rootkits. No harm in using both.

Open a Terminal and enter the following :

sudo apt-get install rkhunter chkrootkit

To run chkrootkit open a terminal window and enter :

sudo chkrootkit

To update and run RKHunter. Open a Terminal and enter the following :

sudo rkhunter --update
sudo rkhunter --propupd
sudo rkhunter --check

7. Scan open ports - Nmap.

Nmap ("Network Mapper") is a free and open source utility for network discovery and security auditing.

Open a Terminal and enter the following :

sudo apt-get install nmap

Scan your system for open ports with :

nmap -v -sT localhost

SYN scanning with the following :

sudo nmap -v -sS localhost

8. Analyse system LOG files - LogWatch.

Logwatch is a customizable log analysis system. Logwatch parses through your system's logs and creates a report analyzing areas that you specify. Logwatch is easy to use and will work right out of the package on most systems.

Open a Terminal and enter the following :

sudo apt-get install logwatch libdate-manip-perl

To view logwatch output use less :

sudo logwatch | less

To email a logwatch report for the past 7 days to an email address, enter the following and replace mail@domain.com with the required email. :

sudo logwatch --mailto mail@domain.com --output mail --format html --range 'between -7 days and today'

9. SELinux - Apparmor.

National Security Agency (NSA) has taken Linux to the next level with the introduction of Security-Enhanced Linux (SELinux). SELinux takes the existing GNU/Linux operating system and extends it with kernel and user-space modifications to make it bullet-proof.

More information can be found here. Ubuntu Server Guide - Apparmor

It is installed by default since Ubuntu 7.04.

Open a Terminal and enter the following :

sudo apt-get install apparmor apparmor-profiles

Check to see if things are running :

sudo apparmor_status

10. Audit your system security - Tiger.

Tiger is a security tool that can be use both as a security audit and intrusion detection system.

Open a Terminal and enter the following :

sudo apt-get install tiger

To run tiger enter :

sudo tiger

All Tiger output can be found in the /var/log/tiger

To view the tiger security reports, open a Terminal and enter the following :

sudo less /var/log/tiger/security.report.*

 

Yeah now that you are through with the all installation and configueation you can test to see if you still have outbound traffix with the following code

sudo tcpdump -i eth0

All request from localhost or your IP is from you, external IP is what you need to watch out for. You can maually add a strange IP with UFW.

sudo ufw deny from <ip address>

Hope this helps someone greatly.