Setting up a new Ubuntu 24.04 server

This article explains how to set up a new server in eclean's infrastructure. Following these steps guarantees that all servers are configured in accordance with our security standards and operational specifications.

Security sensitive

This document includes security-sensitive information about our infrastructure configuration. Before going, make sure to follow security protocols and have correct authorisation.

Prerequisites

Before starting the server configuration process, make sure you have:

SSH key pair produced and ready for usage.
Required firewall rules documented and authorised.
List of required software programs and their versions.

Server Provisioning and Configuration

Prepare the Operating System

You should be starting with a fresh install of Ubuntu 24.04.

1. Update and Upgrade Packages

Start by updating the package lists and upgrading the installed packages on Ubuntu 24.04.

sudo apt update && sudo apt upgrade -y

2. Install Essential Tools

Install ubuntu-advantage-tools to manage subscriptions and services.

sudo apt install ubuntu-advantage-tools

Connect to Ubuntu Pro and Landscape SaaS

Ubuntu Pro might be optional

If you're creating a server for testing or development, you don't require Ubuntu Pro. We strive to use Ubuntu Pro on servers that are predominantly utilised in production or are important to the operation of eclean's services.

Follow Ubuntu's steps to connect the server to Ubuntu Pro Services and Landscape for remote management.

To enable for this connection, eclean facilitates Ubuntu Pro licenses. You may find eclean's Canonical account credentials in Vaultwarden.

Get started with Ubuntu Pro

Install and Configure CrowdSec

Install and configure CrowdSec on your server according to the instructions provided. After installation, follow the steps to setup and connect the server to the CrowdSec Console.

Install CrowdSec on Linux

1. Run Configuration Wizard

Configure CrowdSec with the interactive wizard.

sudo /usr/share/crowdsec/wizard.sh -c

2. Install Bouncers

Add the CrowdSec "firewall" bouncer and the Linux, sshd & iptables collections.

sudo cscli bouncers add firewall
sudo cscli collections install crowdsecurity/linux
sudo cscli collections install crowdsecurity/sshd
sudo cscli collections install crowdsecurity/iptables

3. Connect to CrowdSec Console

Obtain the enrolment key from the CrowdSec Console and enrol the server.

After enrolling, approve it in the CrowdSec Console and restart the service

sudo cscli console enroll <your-enrollment-key>
sudo systemctl reload crowdsec

Network Configuration

To ensure that the server does not use the default DNS servers or upstream DNS servers given by the hosting provider, you should specify the DNS servers.

1. Update DNS Settings

Modify /etc/netplan/50-cloud-init.yaml and /etc/systemd/resolved.conf to use specific DNS servers.

# /etc/netplan/50-cloud-init.yaml
network:
version: 2
ethernets:
    eth0:
        nameservers:
            addresses:
                # https://one.one.one.one/family/
                - 1.1.1.3
                - 1.0.0.3
                # https://www.dns0.eu/zero
                - 193.110.81.9
                - 185.253.5.9

# /etc/systemd/resolved.conf
DNS=1.1.1.3 1.0.0.3 193.110.81.9 185.253.5.9

2. Apply Network Changes

sudo netplan apply
sudo systemctl restart systemd-resolved

3. Confirm that DNS works

nslookup eclean.gg
curl -s https://dns.google/resolve?name=eclean.gg

User Management

1. Create Administrative User

Create a user ecleangg and add them to the sudo group.

sudo adduser ecleangg
sudo usermod -aG sudo ecleangg

2. Configure SSH Key

Add your public SSH keys for user ecleangg and ensure that you can log in with them.

sudo mkdir -p /home/ecleangg/.ssh
sudo nano /home/ecleangg/.ssh/authorized_keys

Edit /etc/ssh/sshd_config to disable root login and password authentication.

sudo sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config && \
sudo sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo systemctl restart sshd

External Monitoring via HetrixTools

This step is only needed for production servers

If you're configuring a server for testing or development, you can skip this step. We only utilise HetrixTools on production servers and services.

We use HetrixTools to monitor the availability of our servers and receive alerts in the event of outage.

HetrixTools also provides blacklist monitoring, which we utilise to see whether any of our IP addresses are included on any public blacklists. This allows us to rapidly identify and respond to any abuse attempts, as well as contact our hosting provider if necessary.

1. Create HetrixTools Monitors

Each production server should have a "Ping/Service Monitor" and a "Blacklist Monitor" via HetrixTools.

Ping/Service Monitor Blacklist Monitor

Provide the following information when creating the "Ping/Service Monitor" uptime monitor.

HetrixTools Uptime Monitor

Setting	Value
Monitor Name	The hostname of the server
Hostname or IP Address	The IP address of the server
Port Number	22
Category (Optional)	Server
Monitor From	New York Frankfurt Sydney Mumbai

Advanced settings

Setting	Value
Timeout	10 seconds
Checkup Frequency	1 minute
Number of tries	3
Number of Triggering locations	50% + 1
Alert only after x minutes of downtime	Alert me after 3 minutes of downtime

Provide the following information when creating the "Blacklist Monitor".

HetrixTools Blacklist Monitor

Setting	Value
IP Address, Hostname, IP Range or IP Block	The IP address of the server
Label (Optional)	The hostname of the server

2. Connect HetrixTools Server Monitoring Agent

Make sure you've already set up the "Ping/Service Monitor" and that the server is active and accessible via the HetrixTools network.

Once the server is operational and accessible via the HetrixTools network, you can install the HetrixTools Server Monitoring Agent from the HetrixTools Uptime Monitor dashboard.

When configuring the agent for installation using HetrixTools, pick the ecl-default template under "Agent install templates".

Core Applications Setup

1. Install Containerization and Web Server Tools

Install podman and nginx.

sudo apt install podman
sudo apt install nginx

2. Enable CrowdSec for Nginx

sudo cscli collections install crowdsecurity/nginx

3. Secure Web Server with Certificates

Use acme.sh to issue SSL certificates from ZeroSSL using Cloudflare for DNS challenges.

cd /home/ecleangg
curl https://get.acme.sh | sh -s email=ecleangg@gmail.com
export CF_Token="your-cloudflare-api-token"
export CF_Zone_ID="dc665c201e8ce0e53a8aca5c97bef20f" # The zone ID of eclean.gg
acme.sh --issue --dns dns_cf -d eclean.gg -d '*.eclean.gg'

3. Connect Server with Cloudflare Tunnel

This step is optional

This step is optional and only required if you want to connect to the server using Cloudflare Tunnel. It could be beneficial if you wish to connect a server that is not accessible over the public internet to services located on eclean's network.

Cloudflare Tunnel Overview

Cloudflare Tunnel is a service for securely connecting your server to your Cloudflare account.

When creating a tunnel, use the eclean Cloudflare account and create it in the eclean.gg zone.

Create a Remote Tunnel via Cloudflare Zero Trust

4. Configure Nginx with pre-made configurations

The ecleangg/nginx-configs GitHub repository offers pre-made Nginx settings that can be used to quickly deploy services.

The main Nginx configuration file should be nginx.conf, whereas core Nginx configuration snippets should be found in the configio subdirectory and its contents.

The sites-available directory includes the real Nginx configuration files for the services, which are subsequently symlinked into the sites-enabled directory.

ecleangg/nginx-configs

Automation configuration

1. Automatic Updates for CrowdSec

Create the following script in the root directory to automatically update CrowdSec.

HealthChecks.io compatible (recommended) Generic

Healthchecks.io account required

This script reports the status, logs, exit code, and script duration to Healthchecks.io. You must have a Healthchecks.io account or a self-hosted Healthchecks.io instance to run this script.

# /root/cron/crowdsec-update.sh
#!/bin/sh
RID=`uuidgen`
HEALTHCHECK_UUID="your-healthchecks-io-uuid"

# Start ping to healthchecks.io
curl -fsS -m 10 --retry 5 https://hc-ping.com/$HEALTHCHECK_UUID/start?rid=$RID

# Script
m=$(cscli hub update && cscli hub upgrade 2>&1)

# End ping to healthchecks.io
curl -fsS -m 10 --retry 5 --data-raw "$m" https://hc-ping.com/$HEALTHCHECK_UUID/$?rid=$RID

Once the script is created, add it to the crontab.

echo "0 */1 * * * /root/cron/crowdsec-update.sh" >> /etc/crontab

Add the following line to the crontab.

echo "0 */1 * * * cscli hub update && cscli hub upgrade" >> /etc/crontab

2. Automatic SSL Certificate Renewal

Create the following script in the root directory to automatically renew the SSL certificates using acme.sh.