klay’s simple cookbook for Linux, v0.10.1

introduction

why a cookbook?

there’s an old joke that using Linux is like ordering a cake and receiving a bunch of flour, sugar, butter, and eggs. many self-hosting tutorials read like the back of a box of cake mix, which is great if you want a cake fast, but unsatisfying if you want to learn about cake making. my goal is for this guide to teach you enough to improvise, to make substitutions.

why all the lowercase?

a few people have asked me this. honestly I have some influences but the main reason is that it reminds me of writing code. in code, uppercase letters are almost always reserved for names. I did my best to capitalize all names in this document the way their owners capitalize them, so the web protocol is HTTP, but the comic strip is xkcd.

who is your audience?

this guide is for you if:

sound good? then let’s get cooking.

materials

I’ll be using a Raspberry Pi 4b and installing Raspberry Pi OS Lite (64-bit). it’s also known as Raspbian, since it’s based on Debian Linux. any Debian-like OS will work for this tutorial. if the Raspberry Pi isn’t available where and when you are, some good alternatives are the Renegade or the ROCK64. If Raspbian doesn’t run on your device, try armbian.

you’ll also need a MicroSD card, a hard drive, and a second hard drive for backup. I recommend a 64GB MicroSD card and two 2TB hard drives. the Pi can power one hard drive via USB, but if you want to plug both in at once you’ll need a powered hard drive enclosure or a powered USB hub.

you can also adapt these directions to existing hardware you may already own, like an old laptop or desktop pc. you can also rent a virtual machine from a remote datacenter! the cheapest virtual machine on DigitalOcean right now (as of 2023 July) costs 6 USD per month, and comes with 25GB of storage. renting a machine in a datacenter means you get the perks of high-speed internet all day and night, which is great if you are running a game server and need a fast connection. however, you’ll pay 10 USD per month for each 100GB of additional storage, so it’s not great if you want to host a huge media archive.

usernames and passwords

at some point you’ll probably want to make your server available on the internet. there are bots out there that do nothing but guess random usernames and passwords all day, and they will find you, so you need to be prepared.

for your username, pick something short and catchy. in my experience, a typical linux username is from 3 to 8 characters long. if you have a short name like ‘alice’ or ‘bob’ you can use that. if you have a nickname online like ‘azure’ or ‘luna’ then that works too. if you have a middle name, you can use your initials, like ‘jfk’. your username should be all lowercase.

for your password, the most important factor is length. maybe you’ve been told that you need to add weird symbols and stuff in it, but that rule is for more limited systems. it’s the 64-bit era, baby! the only thing that matters is how much entropy your password has. entropy is basically a measure of how unlikely something is to happen by accident. like if I flip ten coins and you flip ten coins and we get the same sequence, the odds of that happening are 1 in 2^10, so we say that that particular sequence of flips has 10 ‘bits’ of entropy. it’s recommended that your password have about 60 to 80 bits of entropy. yikes, that’s a lot of coin flips to memorize!

luckily there’s a better way. using a tool like xkpasswd or diceware, you can convert randomness into words. using the diceware list, you can roll 25 six-sided dice to get a random 25-digit sequence, and then convert each 5 digits into a word, to get a 5-word phrase. the odds of two people getting the same 5-word phrase are 1 in 6^25, which is approximately equal to 2^64, so this password has 64 bits of entropy. that’s enough to make most hackers give up and move onto an easier target. go generate a password now, and save it somewhere safe, like a password manager program, or a piece of paper tucked inside a book. don’t be afraid to write things down; sometimes paper is safer.

operating system

time to download and install your operating system. there are lots of guides on how to install Linux already, here’s one for Raspbian. I’ll be using Raspberry Pi OS Lite (64-Bit), with the default settings. with these settings, there is no remote login available; you will need to plug a keyboard and monitor directly into the pi. on the first boot, you’ll be prompted to enter the username and password for the initial user. after that, you’ll be given a login prompt. log in with the username and password you just set. some operating systems display nothing at a password prompt, not even stars. once you log in, you’ll see some system information, and then a prompt that looks like this:

user@raspberrypi:~ $

this is a CLI, a command line interface. from left to right, this contains

if you’ve ever used a bot in a chatroom, you already know how to use a command line interface! it’s a system where you type an instruction, and the computer answers you. here are some examples of commands you can type:

if this is your first time using a command line interface, try some of these commands now. use nano myfile.txt to open a text file, write some text to it, then save with ctrl-o, and exit with ctrl-x. use ls to look at that file, then use cat myfile.txt to have the computer read it back to you. pat yourself on the back, you’re learning so much!

soon we’ll unplug the keyboard and monitor from this computer, making it a ‘headless’ server. before we can do that we need to run a few commands as administrator, or ‘root’. we’ll do this with a very powerful command called sudo. sudo stands for ‘super user do’, and it means ‘do the next thing as an administrator’. you can use sudo <command> for a single command, or enter interactive mode with sudo -i. in interactive mode, all your commands will be admin commands, until you say exit.

⚠️ there is no undo button! if you say something with sudo, make sure you mean it!

throughout this guide, I’ll be using $ at the start of a command if you run it as a normal user, or # if you run it as admin. either way, you don’t actually type this symbol yourself, it should already appear on your command line.

we’ll need to do these things before we can go headless:

first we’ll change the sudo rules for our account. there should be a config file at /etc/sudoers.d/010_pi-nopasswd. we’ll use the special visudo command to edit it, which should launch nano like before, but with special safety guards to catch us if we lock ourselves out of administrator mode.

$ sudo visudo /etc/sudoers.d/010_pi-nopasswd

you should see a single line, like

user ALL=(ALL) NOPASSWD: ALL

remove the NOPASSWD: instruction, so this file reads

user ALL=(ALL) ALL

save and quit. from now on, if you haven’t used sudo in a few minutes, the system will ask for your password. this way if you accidentally leave yourself logged in, and someone else takes over your session, they won’t automatically get sudo access.

we can do the next few steps with the raspi-config tool.

$ sudo raspi-config

the hostname is under System Options -> Hostname. I set mine to ‘teapot’.

if your machine is plugged directly into your home router with a patch cable, you already have network access, otherwise set up wireless with System Options -> Wireless LAN.

to enable remote login, go to Interface Options -> SSH.

select ‘Finish’. you’ll be prompted to reboot, but don’t yet, we’ll reboot later after installing some updates. in Debian Linux, we update packages using a tool called apt. apt update checks for updated packages, and apt upgrade installs them. run both these commands now, using the interactive version of sudo.

$ sudo -i
# apt update
# apt upgrade

finally, if you’re on a Raspberry Pi, then avahi-daemon is installed automatically. if not, you may need to install it yourself. this program broadcasts your hostname to the network, so you can log in remotely without configuring your router.

# apt install avahi-daemon

now go ahead and reboot, to make sure your new hostname is in use.

# reboot

after a minute, you should now be able to log in remotely from another computer, using

$ ssh user@teapot

if this works, congrats! you can now unplug the keyboard and monitor. you’ve created a headless server.

format storage

the Raspberry Pi uses a MicroSD card as the operating system disk. it’s convenient; if the OS breaks you can just pull its brain out, factory reset it, and pop it back in. however, I don’t want to store all my user data on that brain card. I think it’s better if you have a secondary disk that contains only the stuff you create yourself. that way if something goes wrong and you have to reset the brain card, you don’t lose any of your personal data.

there are different formats for a data disk. the format determines exactly where the data and metadata will appear in each ‘chunk’ of the disk. Windows typically uses NTFS, which supports metadata for ownership and last modified time, but not for fine-grained access like whether a file is shared with guests. Linux uses a format that does allow this fine-grained access, called ext4, so that’s what we need to format the data disk as.

first we need to identify the disk’s device file. in Linux, everything you can read or write to is treated as a file, including a USB device like an external disk. note that this device file is not the same as a filesystem mount. we’ll cover mounting later.

with the data disk unplugged, run the command df. plug the disk in, and run df again. compare its output to the previous run. there should be exactly one new entry, and it should look like /dev/sda or /dev/sda2. if you’re not sure which disk it is, don’t risk it, ask a friend for help.

⚠️ warning! this will erase everything on the disk!

format the disk, and label it. this will make it easier to mount later.

# mkfs.ext4 /dev/sda
# e2label /dev/sda teapot-data

mount storage

Linux doesn’t use drive letters like Windows does. instead, every disk’s filesystem lives at some path. the main, or ‘root’ path is /, a single slash. the root path belongs to the operating system disk, in this case the MicroSD card. we’ll create a new sub-path at /data for our data disk.

# mkdir /data

when the system boots up, it looks in the config file /etc/fstab to find out where we want other filesystems to be loaded. since we gave our disk a label earlier, we can mount it using that label. open the file with nano, and add this line to the end of it:

LABEL=teapot-data /data ext4 nofail,x-systemd.device-timeout=5s,x-systemd.automount 0 0

there’s a lot going on here. you can read more about how fstab and systemd work, but basically what we’re saying is

we’re using a delayed mounting process here because we want to make sure our server still comes online, even if the disk fails to load. if the server crashes on boot, we’ll have to go plug the monitor and keyboard back in to fix it. with nofail, we have a chance to fix it remotely.

save and exit the file if you haven’t already, and then check it:

# mount --all --fake --verbose

if all your mounts pass inspection, now is a good time to reboot the machine.

# reboot

backups

I’ll write a longer section about backups later. basically: every month or so, plug in the second disk, and copy everything from the first disk to the second one. it’s a quick and dirty solution and it’s better than having no backups at all. the command you want is

rsync -axHAWXS --numeric-ids --info=progress2 <source> <destination>

explanation here.

website

we’re getting into the fun stuff now. one of the coolest things you can do with your Linux server is host a website. a little chunk of the world wide web that belongs just to you. all you need to be a website is to have a program running and ready to answer HTTP requests with HTML text. HTTP is the HyperText Transfer Protocol, and ‘HyperText’ is just a fancy word for ‘text with hyperlinks in it’.

put briefly: the internet is made of programs that exchange text files with each other!

we could write a web server from scratch, but to get started we’ll use Apache, a free, open-source, and well-established web server. nginx has a free version too, but Apache is good enough for our purposes.

# apt install apache2

it should start itself automatically. go ahead and check http://teapot.local, and you should see the test page! now we’ll write our own page hosted on the data disk. we’ll make a folder for it called /data/teapot.local and a subfolder of that called site.

# mkdir /data/teapot.local
# mkdir /data/teapot.local/site

when making multiple levels of new directories like this, you can use this shortcut to create the whole chain in one go:

# mkdir -p /data/teapot.local/site

we’re about to do a bunch of typing to set up a basic site definition. you can copy and paste this if you want, but I recommend typing it by hand. it’ll train your brain to recognize pieces of the code. remember, use nano to create or edit text files.

contents of /data/teapot.local/site.conf:

<VirtualHost *:80>

ServerName teapot.local
DocumentRoot /data/teapot.local/site

<Directory />
  Require all granted
</Directory>

</VirtualHost>

this is just about the simplest possible website definition. we’re saying “hi, I am a web server listening to port 80, serving pages for the website ‘teapot.local’.” a port is like a post office box for a computer. it allows you to address a specific program inside the machine. port 80 is an old and well-known port which is used for most HTTP traffic.

next we define the DocumentRoot to be /data/teapot.local/site, instead of Apache’s default of /var/www/html. we also tell Apache that it’s allowed to share these files. by default, if no filename is specified, Apache will look for one called index.html, so we’ll write that too.

contents of /data/teapot.local/site/index.html:

<!DOCTYPE html>
<html lang="en">
<meta charset="utf-8">
<title>my website!</title>
<h1>my website!</h1>
lorem ipsum dolor sit amet

HTML is what your web browser sees. in fact, if you’re on a desktop browser, you can press ctrl-u right now to see the HTML text behind this page! what I’ve written for this example is the bare minimum to follow the modern html5 standard.

once you’ve written both of those files, we can tell Apache our site is ready. by the way, when typing these long paths, try pressing ‘tab’ once or twice, sometimes your shell will auto-complete words. neat!

# ln -s /data/teapot.local/site.conf /etc/apache2/sites-available/teapot.local.conf
# a2ensite teapot.local
# a2dissite 000-default
# systemctl reload apache2

now go back to http://teapot.local and hit refresh, and you should see your new website! at this point you can go explore the world of HTML. the nice folks at Neocities are helping to keep this art alive, go check them out! and remember, you can press ctrl-u to view the HTML for any page you’re on.

p.s. up until this point I’ve been writing with pure HTML myself, but as this page is getting rather long, I’m actually switching to a helper tool called pandoc. I may cover this tool in a later tutorial.

networking

at this point, your server is available only on your LAN, your local area network. this network is managed by your router, which is a small computer plugged into your home’s internet cable. this cable goes to an internet service provider, or ISP, and they take the traffic from your network and connect it to other networks around the world. by default, your router will protect your computers from unwanted traffic. people from around the world can’t log into your server, and they also can’t see your website. if you want to change that, you’ll need to change settings on the router.

⚠️ don’t modify a home network without consent!

if you do this wrong it puts everyone on the network at risk, so make sure everyone in your house knows what you’re doing. if anyone isn’t comfortable, then do not open up your home network! you have lots of other options:

if you’re sure you want to open up your home network, read on…

firewall

if you do everything right, the only server that will be available to the outside world is our little teapot, and only people with an account will be able to get in.

⚠️ you did pick a long, random passphrase, right? if not, go back and do that now.

however, for a little extra safety, we’ll use ufw to limit what kind of messages teapot will answer. ufw stands for ‘Ubuntu firewall’ but it works on other systems besides Ubuntu now, so sometimes it’s called ‘uncomplicated firewall’ instead.

💡 I got this information almost verbatim from the Raspberry Pi Foundation, seriously they’re awesome people.

install ufw. unlike other system services, this doesn’t start automatically, since you can lock yourself out with it. that’s why it’s important we configure it before changing settings on the router.

# apt install ufw

we’ll also need to know the router’s local address. we can find this with the ip function.

$ ip route

the first line of the output should look like this:

default via 192.168.20.1 dev eth0

if you’re using a wireless connection, you’ll see wlan0 instead of eth0. underneath the first line, you should see a similar number, but with /24 at the end, like

192.168.20.0/24 dev eth0

this defines the local subnet, the list of IPs that count as part of your local area network. the /24 means that the first 24 bits are fixed. there are 8 bits in a byte, so 24 bits is 3 bytes, meaning that this subnet includes all addresses of the form 192.168.20.x.

to avoid locking ourself out, we’ll allow anything from the local area network to connect:

# ufw allow from 192.168.20.0/24

substitute your own subnet as appropriate. we’ll also allow http and https traffic through. ufw has built-in rules that say how to handle this traffic, and it will open the right ports for you.

# ufw allow http
# ufw allow https

if you want to log in remotely, enable ssh as well:

# ufw allow ssh

you can check the list of rules with ufw show added. if all looks good, turn it on:

# ufw enable

with any luck, you won’t get kicked off your ssh connection, which means the firewall is allowing you through. ufw should now be blocking traffic on all other ports, which will dramatically cut down on the surface area for an attack.

private key auth

chances are, even if you want to allow outside ssh connections, you don’t really need to be able to log in from anywhere. most likely you’ll only be logging in from one or two machines, like a laptop or your phone. we can configure our server to only allow authenticated devices to connect.

there’s a lot of really advanced math behind private keys, math called cryptography, but the important part for our purposes is that each key has a public and a private part. the public key can be used to encrypt files, “locking” them, but only the private key can decrypt, or “unlock” them. we’ll use this to prove that a device is authorized, by sending a locked message that only it can unlock.

on your laptop or other client device, generate an ssh key. if you’re on Linux, Mac OS, or Windows 10, this is probably named ssh-keygen. you can also add a comment to the key, so you remember which device it belongs to.

$ ssh-keygen -t ed25519 -C "my laptop"

follow the prompts to generate a private key. it will be saved in /home/user/.ssh (or your operating system’s equivalent location), as well as a public key with a .pub file extension. you can add a passphrase during key generation, this is highly recommended if anyone else has access to your computer!

we’ll need to get the public key, the one ending in .pub, onto the server somehow so it can recognize us. public keys don’t need to be kept secret, so use any type of file transfer available to you. you could copy it to a USB drive and plug it into the server, or copy it by hand, or even send it to yourself over the public internet. once you get it to the server, create the file ~/.ssh/authorized_keys. on a new line, copy the full contents of the .pub file into the authorized_keys file.

💡 if you need to connect from multiple devices, I recommend repeating these steps for each device, rather than copying your private key between devices. that way you don’t have to worry about accidentally leaking your private key. you can add as many public keys to authorized_keys as you want!

at this time, try logging into the server from the newly-authorized client. it will ask for your key’s passphrase if you set one, but it won’t ask for your account passphrase, since you already authenticated yourself by having the private key. you will still need to type your password to use sudo though, just in case.

if that went well, you can now disable password-based auth completely. as admin, open /etc/ssh/sshd_config and make sure this is set to ‘no’:

PasswordAuthentication no

save the file, and restart the ssh service:

# service ssh reload

it should now be impossible to login without an authorized key. if you want to authorize more devices, repeat these steps for each device. each device should have its own private key. never let a private key leave the device that created it. if you accidentally copy a private key, remove it from authorized_keys and create a new one to replace it.

port forwarding

⚠️ don’t do this until you’ve done everything else!

it’s time to make the big leap. your server is armored up and ready to face the outside world. let’s open the gates. the end goal here is to forward ports 80 and 443, from the outside world to your server. if you want to log in remotely, you’ll also need to forward port 22.

unfortunately, port forwarding is going to be a bit different for every router. I’ll share what worked for my router, a MikroTik running RouterOS.

while we’re configuring the router, we’ll enable hairpin NAT, also known as NAT reflection or NAT loopback. without this rule, the router may get confused if we try to access teapot via its public address while we are inside the local network.

you should now be able to go to http://<your own public IP address>, and your router will forward the traffic to teapot. congratulations, your machine is now a true part of the internet.

https

todo:

file sharing

another neat thing you can do with your server is use it to store stuff! there are some all-in-one solutions for this such as NextCloud, but right now that’s overkill for me; I want to install simple tools to solve simple problems.

sync

Syncthing is a simple tool for duplicating some folders across multiple devices. maybe you want all your photos to be automatically duplicated from your phone to your server, or you have some game save files that you want to be copied between two gaming pcs, or you have some other documents that you access frequently from your laptop and your desktop. Syncthing is one of those rare and beautiful gems of free software that ‘just works’.

as admin, install the service:

# apt install syncthing
# ufw allow syncthing

as your normal user account, enable the service for yourself:

$ systemctl --user enable syncthing
$ systemctl --user start syncthing

that’s it! Syncthing is now installed and running! you can configure Syncthing with a web app that runs on port 8384. if you’re connected remotely via ssh, use this trick to get access. type the special sequence [enter], tilde (~), C (shift+c), and you’ll get a prompt like

ssh>

we’ll use this prompt to set up a temporary tunnel. a tunnel allows you to bind a port on your client machine to a client on the server. specifically, a tunnel passes some traffic through your encrypted ssh connection, as opposed to passing it through a browser via http or https. this way you can quickly get access to an application running on your server, without exposing it to the whole internet. for this example, we’ll bind the client’s port 8000 to localhost:8384 on the server.

ssh> -L 8000:localhost:8384

now, open http://localhost:8000 on your client, and you should have access to the Syncthing console running on the server!

💡 you can also send this command to ssh at login time. for instance:

$ ssh user@teapot -L 8000:localhost:8384

syncthing todo:

file sharing todo:

containers

at this point, we’ll take a detour and talk about containers. everything we’ve done so far has involved making changes to system files. if you need to copy your data to a new system you’ll have to change all those system files again, and this gets more complicated the more things are installed. thankfully, many applications can now be installed as containers.

a container isn’t quite a virtual machine. you’re not running a whole computer-inside-a-computer. it’s more like a facade, a fake computer with a fake filesystem and fake network. containers don’t take much memory to create, and they don’t leave junk all over the filesystem when you remove them. neat!

we’ll be using Podman, a container host service. if you’re familiar with Docker, this is like that, but even more free.

# apt install podman python3-pip
# pip install "podman-compose<1.0"

we’ll test Podman by running a copy of nginx (pronounced ‘engine-x’). we’ll use the image published by Docker Hub. think of an image as a casting mold. you can’t run an image directly, but you can use it to produce a filled container.

$ podman run --rm -p 8000:80 docker.io/library/nginx

💡 if you get an error like “delete libpod local files to resolve”, you may need to do this extra step, then try again:

$ sudo rm ~/.local/share/containers

you should now be able to point a web browser at http://teapot.local:8000 and you will see nginx is running!

finally, we’ll use podman-compose to save this setup. create a folder in /data for this container, and make a new file called compose.yaml.

contents of /data/nginx/compose.yaml:

version: '3'
services:
  nginx:
    image: docker.io/library/nginx
    ports:
    - 8000:80

this does the same thing as our command from before. it starts one service container, running nxing, with port 8000 on the host mapped to port 80 in the container. to run this script, type

$ podman-compose up -d

and to stop it again:

$ podman-compose down

database

todo:

apps

todo:

auth

todo: