Lazy Hosting

Here’s why I started hosting:

Wix is $20/month? That’s crazy! You can host things on WordPress on a server for $5!

I looked around, found myself a $5/month server from Ramnode. Then, I followed a guide for installing WordPress on a LAMP stack.

Solved! The end, thanks for reading. ❤️

Well. No. Because this sent me on a decade+ length of obligation where I began to host not one site, but dozens. Forums, web apps, static sites, etc.

Discovering Linux & Open Source

At the time, I was very interested in Linux. I grew up on Windows XP, then Windows Vista. Windows Vista was a great operating system. Because they had glossy window tiles. I thought they were amazing. I also didn’t know enough about computers to run into issues other people had. Like printers drivers or everything else.

I set up the web server to use a LAMP stack (Linux, Apache, Mysql and Php). However, most of my focus was on fun little things you can do. For example, manually configuring iptables to drop login attempts after 3 attempts.

iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 5 --hitcount 3 -j DROP

I also experimented with honey pot servers. Instead of dropping packets, iptables would route to one after 3 attempts to my ssh server.

I read long discussions on securing your sshd service. There is a lot of debate around this topic. Should you try to reduce the number of login attempts? Should you use port knocking? Why allow access at all? Use a VPN! Eventually, I wanted to move onto other interesting things. So I set it to public key only.

PasswordAuthentication no

I have had some odd pursuits like executing code directly with Nginx. Something I’d switched to because it was “lighter” and “faster” then Apache. But the real reason why I switched? Everyone else was doing it and it was the new “correct” way.

Because of this switch, I spent a large amount of time configuring Nginx. Nginx makes it easy to cache requests. I found the idea of requests never actually touching the app an appealing idea. Of course, this led into dangerous areas. Caching dynamic pages turned out to have several footguns. In one delightful example, users discovered that they would be signed in as someone else!

Migration

Eventually, time passed, and I needed to update my operating system. Due to the VPS I was using, I needed to migrate my server. Now, I have had years of customization’s and tweaks embedded into this system. Packages installed for bits and pieces I’d needed over time. All undocumented. The configuration files are the documentation, right?

This was a painful migration. It took a lot of time having to relearn the tweaks. After all, I’d moved on to more interesting things and I did not want to bother. I was also reluctant, setting up a server took a stressful weekend. I like my servers glossy, with zsh and oh my zsh installed and configured. Time zone updates, little bash functions I’d written, locking down ssh. This took time, and I would make mistakes and forget the right order of things and wipe the system and reinstall until I could get it working. It was a lot of stress, headaches and frustration because I couldn’t find the guide to do X.

Now, you might be screaming all sorts of useful tools that I ought to take advantage of, with you and your experience, hindsight and whatnot. But I didn’t have that, so I took my blunt axe and kept wacking. But, I did discover that something called Ansible and Docker did exist.

I had a different career at this time, Physical Therapy, which required the technical expertise of near 0 (think, typing in documents). So, learning about better ways of doing things took time in years to really have the chance to accomplish them. I just didn’t know what was out there. The truth is, I discovered Hugo before discovering React. The only programming I knew was Python, and I also knew that you couldn’t use Python for anything dealing with the web. The truth is, I’d had nearly 0 interaction with the technical world. I simply did not know what I did not know.

As you can imagine, I felt like this made me very qualified to run a forum. Which, back in 2015 is exactly what I did. I ran this forum the same way I did WordPress. Making all of those fun little tweaks and optimizations.

Unfortunately, this led to mistakes. Nearly everything I did was in discovery mode. I discovered backups out of necessity, I would try to do upgrades, in place, in production. So if the upgrade failed, it was panic time to try and recover the mess. For every upgrade.

These were uncomfortable times for me. As much as I loved hosting, I did not enjoy the deep levels of stress I felt when everything felt broken. I would be in a position where I have no idea why nothing is working and I don’t know how to revert or recover. Multiply this for a large number of services I hosted and the entire endeavor started to become less fun. But still, I did it all. I’m not sure why. It wasn’t fun or especially fulfilling at those points. But it felt right to do. I suppose I have a passion for it. I think without it, I would have quit years ago.

Lazy Hosting

Eventually however, I wanted to move onto other things again. I didn’t want the routine stress and panic. So I began what I called shifting towards “lazy hosting”.

What is lazy hosting? Lazy hosting is where you don’t have to document, or remember anything about your setup. You don’t have to monitor it. It’s so easy to administer it, that you can spend the whole day being lazy and it’ll be fine. It doesn’t mean you aren’t allowed to (you should!), it means that if you don’t, things will continue to function.

To get to this point, I had to move my forum instance around a few times:

• VPS Server
• cPanel Instance
• VPS Server (again)
• Docker Compose instance
• Docker Swarm service

Those last two were part of the transformation that got me into containerization. I especially like docker compose. In one directory, I can once again hack the service to death. Optimize caching (not dynamic!), add in little extras, follow little guides again? Sure! Place those steps into a Dockerfile, and version commit it.

Suddenly, with my tweaking now version controlled I could come back anytime and change it. Do I want to implement Redis and make a dozen overrides in my forum? Sure. A couple years later, can I come back and say “this is terrible, I need to get rid of Redis!” Well, yes! Instead of having to undo changes, I just need to remove those instructions. Each time I ran a docker build, I started from a clean slate.

This was a major step for getting to feel lazy. I relaxed my brain from the task of “make sure to remember how you did x, y, z, aa, ab..” and I run docker build with the configuration I needed. Actually, I use use just build_image myapp now.

Thanks to my effort with Docker, I now had docker compose running my services. Despite moving to docker swarm for most of my hosting, I still see this as near peak for hosting. Got a single server and want to host 1-5 things? Use docker-compose. It is absolutely fantastic. Don’t use named volumes, just do something like data/db and volume mount your database server to it.

volumes:
    - ./data/db:/db

People will say that’s wrong, but it is really hard to compete with doing the following for upgrades:

docker compose down
tar -zcf ~/my_app.tar.gz .
# Do upgrade stuff
docker compose up -d

Following this method allows you multiple benefits. You can move to different hosts, restore from backup, or change your software version. I still use this setup for some services, because I can trust that simplicity so absolutely.

I did want to automate my server deployment. So I finally gathered my strength and began reading the documentation for Ansible. I mulled and thought about it, and built myself a system for automated server deployment. Using roles, I chunked my setup. Now I could setup a server in minutes, instead of weekends.

- name: Init Server Setup
  hosts: all
  remote_user: root
  vars_files:
      - vault.yml
  roles:
      - common
  tags:
      - setup_init
      - setup_common

With these two tools of autoconfigured servers and containerization, I finally made the leap to docker swarm. This took a few years by itself and even then, I didn’t trust it for some time. I felt wary that something would go haywire. I didn’t want to lose critical files because I did something like docker volume prune and erase my databases. But with time I felt more at ease and eventually moved my forum to it.

It’s been great, except for when it’s not. If you only have 3 nodes, if one goes down you have 2 nodes and a split brain problem. And it happened a lot. My docker swarm would get confused, repeatedly and just crash. Eventually, I upgraded to 5 nodes and this problem went away for the most part. But things are fickle and changes have to be incremental. Want to change the vpn your docker swarm is part of? Nope. To resolve that I had to manually rewrite the networking from scratch for the whole swarm and pray (it worked).

Epilogue

If you want my advice, these are the laziest setups I’ve had:

• Zero configured LAMP stack. No changes, leave it alone.
• Docker Compose, with all files in one directory.
• Docker Swarm, with Ansible for server deployment and a repo for the configuration for all of my services.

So, did I accomplish what I wanted?

• I don’t think about individual servers anymore.
• Server setup takes minutes, not days.
• All of my tweaks and changes are version controlled, revertible and visible.
• If my servers get “full”, I set up a new server and have it join the swarm.

So, I think so, hosting has now taken a backseat for me. I now spend more time learning programming and building things. I go out on weekends and rarely think about it.

It’s also convenient, because I can host my own applications now ❤️