So it seems I’ve managed to get something working with my homelab in a way that doesn’t disgust me (too much). In fact, some of it is really quite cool. The main downsides however are pretty annoying, but I’ll get to them. Also this is nowhere near finished, but there is now a finish line.

Isn’t this the nth time you’ve done this?

Yes! And what always happens is that I give up having to deal with it for one reason or another, usually caused by a stupid decision I made at some point. Thankfully, with all that prior experimentation I’ve arrived at a few nice decisions:

1. You don’t need to use ArgoCD/FluxCD for something this simple
2. Maybe I don’t need to self-host everything I think looks cool
3. Paying a little bit of cash to make things easier is perhaps fine

1

This took me by surprise. The plan was to use FluxCD since it makes more sense for a small setup like what I’m doing. But the sheer amount of stuff that needed to be done just for that was too much. I dialed my requirements back a bit and decided that a well documented README with everything handled by helm manually would be sufficient. We will see in a few months how that decision has been.

2

This is a me problem. There are so many things that could be self-hosted. Taking a few seconds back to ask “why” and “how” together can usually stop any sort of impulse. To this end I started trying to devise a list of what is essential:

• A personal wiki
• CI for Codeberg
• RSS Aggregator (for reasons that will be discussed in a later blog post)

This is a weirdly short list, but that wiki (an Otterwiki) is doing a lot of heavy lifting (like where I am currently writing this) that I might write up exactly how at some point. My main questions to help narrow things down are:

• Does it replace a paid/dodgy/sloppy product?
• Is it allowed to go down for X days?
• Does it need to be hosted?
• Is it easy to recover after catastrophe?

Generally if the answer is all yeses, it goes onto my shortlist. A key word in that first question is ‘replace’. I don’t particularly want to add a new application to what I already have UNLESS it’s absolutely critical. In this case, the ForgeJo Actions for use on Codeberg is one of very few exceptions I will be making.

Alas, while still paring everything down, there are some required essentials on top of this:

• Status and alerting system
• Some SSO provider
• A tunnel

But these are running on the VPS machines (yes, plural) so maybe it doesn’t count.

Of course in addition to these I will host my own services here (currently just LFM-GRID and snagg). Hopefully more coming soon…

3

Unfortunately homelabbing isn’t free. Even with an old ‘dead’ laptop. And it gets less free when you want to make it more secure. So I’ve decided to pay the 11€ a month for a pair of VPSs at UpCloud (that’s a referral link BTW just in case I’ve convinced you :) ). One VPS runs the SSO, status, notification, and health services. The other runs Pangolin. They are separated for 2 reasons: firstly, Pangolin is hungry. Secondly, I would rather isolate my health checks and notifications so I can be informed when anything dies (which reminds me, I need a Dead Mans Switch on the status VPS).

Newt

How I’m using Pangolin is by running Newt at the system level and exporting my services by port-forwarding (automatically) on those machines. However, running AlmaLinux (with SELinux) proved to be a bit of a shit for Newt, mainly because I’m a SELinux noob. chcon -t bin_t newt did the trick. If I decide to also forward my music server (on a different machine) it’ll be quite a bit easier since that’s just boring-old-Debian.

Iocaine

I had an idea to put Iocaine everywhere. I ’liked’ Anubis, but considering how ‘sloppy’ it now is and how it interrupts humans instead of scrapers. For my status VPS I’m using Caddy as the reverse proxy because it was easy. Putting Iocaine on there was also very easy. How nice. On the Pangolin VPS I’m using Traefik (since Pangolin defaults to that). Thankfully there was a guide on installing Iocaine as middleware there. So now every site I forward via Pangolin has Iocaine running with it! I currently have it configured at its most basic but I am thinking of switching it to use ‘Nam Shub of Enki’ instead of the default QMK. I also need to forward the metrics somewhere easy so I can check them out easily.

Future

I will likely add some more things to my “essential services” list soon, with the current contender being a todo/planning tool, and a GTS instance is looking might tempting. I’m sure there’s many more quirks I will have to deal with (like proper secrets management, which I think I’ll be going down the SOPS road again) but right now things seem nice.

An idea

Someone I follow on Mastodon had the idea of powering off services when you don’t really need them, but for mental health reasons. I like this idea quite a bit, but since I plan on hosting some publicly accessible things I would have to focus on scaling down based on load instead of outright killing things. I will probably give Keda a go at some point, although not sure how to get that to behave with the Forgejo runner. Although making some ‘power-off/power-on’ endpoint could be kinda neat….