(I am bad at updating this blog but whatever)

I took quiet a break from the self-hosting stuff mainly due to reorganising everything and work getting mega busy from late 2023 to late 2024. So what’s it look like now?

• k3s instead of docker, because its so much easier to manage
  • tried FluxCD but managed to break it almost immediately, so switched to ArgoCD which I know very well
• Server is now running Proxmox with 2 AlmaLinux nodes. Server is struggling a bit and will probably need to be replaced :(
• Outside access is still missing mainly due to the new need to tunnel to a remote server and expose via that. Why? Security basically

Most of the same goals apply really. Having a Wiki to store long-term useful info. CI for Codeberg. Media hosting and so on and so forth. I don’t really want to make the whole list here but instead talk about little bits and pieces, because hoo boy there sure are a lot of little problems everywhere.

Let’s start with the latest one: filesystem consistency. One goal I have is that I can sync my Bandcamp library and allow uploading of music to a central place, which is then accessible either via browser or SubSonic API. Additionally, it requires the ability to be backed up and accessible as ‘regular’ files. The way this was done:

• Create a local-path PersistentVolume to a specific path (e.g. /mnt/storage/music) on a specific node
• Force a PVC to use this PV (extremely easy surprisingly)
• Have a CronJob run the ‘Bandcamp Sync’ tool periodically and save into that PV
• Music server uses that PV as it’s storage
• Backup script on VM host copies the dir to backup storage

This actually worked for a little while, say 2 or 3 days. Until one day I tried to ls the dir and was met with reading directory '.': Bad message. Initially terrifying but straight forward to fix thankfully. Except the fix lost+found‘ed the entire ‘music’ dir, which is fine but I don’t want to have to run fsck every few days. For now I have scrapped the music stuff and now I need to find the culprit since there seems to be nothing online about this type of issue.

But what are the potential culprits? Let’s go from bottom up:

• The disk/server is shot (unlikely)
• The disk being written to by both Proxmox and a VM caused issues
• Treating part of the disk as both a Kubernetes PersistentVolume and a regular volume led to various inconsistencies
• K3S’s local-path StorageClass has issues with the VM disk setup

I am not great at fixing Linux filesystem issues so I don’t really know but to me any inconsistency in data is brought about by either bad concurrent writes or misunderstanding of the destination. There’s a lot of slow testing here but thankfully it’s quite easy to do: just run the ‘Bandcamp Sync’ for a few hours then ls the target dir.

There’s a pair of ’easy’ solutions in my mind but they make either backing up harder or create too much overhead for my tiny 9 y/o server. Thankfully, part of me wanted to heavily document this homelab set up this time, so once I fix this I can write up the solution (iff I remember).

Unfortunately, I still have 3 or 4 more other problems beyond this that also need to be tackled (including one really annoying physical one) but this I think is the most important as it’s one of the key features I want in my homelab.