Reading earlier comments in this community made me consider documenting the workings of my homelab to some extent, ie. docker configuration, credentials, ports and links of my services. I’ve tried to make it consistent and organised but it still feels half baked and insufficient. Everyone suggests documenting everything you do in your homelab but don’t state how. Since I’ve hardly had experience running my own server, I would really appreciate observing the blueprint of some other fellow selfhoster for copying or taking inspiration from rather than considering documentation to be ‘left as an exercise for the reader’.
Edit: I already have a note-taking solution with me. What I wish to ask is to know what needs to be documented and what the structure of the documentation should be to accommodate the information.
You must log in or # to comment.
Everyone will have their own system.
I save all my credentials in Bitwarden/Vaultwarden and take notes in Joplin.
The good thing about YOUR homelab is that YOU’RE taking notes solely for YOURSELF and only YOU know how YOU work and how YOU organize YOUR thoughts.
I save all my credentials in Bitwarden/Vaultwarden
Yeah, I don’t put key phrases, passwords, etc in my notes.
I created something similar to this. It got a lot of love during interviews later down the line. https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fi.redd.it%2Fvmd34mabi4r91.jpg&f=1&ipt=2dde77fd04d48156bc514ad4b1f090c8473f4e666ead0e16906eeed55a79aca6
What tool did you use to create the diagram?
Well I guess I was not clear enough, this one is not mine. Lots of identifying info in mine. I go into a bit more details and made mine a but cleaner and easy to read.
For mine, I used Draw.io not amazing, but did what I needed it to and have ot self-hosted, so it is easy to edit.
(Bookmarked for when I have the mental capacity to …)
Do y’all also document backup/restore procedures?
How often do you test it?Document everything as if it were a step by step tutorial you will give to someone so that they can duplicate your deployment without any prior knowledge. I’ll even include urls to sites I consulted with to achieve production deployment.
ETA: I absolutely care nothing about points. Up voting and down voting used to be a way to weed out bad info. So it always leaves me wondering 'Did I give erroneous advice? What was the reason for the down vote? I mean, if you down voted and said ‘I down voted you because I hate your guts’, I can deal with that.
This is what I like about git ops and infra/config as Code personally.
Ideally everything is an a tofu/ansible/helm chart and git lab pipeline/Fleet job. I add comments for anything that I had to learn to make work to those files. Follow good commit hygenine (most of the time). And bam I can almost a year later half asleep stumble back into a thing I did.
Do you use this for physical machines too?
Yep! Metal3 for servers with BMCs Tinkerbell for everything else.
I also have an ansible playbook that templates everything into a cloud init scripts as a boot strap server.
About 12 nodes in total now, from new servers to freebee junk laptops in it.
Interesting, so Metal3 is basically kubernetes-managed baremetal nodes?
Over the last years I’ve cobbled together a nice Ansible-driven IaC setup, which provisions Incus and Docker on various machines. It’s always the ‘first mile’ that gets me struggling with completely reproducible bare-metal machines. How do I first provision them without too much manual interference?
Ansible gets me there partly, but I would still like to have e.g. the root file system running on btrfs which I’ve found hard to accomplish with just these tools when first provisioning a new machine.
Yep! It uses open stacks Ironic under the hood, but tracks config and stack via k8s.
For OS building I’ve been moving to Elemental which builds OS images from container images and cloud init scripts into Suse Micro immutable OSs (which use btrfs for the snapshot management under the hood for updates).
I agree with the advice that says “Document your setup such that you could recreate it from your notes from scratch” but I’d take it another step further — consider that someone may have to do some work on your system when you are unable or unavailable. The kind of thing you’d keep with your will, or power of attorney. Just a suggestion.
…and to my family I bequeath my entire collection of Linux iso’s
You jest but if I left my wife my Home Assistant setup undocumented she would pee on my grave.
Ansible and Nix. Code is the document.
I’ve been in the process of migrating everything over do Nix. Love it so much.
What hole does Ansible fill for you? I haven’t looked into it in the past really, so just curious. I have a single Paoxmox node so don’t really need horizontal scaling orchestration.
I don’t use NixOS for my home server mainly because of lack of MAC (SELinux or AppArmor). I use Ansible to configure AlmaLinux from package installation to firewall to systemd services.
I use NixOS for desktop and development machines.
How do you upgrade your AlmaLinux from version X to Y? Do you install a new instance or do you upgrade it? I’m asking because I remember that RedHat recommendations is to reinstall.
I’m not real clear what exactly you need to document.
Infrastructure documentation starts with an IPAM.
A good IPAM can help you document all kinds of stuff.I use NetBox.
https://github.com/netbox-community/netbox?tab=readme-ov-file#getting-startedI’m running it as a Docker container on a Linux VM.
I just looked at their latest screenshots, and it appears they’ve done quite a bit with it since I stood up my copy.
It does even more now. I’ll have to upgrade.Yeah,Netbox is also my main solution, combined with forgejo repo.
Works very well.
Netbox is a hell of a package, of which I’ve essentially only touched the IPAM, and I don’t even use it programmatically. I just use the web console to keep track of 4 subnets and about 50 IPs.
It’s got a whole virtualization section that I haven’t touched, although that would make my device mapping more sensible. I just treat em like they are all real, and only map the physical nics on the hypervisor hosts.
I do keep text notes in Netbox entries, but that’s sort of a backup. If its something I’m likely to need to know, I’ll have a note in Proxmox. Usually login links for apps hosted there and the like. And of course I’ve got a folder full of text files with all my deepest secrets.
Netbox,especially when combinded with Plugins is so incredibly good and might,that’s it’s almost funny how good it is. What I do Plugin wise:
-
Documents: not implemented yet by me,but one could store manuals,etc. directly within netbox.
-
Lifecycle and Inventory: While it’s not as good as snipe-it (tbh, inventory is imho one of the worst plugins) it does the job for my small deployment
-
Slurp it to scan automatically
-
QR Code for obvious reasons
-
Floorplan as well
Of course that sounds overkill for a small deployment, but I simply forget too many things after a few months otherwise and it’s something my family (wife is in IT and far more qualified than me) would need if something happens to me,so a proper documentation would be essential for that as well.
Alright already! I’ll work on my upgrade.
I’m wondering if I should just build a new docker and then migrate the data instead of upgrading in place. I bet that’s the easier thing to do in the end. Sounds safer too. I got backups and all, but …
-
