LOL Not really, but boy it has been a day. Started at 7:00 am and I finally resolved (?) the issue. In fact I’ve got through every last bit of my network, and at this point in the evening, I actually don’t have a solid reason why the issue was present. Something in my VPN settings glitched, or something got triggered on pFsense and got hung up…something, something with Tailscale. It wasn’t CLoudflare this time. LOL
You ever do so much to a problem that when you ‘fix’ it, you have no real idea what the fix truly was? You ever have a problem and find all the shit you cobbled together in the name of ‘just get it running and back online’? I did, and decided that I would fix that shit too. It took all flippin’ day.
You guys that do this for a living…I salute you! jebus crispies!
ETA: 8 bells and all’s well today.
“So what was the problem in the end?”
“Man, I don’t fucking know.”
- me, every goddamn time
It’s always DNS
Touch it until it works, then never again while it still does.
Sometimes the fix has been done but the effect takes a while. For a cache to age out or a change to propogate. It all depends on what you are working with/on. Or you made a change but forgot to restart a specific service.
Meanwhile even though you did a fix correctly and aren’t aware of it, since it doesn’t seem to work you change something else and break it again inadvertently.
Man, I know that feeling. One thing that helped me better deal with issues like this, was to have a changelog. Basically I write down what a setting was, what I changed it to and a reason. If something goes wrong, I can at least undo what changes I’ve made and see if it helps. It’s not perfect, but it might shave some hours off a RCA.
This is what led me to set up SnipeIT and keep a log on every change on each device. It gets wild as the devices and services increase.
SnipeIT
I’ve seen the app rolling around in Awesome Lists, but always thought it was for larger operations with a lot of infrastructure and not necessarily a home lab. I might look into it seriously.
I have it for my home and office. Nice piece of software honestly.
I’ll check it out. I do keep prolific notes on everything I do, however they need to be more structured and searchable. Right now I’m using Trillium but it has it’s limitations for what I want it to do. Nice piece of software tho.
I tried trillium, but turns out I can’t organize anything to save my life, so now I run Obsidian for notes and snipe for all my networks and devices.
I tried Obsidian, and it is a very capable piece of software. However, I wanted it to run in a docker container. Running in a container, it uses KasmVNC. So, every doccument you wish to save in Obsidian, you first have to paste into KasmVNC’s clipboard, then paste it into the doccument. I found that quite clumsy.
If I’m doing it wrong, please share. Obsidian was my first choice.
Oh, I just install Obsidian in all my devices (laptop, PC, phone and tablet) and then sync them all via Syncthing. No docker, no server, just sync the Obsidian folder on each device.
I hate, hate, hate when I fix something and I don’t know why the fix worked (or what the fix even was…). I want my suffering to result in something learned so it doesn’t happen again.
My mind went to this one
OK, here’s a somewhat famous case of email that could only be sent within something over 500 miles, but no further: https://web.mit.edu/jemorris/humor/500-miles
When you do it for work, you log what you have changed each time you make a change to try to fix it, and you log what you revert, so you can keep track of what you have tried, what worked, and what didn’t and have a clearer idea of what the solution was.
Sometimes it really does take a while to nail down though, and sometimes it isn’t entirely clear why what worked worked. Especially if you’re a junior network engineer without as much experience.
I made a self-hosted forgejo repository of /etc. Commit messages aren’t always informative, and I’ve never actually gone back to the repository to figure something out, but it’s there, just in case. Me cosplaying a sysadmin.
