Sorry everyone, I opened a new branch off the 2003 release while I was researching some of the bugs in the legacy code. That wasn’t supposed to get merged in…
yep, the bug triggers when a political party overflows its buffer. The funny thing is, the system still catches the error and spams the logs with FATAL error messages, but the system has been throwing them for so long and seems to still be chugging along so we kind of ignore them. One of the original devs made a comment about how someone should fix it right before leaving the team. Anyway, the effect tends to accumulate, so if you deploy it to prod and don’t fix it, eventually you get… this.
Trying, I’m stuck on a bridge with some of the seniors right now. One of them wants to “just push through and see what happens” and roll forward with the change, the other one disapproves but clearly has no idea what else to do. Nobody seems to want to even propose a plan because then they’d be on the hook for it, so we just keep going back and forth and not. ending. the call.
It’s related. All over the codebase different devs have put in catches for it, but often as not instead of handling it they just wrap it in a new warning, dump it to the log, and move on. Dwight wanted to decouple those modules, correctly, but he didn’t know how because the reason they’re coupled in the first place is because we didn’t handle this problem correctly in the initial design, so he slapped a warning on it and called it good enough. Every once in a while some hotshot dev comes in with a proposal that exploits it (and introduces tons of technical debt that we don’t see because we don’t measure it) and because it looks good for the shareholders the PR gets approved and merged. If they’re lucky then they get promoted out of development, or occasionally fired, but in either case the problem in the code gets exacerbated and never really fixed.
This just further confirms that the world simulation actually did break from the y2k bug. We fixed the bug in here, but no one fixed the simulation. The cascade failure has just been spreading out of control since then. Now we have recursion errors! The sim tried to restart but it’s all corrupt lol.
I think the root problem lies in some of the really gnarly, pre-1.0 code that we are afraid to touch because it might break something, that’s why it keeps cropping up even though we keep slapping patches pell-mell on top of it. It just happened to pop out pretty bad in 03.
This is why I’ve been arguing against rebase merges. If we aren’t super careful with older branches, we end up with messes like this.
Maybe just squash and merge in the future?
I don’t know, this whole situation could have been avoided if people would just check with the team before merging stuff. I think people see revisions from old releases so they just merge them because they “worked before” so they should work now. It doesn’t work for the current userbase, frankly it didn’t work all that well for most users back when it was released the first time. Just merging any old crap straight into main without reading it so they can brag about how many PRs they close.
As usual, policy and procedures without properly defined controls burn us.
Honestly, why did we even go through that whole re-org after the WW2 outage if we’re not actually going to adhere to our own policies?
Sorry everyone, I opened a new branch off the
2003release while I was researching some of the bugs in the legacy code. That wasn’t supposed to get merged in…Ugh, just tell me you didn’t pull
maininto your working branch again, instead of rebasing. We talked about this.But you made it worse!
yep, the bug triggers when a political party overflows its buffer. The funny thing is, the system still catches the error and spams the logs with
FATALerror messages, but the system has been throwing them for so long and seems to still be chugging along so we kind of ignore them. One of the original devs made a comment about how someone should fix it right before leaving the team. Anyway, the effect tends to accumulate, so if you deploy it to prod and don’t fix it, eventually you get… this.My bad.
Fix it Fix it Fix it Fix it!
(I tried to find the Futurama meme of that, but my phone said No)
Trying, I’m stuck on a bridge with some of the seniors right now. One of them wants to “just push through and see what happens” and roll forward with the change, the other one disapproves but clearly has no idea what else to do. Nobody seems to want to even propose a plan because then they’d be on the hook for it, so we just keep going back and forth and not. ending. the call.
wait, this is the log Dwight wrote that keeps spamming “military industrial complex”?
It’s related. All over the codebase different devs have put in catches for it, but often as not instead of handling it they just wrap it in a new warning, dump it to the log, and move on. Dwight wanted to decouple those modules, correctly, but he didn’t know how because the reason they’re coupled in the first place is because we didn’t handle this problem correctly in the initial design, so he slapped a warning on it and called it good enough. Every once in a while some hotshot dev comes in with a proposal that exploits it (and introduces tons of technical debt that we don’t see because we don’t measure it) and because it looks good for the shareholders the PR gets approved and merged. If they’re lucky then they get promoted out of development, or occasionally fired, but in either case the problem in the code gets exacerbated and never really fixed.
https://gifs.cackhanded.net/invader-zim/the-nightmare-begins/worse-or-better.gif
This just further confirms that the world simulation actually did break from the y2k bug. We fixed the bug in here, but no one fixed the simulation. The cascade failure has just been spreading out of control since then. Now we have recursion errors! The sim tried to restart but it’s all corrupt lol.
I think the root problem lies in some of the really gnarly, pre-1.0 code that we are afraid to touch because it might break something, that’s why it keeps cropping up even though we keep slapping patches pell-mell on top of it. It just happened to pop out pretty bad in 03.
This is why I’ve been arguing against rebase merges. If we aren’t super careful with older branches, we end up with messes like this. Maybe just squash and merge in the future?
I don’t know, this whole situation could have been avoided if people would just check with the team before merging stuff. I think people see revisions from old releases so they just merge them because they “worked before” so they should work now. It doesn’t work for the current userbase, frankly it didn’t work all that well for most users back when it was released the first time. Just merging any old crap straight into main without reading it so they can brag about how many PRs they close.
As usual, policy and procedures without properly defined controls burn us. Honestly, why did we even go through that whole re-org after the WW2 outage if we’re not actually going to adhere to our own policies?