• 0 Posts
  • 28 Comments
Joined 1 year ago
cake
Cake day: July 9th, 2023

help-circle



  • I agree with the sentiment regarding being woken up, but I used to look forward to being on call. I could go to bed happily, knowing I was earning a significant premium and I’d still get a good night’s sleep because the systems just didn’t go down. I had the advantage that most of the customers I supported had similar requirements, so I had their systems locked up pretty well. Minor problems (disk space. Why is it always disk space?) would self heal, catastrophic failures (hardware failures or the engineer who supposed to replace a component unplugging the wrong server) would fail over to the rest of the cluster. I never had much trouble with logging either, it was typically one of the first things set up, and I had most of the setup automated to avoid missing anything. I suppose the thing was I was supporting systems I’d built, and I’d built them to ensure I didn’t have to be woken up.

    I do a lot more troubleshooting and rescue type work nowadays, and the number of times I run into systemd components just not doing what they should is frustrating to say the least. Being able to pull the logs by knowing the service name would be nice, but a) you could already do that because you set up different services to log to different places and b) you don’t always know the service name in question. Being able to just grep the log directory is a lifesaver. You can still do that, but only because distros set systemd up to log to file as well as it’s binary format. I loathe the way systemd ends up spreading it’s unit files over about a dozen different directories, with overrides increasing that even further. I just want to know what services I’ve got and what will start up, in exactly what order, on the next reboot, dammit! The last one is particularly tricky as, due to services being started in parallel, you can’t predict exactly what order things will actually start between targets. That shouldn’t matter, units should have all their dependencies properly listed, but it’s no fun tracking down a race condition that only happens once every x reboots when a particular network service takes a few hundred milliseconds longer to come up. Give me sequential boot any day. It might take a few tens of seconds longer, but it happens the same way each time, and I only need to look in one place to know what that is.

    As to systemd’s dominance, once Redhat, where Mr Pottering worked, chose it, it became hard for other distros not to. Derivative distros obviously went with it, and if you look back through the various email discussions, it was far from a unanimous choice for distros like Debian to choose it. They did so eventually mostly, as far as I can see, because it would theoretically make packaging easier. Fortunately they still support sysvinit, so all is not lost for those of us who want a mainstream distro without systemd bloat.

    Shifting stuff to kube is definitely goot for making things more robust, so long as you’ve got the underlying clustering working, and I quite like working with it too. Once you realise it’s basically just a database and message queue with a bunch of controllers for managing storage, networking, containers and the like, and the ability to extend that, you can do all sorts of fun things with it.

    Anyway, I’ve gone on for long enough. If you’re a sysadmin and the number of trouble calls is going down, then you probably don’t hear this often enough: well done, you’re doing a great job.


  • Ok, fair point on the capital D, I must have read it like that years ago and it stuck. I shall have to make an effort to unlearn it.

    As to the rest, systemd has been a constant thorn in my side ever since L. Pottering published “Rethinking PID 1” back in 2010 or so. I found, and still find, that most of the assertions and actions in that document either don’t really hold, or just aren’t really relevant. Basically it’s trying to solve a problem that really wasn’t an issue in the real world, and does so in such a massively overbearing way that everything actually becomes more laborious than it otherwise would be. From my perspective it’s an unnecessarily complex and poorly architected attempt to answer a need that was better served in different ways. That it’s become a near mono-culture is deeply concerning.

    I’ve also run into all sorts of awkward edge cases and misfeatures over the years, from the automounter that occasionally didn’t to race conditions that only manifest at the worst moments, none of which would have occured had the basic tenet of “do one thing and do it well” been followed. The extreme verbosity of the configuration, and unnecessarily large number of places it can be spread just serve to make it even more unpleasant to deal with compared to the simplicity of init scripts, crontabs and the like.

    The sad thing is, there’s undoubtedly some good ideas buried in it, but they could all have been implemented much more lightly and in a way that worked with the rest of the ecosystem rather than fighting it. Things like starting daemons in what is essentially a repeatable sandbox, or being able to isolate logging per service. They could, and had both been implemented already, but systemd has a real “not invented here” problem, so everything was built again, with all the attendant bugs, and design issues that inevitably brings.

    Ultimately clients pay good money for me to look after their systems, systemd or not, so I probably shouldn’t grumble, but I miss the days when Linux was a clean and elegant system, without this multi-tentacled thing sitting on top of it.


  • SystemD is far too much of a poorly thought through mess to have anything like a sane GUI configuration, it doesn’t even have a sane textfile based configuration. We’re going to have to wait fir SystemD to crumble under it’s own weight and be replaced with multiple, simple, cleanly designed components before we have any hope of a sane config again. Sort of like we used to have before a certain someone/some company (depending on how conspiratorial you’re feeling) decided to come along and muck it all up.

    /rant

    Thank you for coming to my Ted Talk Rant. You may gather I dislike SystemD quite a lot.



  • That’s the thing, you wouldn’t have the power to do any of that before you were booted out. CEOs do have a lot of power over the board, and the board has power over the company. The net result is that if the CEO pushes too fast or too radically they get removed before any change occurs. As the poster above said, in situations like this the CEO is paid to be the fall guy; the people who wield the actual power are the board members and the large shareholders. The CEO deserves a chunk of the blame for being the face of the organisation and legitimizing it, but killing one, or even a few, off wont significantly change the direction these companies are headed in.




  • Sit by the bedside of a loved one as they die in agony that can only be even partially controlled by keeping them comatose. You’ll likely soon come to the conclusion that we shouldn’t be trying to just live ‘as long as we can’, but as long as we can well.

    There often comes a time when the rest of a person’s life will consist only of barely managed pain, suffering, indignity and imminent death. It should be up to the person living that to decide if it is worth it, and and up to the medical profession to deliver a peaceful end if that is what they want.

    There are plenty of issues that need to be worked through before it is possible, particularly around coercion, deliberate or accidental, and how it is delivered, but they must be worked through if we are to consider ourselves humane. When an animal we care about is suffering, with no hope of relief, we can make the choice to end their lives to alleviate the suffering, we should be able to do the same for ourselves.





  • Vim is running as you, rather than root, so you wont be able to edit other files as root, and any rogue plugins wont be able to either, which is good.

    Sudoedit has various guards around what it’ll let you edit, in particular, you can’t edit a file in a directory you already have write permission on as doing so allows the user to bypass restrictions in the sudoers setup (there’s more detail in their issue tracker. If the directory is already writable though, you don’t need sudoedit anyway.



  • Pre-ordering something would usually cause a $0.00 transaction to confirm the card details are valid. It would be a ‘pre-auth’ transaction where the merchant reserves an amount on the card for payment at a later date, when they ship the item. If a fraudster makes a pre-order they xan validate that the card details are valid, then cancel the order, usually leaving the victim none-the-wiser. In your case, the bank noticed the transaction and notified you, but that seems to be rare. Once the fraudster knows the details are valid, they can sell them on.

    It’s just a theory, and unless your bank and Blizzard work together to track the transaction, why it happened, and who instigated it, its going to be difficult to get to the bottom of it.




  • This is fundamentally true. However it is possible to limit the bandwidth of data the employee can exfiltrate.

    That’s fair, but the OP was talking about having the sensitive data directly on the laptop, which rather limits your ability to control access to it, and why I was suggesting keeping the data on a server and providing access that way, so limits can be put in place.

    Assuming a privileged employee suddenly becomes a bad actor.

    Your threat model probably isn’t the employee who suddenly goes rogue and tries to grab everything, but the one who spends and extended period of time, carefully, extracting key data. As you, the former can be be mitigated against, but the latter is very much harder to thwart.

    But I couldn’t for example download our entire customer database, I could get a specific record, I could maybe social engineer access to all the records of a specific customer, but there is no way I’d be able to extract all of our customers via an analog loophole or any standard way. The data set is too big.

    That’s well set up, but, whilst your competitor would love the whole database, what they’re really interested in is the contact details and contract information for maybe your largest three customers. When the dataset to extract is small enough, even quite advanced rate limiting can’t really help much. Making sure no one person has access to all of the most valuable data is a good start, but can present practical problems.

    And this is what you are trying to limit. If you trust your employees (some you have to), you can’t stop them from copying the keys to the kingdom, but you can limit the damage that they can do, and also ensure they can’t copy ALL the crown jewels.

    I think we’re basically saying the same thing. The OP talked about putting all the sensitive information on the employee’s laptop, and that’s what I was trying to steer them away from.

    In the past I’ve been asked if we can provide our developers access to pull the full source tree, but not copy it anywhere, and, to quote Charles Babbage, “I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.”

    I enjoy the security side of sysadmin work, but I find it rather depressing, as all you can hope to do is lose slowly enough that it’s not worth attacking you.