It’s way past time to get the National Popular Vote Interstate Compact to 270.

Oh good. Missouri in the national news again. :\

Java, Postgres mostly but also LDAP and random in-house-written RESTful services, almost 20 years.

• The objects we store in the Postgres database are very “hierarchical” in nature, with one top-level object and lots of child/grandchild/great-grandchild objects. (We asked for a Mongo database but the infra team at the time said "make do with Postgres.)
• As I mentioned, some of that hierarchy is in LDAP or RESTful services, not in Postgres, so we needed something capable of dealing with multiple storage backends that would stitch the objects together as necessary. So the “ORM” needed to have backends for multiple backend systems.
• We knew clients would need a vast number of different queries. So we made a RESTful endpoint that gave the full power of the ORM to (authorized) clients. If they needed different data, we’d be like “change your query like this” and they didn’t have to wait on us.
• Early in the project, we consciously designed an extensible JSON representation of our hierarchical objects. That is what’s returned from the aforementioned RESTful endpoint.
• However, we also created a “shortcuts” system to allow us to “balance” how much of the logic lived on the server vs in the client. (It can mix and match. Like “apply this shortcut, but also filter this way and paginate” or whatever.)
• We made the API of the ORM such that it could both be used to query from the database/LDAP/RESTful systems, or be used as a client SDK for the aforementioned RESTful query endpoint that the application exposed.
• It’s both “more than an ORM” (querying from non-database sort of backends) and not fully an ORM (read only, doesn’t handle schema evolution.) But it’s fair to say it’s more “an ORM” than “not an ORM”.
• The implementation of the Postgres backend part of it is heavily inspired by Django’s ORM.

We couldn’t have pressed Hibernate into this use case. It doesn’t really deal with hierarchical data and sure as hell doesn’t know how to query from LDAP. I don’t know that anything existed at the time (nor am I sure anything exists now) that would fulfill our use case.

And the alternative to what we built was a massive, unmaintainable DAO with ridiculous numbers of individual queries in it that would have to be modified or added to endlessly every time someone needed to filter a bit differently or whatever.

This was a developed-in-house e-commerce web application at a major e-retailer. So fortunately that monstrosity of a cookie-handling mess was only ever used by one company.

You know what, though? Talking about this reminds me of another story about the same e-commerce application.

After a customer placed an order on this e-commerce site, the company’s fraud department had to evaluate the order to make sure it wasn’t fraudulently placed. (As in, with a credit card not owned or authorized for use by the purchaser.) Once that was done, the order had to be communicated to a worker at the warehouse so they could pack the right items into a box, put on a shipping label, and set the box aside to be picked up by the UPS truck which would come once a day near the end of the day.

The application used by the fraud department and the application that displayed new orders to warehouse workers was one and the same application. Whether a user had fraud-evaluating powers or pack-items-in-boxes powers just depended on what permissions their particular user had. (That may have been decided by LDAP groups. I don’t remember for sure.)

Meanwhile, the e-commerce site offered gift cards for sale online. The gift card would be shipped to the customer. And there was a box where you could write a message associated with the gift card. So, for instance, someone could buy a gift card to be sent to their nephew’s address or whatever and include a little note like “Happy Birthday. Don’t spend it all at once.” or whatever. And the fraud/pick-and-pack application would display all details of the order including any messages associated with the gift cards.

Well, I found a stored cross-site scripting vulnerability where if you put <script>...</script> tags with some JavaScript in the gift card message box and completed the order, the JavaScript would execute any time someone viewed the details page for the order in the fraud/pick-and-pack application. And of course, the JavaScript could do within that application just about anything the user could do with their given permissions.

The main danger was that a malicious actor with sufficient knowledge of how our fraud application worked could place an order fraudulently with someone else’s credit card and include in the order a gift card with a malicious JavaScript payload in the message box, and then that malicious JavaScript could automatically mark the order “a-ok, no fraud here” when a fraud department worker loaded the order details page, letting the order be fulfilled without any actual fraud review.

The fix was pretty simple. Just stick a <c:out>...</c:out> in the appropriate place in the fraud/pick-and-pack application code. But it was an interesting example of a vulnerability in a not-customer-facing application that could none-the-less be exploited by any public customer/user without any particular special access.

If you’re interested in one more interesting story about the same e-commerce application, see this comment I made a while ago.

Never roll your own ORM

I’ve done this. Probably 10 years ago. Even today, I maintain the same application that has the ORM in it that I designed. If I could go back in time and do something else, I’d do the same thing again. Honest to god. For my use case, I feel it was warranted. It was risky, but it worked out surprisingly well.

Java webapp. Customer facing. E-commerce application, so in PCI scope and dealt with credit card info and such.

There was one specific cookie that stored some site-wide preference for the customer. (Why not just put that preference in the database associated with the user? Because that would make too much sense is why.)

But the way they encoded the data to go into the cookie? Take the data, use the Java serialization framework (which is like Python’s “Pickle” or Go’s “Gob”) to turn that into a string. But that string has binary data in it and raw binary data is kindof weird to put in a cookie, so you base64 encode the result. (The base64 encoding was the only sane step in the whole process.) Then you do the reverse when you receive the cookie back from the browser. (And no, there was no signature check or anything.)

The thing about the Java serialization framework, though is that decoding back into Java objects runs arbitrary object constructors and such. As in, arbitrary code execution. And there’s no checking in the deserialization part of the Java serialization framework until your code tries to cast the object to whatever type you’re expecting. And by that point, the arbitrary code execution has already happened. In short, this left a gaping vulnerability that could easily have been used to extremely ill effect, like a payment information breach or some such.

So all a malicious user had to do to run arbitrary code on our application server was serialize something, base64 encode it, and then send it to our servers as a cookie value. (Insert nail biting here.)

When we found out that there was a severe vulnerability, I got the task of closing the hole. But the existing cookies had to continue to be honored. The boss wasn’t ok with just not honoring the old cookies and developing a new cookie format that didn’t involve the Java serialization framework.

So I went and learned enough about the internal workings of how the Java serialization framework turned a Java value into a binary blob to write custom code that worked for only the subset of the Java serialization format that we absolutely needed for this use case and no more. And my custom code did not allow for arbitrary code execution. It was weird and gross and I made sure to leave a great big comment talking about why we’d do such a thing. But it closed the vulnerability while still honoring all the existing cookies, making it so that customers didn’t lose the preference they’d set. I was proud of it, even though it was weird and gross.

The value that was serialized to put into the cookie? A single Java int. Not a big POJO of any sort. Just a single solitary integer. They could just as well have “serialized” it using base-10 rather than using the Java serialization framework plus base64.

They’re openly disdainful of all things Open Source and OSI, and their software is not Open Source.

They’re a fully for-profit company using consumer rights kind of rhetoric to manipulate people in service to FUTO’s own pocketbook. Don’t fall for their use of the .org tld or promises to “never abuse customers”.

I’ve written many times about what assholes FUTO is. I’m fully convinced they’re way more part of the problem than part of the solution and they’re not to be trusted.

They’ve soured me on Rossman. Back when he was doing coverage of things like the court cases around right to repair and tractors and stuff, I watched a few videos of his and thought he was awesome. And then he got involved with FUTO and started billing Grayjay as “Open Source” and I’ve gotten sufficiently disillusioned with FUTO that I haven’t followed Rossman at all. I’ve got coworkers changing their avatars to Clippy and shit, and I believe their hearts are in the right place, but I can’t in good conscience really get behind anything or anyone associated with FUTO.

Edit: Oh. I honesty didn’t even see this OP was a link to an article. Lol. Now I’m very interested to go read that article.

Edit 2: Shite. I’ve now read the article. I knew FUTO was assholes in a lot of different ways, but now I know how deeply, completely irredeemable they really are.

If you get caught, your professional life is over.

That seems hyperbolic. Maybe your workplace is super draconian and will immediately fire you in such a case. But different employers have different cultures. Where I work, there are running jokes among the employees about how hard it is to get fired. One of the few cases of a firing we know of involved someone who was so passed-out drunk at his desk that he couldn’t be awoken. And that was after he was given multiple stern talkings to.

I’ve seen people play WOW and Counter Strike on their office computers in the office in very visible areas.

Lest you think “yeah, but no place where it’s that hard to get fired is going to have a locked down firewall” this is the same place where I had to make a special request to have http://portswigger.net/ , the official site of Burp Suite Pro, the web application security tool, unblocked so I could evaluate it’s suitability to replace the tool we were using previously. (From what I’ve seen, Burp Suite Pro is kindof the de facto tool for web app security among pen testers, or at least was at the time.) The reason given on the “this site is blocked” page the corporate proxy gave was because it had something to do with alcohol.

In my time here, I’ve gone to lengths to curcumvent corporate firewalls multiple times. Both for personal aims and because it was necessary to do my job. I’ve never once been repremanded for it.

OP knows their workplace. OP, be smart, but do if you can get away with it, go for it.

Funny you should mention the company network.

To tell the next part of my story, when I did all of what I described, I first backed up the KDE neon install onto a tiny little partiton. So I still had it to go back to if I needed to.

And after I’d been using Arch for a good while, the VPN folks decided to retire OpenVPN and switch to something called “GlobalProtect”.

They run BMC, a remote machine management program, on all freshly-imaged machines. That lets them (un)install shit without the user’s knowledge and stuff. Windows users had lots of horror stories about “the great Java uninstall of 2018” where the PC Support folks just randomly decided one day to uninstall OpenJDK from every Windows user’s machine. While we were trying to write/maintain Java software written in-house. (This happened multiple times within a few years.)

One of the biggest benefits to running Linux (even if it was KDE Neon) was that the PC Support folks were scared of Linux and stayed very hands-off. They never (un)installed stuff remotely for KDE Neon users.

…until they switched to GlobalProtect. They wouldn’t give out the .deb for GlobalProtect to let folks install it themselves. They’d only install it for you via BMC.

But since I was running Arch and had never installed BMC, (actually I have another story about BMC on Arch, but I’ll save it for when I have more time), my machine was passed over when they installed GlobalProtect on all the KDE Neon machines.

So I rebooted into KDE Neon, asked pretty please that they install GlobalProtect, and have been using KDE Neon ever since.

Now, I’ve done nothing to disable the TPM or anything on Arch. I don’t think even if GlobalProtect uses the TPM that there’s any reason it couldn’t do so while on Arch. But I tried just copying the install from KDE Neon to Arch file-for-file and running it. It didn’t work. I had to strace it to get more info and… don’t remember what the error was about now. Some inter-process communication thing I had never heard of before wasn’t able to talk to the daemon process.

I keep telling myself I’m going to get GlobalProtect running on Arch again so I don’t have to keep using KDE Neon, but it’s been a while since I’ve worked on that any.

Also, one of my coworkers had been working for years by connecting to the company VPN from a personal machine. And I told him he needed to figure out his VPN situation months before they actually turned off OpenVPN. But he didn’t heed my warnings and when they shut off OpenVPN, he was screwed. He took the Mac they’d sent him when he was first hired off of mothballs and tried to get it running. They ended up just telling him they needed to send him a new machine. So he basically couldn’t work for almost two weeks while he waited for the new KDE Neon machine he ordered to get set up/imaged/etc and then shipped halfway across the country. He uses KDE Neon on a company laptop now.

There are some great stories about how we’ve messed with PC Support at this company. Lol.

Edit: Ok. I’ll tell the BMC-on-Arch story now.

Same company. Back before they were issuing secureboot’d machines, and before they offered the option of a Linux machine (or without special manager approval, a Mac, actually), I installed Arch on my host on a forgiveness-rather-than-permission basis.

When they started supporting Linux, they got BMC set up for Linux. (It had worked on Windows prior, of course.) And then they started sending me nagging emails about installing BMC. They knew my boss would back me up if they pressed me to switch back to Windows, so they didn’t push for that. But they wanted me to install BMC just to get the feature that it periodically phoned home to let PC Support know it was still in use and all that. (I think it also offered features like if I ever reported it stolen, they set it up so it would wipe its own hard drive next time it phoned home. To protect any trade secrets.)

I kindof ignored them for a while. Eventually they visited my desk in person. (This was before I was working remotely.) I was like “yeah, ok, tell me what to do” (I figured it was a good compromise that would let me keep Arch) and they were like “we’ll send you the installer.”

Now, the Linux distro they supported at the time wasn’t KDE Neon. It was Ubuntu. And I was on Arch. And I asked “the installer was probably was packaged for Ubuntu, right? BMC is supposed to run as a daemon and Arch doesn’t even use the same init system. I’d be surprised if it worked.” And one of the PC support guys looked me right in the eye and passed his hand over his head in a “you’re talking over my head” gesture. And then walks away.

I received the installer. Tried to run it. It immediately choked for exactly the reason I suspected. Basically it looked at my system, didn’t find the init system it expected, and aborted before extracting the files to be installed.

So, was I going to give up and switch to Ubuntu? No! I wasn’t daunted.

So I broke out strace and gdb and managed to trick the installer into extracting the files. (Basically when it checked for the init system, I altered a variable from false to true to make it not abort before extracting.)

And then I just had to stick it at the right place on the filesystem. I never made a service file for it. I just manually ran it every now and then. And killed it a little while later. No one nagged me again.

Now, I wasn’t the only one who ran Arch. I had a coworker there who also ran Arch and somehow he was never nagged to install BMC. Not sure why. But when I left the company, I left all my work with this other coworker in case he ever needed it.

And then I returned to this company. It was after that that I did the Archbunkenstein thing because they’d started using machines that enforced secureboot. The coworker who was still running Arch when I returned had lost my BMC installer reverse engineering work. And still had never been nagged by PC Support. I expected to be nagged again, but I ran Archbunkenstein for a good year or so without anyone nagging me. When I switched back to KDE Neon for the VPN, it had BMC installed, so I’ve been using BMC ever since.

This is a great question!

It’s hard to really wrap your head around it without doing a ton of low-level taking things apart and putting them together differently.

But to answer, it’s pretty impressive the extent to which a full Linux install of any distro tends to just be like a bunch of legos put together in one particular way.

Theoretically, there’s no reason why you couldn’t ship-of-Thesius one distro into another. You’d have to have a good idea of what the differences between the two are, but it can certainly be done.

There’s a thing called a “chroot.” It’s basically a whole OS installed in a subdirectory on another whole OS. And there’s a command (also called “chroot”) that can be used to tell the parent OS to “give me a shell in the chroot OS – as in run the /path/to/chroot/bin/bash (or whatever) executable in ‘The Matrix’ such that that process thinks that the chroot is the root OS.”) That lets you do some pretty cool stuff like building an OS to be installed on another box. But when you run in the chroot, it doesn’t load the guest OS’s kernel or (typically) init system or anything. The processes run on the host system’s kernel.

And it’s entirely possible to have the guest chroot system be a whole different distro than the host. (Though some distros will have tools that make it easier to chroot into a guest chroot of the same distro.) Which implies that you can just kindof substitute one distro’s kernel for another distro’s, right?

Turns out the answer to that question is “at least mostly yes.” Quick funny personal story. I started working somewhere recently where they allowed new hires a choice between Windows, Mac, or Linux on their work laptop. I chose Linux, but didn’t like the distro they pre-installed on it. (KDE Neon. I preferred Arch. Insert hate here.) But the laptop had secure boot enabled and the PC support department wasn’t willing to let me disable that. The laptop would only allow certain kernels to boot. Windows and some kernels from some unknown set of Linux distros.

Just as a quick aside, the way it knew how to deny a specific kernel from running or allow another to run was with signatures. Canonical which makes Ubuntu includes cryptographic signatures in the kernel file identifying that kernel image as made and certified by Canonical. (Microsoft does roughly the same thing for Windows kernels.) The secure boot system on the laptop has a list of trusted certificates. If the kernel that the bootloader (which is also signed, by the way) asks the secure boot system to boot is signed by one of those certificates, it boots. If not, secure boot denies the request. Theoretically more certificates can probably be configured/trusted, but that wasn’t an option in my case.

But I still wanted to run Arch! Now, KDE Neon uses the Ubuntu kernel, so I knew that was one I could boot without access to the secure boot config. So I grabbed the .deb for the Ubuntu kernel, wrote a script to convert the .deb for the Ubuntu kernel into an Arch package. (Arch doesn’t use .debs or .rpms. It uses “pacman packages”.) I installed that arch package, configured the bootloader to point to the arch install including that Ubuntu kernel, and booted it. Viola! Arch (mostly) without secure boot access!

What I was running was really kindof 95% Arch and 5% Ubuntu kernel. Kindof a Frankenstein’s monster of OS’s. But it worked perfectly.

And theoretically, just about any part of a distro can be replaced with the equivalent from another distro. (Or from the upstream/source version.) You could technically take a Fedora system and replace the package manager with apt (I’m guessing there isn’t an rpm package that would install apt on your Fedora, so you might have to make it yourself or just build it from source and install it manually) pointed at Ubuntu repositories and transform Fedora piece-by-piece into Ubuntu. It’d be a pretty wild and messy process. And it would probably be easier to just reformat and install Ubuntu. But it could be done.

Similarly, you could replace the init system. Artix is a fork of Arch that gives a choice of init systems whereas Arch only supports Systemd. And it’s kindof another Frankenstein’s monster of an OS because it still relies heavily on the Arch repos. But it works.