Its because the contents he made are inconsistent with common conventions in data engineering.
- It is very common not to deduplicate data and instead just append rows, The current value is the most recent and all the old ones are simply historical. That way you don’t risk losing data and you have an entire history.
- whilst you could do some trickery to deduplicate the data it does create more complexity. There’s an old saying with ZFS: “Friends don’t let friends dedupe” And it’s much the same here.
- compression is usually good enough. It will catch duplicated data and deal with it in a fairly efficient way, not as efficient as deduplication but it’s probably fine and it’s definitely a lot simpler
- Claiming the government does not use SQL
- It’s possible they have rolled their own solution or they are using MongoDB Or something but this would be unlikely and wouldn’t really refute the initial claim
- I believe many other commenters noted that it probably is MySQL anyway.
Basically what he said is incoherent to anybody who has worked with larger data.
In terms of using SQL, it’s basically just a more reliable and better Excel that doesn’t come with a default GUI.
If you need to store data, It’s almost always best throw it into a SQLite database Because it keeps it structured. It’s standardised and it can be used from any programming language.
However, many people use excel because they don’t have experience with programming languages.
Get chatGpt to help you write a PyQT GUI for a SQLite database and I think you would develop a high level understanding for how the pieces fit together
zalgotext@sh.itjust.works 6 days ago
Great explanation, but I have a tiny, tiny, minor nit-pick
I’m being pedantic, but I disagree with your wording. As a backend dev, I work with relational databases a ton, and what Musk said wasn’t incomprehensible to me, it just sounded like something a first year engineer fresh out of college would say.
Again, the rest of your explanation is spot on, absolutely no notes, but I do think the distinction between “adult making up incomprehensible bullshit” and “adult cosplaying as a baby engineer who thinks he’s hot shit but doesn’t know anything beyond surface level stuff” is important.
Hawk@lemmynsfw.com 6 days ago
Fair point, I’ve edited the answer to be clearer for future readers.