So this xyz company had a guy who built the entire data infrastructure on his own but with zero documentation, no version control, and he named tables like temp_2020, final_v3, and new_final_latest.

Pipelines? All manually scheduled cron jobs spread across 3 different servers. Some scripts run in Python 2, some in Bash, some in SQL procedures. Nobody knows why.

He eventually left the company… and now they hired my friend to take over.

On his first week:

He found a random ETL job that pulls data from an API… but the API was deprecated 3 years ago and somehow the job still runs.

Half the queries are 300+ lines of nested joins, with zero comments.

Data quality checks? Non-existent. The check is basically “if it fails, restart it and pray.”

Every time he fixes one DAG, two more fail somewhere else.

Now he spends his days staring at broken pipelines, trying to reverse-engineer this black box of a system. Lol

Nothing says “data engineer” like celebrating a 0.0000001% improvement in data quality as if you just cured cancer. Lol. What’s your most dramatic small win?

6 years ago I was homeless. I landed this internship as a data engineer and today by my bosses boss was told I am the best intern they have ever had! I don't know how to take it they are extending my internship till I graduate and Hopfully I'll get a full time offer!

We are hiring for a data engineering position, and I am responsible for the technical portion of the screening process.

It’s pretty basic verbal stuff, explain the different sql joins, explain CTEs, explain Python function vs generator, followed by some very easy functional programming in python and some spark.

Anyway — back to my story.

I hop onto the meeting and introduce myself and ask some warm up questions about their background, etc. Immediately I notice this person’s head moves a LOT when they talk. And it moves in this… odd kind of way… and it does the same kind of movement over and over again. Odd, but I keep going. At one point this… agent…. Talks for about 2 min straight without taking a single breath or even sounding short of breath, which was incredibly jarring.

Then we get into the actual technical exercise. I ask them to find a small bug in some python code that is just making a very simple API call. It’s a small syntax error, very basic, easy to miss but running the script and reading the error message spells it out for you. This agent starts explaining that the defect is due to a failure to authenticate with this api endpoint, which is not true at all. But the agent starts going into GREAT detail on how rest authentication works using oAuth tokens (which it wasn’t even using), and how that is the issue. Without even trying to run it.

So I ask “interesting can you walk me through the code and explain how you identified that as the issue?” And it just repeats everything it just said a minute ago. I ask it again to try and explain the code to me and to fix the code. It starts saying the same thing a third time, then it drops entirely from the call.

So I spent about 30 minutes today talking to someone’s scammer AI agent who somehow got their way past the basic HR screening.

This is the world we are living in.

This is not an advertisement for a position, please don’t ask me about the position, the intent of this post is just to share this experience with other professionals and raise some awareness to be careful with these interviews. If you contact me about this position, I promise I will just delete the message. Sorry.

I very much wish I could have interviewed a real person instead of wasting 30 minutes of my time 😔

Our company is bmp-ing up against some big Databricks costs and we are looking for alternatives. One interesting idea we’ve been floating is moving all of our data operations to MS Paint. I know this seems surprising but hear me out.

1. Simplicity: Databricks is incredibly complex but Paints interface is much simpler. Instead of complicated sql and spark our team can just open paint and start drawing our data. This makes training employees much simpler.

2. Customization: Databricks dashboards are super limited. With Paint the possibilities are endless. Need a bar chart with 14 bars, bright colors and some squiggly lines? Done. Our reports are infinitely customizable and when we need to share results we just email bmp files back and forth.

3. Security: with Databricks we had to worry about access control and mfa enablement. But in paint who could possibly steal our data when it’s literally a picture. Who would dig through thousands of bmps to figure out what our revenue numbers are? Pixelating the images could add an extra layer of security.

4. Scalability: Paint can literally scale to any size you want. If you want more data just draw on a bigger canvas. If a file gets too big we just make another.

5. AI: Microsoft announced GPT integration at Paintcon-24. The possibilities here are endless and just about anything is better than Dolly and DBRX.

Has anyone else considered a move like this? Any tips or case studies are appreciated.

Over the past year I have been working primarily in the Palantir Foundry system. My CEO unilaterally decided to pursue an enterprise agreement after being sold the AI dream. Palantir sales engineers did ‘analyses’ which suggested that the multimillion dollar price tag would result in 10x savings due to process and decision optimization. Our IT team cautioned no, but were steamrolled.

The project I am on was estimated to take four months and require 5 additional contract engineers. 2 of those were directly employed by Palantir as part of an additional contract (read: more $$) and the other 3 were a separate agency because Palantir said they don’t do XYZ work (again, more $$$). It took the externals plus me 15 months to deliver an MVP. This is primarily because we were building a complex enterprise grade app (which we previously subscribed to) on a low code platform. The Palantir engineers left as soon as MVP was deemed complete with just 30 days notice and since then I have taken on their SOW.

The work completed by the “brilliant” Palantir FDEs has been consistently failing. I’m finding they hardcoded dates. They hardcoded accounts. They used different inputs for the same business concepts. They ‘fixed’ issues that were earlier pointed out with hardcoded logic. They used AI FDE to code spaghetti mess logic. This has been a freaking nightmare.

My company had received 0 ROI to date and the CEO blames our IT team for the lack of delivery.

If you have the ability to run away from working with this god awful company and their charlatans RUN.

Had a new hire “hit the ground running” and shipped a significant number of PRs, spending well over $20k on tokens in their 3rd week on the job…

And this was celebrated by leadership as productivity.

Am I going insane to think in what world does an onboarding dev have judgement and context to decision that much code generation. Particularly on top of some slop generated codebases that even I have a hard time keeping up with.

End rant

So happy I don't work on reports anymore

Then today our SLT told us that we should replace our BI tools with Claude lol

• Ebooks next week according to Kleppmann at https://bsky.app/profile/martin.kleppmann.com/post/3mf4wvtjg7s25

• Available at online O'Reilly https://www.oreilly.com/library/view/designing-data-intensive-applications/9781098119058/

• Print 3-4 weeks.

One of the best books (IMO) on data just got its update. The writing style and insight of edition 1 is outstanding, incl. the wonderful illustrations.

Grab it if you want a technical book that is different from typical cookbook references. I'm looking forward. Curious to see what has changed.