Skip to content
Episode from the podcast The Databricks Data Engineer
Why senior Databricks engineers write less code than mid-level ones artwork

Episode

Why senior Databricks engineers write less code than mid-level ones

Released Tuesday, June 2nd 2026 11 min

Show notes

About this episode

Two engineers, same team, both five years in. Last quarter Mark shipped forty-seven pull requests across three pipelines. Sam shipped nine. On any dashboard, Mark wins by a mile. Sam got the staff offer. Mark got a kind note about continuing to demonstrate impact. This isn't politics, and it isn't luck. It's a pattern that specifically catches the engineers who are best at shipping, because the most valuable work a senior Databricks data engineer does is invisible by construction. You can't put a ticket number on a problem that never happened. In this episode: - Why the exact behavior that makes you great at mid-level is the behavior that keeps you stuck there - The three categories of senior work that produce zero lines of code but move the entire platform - How to tell leveraged work apart from work that just feels safe to ship - Why "less code" is a symptom and not a goal, and the failure mode of engineers who get that backwards - The one question to ask before your hands hit the keyboard that changes what you volunteer for next sprint This episode is for Databricks data engineers who ship more than anyone on the team and quietly wonder why it isn't landing at review time. Whether you're a mid-level engineer optimizing the wrong line on the chart, or a senior tired of watching your highest-leverage work go uncounted, you'll walk away with language to make prevented work legible and a lens for spending your hours where they actually compound. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

More from The Databricks Data Engineer

All episodes

Leave or stay? The 4 questions Databricks engineers keep asking (and the 10-minute test that answers them)

11 min

It's late. A recruiter message sits unanswered, and the question you keep circling is short: I'm bored. Is bored a good enough reason to leave? It almost never travels alone. Four questions about leaving keep coming up from Databricks data engineers, and only the wording changes. Is boredom enough? Is it too late to stay for the migration? What is a good stack worth against twenty thousand more? Am I running toward something, or just away? They sound like questions about salary, stack and timing. They're one question, and from the inside its two answers feel identical. In this episode: - Why boredom says more about the next six months than the last two years, and where to look before you act on it - How joining a Databricks migration late becomes either a strong senior interview story or an answer you can't give in a final round - What a good stack is actually worth against a bigger offer, and why the worse platform might have the higher ceiling - The question to ask in every interview that shows how high a company lets anyone go - A ten-minute test built on the decisions your company made in the last six months This episode is for Databricks data engineers with a background thread about whether to move. Whether you're bored in a stable seat or weighing an offer on a different stack, you'll walk away able to name which situation you're actually in, and what to do about it this week. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Medallion architecture vs. reality - when your Databricks bronze, silver and gold are just folders

12 min

Three schemas. Bronze, silver, gold. The diagram got nods in the architecture review. Then two dashboards land in the same leadership meeting with two different active customer counts, and the head of finance asks which one is right. The trail ends at bronze, where rows that failed parsing were quietly dropped. The vendor keeps thirty days. Every label was there. What the medallion was invented to fix was not, and the pipeline had rebuilt the old mess across three schemas. In this episode: - What pipelines looked like before bronze, silver and gold, and the three pains the diagram was drawn to fix - Why dropping bad rows at bronze feels clean and quietly decides how far back you can rebuild - When skipping a layer is the right call, and why seven layers is not a safer bet than three - The question to ask about every gold table before your next architecture review This episode is for Databricks data engineers who build or inherit a medallion architecture and have wondered what silver is actually for. Whether you're designing a new lakehouse or maintaining one that looks textbook, you'll walk away with a way to check that each layer does a job no other layer does, and to defend it with more than "it's best practice." --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

How the always-sure Databricks engineer caps their own promotion (and never sees it)

10 min

Mark's manager asks the room one question: if we move the biggest fact table onto Liquid Clustering, will it fix the slow dashboards? Mark answers first: yes, problem solved. Sarah says what she'd bet on, what she doesn't know yet, and when she'll know. Sarah claimed less. The decision goes to Sarah. Nobody tells Mark why. The instant-answer reflex that got him to mid-level is the same one that quietly keeps him there, and there's a name for what he's paying. In this episode: - Why a fast, confident answer is right early in a Databricks career, and when it stops paying - What the room needs from you when nobody knows for sure - The difference between being decisive and sounding certain - One sentence, two parts and a date, to carry into your next design review This episode is for Databricks data engineers who answer fast and still watch the risky migration and the design review invite go to someone else. You'll walk away with a way to make your "I'm sure" mean something again. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Two Databricks jobs, one Delta table, same moment: 3 endings (and the 1 that needs a human)

11 min

Two in the morning. A nightly merge is folding yesterday's orders into a Delta table. A backfill someone kicked off before logging off is correcting March. Neither job knows the other exists, and nothing in Delta would tell them. A lot of design around this runs on superstition. Either "it's Delta, it's ACID, relax." Or "never let two jobs touch that table." Both are guesses, and the tidy middle version, same data collides, different data is fine, falls apart the first time the error shows up. In this episode: - What Delta actually does when the second job tries to commit to a table that moved under it - Why two jobs writing completely different months can still collide, and what that error message is really telling you - Why scheduling jobs hours apart is a lock you built by hand, and the cheaper fix - When a retry quietly does damage instead of fixing things This episode is for Databricks data engineers who own a Delta table with more than one writer: a nightly merge, a backfill, a plain append. Whether you're designing a new pipeline or defending one in a design review, you'll walk away with a way to say which endings each writer can produce, instead of hoping. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Unity Catalog Can't Manufacture Trust: Your Most-Read Databricks Table Might Be the Least Trusted

11 min

The rollout is done. Every production table migrated to Unity Catalog, every one owned by a team group, and the revenue table carries a certified check mark. The platform lead posts in Slack: fully governed. Monday morning, finance replies in the thread. The revenue on the finance dashboard and the revenue in the sales deck are about three percent apart. Which one's real? The catalog has something to show for every field: an owner, a check mark, a health indicator that says healthy, a beautiful lineage graph. None of them answers that question. Unity Catalog might be the best thing your platform team ever shipped, and it still can't make one person believe a number. The missing piece isn't a feature. It's the habit of treating every trust complaint like one. In this episode: - Why an owner, a certified check mark and a healthy indicator can all look right while the number is still wrong - Why the most-read table in your catalog might be the least trusted one - What a metric view settles when two teams define revenue differently, and what it leaves to humans - The one question to ask before you touch the platform for any governance request This episode is for Databricks data engineers who work in Unity Catalog and keep getting governance requests another tag won't fix. Whether you just finished the migration or you're the one answering "which number is real?" in Slack, you'll walk away with a way to sort every request before anyone opens a ticket. --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

This podcast, its content and its artwork are not owned by, affiliated with, or endorsed by Castquire.

Up next

  1. Nothing queued. Use + Queue on any episode to line it up.

0:00 0:00