jesseryoung

Every commit in Delta is time travel-able. Defaults in Delta clean up logs in 30 days (and data in 7 days but requires a VACUUM be issued.) You can set those retention periods to any arbitrary time period, but it's typically advised against due to the cost to store all the historical data.

Welcome to Databricks!

I've been an SA at Databricks for the past two years and love it here. The people you get to work with here are world-class and our customers legitimately love our product.

I too am a little confused about comments in threads on HN about Databricks, they seriously don't reflect what I see internally and what my customers say. I don't think I'd be working here if they did.

Thank you for FullPageOS. I was thinking of building something for a project I had. This will save me tons of time!

Programming languages will always be better, more feature rich and more capable than any DSL. However, the whole reason things like terraform or puppet become popular is because the people who write the DSL don't consider themselves "programmers" and they don't want to be thought of as "programmers".

As somebody who identifies as a "programmer" and often has to manage infrastructure I would absolutely love for a serious IaS library to pop up in my favorite language. I would switch to it immediately. However unless we start hammering home that if you are regularly writing in these DSLs that you are actually a programmer and there's nothing wrong with that people will still flock to these tools.

I tried to organize a corporate hackathon at one point in my career. My initial idea was to make it freeform. However after talking to people who wanted to participate, I found that nobody really had any ideas and just wanted to join and hack on somebody else's idea.

My (possibly incorrect) take away - people with great ideas will build them themselves without a hackathon.

I still love the idea of hackathons, I just don't know the right way to organize them.

I started in the software engineering space and move into data engineering, and I was floored with the complete lack of tooling. There is a HUGE gap between software engineering and data engineering when it comes to both tooling and practice. Even the simplest "unit" test of "Is the SQL statement valid" is not all that common in frameworks and tooling but in practice is like 90% of the production failures that I've seen.

Starting with a framework that is programming language first (IE Spark) can help you build your own tooling to help you actually build unit tests. It's frustrating though, that this isn't just common across other ETL tooling.

Ran into a similar issue last year in the East US region. We contracted support and they gave a similar response. From my understanding talking to people who use AWS and GCP this isn't uncommon across cloud platforms.

While we could've just swapped a deployment parameter to deploy to another region, we opted to just use a different SKU of VMs for a short period and switch back to the VMs when they were available again.

We haven't seen issues since.

I'm with you, I'm a dev who prefers Windows over Mac. I'm convinced that most Mac users don't use multiple monitors and they're completely fine with the really poor window management. It boggles my mind; I am objectively less productive with a single monitor than I am with multiple.

Ah - that makes sense. This might be a bit easier in healthcare as I believe it's pretty common to have many different ops teams each responsible for different parts of the business.

I feel like most of the time "compliance" is blamed when really, it's your first point in that section (Absent an existential threat, the necessary organizational changes were more difficult to make) that is the real holdup.

Maybe I should rephrase that: Is it impossible for a company that defines Dev+Ops as a single responsibility to be compliant?

What exactly is a "Conflicting duty"? What's stopping a company from stating that developing, deploying and supporting software is a single duty?

The author calls out a few reasons why DevOps fails for organizations all of which I agree with - however the one that I've never completely understood: Regulatory reasons for keeping Ops centralized.

I work in healthcare which I guess should fall under this rule - but in practice I haven't really seen that impeding DevOps. Teams that have the capabilities to build the full stack get handed a subscription to a cloud provider and they go off and do so. They still fill out and track change logs, audit changes and seek approvals - but after that's done, it's still the team who presses "the button".

Anybody in a regulated industry where you've hit hard walls that prevent you and your team from going full on DevOps? If so, what rules were quoted that stopped you.

I was getting ready to disagree with you - but then I tried to think of any time I've actually pushed code to production with the "DELETE" keyword in it. The problems that I've had to solve in my career very rarely call for deleting something.

"Soft deletion" and "audit trail" are technical terms we developers come up for solutions the business wants but maybe hasn't asked for yet. It's not really a soft deletion it's a "deactivate" or "hide". Likewise, it's not an audit trail it's a "history" or "undo". Most of the time your stakeholders and users actually want these features, but don't ask because they perceive this as more expensive to build then just a "delete" button.

I've worked in healthcare IT for my entire professional career - It's A LOT more complicated than most people think. For the last 5 years I've focused on the data side of healthcare and I think that deep learning is 100% possible - it's just not achievable by a single person and it's likely VERY expensive. There are so many facets to healthcare data that's it's just impossible for a single individual to achieve something meaningful by themselves without the help of teams of doctors, data analysts, data engineers and data scientists. Just dealing with data quality issues (such as the ones called out in this essay) require a team of people to determine if metrics you are trying to measure are legit or not.

On billing: I'm convinced that the primary reason why healthcare (at least in the US) is so complex - is because of the dichotomy of saving people at all costs, while doing so fiscally responsibly. It is fairly common for large healthcare organizations to have ACTING doctors in their c-suite, who's primary goal is not to make money - it's to save lives. The people who care about saving money, reducing cost and increasing efficiency have no control over the organization. I'm not saying this is a bad thing, but IMO it's the largest contributing factor as to why healthcare billing is so complex, and healthcare costs get as high as they do (at least in the US).

I doubt that Microsoft was looking to get into the "advanced text editor" business. Their biggest money maker is Azure, they made a text editor that promotes Azure.