ironchef

"Here’s why: There is absolutely no benefit for you to gain by talking in an exit interview, and plenty of negative consequences to come out of it. At best you’ll be remembered as a complainer, and you may make enemies."

I guess I would counter with if I have friends there, I would like their lives to be better. If my exit interview is able to do that, then I would take that as a net positive.

Folks often use things like LoRAs for that.

It's at least CUI. At _least_. I would suggest material that is operational in those regards would be secret or top secret. It had:

* timelines of kinetic resource missions * conops including order of battle, etc.

Per normal ODNI bits, it's pretty clearly within bounds for classification.

Lots of folks out there use evidence.dev. It's a simple way to get some BI up and running without needing to deal with licensing / corporate IT, etc.

Remember all it takes is 1 employee to put that claim up there (although I do like evidence.dev).

The libraries out there lack the breadth and maturity of some of the other ecosystems (as a simple example).

Or at least they did for some of my corners of the world.

> Look into datalake architectures.

Yup .. comfy with iceberg/delta/hudi

> RDBMS based data warehousing is obviously not economical at the petabyte scale.

I never said it was .. I'm simply responding to "I simply cannot understand how anyone chooses this over running your own Spark clusters with Jupyterlab". I'm trying to help you understand why folks would choose a SaaS over run your own.

> But storing all that data in S3 with Delta Lake/Iceberg format and querying with Spark changes things entirely. You only pay for object storage, and S3 read costs are trivial.

No. You don't just pay for object storage + minor S3 read costs.

You pay for operations You pay for someone setting up conventions You pay to not have to optimize data layouts for streaming writes You pay to not have to discover race conditions in s3 when running multiple spark clusters writing to same delta tables You pay to not have to discover that your partitions/clustering needs have changed based on new data or query patterns

But look .. I get it. You have chosen to optimize for cost structures in one way .. and I've chosen to optimize in a different way. In the past I've done exactly as you've said as well. I think being able to seeking to see _why_ folks may have chosen a different path may help you understand other areas to consider in operations.

Here was my situation. Occasional queries. Over a couple petabyte of data. Customer facing so response in seconds per SLA but > 95 percent of the time the warehouse isn’t running. Cached queries from within 24 hours which don’t require the warehouse to even spin up. Our snowflake costs were significantly less than an FTE.

Would that potentially be a situation which “running your own” doesn’t make sense?

“I can’t imagine being a traditional teacher, and having to teach kids that don’t care.”

What I’ve seen is igniting that spark and moving folks from not caring to caring (or realizing they can affect change) can be even more rewarding than the other mentoring being discussed

There are some huge deployments out there. Snowflake (the database) has a big dependency on it for how they do metadata (for example)

Let’s say you work for a SaaS doing analytics. Your boss says “hey! We need to start reporting on new logos. Can you snag those from the DB?”

But what counts as a new logo? Does a pro serve engagement that doesn’t use the product count? What about a business using the SaaS but still in a trial period? Etc.

A semantic layer helps provide common agrees upon definitions to the business. So any one looking for common data entities can just look those things up… and can come to published definitions (which are backed by queries to databases, data lakes, etc).

Does that help? Another example of this would be dbt for example

Yeah. The frameworks solve the 80 percent problem quite well but once you start trying to be as specific as the business often wants, I would find them lacking and end up popping JSON to d3 (or some such)

There's a lot of ... sub par BI devs who want to point and click through horrendous PowerBI interfaces and such. Personally i find that style to hamper my ability to express what the business units need out of data products.

I'm curious what you hit in dash and plotly that evidence let you overcome.

(FTR, I use all of them, lightdash, streamlit, (as well as drop down to flask) etc.)