runT1ME

Let's say your immutable DAG has four different 'transformations' (think either a map or a fold on the data). Beam or Spark will take a chunk of your data, partition it somehow (so you get parallelism), and do a transform on the data.

Now if you're doing a bunch of computations off of tens or hundreds machines you don't want to fail the whole thing because one hypervisor crashes or there's a JVM segfault or whatnot. So each step is usually saved to disk via checkpoint and then moved along to the next transformation in batches, so if the subsequent transfomation is where it dies, you have a safe spot to restart that particular computation from.

In addition, your (chunk of) data might need to be shipped to an additional machine to be transformed so you're not so much 'updating' data as you are creating new data in a space efficient way of the changes and shuffling those along the various steps.

It's awkward to work with fully nested structures. Think about having a map of customer objects which have a list of addresses and you need to update everyone's first address to be capitalized for some raeson. You'd really want a fully fleshed out lens library and maybe even macros for auto generating them on your concrete data structures to make it easy to 'update' (via a persistent structure) without having to deal with all the cruft of doing it by hand.

I really think this is where pure FP shines.

If you look at the architecture of something like Apache Beam, while you describe your computations in a language like Java or Python, you're really using a DSL for creating an immutable DAG that processes chunks of immutable data, persisiting each chunk (being loose with terms here) for durability, 'modifying' it in the immutable sense and then passing it on to the next edge and so on.

In a single standalone system, many argue that a purely immutable conceptual approach has pros and cons. In the Big Data world, I've never heard anyone argue for an imperative approach. Immutability is necessary, and as soon as you need immutability you want all the other goodies that facilitate it in a clean way like monads, lenses, and the like.

I don't have time (nor the endurance) for a 90 minute run, so could you combine the two? I often do a 14 hour fast and then go for a 20 minute run combined with some calisthenics on my lunchbreak before eating. How can you tell your body is in a ketogenic state?

I can definitely say the first few times I exerted a ton of energy after not eating for 12+ hours I felt like I had the flu for a few hours, but now I don't really get that.

Would love to get your take on if python is popular in large organizations because of the existing libraries or if it is the language asethetic itself?

To an outsider who knows a bit of python because of Airflow and Spark, it seems Python has become a popular metaprogramming language that various disparate ecosystems have all adapated.

Pyspark is python but kind of its own language and much of the processing is happening outside the python runtime. I think you could say the same for people doing Numpy or Pandas.

Tensorflow probably even more so where I believe Python is the most popular interface, but you're really just programming a program to run somewhere else. Again this is the same with Airflow conceptually, though I believe the runtime is python.

If my hypothesis is overall correct, and that a majority (or large share) of python programmers aren't sharing the same ecosystem, libraries and packages, then shifting to a different runtime or language that just encapsulates a subset of the language is much less challenging.

This is the opposite problem of the JVM, where no one really likes Java the langauge, so everyone tries to create enjoyable (and productive) languages for the jvm to keep using the libraries and the runtime optimizations.

This isn’t legal. A corporation that leases someone a car and a house is paying them a taxable income, regardless of it not being a cash income.

When I would commute from Orange County so Santa Clara the flight was often ~40 minutes. Combine with takeoff and landing it didn't make a lot of sense to pull out an addition device but it was easy enough to watch a short TV show on my phone.

Depends on the language (I don't use it for Java), but for Scala and python I definitely prefer it. I'd definitely recommend using TMux with Vim, I've mapped my tmux navigation to mirror Vim's window navigation, and I can keep a terminal and REPL open and easily copy-paste between Vim and my tmux windows.

We can write a general procedure to see it any given program halts within n-steps however.

>Sure, that's possible, but the opposite is also possible: it might be wrong for most programs we actually write.

We could confirm this though! It's not like we can't find out if a given program halts or is inconsisent. Godel talks about it in his letter to Von Nuemann.

It might only be wrong for programs that have never existed and will ever exist.

>Only one thread actually mutates anything, and its the same one over and over

Sounds great in theory and usually results in fantastic average case, but you get a hot partition and suddenly you can't work share and things go south. It's not a perfect solution.

I think it's important to point out that you can do calculus and arithmetic and only construct proofs that are provable. The possibility of deriving unprovable but true statements in an axiomatic system doesn't mean they're common or you can't prove all the satatements one has encountered thus far.

> The day rates look high, but they have to pay self-employment taxes and health insurance

Most SWEs doing 'contract work' aren't actually 1099 contractors, what it means is they work for an intermediary and then 'contracted out' to a big company with a recognizable name (Google/BofA/Verizon/IBM). These jobs are W2 so you don't pay self-employment tax, they usually have mediocre insurance options, and you lose your job as soon as the work dries up.

It's simply a way for a company to staff up quickly without going through as strict of hiring committee and having any obligation to keep said staff around if plans change.

It's often easier for a department to get a req for a senior/staff contractor (and make salary exceptions) than a FTE, so what happens is departments will offer 200, 250 an hour for temp work but only able to hire engineers for 150k a year because their 'employee' req is slotted lower.