arman_ashrafian

"The Dream Machine" by M. Mitchell Waldrop.

It tells the history of computing by following J.C.R Licklider. As one of the directors of ARPA, he was responsible for funding research labs to work on computer research. He had a major impact on which projects got funded, and in-turn which systems are now being used 60 years later. I honestly love this book so much. If you love computers and history, its a must-read.

Oh cool need to think about this a bit more. I’ll see you in the slack channel :)

Nice I'll check it out!

I'm just going to throw out another problem since you guys are here :)

The current batch process joins some tables before sending data to elasticsearch. This means that the debezium connector doesn't write all the data I need into kafka. I was thinking I could create a materialized table in ksql with infinite retention for the other postgres tables I need to join on. Then when I stream in an update for the data I want in elasticsearch, I can join on these tables.

The issue is that a Stream-Table join only gets triggered when the stream changes. This means that when the data in the tables change we will not see these updates in elasticsearch.

I guess my only option is to join everything in our app and then produce the full message to the elastic sink topic?

I just started using kSQL at work to build a data pipeline from Postgres to Elasticsearch. This will be replacing our error-prone batch processes. So far it has been a really nice experience, however I don't have it running in production yet.

Little bit concerned with how we will update the schema of the streams in production. At some point we will need to drop a stream and recreate it with a different schema, and I'm afraid we will miss some incoming data during the downtime. Confluent just released a migration tool, but I'm pretty sure this only works for adding/dropping columns and not changing the datatype of a column.

I have been using CDK at work and it is fantastic. It feels like React for Infrastructure. I used to write raw Cloudformation at my last internship, and being able to write Typescript instead feels like moving from assembly language to C.

I'm setting up a blog soon. I'm planning on using pandoc and some shell scripts to generate the html from markdown and host on github pages.

The Dream Machine might be my favorite book of all time.

I would like to see him on lex fridman's podcast

Coachella is being delayed till October.

This is not an answer but on a similar note, has anyone watched the latest Joe Rogan Experience with Daryl Davis. He is a black man who convinced Klan Members to leave the KKK. Amazing podcast. Maybe the only way to stop toxicity/polarity is to show real conversations with real people.

Have you tried the CDK yet instead of using Troposphere?