amypinka

Yes, businesses only seem to care what you've been up to lately. If you jump onto something else, this 3 month stint will be meaningless within a few years.

The Steve Jobs biography. Listened to it while backpakcing in SE Asia.

I trust it more than I trust myself to spot issues.

I watched ~25 Moog Grandmother videos by Once Upon a Synth on YouTube. It really helped me drive deep into subtractive analog synthesis.

Libc, zlib, sqlite3

Just stop paying devs by the hour.

Kdb

The full path calls a GNU tool, the other calls your shell's builtin function.

You should only go to work to make money and further your career. Your next interview should know they've only got you in front of them because there is a pay rise on offer.

Don't spend more than a sentence or two discussing your previous role. Describe their business model and leave it at that.

90% of your interview should be finding out what skills and tech your new employer needs and explaining to them why you're perfect for delivering those skills.

Its fine to leave gaps, the latest role is the most important. The .net work doesn't add much to a DS role so there isn't much weight in mentioning it and it shows you spent less than 100% of your focus on DS. If there aren't other candidates then that's fine but there are 10s of candidates that are more or less DS-only on paper.

If candidates could see the other applicant's CVs it would kick off an arms race.

If you put a job spec out in London for anything technical you can easily get 1000s of CVs coming back. Location is super easy to filter on.

If you apply for a job, the experience you list should match the tech stack in the job spec. Don't list roles that aren't relevant (i.e. no .net if it's a Python role).

I helped hire a DS this week. Their CV was among 20 excellent candidates all in London with their latest role matching most of the tech stack we're looking for. All the candidates are in London. This is what you're competing against.

In benchmarks I've seen CStore is about 50% slower than Parquet on Spark.

Where is the transactional requirement? This person is working with a copy of the real data.

ETLs only need to be written once and if he decided on a PSQL approach he'd be writing ETLs to send the data there too. He's probably going to find a number of consistency problems so trying to normalise all this data again will just result in more work that won't make his team of DS' more productive.

If he's at ~1 TB of data today, where will he be in a few years time? What's the point of putting infrastructure in place that won't last for the next 10+ years?

I don't know your ratio of HDF5 to Parquet files but remember for every GB of parquet you have it will equate to about 10 GB of space needed in CSV or PostgreSQL's internal format. So your data set is probably closer to 1 TB than 100 GB.

Storing that data on S3 is probably 50% the price of storing it on EBS and you won't have the durability guarantees of S3 when you're using PostgreSQL on EBS volumes.

If you're both exploring data and building models then Spark is fine. Its APIs are no more complicated that anything else out there for these tasks.

Hive is doing nothing more than offering schema on read and shouldn't be something you're thinking much about.

PostgreSQL is row-oriented and won't be able to offer features like row-group statistics that allow queries to get minimum and maximum values for every 10-15K rows of data for the columns their interested in. This gives queries a huge speed up over needing to scan over rows rather than just the statistics for the columns their interested in.

Remember that you can have a single engineer run a single query on Spark and distribute it across several servers. This allows you to scale CPU and memory bandwidth in a way you won't be able to with PostgreSQL.

It sounds like your data isn't well organised. If you moved it around and put some consistent naming conventions in place that could help. You could also look to build an atlas of the data for newcomers to get an overall picture of what data you're storing and where it lives.