belmarca

I would pay for such content. Do you have course notes? Have you written a book? I'd pay to get expert knowledge from... an expert! LLMs are fine, but still not replacing actual human experts IME.

There is a lot of confusion around those words. I personally make a distinction between a "compiler" and a "transpiler".

A compiler preserves the semantics - if you compile Scheme to C, you get a real Scheme program in C, with full semantics.

A transpiler does not have to guarantee that. It is much easier to transpile Python to JavaScript than it is to truly compile Python to JavaScript. A transpiler can tolerate some amount of leakage between worlds (think 'arguments' as a variable name in Python vs JS).

It really depends on your needs. Sometimes a transpiler is okay, sometimes totally inadequate.

You will need a combination of the two. The filename should contain a compound business identifier, essentially a tuple of appropriate facts, as well as a (truncated) hash of the content.

And yes, the reason is auditability but also bare necessity.

Imagine vendor A publishes fresh files every day, but you don't exactly know when. And suppose that the vendor can (and does) republish some of them at any time, with or without notice. You need to watch those files for changes and update your own sources accordingly.

For the collection and dedup, you can construct a tuple of meta-information that will give you a good idea if the file is the same (name, time last modified, file size, etc). Then only if you're still unsure do you download the file and hash it, compare with what you already have, and make a decision. (What if the file name changed slightly and now you must re-categorize the data in the file, or what if a single row was added or removed, etc).

It's also useful when auditing (tamper-proof) or when re-running or debugging calculations that used the data in question. If a user used the data available at 9AM, which was unfortunately incomplete and led to issues, having a full data lineage really helps. You can trivially re-run the computation with the information available at that time to see what happened. Or you can mark some results as stale, etc.

There are different ways to do this and one is keeping track of your content-adressed files in a relational table and give each row a unique identifier, so that you can tell that this particular row in your AAPL OHLCV data comes from this particular file which was fetched at this particular time etc, etc. Pair that ID with a timestamp and when doing computations you can query for "the latest" AAPL data while storing the exact file ID (time is fuzzy, content hash is not). Look into temporal tables and the like.

Nice. The book contains a bunch of good information that could already be found elsewhere but collecting it is quite practical. I highly suggest to read Kleppmann's Designing Data-Intensive Applications. The first edition was very good, a second one came out recently.

I was CTO of a FinTech where I built the whole software stack from scratch: the lessons in the book are mostly correct. I say mostly, because as always, there is a lot of "it depends" to take into consideration for your particular project. For example, I chose to not use event-sourcing to avoid the whole state computation issue. A standard append-only audit trail can do the job.

You can't guarantee exactly-once delivery but you can construct effectively-once processing, and that is what you really want.

Store every request and response : absolutely, and not only when consuming APIs, but when collecting any information from the outside world (and, if you can, also log every intermediate transformation step within your perimeter). Content-adressed buckets + a relational table are great for this.

The text also does not mention anything about data lineage. What happens if a vendor updates some data mid-day that you absolutely need to be aware of? You need to be able to account for that, while also re-playing computations that used the old values and get the same result. It's not a particularly hard problem to solve, but it takes some thought.

OK, sure the concepts might be harder than setting up an environment. But setting things up is, by experience, a very real, very annoying and time-consuming blocker. We have students come in with tablets and it "just works"!

We started this project before LLMs were popularized :) And surprisingly enough, experience says a lot of students would get stuck anyway.

This is not typically how Who is Hiring works :)

I was CTO/engineer at a FinTech up until last month. You can find my email in my profile.

Location: Montreal

Remote: Yes

Willing to relocate: No unless very strong commitment.

Technologies: Python, JavaScript, Scheme, PostgreSQL, ClickHouse, compilers, interpreters, foreign function interfaces, FastAPI, Flask, Celery, RabbitMQ, Redis, Pydantic, SQLModel, technical strategy, team management (6+ devs + interns), SDLC best practices, etc

Résumé/CV: Sent by email

Email: See profile!

I have over a decade of experience working in various roles and industries, from healthcare to education to finance. My last role was CTO of a FinTech that I brought from zero to one, architecting and implementing all of the tech stack from compliance to distributed locks while managing a team of 6 developers.

I am looking for interesting problems and a good team. I strongly believe that good, thoughtful work pays dividends when things break or need to move fast. Boring, proven tech before shiny new thing.

I would say about 80% of the code was in "production". Obvious hiccups were fixed after a few weeks of running. The platform was collecting, transforming, distributing data as designed without any major issues. The financial code was heavily tested by a PhD Wall Stree quant. The less tested parts were the front-end and the swarm itself under load.

And to clarify: I didn't "forget" how to code. I froze for a moment, after relying on a tool to write for me for most of a year. More like rusting.

My first task at my last job was removing access to an employee being let go. I had just gone through onboarding so I knew every (documented) service we needed to handle. We live tested it on my own accounts, measured the time before I noticed, and then proceeded to successfully go through the checklist.

Except not everything was properly documented, and it turned out the employee had given admin rights on some resources to a contractor which proceeded to wreak havoc on their behalf (the 'rm -rf' kind). Eh!

I didn't "do the wrong thing" for 8 months. I built reliable, robust software that did what it was designed for.

Would you say calculators rot brains?

Did the invention of writing rot the brain?