yonl
Joined 8 years ago
Joined 8 years ago
I hope, since it’s opensource, you are thinking about exposing api / hooks for downstream tasks.
Speaking of implementation, i don’t mind if a browser extension forward cookies from my browser to the automation (privacy and security is an issue of course, and i’d ideally want the cookies to not leave my device, but personally i’m okay with some trade off).
Our main requirement was querying recent operational data across daily/weekly/monthly/quarterly timeframes. The data sources included OLTP binlogs, OLAP views, SFDC, and about 15 other marketing platforms. We implemented a datalake with our own query and archival layers. This approach worked well for queries like "conversion rate per channel this quarter" where we needed broad data coverage (all 17 integrations) but manageable depth (reasonable row scanned).
This architecture also enabled quick solutions for additional use cases, like on-the-fly SFDC data enrichment that our analytics team could handle independently. Later, I learned the team integrated LLMs as they began dumping OLAP views inside the datalake for different query types, and eventually replaced our original query layer with DuckDB.
I believe approaches like these (what I had done as in house solution and what definite may be doing more extensively) are data and query-pattern focused first. While it might initially seem like overkill, this approach can withstand organizational complexity challenges - with LLMs serving primarily as an interpretation layer. From skimming the Dustt blog, their approach is refreshing, though it seems their product was built primarily for LLM integration rather than focusing first on data management and scale. They likely have internal mechanisms to handle various use cases that weren't detailed in the blog.
- if the key is present
- if the value of the key is in given range (any filter function you can imagine)
- conditional key presence (Ex: if key1 is present key2 should be less than 10)
- complex nested structure of the payload
- array validation (length / type / property)
I end up write a significant amount of if else block just to validate if the payload is acceptable. So I wrote a tool a long time back which I use in all my JS based server project (My gateway servers are always in Node for past 3-4 years because of the amount of instrumentation / ease of debugging)
I wrote helson 2 years back, where a dev would define schema of a payload
``` typedef Payload { str "url": pass, []str "tags": pass, bool "isCollection": pass, obj "content": { str "body": strShouldNotBeEmpty | shouldBeOfMinimumLength 15 | shouldHaveMinimumWordLength 5, }, } ```
with primitive type support of str, bool, number and compound type support array, enum, ref (https://github.com/adotg/helson/blob/develop/test/helson.tes...) for type checking and custom function (strShouldNotBeEmpty, shouldBeOfMinimumLength above example) as value checking
And it tests against incoming payload
``` { url: '#/e/hash-of-a-link', tags: ['a1', 'a2', 'a3'], content: { body: 'This is a body', } ```
Now that I use typescript, I was in the verge of deciding (I don't have enough time solving just for myself as the value addition is not justified for myself, but if enough people wants it I'll build it) should I also build a typescript supported workflow. Like from a schema like above generate interfaces (and vice versa). Or is this lib meaningful to you at all.
It also throws meaningful error automatically (which you can override) to return to client directly