janchorowski
Joined 3 years ago
Karma
238
Submissions
22
Member since
Jul 27
CTO @ pathway.com
View on Hacker NewsRecent submissions
Comment · 1 year ago
Yes, we hope that the YAMLs will provide a clear separation between configuration and code allowing easier deploys of many apps from same codebase and principled experiments.
Thanks for the hint, we'll do a large summary of all options.
Comment · 2 years ago
An index that combines multiple indexing techniques, e.g. vector search with more classical information retrieval techniques such as BM25. We found that they have different strengths: vector indexes are very good at getting synonyms and indirect matches, while classical term-based indexes are better for direct queries. Hybridization gives you the best of both worlds.
Comment · 2 years ago
Thanks! Can you share some more details on the usecases and features used?
Comment · 2 years ago
Thanks for the kind words!
Comment · 2 years ago
Hey, I'm Jan, a Machine Learning researcher, and CTO of Pathway. I've been working on RAG for the last year and I'm excited to share a new RAG optimization strategy that adapts the number of supporting documents to the LLM behavior on a given question.
The approach builds on the ability of LLMs to know when they don’t know how to answer. With proper LLM confidence calibration, the adaptive RAG is as accurate as a large context-based RAG, while being much cheaper to run.
What was really interesting for us here is that the basic idea is "geometric doubling" but it needs to be put into place with so much care, because of the counter-intuitive correlation effects of mistakes produced by LLM's for different prompts.
We provide runnable code examples, you will also find a reference implementation of the strategy in the Pathway LLM expansion pack:
https://github.com/pathwaycom/pathway/blob/main/python/pathw...
Comment · 2 years ago
Now you can use Airbyte source connectors to process data in memory with Python.
We integrated Airbyte connectors with Pathway, a Python stream processing framework, using the airbyte-serverless project. We believe ETL pipelines are coming back with many use cases in AI (RAG pipelines), ETL for unstructured data and pipelines that deal with PII data. In this article, we show how to stream data from Github using Airbyte and remove PII data with Pathway. We are curious on your feedback on the implementation and other use cases you may think of from decoupling the extract and load steps.
Comment · 2 years ago
Originally from 1998: https://web.archive.org/web/19980206223800/https://norvig.co... I didn't want to mislead with a newer date. Sorry if it confuses you.
Comment · 3 years ago
That's a good idea, the deduplication criterion is easy to change, using an llm is faster to get started, but after a while a corpus of decisions is created and can be used to either select another mechanism, or e.g. train one on top of bert embeddings.