klavinski

Here is the list of technological problems:

1. When is a page ready to be indexed? Many websites are dynamic.

2. How to find the relevant content? (To avoid indexing noise)

3. How to keep an acceptable performance? Computing embeddings on each page is enough to transform a laptop into a small helicopter with its fans. (I used 384 as the embedding dimension. Below, too imprecise; above, too compute-heavy).

4. How to chunk a page? It is not enough to split the content into sentences. You must add context to them.

5. How to rank the results of a search? PageRank is not applicable here.

    type A = { kind: "kindA", a: "dataA" }
    type B = { kind: "kindB", b: "dataB" }
    type Sum = A | B

    const match = <
        const V extends { kind: string },
        const C extends { [ kind in V[ "kind" ] ]: ( value: V & { kind: kind } ) => unknown }
    >( value: V, cases: C ) => cases[ value.kind as V[ "kind" ] ]( value ) as ReturnType<C[ V[ "kind" ] ]>

    // You check the type of result, change the type of value to A or B, make the cases non-exhaustive...
    const howToUse = ( value: Sum ) => {
        const result = match( value, {
            kindA: _ => _.a,
            kindB: _ => _.b
        } )
    }

None of these could spark enough interest to get feedback on what users want. I am sharing this experience so that you may study my attempt, if you want.