One idea you could try to make the repeat load less of a burden, and enable more notes or items to be included, is to not just track whether they've been shown before, but to use a spaced repetition algorithm to try to estimate whether an item has been forgotten, and surface the most likely-to-have-been-forgotten items. Those are the ones you would benefit from serendipity, because presumably, if they had become relevant and you already remembered them, you'd've made use of them, and it's the forgotten ones where serendipity strikes.
I call this idea 'anti-spaced-repetition': https://gwern.net/note/statistic#program-for-non-spaced-repe...