lmcinnes
Joined 11 years ago
Karma
112
Submissions
57
Member since
Dec 3
Recent submissions
Comment · 10 months ago
Comment · 1 year ago
> And algorithms can only predict content that you've seen before. It'll never surprise you with something different. It keeps you in a little bubble.
This is not true at all, algorithms can predict things you haven't seen before, and can take you well outside your bubble. A lot of the existing recommendation algorithms on social media etc. do keep you in a bubble, but that's a very specific choice 'cause apparently that's where the money is at. There's enough work in multi-armed-bandit explore/exploit systems that we definitely could have excellent algorithms that do exactly the kind of curation the author would like. The issue is not algorithms, but rather incentives on media recommendation and consumption. People say they would like something new, but they keep going back to the places that feed them more of the comfortable same.
Comment · 2 years ago
Assuming you have a dimension-reduction or manifold learning tool of choice (UMAP,PacMAP,t-SNE,PyMDE,etc.) then DataMapPlot (https://datamapplot.readthedocs.io/en/latest/) is a library specifically designed to make visualizations of the outputs of your dimension reduction.
Comment · 2 years ago
If you just want in-memory then PyNNDescent (https://github.com/lmcinnes/pynndescent) can work pretty well. It should install easily with pip, works well at the scales you mention, and supports a large number of metrics, including cosine.
Comment · 3 years ago
For suitable specialized cases thins can be quite efficient. For persistent H_0 of VR-complexes in low-dimensional space there is an O(N log(N)) algorithm for N data points; that's decently fast. If you want H_1 I believe (but cannot prove) that there exists an O(N^2 log(N)) algorithm. Beyond H_1 things get painful. Since most PH software is written for general cases they don't tend to avail themselves of these special case shortcuts. Given that H_0 and H_1 of VR-complexes of low dimensional data covers a vast amount of the use cases I think specialized code for this would be worthwhile.
Comment · 5 years ago
It is really not that much slower for training (see the paper), and if you are interested in pipelines the difference is not so great considering you are looking at a one off training time vs. lots of inference.
Comment · 5 years ago
If this is a thing you want to be able to do efficiently then ParametricUMAP (see [docs](https://umap-learn.readthedocs.io/en/latest/parametric_umap....) and [the paper](https://arxiv.org/abs/2009.12981)) will be very effective. It uses a neural network to learn a mapping directly from data to embedding space using a UMAP loss. Pushing new data through is only slightly more expensive than PCA, so being part of an inference pipeline is fine.
It is probably not all the things you want, but AlignedUMAP can do some of this right now: https://umap-learn.readthedocs.io/en/latest/aligned_umap_bas...
If you want to do better than that, I would suggest that the quite new landmarked parametric UMAP options are actually very good this: https://umap-learn.readthedocs.io/en/latest/transform_landma...
Training the parametric UMAP is a little more expensive, but the new landmarked based updating really does allow you to steadily update with new data and have new clusters appear as required. Happy to chat as always, so reach out if you haven't already looked at this and it seems interesting.