peadarohaodha

Would love to have a good selection of keyboard shortcuts! Power Vibe Kanban

Thanks for the feedback! It's a good question re compute. There are some fun engineering and ML research challenges that we are constantly iterating on that are related to this. A few examples - how to most efficiently share compute resources in a JIT manner (e.g. GPU memory) during model serving for both training and inference (where the use case and privacy requirements permit) - how to construct model training algorithms that operate in a more online manner effectively (so you don't have to retrain on the whole dataset when you see new examples) - how to significantly reduce the model footprint (in terms of memory and flops) of modern deep transformer models given they are highly over-parameterised and can contain a lot of redundancy.

this stuff helps us a lot on the margins point!

We've found you can get an order of magnitude improvement in the amount of labelled data you need - but there is some variance based on the difficulty of the problem. Because you are retraining the model in tandem with the data labelling process, there is additional compute associated to an active learning powered labelling process versus just selecting the data at random to label next. But this additional compute consideration is almost always outweighed by the saving of human time spent labelling.

adding to what Raza said - a consideration for both active learning and weak supervision is the need to construct a gold standard labelled dataset for model validation and testing purposes. At Humanloop, in addition to collecting training data, we are also using a form of active learning to speed up the collection of an unbiased test data set.

Another consideration on the weak supervision side (for the snorkel style approach of labelling functions) is that creating labelling functions can be a relatively technical task, which may not be well suited for non-technical domain expert annotators for providing feedback to the model

Adding to what Raza as said - to your point on "real active learning" hardly working. I would be interested to hear what approaches you took? We've found that the quality of the uncertainty estimate for your model is quite important for active learning to work well in practise. So applying good approximations for the model uncertainty for modern sized transformer models (like BERT) is an important consideration

Our default deployment option is cloud first for both training and inference at the moment, but we have thought about the ability for users to export a trained model. Either exporting the model parameters in some standardised format, or a compiled predict function, or a docker image that encapsulates a full inference service, etc. So if you could use this kind of export within your application, this would allow on-premise inference. This is something we could probably make available pretty quickly if necessary for your use case.