There are three places to accumulate latency, the input from the instrument to the computer, the processing of filters, the output to the monitor (headphones/speakers)/recording. Sometimes you can get away with 2 or 3 ms of latency, but anything over 5 ms is super frustrating. Remember, you're fighting the latency between plucking a string or hitting a key and when the computer acknowledging the data and sending it back out to the monitor which you're using as your guide. Best case you go and "massage" the new track to line up with the existing tracks, worst case it sounds like an out of sync high school marching band.
EDIT: The concerns here are primarily with input latency. Between plucking a string and heading it in your monitor it has to go through: your input hardware, the USB interface, the OS, the browser (which doesn't have explicit low latency capabilities), and JS. Most platforms support ASIO which is a low-level driver for reading audio data from devices. About as close to reading the ADCs yourself. Without a low-latency driver working with the OS there's so much latency overhead it's audible.
EDIT: The concerns here are primarily with input latency. Between plucking a string and heading it in your monitor it has to go through: your input hardware, the USB interface, the OS, the browser (which doesn't have explicit low latency capabilities), and JS. Most platforms support ASIO which is a low-level driver for reading audio data from devices. About as close to reading the ADCs yourself. Without a low-latency driver working with the OS there's so much latency overhead it's audible.