leod
Joined 14 years ago
Joined 14 years ago
Personally, I think that this is a price worth paying!
Would you be willing to share details about the fine-tuning procedure, such as the initialization, learning rate schedule, batch size, etc.? I'd love to learn more.
Background: I've been playing around with generating image sequences from sliding windows of audio. The idea roughly works, but the model training gets stuck due to the difficulty of the task.
(Full disclosure: I have only read the abstract so far.)
[1] https://archive.org/details/14566367HackerNewsCommentsAndSto...
If I understand nshepperd's code correctly, it uses a constant and small learning rate. Do you know if this works better than the learning rate schedule that is usually used for Transformer models (https://www.tensorflow.org/alpha/tutorials/text/transformer_...)?
The generated titles are great! You can put them into hncynic (https://github.com/leod/hncynic) to get closer to a fully generated HN experience.
As far as HNSW implementations go, this one appears to be almost entirely unfinished. Node insertion logic is missing (https://github.com/swapneel/hnsw-rust/blob/b8ef946bd76112250...) and so is the base layer beam search.