spidy__

Interesting, are there any implications for converting the whole model to integers? Maybe degraded inference speed? Or overflowing issues?

yep!!

Ummm I don't have a lot of idea about crypto but in compression the idea was that you just share the model along with the metadata file to the other user and they de compress it, not sure how things will differ in crypto

If I were only using a transformer that would have been true, but we use arithmetic coding alongside our transformer to fix those mistakes (layman terms). You can read about arithmetic coding, its a pretty cool topic.

Ohh that "what I tried that didn't work" was a section that I specifically wrote myself (and then polished with AI) because I wanted to document what are the different approaches I tried to compress more but failed.

Also thanks for the reference looks like a interesting read.

I mean neural compressors provide great compression, BUTT the issue is they are really slow like in my project it takes around 45 minutes for de compression of 100 mb so I doubt if it would be useful, also using a transformer in user's browser sounds like a heavy task.

I tried with zstd 19

Ofcourse, am doing this just because I enjoy it.

While we are on the discussion I have mentioned a question at the end of the discussion around an assumption am trying, can you please check it out and see if you have any suggestions?

Would be awesome if someone can validate or help.

Ohh yeah , I took it from Project Nayuki as mentioned in the file as well, i tried to pip install it but there were some issues so just took the file and kept the copy right as it is.

Its not an issue is it? I am not sure.

Oh, sounds interesting. I hadn't considered using a diffusion model for this. My current approach generates the document byte by byte with an autoregressive transformer, so I'm curious how a diffusion model would improve memorization or reconstruction quality.

Can you point me to something that i can read? I really wanna try this approach , diffusion model does sounds interesting for compression.

I might be confused by the question, but I overfit the model on a single file and then transport the model along with the arithmetic coding file. There have been ideas where you generalize a model (constant weights) and then pass the arithmetic coding file along with it. So that way you only pass the arithmetic coding file.

BUT my model size is just 900KB (for 100mb file atleast) so it is negligible

Yeah yeah, I just found the idea kinda interesting so wanted to implement it

Sounds interesting man, soo am a bit confused maybe but can you run this on enwik9?