selfup
Joined 10 years ago
Joined 10 years ago
I was replying to a comment asking why PII data shouldn't be stored in git vs a database. Just tried to give a general rundown of how to think about the tools.
Totally agreed that the data should have been synthetic/scrubbed and that a small sample set (large enough to validate) would be ok.
Sometimes a PDF or a few pictures are necessary (or just much simpler). I get that. I have seen some repos with an unreasonable amount of PDFs/Docs/pictures/etc.. and that's when a script that copies them into a gitignored directory from someplace (Dropbox/S3/etc..) is a better fit.
If a diff is not useful, don't store it in git. Doing a code review on PII is not useful.
You also shouldn't add packages in a git repo if they can be downloaded immutably in CI/locally. It adds a lot of space. Every commit is a snapshot + new code. Commits are not diffs.
I have found that async or even in Go with goroutines, a large amount of small threads are not faster even if they should be in theory when it comes to CPU intensive work.
I use it to generate a 3D universe (at the atomic level) but store the data as enums.
Here is the repo: https://github.com/selfup/oxidizy
Big help from rayon:
1) https://github.com/selfup/oxidizy/blob/master/crates/unigen/...
2) https://github.com/selfup/oxidizy/blob/master/crates/unigen/...
3) https://github.com/selfup/oxidizy/blob/master/crates/unigen/...
Example perf generating an obscene amount of data on a 32GB machine with 16 threads (3800x):
Atoms: 110592000 Baryons: 26099712000 Quarks: 78299136000
real 0m12.779s user 0m0.000s sys 0m0.000s
Essentially something that both quantum and classical computers cannot crack/brute force.
I would look into lattice-based crypto. It is one of the strongest candidates under review by NIST for post-quantum public/private key mechanisms.
Also Apple buying into using 1Password company wide helps gain some trust (I am sure there was some serious auditing): https://medium.com/enrique-dans/apple-and-1passwords-deal-sh...
https://selfup.me
https://github.com/selfup
https://gitlab.com/selfup
https://codeberg.org/selfup
[ my public key: https://keybase.io/selfup; my proof: https://keybase.io/selfup/sigs/zeb7OPc4pQ6U2r7tM07W7uCSu54nzt6bnF6-gJ9onPE ]