It really is great that Meta released the notes/logbook. Credit where credit is due. Very few other academic or industry labs release materials like this, especially when the reality is so messy.
Some interesting takeaways:
- Meta aren't using any software for scientific logbooks, just prepending a document
- So many hardware/cluster issues.
- Hot-swapping algorithms is common and likely underreported (in this case activation functions and optimization method)
- A well resourced team didn't solve enough issues to fully utilize compute resources until >50% of the total time into the project
Some interesting takeaways:
- Meta aren't using any software for scientific logbooks, just prepending a document
- So many hardware/cluster issues.
- Hot-swapping algorithms is common and likely underreported (in this case activation functions and optimization method)
- A well resourced team didn't solve enough issues to fully utilize compute resources until >50% of the total time into the project