pcmoritz
Joined 9 years ago
Karma
28
Submissions
14
Member since
Oct 16
Recent submissions
Comment · 12 days ago
I first built my own to try it out (the exact hardware is described in https://relativity.cc/hardware/ under "one Blackhole card") and when I liked the hardware I got a quietbox 2. It is very high quality but a little expensive (but inexpensive compared to other LLM hardware and it has excellent interconnects which is very rare in the consumer/dev space).
Comment · 12 days ago
Thanks for sharing, this is super cool! Right now my approach to correctness it to just run as many tests as possible and fix them (torchtpu greatly expands the amount of available tests), but formal verification is also super cool. Having a look at your book https://lean.brettkoonce.com/blueprint.pdf
The Tenstorrent ISA is documented btw https://github.com/tenstorrent/tt-isa-documentation, I wonder if one could do the verification end-to-end with the hardware.
Comment · 12 days ago
Thanks for the encouragement, if you have the hardware and end up trying it out I'd love to get your feedback (feel free to drop an issue in the repo) :)
Comment · 12 days ago
For people interested in this: I've been working on an open-source libtpu implementation for https://tenstorrent.com/ accelerators as a side project, it can already run most of torchtpu, see https://github.com/pcmoritz/libtt (based on the tenstorrent software stack). It shows all the things involved in lowering torch to tile based accelerators.
Comment · 8 months ago
You are right, it is fixed now!
Comment · 8 months ago
Thanks, fixed!
Comment · 8 months ago
We are excited about the Tinker API, it exposes the primitives forward_backward, optim_step, sample and checkpoints of an LLM as a REST API, which can be used to implement pretty much arbitrary training recipes while hiding all the infra challenges of running the model and also abstract the underlying accelerator with very small surface area. We hope it can emerge as a standard. If you have any feedback for our open-source implementation we would love to hear about it!
Comment · 7 years ago
If you are interested in learning more about this, there is also a recent PyData talk: https://rise.cs.berkeley.edu/blog/modin-talk-at-pydata-nyc-2...
Comment · 8 years ago
One of the authors here! We are glad you like it!
The project is indeed driven by the authors listed on the paper and also the knowledge and experience that was accumulated in the AMPLab (the predecessor of the RISELab, see https://amplab.cs.berkeley.edu/). If you look at the github history, we've been working on it for longer than a year and had various prototypes before that, so it doesn't come out of "thin air" ;)
The lab's sponsors are also helpful, some of them have been experimenting with the system internally and giving us feedback.
Comment · 9 years ago
There are two considerations here.
(1) Python single threaded performance: Here, most of the libraries we are using are implemented in C++ (like numpy, TensorFlow, Cython to speed up the code, etc.). Ray is orthogonal to that.
(2) Python parallel performance: Here Python is mostly problematic because of its lack of support for threading (the GIL is one problem here); we handle this problem by using multiple processes and shared memory throughout. Efficient serialization makes this feasible.
The core of Ray is implemented in C++, so performance is not an issue for that; also all of the serialization is implemented in C++.
Comment · 9 years ago
For Ray, the main use case at the moment is parallel/distributing machine learning algorithms, people have been using it for parallelizing MC(MC) style applications, doing hyperparameter search, (pre-)process data, we are using it for reinforcement learning (and have a library for that, see http://ray.readthedocs.io/en/latest/rllib.html)
More broadly it is useful for many parallel/distributed Python applications where low latency (~1ms) and high throughput of tasks are a requirement.
Comment · 9 years ago
Author here. For CapnProto see the answer to the other comment above.
Concerning dill, we have been using it for serializing function and classes (and then switched to cloudpickle because it supports some Python functionality better and the community around it is very active responsive); cloudpickle/dill are great in that they support a very wide variety of Python objects, especially concerning code (functions, lambdas, classes); it is less ideal for large data, because there is no zero copy mechanism, the format is not standardized and serialization/deserialization can be slow, sometimes it is slower than pickle. We do fall back to cloudpickle for objects we don't support like Python lambdas. We also use it to serialize class definitions, so the data associated to classes is serialized using the solution presented above and the code/methods are serialized using cloudpickle. This combines the advantages of both solutions.
Comment · 9 years ago
Author here. From our perspective, CapnProto has similar characteristics as Flatbuffers and the reasons to prefer Arrow over it are the same: We would need to develop a mapping from Python types to CapnProto from scratch and Arrow has many facilities that are useful for us already built (Tensor serialization, code to deal with some Python types like datetimes, zero copy DataFrames, a larger ecosystem for interfacing with other formats like parquet, reading from HDFS, etc.). And it is designed for Big Data. So Arrow was a very natural choice (and it also supports Windows). Wes is doing some amazing work here!