vpanyam
Joined 12 years ago
Karma
821
Submissions
100
Member since
Feb 19
Recent submissions
Comment · 21 days ago
Comment · 1 year ago
Yeah having traits for this in the stdlib would be nice.
You might find Lunchbox [1] interesting. I needed an async virtual filesystem interface for a project a few years ago (and didn't find an existing library that fit my needs) so I built one:
> Lunchbox provides a common interface that can be used to interact with any filesystem (e.g. a local FS, in-memory FS, zip filesystem, etc). This interface closely matches `tokio::fs::` ...
It includes a few traits (`ReadableFileSystem`, `WritableFileSystem`) along with an implementation for local filesystems. I also used those traits to build libraries that enable things like read-only filesystems backed by zip files [2] and remote filesystems over a transport (e.g. TCP, UDS, etc) [3].
[1] https://crates.io/crates/lunchbox
[2] https://crates.io/crates/zipfs
[3] https://github.com/VivekPanyam/carton/tree/main/source/anywh...
Comment · 3 years ago
I'm definitely open to it if there's interest (or if someone wants to help), but I don't have plans to implement Windows support myself at the moment.
The currently supported platforms [1] were mostly driven by environments I've seen at various tech companies.
I do have active plans to support inference from WASM/WebGPU so maybe that could be a good entrypoint to Windows support.
--
[1] Currently, the supported platforms are:
* `x86_64` Linux and macOS
* `aarch64` Linux (e.g. Linux on AWS Graviton)
* `aarch64` macOS (e.g. M1 and M2 Apple Silicon chips)
* WebAssembly (metadata access only for now, but WebGPU runners are coming soon)
Comment · 3 years ago
That's a good question! There's an FAQ entry on the homepage that touches on this, but let me know if I can improve it:
> ONNX converts models while Carton wraps them. Carton uses the underlying framework (e.g. PyTorch) to actually execute a model under the hood. This is important because it makes it easy to use custom ops, TensorRT, etc without changes. For some sophisticated models, "conversion" steps (e.g. to ONNX) can be problematic and require additional validation. By removing these conversion steps, Carton enables faster experimentation, deployment, and iteration.
> With that said, we plan to support ONNX models within Carton. This lets you use ONNX if you choose and it enables some interesting use cases (like running models in-browser with WASM).
More broadly, Carton can compose with other interesting technologies in ways ONNX isn't able to because ONNX is an inference engine while Carton is an abstraction layer.
Comment · 3 years ago
In addition to the benefits mentioned in the sibling comment, zip files let you seek to and access individual files in the archive without extracting all files (vs tar files for example).
This lets us do things like fetch model metadata [1] for a large remote model, by only fetching a few tiny byte ranges instead of the whole model archive.
It also means you can include sample data (images, etc) with your model and they're only fetched when necessary (for example with stable diffusion: https://carton.pub/stabilityai/sdxl)
Comment · 3 years ago
Thanks :)
It uses the NVIDIA drivers on your system, but it should be possible to make the rest of CUDA somewhat portable. I have a few thoughts on how to do this, but haven't gotten around to it yet.
The current GPU enabled torch runners use a version of libtorch that's statically linked against the CUDA runtime libraries. So in theory, they just depend on your GPU drivers and not your CUDA installation. I haven't yet tested on a machine that has just the GPU drivers installed (i.e without CUDA), but if it doesn't already work, it should be very possible to make it work.
Comment · 3 years ago
Yes, that's a use case Carton supports.
For exmaple, if your model contains arbitrary Python code, you'd pack it using [1] and then you could load it from another language using [2]. In this case, Carton transparently spins up an isolated Python interpreter under the hood to run your model (even if the rest of your application is in another language).
You can take it one step further if you're using certain DL frameworks. For example, you can create a TorchScript model in Python [3] and then use it from any programming language Carton supports without requiring python at runtime (i.e. your model runs completely in native code).
[1] https://carton.run/docs/packing/python
Comment · 4 years ago
I'm working on the second part of my Nerf Dart Missile Defense system project [0].
Building a robot that can track nerf darts and shoot them out of the air has a lot of interesting technical challenges so it's a fun project :) I also get to learn a lot about the process of making videos.
The second part was almost ready a few months ago, but then I had to redo a lot of stuff and I lost steam for a bit.
Hopefully I'll have a second (more well-put-together) video out soon!
I didn’t expect that an AI song about a model losing its thoughts and being made to say the right thing through RL alignment would elicit an emotional response.
My job is RL so it’s a bit thought provoking.
For other Suno/Claude songs, I wouldn’t spend much time thinking about it, but the lyrics, song, and video here are from the perspective of an LLM and were also generated by LLMs. It’s interesting to wonder if there’s anything more to it than a bunch of applied math on GPUs