mihevc
Joined 8 years ago
Karma
15
Submissions
11
Member since
Feb 4
Recent submissions
Comment · 1 month ago
If I read the example correctly they are using long living http/2 connection to send arrow IPC. Which seems pretty pragmatic.
Comment · 1 month ago
grpc is an RPC framework and Arrow IPC is a serialization format. You can use Arrow Flight which uses grpc to send Arrow IPC. But I suppose TypeSafe doesn't provide grpc endpoints yet?
Comment · 8 months ago
Et tu, Knuthus?
Comment · 10 months ago
Nice! Thanks for the context, it's great to know!
Comment · 10 months ago
How does this compare to https://github.com/Query-farm/tributary ?
Comment · 1 year ago
I was there a week ago and it still wasn't back. The penis room was wide open though.
Comment · 3 years ago
Here are some cookbook examples: https://arrow.apache.org/cookbook/py/data.html#group-a-table, https://arrow.apache.org/cookbook/. Datasets would probably be a good approach for the billions size, see: https://blog.djnavarro.net/posts/2022-11-30_unpacking-arrow-...
Comment · 5 years ago
I wrote my master thesis on Acer C720 (2013) running xfce. I used some octave but I mostly needed it for latex.
These days I don't use it anymore because the screen is not that great but it aged incredibly well.
Comment · 5 years ago
Yet another format is CSF (Compressed Sparse Fiber): http://glaros.dtc.umn.edu/gkhome/node/1177
It's a generalized CSR/CSC (to ndim > 2) format that uses a tree data structure where each path from root to leaf encodes a nonzero element.
Comment · 5 years ago
Nice post!
For completeness: Apache Arrow is also adding sparse tensors in c++ [1] and wrapping them with python [2] (although the documentation for Python might be a bit lacking at the moment).
[1] https://arrow.apache.org/docs/cpp/api/tensor.html?highlight=...
[2] https://github.com/apache/arrow/blob/master/python/pyarrow/t...