enduku

Joined 8 years ago

Karma

325

Submissions

101

Member since

Mar 16

Blog: www.densebit.com Twitter: @densebit Email: [email protected] GitHub: https://github.com/xtellect

View on Hacker News

Recent submissions

Comment · 13 days ago

Cool. superoptimizing-jit ideas are exciting to me. Auto parallelism, swapping faster code/algos based on data distribution, autovectorization etc. Several years ago I had worked on an idiom based optimiser for assembly (https://github.com/xtellect/osm) and I hoped to extend it to binaries like what this team did. Couldn't find time to work on it further, but what this team achieved looks admirable. Congrats

Comment · 1 month ago

I think tcmalloc gains on thread churn and oversubscription by going the cpu-cache route on Linux. on other platforms, I am not so sure but that can be offset by say a treiber-stack like setup for cross-thread frees/teardowns. So lesser code for Linux for similar fastpath design I guess.

Comment · 3 months ago

yeah the site's clearly vibecoded and isn't opensource, but i also think this is a genuinely interesting design space and more people should be building in it. APL (https://www.dyalog.com/), BQN (https://mlochbaum.github.io/BQN/), J/Jd (https://code.jsoftware.com/wiki/Jd/Overview), Klong (https://news.ycombinator.com/item?id=10586872), Kerf (https://news.ycombinator.com/item?id=9782520), RayforceDB (https://news.ycombinator.com/item?id=45889607), k/q (https://kx.com/) glad there's a new entrant.

Comment · 5 months ago

The `Cilk` angle is interesting. There’s still room for small runtimes focused just on fork/join recursion.

I’ve been working on one for C: https://github.com/xtellect/cactus

It’s narrower than Silk/SeaStar: continuation stealing for CPU-bound recursive code, not a general async I/O fiber runtime.

Comment · 5 months ago

Fair points on both. Thanks for aasking.

The 120 MiB/s encode ceiling is the cost of the mode competition. that's where the ratio comes from. At 800-1600 MiB/s off a digitizer, fc is the bottleneck no matter what transport sits behind it; for that regime zstd-3 or lz4 are the better fit, or fc further down the pipeline on aggregated/decimated data.

You're also right on int/short. fc's modes look for IEEE-754 bit patterns, so doubles that started life as rescaled ints lose the structure those modes exploit. A native int16/int32 path is on the list.

For the wiring itself: I have a sister single-header library, vibe (https://github.com/xtellect/vibe), built for this exact pattern: length-prefixed TCP/IPC framing on Linux, with a `telemetry_sink` example close to the edge-sensor --> cloud-ingest case. Producer compresses with fc, ships framed bytes through vibe, consumer decompresses. Doesn't solve the throughput ceiling, but handles the producer/consumer setup cleanly.

edit: i think the comments is flagged automatically because I used `vibe` (bad name I know) :)

Vibe, A single-header lock-free networking library for Linux github.com

4 enduku 5 months ago 0 comments

Comment · 5 months ago

Yeah, and also approximating a double (within range) to int32 :)

https://x.com/Densebit/status/1839705674378613043?s=20

Comment · 5 months ago

Thank you. Fuzz safety is definitely on my list. Current focus is to broaden the benchmarks , predictors and preprocessors and see what sticks

Comment · 5 months ago

I need to add it to the benchmark. My expectation is that OpenZL should be strong when the enclosing format is known and SDDL can separate typed fields cleanly. Running both on the same f64 arrays will give some information

Comment · 5 months ago

Yes it is. The mismatch is mainly representation and purpose: audio is usually int16/int24/float32 PCM, and audio codecs often exploit perceptual loss. fc is lossless and currently tuned for float64 streams.

Comment · 5 months ago

Agreed; pcodec is probably one of the most relevant comparisons. I will add pcodec to teh benchmark

Comment · 5 months ago

I have an XOR128-style mode and a byte-transpose/byte-split-like mode, but I should not claim that as a proper Chimp128 or Arrow Parquet byte-stream-split comparison yet. I willadd direct baselines for Chimp128 and Arrow/Parquet BSS+zstd to the harness.

Comment · 5 months ago

Agreed. will work on that :)

Comment · 5 months ago

I’m regarding that term loosely here- in this case it is 'try several representations/codecs for a block and store the winner.' Similar ideas show up in columnar formats choosing encodings per column/page, OpenZL selectors (asother commenters pointed here), and shuffle/transpose + backend-compressor pipelines. fc’s version is much narrower: a tournament among f64-specific modes per block.