cliffcrosland
Joined 16 years ago
Karma
198
Submissions
49
Member since
Sep 23
Co-founder of https://scanner.dev
View on Hacker NewsRecent submissions
Comment · 2 years ago
tl;dr don’t use an architecture from the on-premise era where storage and compute are coupled together.
Comment · 3 years ago
Great project. Zig and Rust working together in harmony to make it easier to cross-compile Rust for different targets. We use this to great effect to build for x86 and ARM chips in AWS.
Comment · 3 years ago
We're really liking the combination of using Rust, AWS Lambda functions, and ECS Fargate. Everything runs well in ARM, but make sure to use `musl` instead of `glibc` - we've found memory usage to be almost 10x lower with `musl` than `glibc` for our use case (indexing and searching logs). If anyone wants to talk about the tech, happy to answer questions!
Comment · 3 years ago
Yeah I think this is where Datadog excelled and Splunk struggled. Design is clearly better at Datadog, especially the onboarding UX and workflow.
Comment · 3 years ago
Maybe, but the Splunk query language is reasonably well liked by its users, at least in the security space. Much more approachable than SQL, which seems to be what all new tools these days are forcing users to use due to their dependence on Snowflake and Presto/Trino. In Splunk, you can type free text queries, and you can also add structure. Fairly flexible. We’ve been asked many times to make Scanner’s query lang more like Splunk’s.
Comment · 3 years ago
I worked at Cisco following an acquisition. I tend to agree with this comment about the upper stacks :)
However, AppDynamics and Duo seem to be doing well at Cisco from what I can tell. I think observability and security tools are a good match for Cisco and bundle well with hardware. For this reason, I’ll bet Splunk does reasonably well under Cisco too.
Comment · 3 years ago
We’re also giving this a shot. The annual Splunk bill at our last startup exploded from $10k to $1M when we reached 1TB of logs generated per day, which is actually an easy threshold to hit when you have decent traction and aren’t proactively reducing logs. So we built Scanner.dev to drop these costs by 10x.
Decoupling compute and storage is definitely the way to go. We’re using Lambda functions and ECS Fargate containers for compute that scales up and down rapidly, and S3 for storage. Getting ~1TB/sec log scan speeds, which feels fairly good. We keep sparse indices in S3 to narrow down regions of logs to scan. Eg. if you’re searching for an IP address that appears 10 times in a 25TB log set, the indices reduce the search space to around 300MB. Takes a few seconds to complete that query, whereas Athena and CloudWatch take like 20 minutes.
We’re also using Rust to maximize memory efficiency and speed - there are lots of great SIMD optimized string search and regex libraries on crates.io.
We’re early, so there are a lot of SIEM features like detection rules that we are still building. But Splunk/DataDog users might find it useful if costs are a problem and they use mostly log search:
Comment · 3 years ago
How do you think Splunk will fare as more companies move to public clouds? Seems unbeatable on-prem, which probably means that the company is a good match for Cisco.
Comment · 3 years ago
Agreed. Costs plummet when you use S3 as the storage medium for these massive log data sets. I think S3 is much faster to query than most people realize. Just have to be smart about how you organize things.
Comment · 4 years ago
Looks really cool! Great work!
Had a pricing question. Say we connected a log forwarder like Vector to send data to Clickhouse Cloud, once per second. If each write unit is $0.0125, and we execute 86,400 writes over the course of the day, would we end up spending $1080? Do you only really support large batch, low frequency writes?
Comment · 4 years ago
We are using Rust to build a fast log event database. The workload includes a lot of asynchronous networking I/O and parallelized CPU-intensive work, like indexing and SIMD-accelerated string search. There are great libraries for what we need. The performance and low memory footprint are quite useful.