scottcha

Joined 16 years ago

Karma

1,000

Submissions

162

Member since

Jan 18

https://neuralwatt.com [email protected] https://www.linkedin.com/in/scott-t-chamberlin/

View on Hacker News

Recent submissions

Comment · 12 hours ago

I agree and we charge by kWh though we are constantly working to allow users to compare to various token rates or sub amounts. Token subs are all very obfuscated and I don’t feel it’s for the benefit of consumers.

Comment · 7 days ago

We can push 1k tps on our speed lane with the right request parameters with deep seek v4.1 flash. Is 2k a hard requirement?

Comment · 29 days ago

We publish live stats comparing model energy here: https://portal.neuralwatt.com/energy-pricing though since we are ZDR we don't really have any information about what effort levels these are run at but I'll try and put together some sort of analysis. Its not entirely clear cut as higher effort certainly uses a lot more energy but you might take fewer requests to solve a task.

Comment · 29 days ago

Awesome, glad it worked well. Amazing writeups!

Comment · 1 month ago

On our journey at Neuralwatt (we are likely the provider he's using as we are the one that does all the energy observability and reporting in our cloud and I'm the CTO there) we quickly discovered that there is a large disconnect between what is in the press (like the rest of the AI narrative very subject to worst case but possibly unlikley future extrapolation of current trends) and what we see on the ground. We actually spend most of our time focused on various way energy constraints (including maximizing tokens/joule while maintaining perf) manifest in current datacenters and provide better ways to get more tokens out of the energy that are already in these data centers or already available but hard to utilize on the grid. So its really a technical constraint problem <it>today</it> rather than an impact problem and I think the press narrative could be better about this.

Regarding total costs relative to the pure energy costs it is multiple orders of magnitude different but also realize in the datacenter the energy is the pure commodity while almost every other component has huge margins driven by lack of supply. I do think over time this might get closer together (more competition on HW might lower margins) while energy might become more of a bottle neck (raising the energy prices).

Comment · 1 month ago

Very cool paper and some interesting things for local serving. Though its a little apples-to-oranges we do publish live energy stats for all models on our service here https://portal.neuralwatt.com/energy-pricing in case you are interested in what this looks like on the cloud side. FWIW DSV4.1 flash is really getting popular due to its IPW.

Some of the items like model routing, if you do it per request instead of per session, can break down on the cloud from an energy and cost POV since one of the best things you can do for both is to maintain the KV cache which both reduces time component of energy and the quite expensive prefill energy.

I am keen on the future where we have local/cloud hybrid serving which is cache aware. I do think that could be the best use of energy resources for AI.

Comment · 2 months ago

Neuralwatt | https://neuralwatt.com | REMOTE | Seattle or Denver/Boulder metros | Full-time

Hiring: 2 Engineers + Director of Sales & Operations

Energy is becoming one of the biggest constraints on AI infrastructure. Neuralwatt is building software to help datacenters get more useful AI compute and revenue from every kilowatt.

We're a VC-backed early-stage startup working on GPU optimization, AI infrastructure, energy-aware control systems, and the Neuralwatt Cloud.

$180k to $220k DOE + equity

You'll work on: Rust and Python, GPU, inference, and datacenter optimization, Production AI infrastructure, Energy-aware scheduling and control, Neuralwatt Cloud

Director of Sales & Operations

We're looking for someone with datacenter industry experience to build sales channels, partnerships, and customer relationships, support company operations, and help drive growth. Comfort using modern AI and agent tools is required.

Location: Remote-first, but you must be based in the Seattle or Denver/Boulder metro areas.

No visa sponsorship available.

To apply: [email protected] Subject: HN Hiring (<your metro>/<engineering or sales>)

Include your resume, GitHub or relevant work profile, and a short note on why you're interested in AI and energy.

Comment · 3 months ago

We use vllm as it generally has the best ecosystem support. Parameters are largely dependent on what type of requests you are serving (concurrency, input/output ratios, cached hit patterns). We've never had a limitation at the tokenizer step. Limitations at peak tend to manifest more on slower time in vllm doing prefill or decode though we actively try and minimize this.

Comment · 3 months ago

I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.

Comment · 3 months ago

Hi, I'm co-founder of Neuralwatt. While there aren't other providers selling by energy we also produce the same tokens stats you get at others (input, output, cached) and you can compare using those. The CO2 number is really just multiplying the energy by the CO2-intensity at the time/place we served the request. So everything is actually measured and can be derived externally.

The main observation when you compare to token prices our input/cached energy is much lower than the equivalent token prices while the output energy is generally higher. One of the reasons this is a bit cheaper, especially for agentic systems, is that in an fully built/cached session your input & cached to output ratio is really massive input to very small output (multiple orders of magnitude usually). The other thing we do is we actively work to optimize the system around tokens/joule with the goal of making the most energy-optimal system.

Happy to answer any other questions.

Comment · 3 months ago

They do have a growing amount of Scope 1 emissions (emissions from their on site sources) which originally was primarily on site diesel but due to grid interconnect delays have been growing number of on site gas turbines.

This certainly wouldn’t be necessary with adequate generation and transmission capacity.

Comment · 4 months ago

Thanks for the feedback! Our primary focus is charging by energy, for token pricing we really just try to be close to the market. That being said I'll take a look at our token pricing to see if we need an update there https://portal.neuralwatt.com/energy-pricing Generally our users get much lower cost on energy than token pricing though on a typical request with a high prefix cache hit the input, cached costs is very small and the output energy cost is higher.

We definitely don't have any intention to obfuscate and in fact we actually try and provide more data than any other provider out there about both an individual request, as well as the fleet behavior. Since we tend to focus directly on our energy pricing and optimizing that the issue is likely where the ROI lies on energy optimization versus token optimization (totally correlated but we have other levers to reduce energy while keeping token counts the same).

Comment · 4 months ago

Hi I'm the CTO of neuralwatt, would love to hear your feedback on what your experience was. Feel free to email me [email protected]. Also for GLM5.2 we run the FP8 quantization at 1M context which is a common deployment target.

Comment · 4 months ago

Yes way better. We host both and while qwen3.6 is over 100tps we usually can do glm around that too.

Comment · 4 months ago

I use glm5.1 plus pi with a few customized skills and am very happy with it. I hadn’t touched my Claude 5x plan for a couple of weeks but opened it back up in Claude code when fable was released and did a few tasks and still was happy to return to glm/pi.