Comment

Comment by SloopJon

I've always been fuzzy on Excel's precision. It's frequently described as preserving or storing fifteen significant figures. I wasn't sure if it rounds before storing, or if that's just how it displays. After some experimentation, I've satisfied myself that it can store more than it can display.

For example, =PI() displays at most 3.14159265358979, no matter how many digits I ask it to display. If I add 1.999E-15, I get 3.1415926535898. If I type 3.14159265358979 (omitting the invisible trailing 3), then add 1.999E-15, I get 3.14159265358979.

In other words, Excel uses double-precision floating point, but never displays more than fifteen significant figures. You may be in for a surprise if, say, you round trip a spreadsheet through CSV.

parent SloopJon 4 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton