> Of all the float-pointing point numbers your computer can represent, a quarter of them lie in [0,1].
Many people think floating point numbers are magically precise. They're not. They're far more accurate at lower magnitudes and precision fades as you work with larger values.
Or you can think of it as precision in the number of significant digits for a number. Each floating point number can only store a certain amount of significant digits. The fractional precision on each float is the same, but the absolute magnitude of the error in the float value increases as the value increases.
>Many people think floating point numbers are magically precise.
Even worse, it kicks in earlier than one might naively think. People are often surprised to hear a value as simple as 0.1 has no matching floating point representation.
No, that's explicitly a floating point format. It's one of the representations in IEE-754-2008. The three decimal floating point formats are decimal32, decimal64, and decimal128. That people often use them where fixed-point math would work doesn't mean they're not floating point formats.
It does a variable number of fraction bits, so that the values near 1 are even more densely represented than under the usual IEEE floats, while also providing greater dynamic range (but at reduced precision).
Out of curiosity, I started watching this talk, but right there in the introduction, the very first example is a dot product where supposedly IEEE 754 double precision gets the wrong answer; I stopped to check the result, and I got the correct answer with double precision (even without binary sum collapse). Then, he says the x87's results are nondeterministic due to being affected by cache, which is incorrect. Then, he trots out a series of half-truths about IEEE 754, which are somewhat true but don't mean what he's presenting them to mean. He even cites a Cray 1 quirk as an example of IEEE 754 weirdness, when in reality, the Cray 1 (1975) predated IEEE 754 by about 10 years.
Designing a good arithmetic system takes a lot of attention to detail, and he doesn't make a good impression by being that loose with details in his introduction.
Doubles on my Intel give correct answer too, but change the exponent used everywhere to 8 and you'll see the problem. Gustafson may have used different implementation of IEEE 754 that gave him different result.
There are problems with repeatability of float operations such as loss of precision when moving value from registers to memory and data alignment issues, but it seems that these can be avoided with proper compiler options, at the expense of speed.
IEEE 754 is a standard, and it requires a specific, bitwise-reproducible, answer for the computation in question. I'm the author of most of the floating-point tests in WebAssembly's conformance testsuite, which tests such things in practice across several hardware platforms.
One of the half-truths in the presentation (in the intro) is that IEEE 754 is a mixture of requirements and recommendations. IEEE 754 does have both requirements and recommendations, however what the presentation doesn't say is that, within a given format like double precision (aka binary64), the basic operations like add, subtract, multiply, divide, squareRoot, etc.) have exactly one possible result for any given input (except that NaNs may have some implementation-defined bits, though this is usually unimportant).
Only a subset of scientific codes benefit from denormals, and it does not appear that implementing them is that big of a deal, given that 4-stage pipelines can do it. The pipeline depth to memory is a lot bigger than 4.
Only a subset of scientific codes benefit from denormals, and it does not appear that implementing them is that big of a deal, given that 4-stage pipelines can do it.
Maybe I'm misunderstanding your terminology, but it seems like you are saying that operations involving denormals have the same latency as normal floating point multiplications and additions. At least for multiplication on Intel chips through Haswell, I think the current case is that subnormals between 0 and FLT_MIN still have abysmal performance --- 100+ cycles of penalty.
Here's Bruce Dawson from a few years ago:
Performance implications on SSE
Intel handles NaNs and infinities much better on their SSE FPUs than on their x87 FPUs. NaNs and infinities have long been handled at full speed on this floating-point unit. However denormals are still a problem.
On Core 2 processors the worst-case I have measured is a 175 times slowdown, on SSE addition and multiplication.
On SandyBridge Intel has fixed this for addition – I was unable to produce any slowdown on ‘addps’ instructions. However SSE multiplication (‘mulps’) on Sandybridge has about a 140 cycle penalty if one of the inputs or results is a denormal.
And here's the overview from a slightly more recent paper:
C. Subnormal Performance Variability
Due to the complex nature of the floating point numbers, processors struggle to handle certain inputs efficiently. In particular, it is well understood that operating on subnormal values can cause extreme performance issues, including slowdowns of up to 100× [19]. As an example, on a Core i7 processor using SSE instructions, performing standard mul- tiply between two normal numbers takes 4 clock cycles, whereas the same multiply given a subnormal input takes over 200 clock cycles.
If you consider every iterative algorithm that solves systems of nonlinear equations a subset you cam ignore.... SPICE, linpack, fishpack.... sorry, denormals are essential to today's algorithms.
As to "no big deal".... show me your code.
Don't worry, I'll be able to understand it. After 20 years as a CPU designer I've learned how to understand bit-bashing.
The iterative algorithms for all of the scientific fields that I've worked on don't fall within the subset that you're describing, so no, I'm not going to agree that you're speaking about the majority of today's algorithms. Weather, climate, finite differencing, FFT... nope. Implicit systems, sure. Radar cross-sections, sure.
And as a CPU designer, are you disagreeing that most FP units today take 4 cycles (fully pipelined) (add and multiply, of course)? And that main memory is a lot farther away than that?
Given 20 years of experience, you missed out on the Cray PVP machines that were the start of this sub-thread. But the cycle counts I'm giving are modern ones. Cray did eventually implement IEEE in these machines without any significant problems, but that was more than 20 years ago.
Every number system in which all numbers occupy the same amount of space must have this property (namely, that there are roughly as many positive numbers above 1 as below 1, and therefore that there is greater precision near 0) or the accuracy of the function x->1/x will suffer.
Correct me if I'm wrong, but my understanding is that a consequence of this is that it's better to store, for example, angles as radians from -pi to pi rather than as degrees between 0.0 and 360.0, in order to take advantage of all that precision between 0 and 1.
The most efficient/uniform representation is to store angle measures as fixed-point numbers in [–1, 1) representing [-π, π) radians, using two’s complement for negative numbers (so you can also think of this as [0, 2) with unsigned fixed-point numbers). This representation is called “binary angles”. You can think of it as an extended version of magnetic compass directions, or the corners of a regular polygon with 2^n sides. https://en.wikipedia.org/wiki/Binary_scaling#Binary_angles
Using IEEE floating point to represent angle measures is horribly wasteful of bits, regardless of how you do it.
As an alternative, consider not storing angle measure at all, but instead using a stereographic projection (“half-angle tangent”) of the circle onto the whole line, and then using floating point numbers in the range (–∞, ∞] for your representation. https://en.wikipedia.org/wiki/Stereographic_projection
What you really want as a working format for rotations, ray directions, or points on a circle is to use a 2-dimensional square-grid (Cartesian coordinate) format like unit-magnitude “complex numbers”. The angle measure or stereographic projection is primarily useful as a compression format when you need to save bits transferring or storing data. Angle measures work okay for this, but can be annoying for requiring transcendental function evaluations to convert them to/from Cartesian coordinates, whereas to take the stereographic projection or its inverse only requires a single multiplicative inverse (i.e. division operation) plus some addition and multiplication per point.
I don't see why that would help. Of course there is more precision in terms of decimal places around 1 than around 360. But if you use 360 each 1 represents a much smaller piece of the circle.
To put it another way...it's about significant digits. A float has about 7 and a double about 15. It does not matter whether you use 0 to 360 or 0 to 3.6 million...the number of digits that are meaningful remain the same.
For binning data in spreadsheets for histograms etc, I typically use text() to set significant digits, and then value(). It's also good sometimes to make sure that 0 is really 0.
It's not really relevant there, that's just scaling the error factor. The potential downside to using degrees is that the values in the 0-1 range are going to have slightly more accuracy than those in the 358-359 range. The difference will be very minor if these aren't used in compounding calculations.
It's more of an issue when you're dealing with large sums composed of very tiny ones. If you add them together incorrectly the number becomes so large the tiny values stop mattering even if in aggregate they're important.
Like adding 1e-6 to itself a hundred million times gives you a result different than 1e-6 * 1e8. In the first case I get 999.999998191639 when the multiplied version is 1000.
These errors can accumulate to a dangerous degree if you don't do your operations in the right order.
Really, 52 bits of mantissa is more than enough for representing anything. You have to be wary of error building up on calculations, not of original representation errors (unless you are living on the edge; and I'd tell you: don't).
Useless answer: No, the precision is fixed for any magnitude. When you multiply your values, you will also multiply the interval, and there are as many useful numbers inside that new, larger, interval than were inside the old, smaller one.
If all you're dealing with is angles, 64-bit quantization of the 360° space will give you far more accuracy than you'll ever need, and it will be absolutely uniform throughout.
Your worst-case relative precision is going to be the same no matter what your end point is. You get a little more best-case precision, but most of that is crammed into a tiny fraction of the circle so it doesn't really help you. You get an extra bit of precision by using negative numbers too, but -180 to 180 would also do that.
Depends on how big are the angles you want to store. May be you're using a variable to measure how much the wheel have rotation, totally, while driving (i.e. millions of degrees)? Or angular diameter of stars on the night sky?
What's hilarious is that this is what is meant by calling them "floating point". The counterpoint "fixed point", in comparison, does not necessarily imply integers.
Many people think floating point numbers are magically precise. They're not. They're far more accurate at lower magnitudes and precision fades as you work with larger values.