View RSS Feed

Development Team Blog

Numeric masks and rounding, floating point values and conversions

Rate this Entry
Frank is asking about a peculiar behavior with numeric masks and rounding, where it seemingly sometimes rounds the value to the specified number of decimals in the mask, but other times it seems to truncate instead. What's going on? Is this a bug?

Frank demonstrated the problem in an excellent manner using a tiny program like this:
Code:
Use DfAllEnt

Object oMain is a Panel
    Object oForm is a Form
        Set Size to 12 50
        Set Numeric_Mask Item 0 to 10 4
    End_Object

    Object oButton is a Button
        Set Location to 12 0
        Procedure OnClick
            Set Value of oForm Item 0 to 79.84495
            Send Info_Box "Truncating"
            Set Value of oForm Item 0 to 179.84495
            Send Info_Box "Rounding"
        End_Procedure
    End_Object

End_Object

Start_UI
If you run this program, you'll see that in one case it appears to truncate the value, and in the other case it appears to round it. It obviously needs to do something to take 5 decimals down to 4 as specified in the mask, but it seems inconsistent. The truth is that it's complicated, it's not as it seems, and it's not a bug, it's basically the expected behavior from floating point values, which is used internally with the form mask GUI control.

First, to demonstrate this, you will see the same behavior with the following C++ program:
Code:
#include <iostream>
#include <cmath>

int main()
{
    double dblVal=atof("79.84495");
    std::cout<<dblVal<<std::endl;
    dblVal=atof("179.84495");
    std::cout<<dblVal<<std::endl;
    return 0;
}
The output on my machine is:
Code:
79.8449
179.845
Which incidentally is the exact same values seen in the VDF test program with the masked form object. So, what's going on? Is the masking rounding or is it truncating to 4 digits to the right of the decimal point, or what? The short answer is the masking is truncating. But, but, I hear you say, wait, I'll get to that. The longer answer is a little complicated. As explained in here, floating point values are approximations, and not always exact representations of the expected value.

Indeed, the specific value 79.84495 cannot be represented exactly in a double. It's actually represented something close to 79.844949999999997 or similar on my machine. Different code will do different rounding and different truncations. The VDF masking library logic is different from the VDF conversion logic, etc. etc. In the end you end up with somewhat unpredictable results. And that's exactly the central point here, floating point values are approximations, they're not always exact values.

If such very close approximations like 79.84494 vs. 79.84495 are not close enough for your program, then floating point values are not for you. This may seem absurd at first, but it's often less of a problem than it appears. Often the difference between 79.84494 and 79.84495 are negligible. For example, the distance between Los Angeles and San Francisco in fractions of miles is typically negligible in most situations. When performing computer calculations for graphics on screen or printed paper, the tiny inaccuracy introduced by floating point values are often negligible, and the performance gain of floating point calculations usually far outweigh it. One simply has to accept the difficult notion that floating point values are approximations, you cannot chase the goal of exact numbers with floating point, it's futile.

Unfortunately the form mask GUI control is using floating point values (the C double type). That means you will be exposed to these problems with approximate values. It's unfortunate, but luckily it's not a frequent issue with VDF programs. Most people don't use that kind of accuracy together with masking. After all, demanding accuracy while at the same time using a fixed mask which will silently round or truncate seems a bit contradictory.

If you're one of the few people who do calculations and need to display a rounded value in a masked form control, the recommendation would be to perform the rounding in code, rather than rely on the GUI control to round the number for you. This way it will also work correctly without first going in and out of a GUI control to perform a rounding transformation.

So what about the original question, rounding vs. truncation? The masking logic is doing the same thing in both cases, it's just that the actual value is not the value you think it is. For example, 79.844949999999997 becomes 79.84494 and 179.84495000000001 becomes 179.84495. So you see, the problem is that the double type is not able to hold an exact representation of the value you tried to assign, and because of that, the masked value will appear to round/truncate differently in the two separate cases.

Can the form mask behavior be changed? The only way to change this behavior is to change the way the form mask control works, making it use a BCD type or similar instead of floating point. As you can imagine, that's not a small task. The masking/widgets library used by this control was acquired a long time ago, and it's been in use with VDF for well over a decade. Since it's not a frequent issue (most people don't use that kind of accuracy together with masking and rounding), therefore it's unfortunately also not something with a very high priority, and, as with any such major change, the risk of introducing new problems and bugs that will impact far more customers is very high.

In fact, there have been similar other minor tweaks to the automatic type conversion between the VDF Real and String types in the runtime(which is very different from the form mask logic) over the years, they have pretty much all been backed out though. They all resulted in more complaints and issues worse than the original problem. The change may have arguably not been technically incorrect (although whether it was any better was usually debatable), but it changed the behavior of existing program code (which is often very difficult to defend and justify). Basically, we've tried that and there's just no winning in making such kind of tweaks.

Comments

  1. Michael Mullan's Avatar
    Thanks Sonny for the clear and concise description of the reasons for the problem, but what is the recommended fix?

    Is there a DAW supplied rounding function we should be using? I have about 4 of them by different authors, with different limits.

    In my most common case, I have to invoice for fuel, purchased & delivered to 3 & 4 decimal places.. (245.567 gallons @ $2.8764/gal), which then has to be converted to an actually billable amount 706.3489188 ==> $$706.35

    I can see bunches of different ways to handle this, but is there a "normal" way?

    MM.
  2. Garret Mott's Avatar
    Quote Originally Posted by Michael Mullan
    Thanks Sonny for the clear and concise description of the reasons for the problem, but what is the recommended fix?

    Is there a DAW supplied rounding function we should be using? I have about 4 of them by different authors, with different limits.

    In my most common case, I have to invoice for fuel, purchased & delivered to 3 & 4 decimal places.. (245.567 gallons @ $2.8764/gal), which then has to be converted to an actually billable amount 706.3489188 ==> $$706.35

    I can see bunches of different ways to handle this, but is there a "normal" way?

    MM.
    As I read Sonny's blog, you should round the value before masking (maybe before saving too?). As far as using DAW's Round method, I find having to convert a # to an integer, then round, then convert it back to be "fraught with peril" & a lot of work. I realize that there are a # of rounding packages out there, but I have had great luck with one that Dave Martinko wrote that I then tweaked a bit - with his consultation. I thought it was in Open Source, but Search here doesn't turn it up. I'll check with Dave on that if you like.

    I'm currently dealing with a project where I have to have 100% accurate #'s to 4 places, so I'm watching all this very carefully. The numbers are stored in SQL Express as Reals, so I may be in for some fun.....