Results 1 to 3 of 3

Thread: Integer Function has a mind of it's own (not truncating reals)

Threaded View

Previous Post Previous Post   Next Post Next Post
  1. #1
    Join Date
    Mar 2009
    Posts
    131

    Angry Integer Function has a mind of it's own (not truncating reals)

    The Integer function has this documentation: "Real and number values are converted to integer values by truncation of the decimal portion of the value". Mostly this is true, but not entirely true. Try this simple code:

    Code:
    Procedure TestReals    
        Real rVal
        Integer iVal
        Move 657434.999999981 to rVal
        Move (Integer(rVal)) to iVal
        Showln "rVal = " rVal " iPos = " iVal
    
    
        Move 657434.999999980 to rVal
        Move (Integer(rVal)) to iVal
        Showln "rVal = " rVal " iPos = " iVal
        inkey windowindex
    End_Procedure
    Send TestReals
    Results in
    Code:
    rVal = 657434.999999981 iPos = 657435
    rVal = 657434.99999998 iPos = 657434
    So anytime the decimal portion of a real is >= 0.999999981 (which is well within the precision of a real) the truncation isn't really a truncation.

    But wait, if you try it out with a lower value (say, 600,000 for instance) it works.....turns out this odd behavior is limited to reals >= 631087 AND to reals <= -631087. I bet you can imagine how fun it was to track this down.

    This can really screw with any precise calculations you have to do. It is very hard to workaround this bug without changing code all over and it happens in all versions I've tested so far.

    OLIVER
    Last edited by flxkid; 7-Sep-2021 at 04:23 PM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •