Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations derfloh on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
INT function - VFP6
- Thread starter foxbldr
- Start date
Looks like a typical rounding problem that all computers using any language can encounter. Basically, since computers do not understand decimals (no one yet has been able to design a five-way switch), all math computations are binary computations which of necessity MUST be rounded. Therefore, by definition, depending on the computer and the language, rounding errors will ALWAYS occur at some point, just depending upon how many significant digits the computer uses in its math computations. It looks like you have found the point at which VFP6 will not always return values as decimal math would expect.
Do a search in tek-tips for lots of discussion of this issue.
Also see thread184-531793.
mmerlinn
"Political correctness is the BADGE of a COWARD!"
Do a search in tek-tips for lots of discussion of this issue.
Also see thread184-531793.
mmerlinn
"Political correctness is the BADGE of a COWARD!"
Try scheme... I've used it to create values over 65,000 digits long, without rounding issues. Takes some time to learn (6 months or so), but VERY useful for reliable, fast, (accomodates recursion, and iterative recursion, and some versions tail drop recursion) is very accurate, and I've seen no problems with rounding.
Best Regards,
Scott
"Everything should be made as simple as possible, and no simpler."![[hammer]](/data/assets/smilies/hammer.gif "[hammer] [hammer]")
Best Regards,
Scott
"Everything should be made as simple as possible, and no simpler."
VFP 9.0 handles large numbers much better. Rounding doesn't take effect until it surpasses this number, using your *100 example:
? INT(67553994410557.88 * 100)
6755399441055788 && 6,755,399,441,055,788
In English, that final number is over 6755 trillion.
? INT(67553994410557.88 * 100)
6755399441055788 && 6,755,399,441,055,788
In English, that final number is over 6755 trillion.
- Thread starter
- #5
Olaf Doschke
Programmer
AFAIK VFP has the same precision since many versions and VFP9 can't store numbers with more precision, as it does not use more bytes than earlier versions. It's using the floating point standard and that hasn't changed for many many years.
You need to take into account the number stored in memory, not the number displayed.
When displaying it on screen behind he scenes this stored number is converted, depending on settings like SET DECIMALS and it's translated to the decimal system while stored in the dual system. Take aside the decimals, even this conversion alone can change the display compared to the number in memory.
In your case 1333365.89 can't be stored exactly in dual system. It's dual representation is slightly lower. In the calculation *100 without INT this slightly difference is not making a difference in the conversion back to the decimal display. You could say the conversion error forth and back compensates each other. If you calculate INT() it's done on the slightly lower dual representation, which rounds it down, remember INT() is not rounding, its cutting decimals.
Help says numeric precision is approximately 15 decimals. This is only true for numbers <1. As numbers are stored as floating point, as the name says the point floats. If a number is higher as 1, the number is kind of normalized to 1 (it's divided by 2 - shifted in the dual representation - while being >1) and an exponent is stored. So with numbers having more than 15 digits in front of the decimal point the precision even ends before it.
And seen from your example it can even end if you only have 10 relevant digits. The error is in the order of the 15 digits precision, so perhaps in memory 1333365.89*100 is something like 133336588.999999. But despite of that precision it is cut down to 133336588 by int() of course. int can't decide that this is sooo close to 133336589 that the user may really want 133336589 depite of the definition of int(), int() has to take the number passed to it for granted.
Even when setting DECIMALS TO 18 you can't get more precision than floating point numbers offer. This makes only sense for numeric fields, in which numbers are stored as c(20) strings. But everytime you calculate it's done with floating point numbers, just the money/currency field type allows a different way of calculating.
Take a look at the topic "Numeric Data Type" of VFP9 and even more details are said at , which is recommended in this help topic.
Bye, Olaf.
You need to take into account the number stored in memory, not the number displayed.
When displaying it on screen behind he scenes this stored number is converted, depending on settings like SET DECIMALS and it's translated to the decimal system while stored in the dual system. Take aside the decimals, even this conversion alone can change the display compared to the number in memory.
In your case 1333365.89 can't be stored exactly in dual system. It's dual representation is slightly lower. In the calculation *100 without INT this slightly difference is not making a difference in the conversion back to the decimal display. You could say the conversion error forth and back compensates each other. If you calculate INT() it's done on the slightly lower dual representation, which rounds it down, remember INT() is not rounding, its cutting decimals.
Help says numeric precision is approximately 15 decimals. This is only true for numbers <1. As numbers are stored as floating point, as the name says the point floats. If a number is higher as 1, the number is kind of normalized to 1 (it's divided by 2 - shifted in the dual representation - while being >1) and an exponent is stored. So with numbers having more than 15 digits in front of the decimal point the precision even ends before it.
And seen from your example it can even end if you only have 10 relevant digits. The error is in the order of the 15 digits precision, so perhaps in memory 1333365.89*100 is something like 133336588.999999. But despite of that precision it is cut down to 133336588 by int() of course. int can't decide that this is sooo close to 133336589 that the user may really want 133336589 depite of the definition of int(), int() has to take the number passed to it for granted.
Even when setting DECIMALS TO 18 you can't get more precision than floating point numbers offer. This makes only sense for numeric fields, in which numbers are stored as c(20) strings. But everytime you calculate it's done with floating point numbers, just the money/currency field type allows a different way of calculating.
Take a look at the topic "Numeric Data Type" of VFP9 and even more details are said at , which is recommended in this help topic.
Bye, Olaf.
- Status
Similar threads
- Locked
- Question
- Locked
- Question