VFP generating incorrect product 1

[rkolva]

I'm working with a collegue on an algorithm that determines the number of record discrepancies between mirrored databases. The algorithm requires generating some fairly large numbers and we're finding that the VFP end is giving us some weird results. eg:
VFP is giving me:
9980587959 * 10000998 = 99815840216783100
but this should be: 99815840216783082

What am I not understanding about VFP?

Thanks,

Ralph

 

[GriffMG]

Ralph,

You are working in very large numbers indeed!

VFP is limited in its numeric accuracy at the extremes:

When storing floating-point numbers in Numeric fields, numeric precision is limited to approximately 15 digits in Visual FoxPro. Therefore, precision more than 15 digits might be lost when converting from decimal to binary numbers, storing numbers with infinitely repeating decimal values in binary, performing multiple repeated operations, and storing numeric values in Character fields and in memory variables in binary format.

Now, I know your numbers are very large integers - rather than floating point numbers - but I suspect with that size of numeric, VFP is using an equivalent of scientific notation ( N x 10^X) - which will tend to loose accuracy in the least significant digits. Hence your answer is a bit out.

I'm curious as to what you are counting? and what the name for a number that big is! Is it a google?

99,815,840,216,783,100



Regards

Griff
Keep [Smile]ing

 

[Mike Lewis]

Griff,

what the name for a number that big is! Is it a google?

Not even close. A google is ten to the power of 100.

Mike


__________________________________
Mike Lewis (Edinburgh, Scotland)

My sites:
Visual FoxPro (www.ml-consult.demon.co.uk)
Crystal Reports (www.ml-crystal.com)

 

[GriffMG]

Thanks Mike,

Still pretty big numbers, and (unfortunately) beyond the scope of a VFP program to count accurately in a 'native' manner.

I suspect to do maths with numbers this large, Ralph is going to have to use strings and develop his own maths functions for handling them.

Slow mind you

Regards

Griff
Keep [Smile]ing

 

[Mike Lewis]

Griff,

I agree. In fact, it is not a VFP problem. Ralph is up against the limit of the floating-point standard.

Mike


__________________________________
Mike Lewis (Edinburgh, Scotland)

My sites:
Visual FoxPro (www.ml-consult.demon.co.uk)
Crystal Reports (www.ml-crystal.com)

 

[GriffMG]

Sorry Ralph,

There are no shortcuts to resolve your problem, the only workaround will be to store your very big numbers in formatted strings, and then write UDFs for adding, multiplying, deducting and dividing...

I DID play for a bit, but do not have time to write a working example today - I might look again over the w/e

Good luck

Regards

Griff
Keep [Smile]ing

 

[wgcs]

Ralph,

You may find this useful:

http://fox.wikis.com/wc.dll?Wiki~BeCarefulWithTheComputationOfLargeNumbers

- Bill

Get the best answers to your questions -- See FAQ481-4875.

 

[rkolva]

Thanks guys,

I was afraid of that. The algorithm generates a pair of integers off of a key field for every live record in a client data base so it's going to get pretty slow eventually as the recordsets grow. I don't think I can justify slowing it down further while leaving the calculations tied into the GUI. Maybe look at making it out of process in which speed wouldn't be so much of a concern. Since the app transmits the int as an XML string maybe C will give me enough wiggle room but as Mike pointed out I may be out of luck there also

Have you encountered any body who has a class for handling large ints as strings as Griff suggested?

Thanks for your help! I always find this forum to have the best help and info.

Thanks again,

Ralph

 

[wgcs]

The reference above on the Wiki links to several functions for dealing with long integers stored in arrays... it would be trivial to combine them into a class and wrap them to convert from strings to the arrays... (unless you meant a C++ class, but, even then, the logic from the Wiki would work well, since "strings" in C++ are basically just arrays...)

- Bill

Get the best answers to your questions -- See FAQ481-4875.

 

[rkolva]

Thanks Bill,

I'll test the MultiPrecision multiplication and report back how it works for me.

Ralph

 

[rkolva]

I would like to you all again for your responses.

For the time being we are looking at making the integers used in the algorithm smaller so that VFP can handle them and building C/C++ that will do the hard calculations and pass VFP back what it needs.

WGCS, thanks for the link to 'MultiPrecision' on wikis but passing the calculation out to another UDF is going to slow things down and then the algorithm needs to mod the integer so I would be back to calling another UDF. This algorithm is called once a day as the app closes and I think all this processing would slow things down just a little too much for our users.

Griff, you were curious as to what we're counting. The idea is based on a paper by Minsky, Trachtenberg and Zippel on set reconciliation using characteristic polynomials (

http://www.cs.cornell.edu/People/egs/syslunch-spring02/

look for set reconciliation). The basic idea is to figure out record discrepancies efficiently in a peer to peer system. Roughly, the vfp end creates matched pair sets that are reverse engineered at the server to create polynomials. The server creates its own polynomials based on records it owns and the ratio of the two expressions/polynomials identifies exactly which records are discrepant. I've got the easy end that I don't need to reverse engineer the polynomials but VFP couldn't handle the large integer calculations.

It will be interested to see if using a C/C++ dll will speed the process up as doing the calculations was the slowest part of the process.

Thanks again,

Ralph

 

[mspratt]

Hi Ralph,

Can you find the number of non-descrepancies and subtract from the RECCOUNT()? Or would that be even a larger number?

Regards,

Mike

 

[rkolva]

Mike,

I can't recall the reason right now but early on I discarded using RecCount(). The seed value for the algorithm needs to be a prime larger than the maximum key value you expect to use. So even with RecCount() if you expected 10k recs in a given table the ints would still get pretty large quickly, particularly by the end of the algorithm. I'm not sure there is a way around potentially exceeding VFP's limit if I stick with it's native arithmetic functions. We were originally thrown off because the first few pairs checked out but the latter ones became increasingly irregular. I would like to work out a solution that doesn't potentially blow up on me a few years down the road.

Thanks for the suggestion,

Ralph

 

[mspratt]

Hi Ralph,

How about breaking the numbers up?

A/2 = C
B/2 = D
X = C*D*2

Regards,

Mike

 

[rkolva]

Sorry about not responding to your last post earlier Mike.

I didn't try your suggestion as I was working on tying in calls to a dll that a collegue created for doing the calcalution. I'm now creating a text (csv) file of the key values (via SQL Select and Copy To) and making a call to the dll. For the 40k records I'm using for testing it's about 7 times faster calling the dll as it was using SCAN... ENDSCAN.

Can't complain at all. Better, faster, we have the capability to create a bionic vfp app for well under 6 million.

If I ever have another need to do some other hard number crunching I'll defintely consider passing it out to a custom dll.

Thanks to all that responded.

Ralph

 

[Olaf Doschke]

Hi Ralph,

There is a library for Foxpro called "math unlimited" from Hilmar Zonneveld at the Universal Thread. Download-ID 10044.

It's making use of arrays. One of the downers is the speed, indeed.

There is a free c compiler called lcc, which I recently used for experimenting with DLLs. It has a nice bignum library, which works with vectors of integers, each worth 30 bit of precision. You can define the precision as you like with the number of integers, that should be concatenated, giving you eg 120 bit precision for 4 integers...

In- and output to a lcc-DLL using the bignum library could be strings of numbers. Speed should be no problem and the math is not limited to basic operators.

http://www.cs.virginia.edu/~lcc-win32/

Bye, Olaf.