×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

I'm pretty sure this math is wrong....

I'm pretty sure this math is wrong....

I'm pretty sure this math is wrong....

(OP)
I'm scraping the screen to obtain some monetary values, annoyingly listed with commas, so I'm reading each "section" and appending them, skipping the commas.

I'm then adding them together and displaying them in a message box.

It seems that Extra basic can't add up!?

This is the code:

CODE -->

net=Csng(trim((Sess0.Screen.GetString(6,66,3)+Sess0.Screen.GetString(6,70,3)+Sess0.Screen.GetString(6,74,6))))  
                        pen=Csng(trim((Sess0.Screen.GetString(7,66,3)+Sess0.Screen.GetString(7,70,3)+Sess0.Screen.GetString(7,74,6))))
                        fee=Csng(trim((Sess0.Screen.GetString(8,66,3)+Sess0.Screen.GetString(8,70,3)+Sess0.Screen.GetString(8,74,6))))
                        arr=Csng(trim((Sess0.Screen.GetString(14,66,3)+Sess0.Screen.GetString(14,70,3)+Sess0.Screen.GetString(14,74,6))))
                        arrexvat=(arr/1.2)
                       
                        cust=(net+pen+fee+arrexvat)
   
                        message$=(str$(net)+" "+str$(pen)+" "+str$(fee)+" "+str$(arr)+" "+str$(arrexvat)+" "+str$(cust))  
               
                        MsgBox message$ 

The output values in the message box on a test run are:

3487.97
0
100
0
0
3587.96997070313

How is 3487.97+100 resulting in a load of decimal places?!

Thanks in advance!

RE: I'm pretty sure this math is wrong....

hi,

When you are performaing FLOATING POINT MATH, there is a known loss of precision. It's a fact that you have to 1) recognize and 2) deal with.

If you want precision to a certain number of decimal places, then make appropriate conversion and use INTEGER MATH.

Skip,

glassesJust traded in my old subtlety...
for a NUANCE!tongue

RE: I'm pretty sure this math is wrong....

Hi,

As an alternative to the commas, you can do this

CODE

data = Sess0.Screen.GetString(6,66,14)
net=trim(Getfield(data,1,",") + Getfield(data,2,",") + Getfield(data,3,","))

data = Sess0.Screen.GetString(7,66,14)
pen=trim(Getfield(data,1,",") + Getfield(data,2,",") + Getfield(data,3,","))

data = Sess0.Screen.GetString(8,66,14)
fee=trim(Getfield(data,1,",") + Getfield(data,2,",") + Getfield(data,3,","))

data = Sess0.Screen.GetString(14,66,14)
arr=trim(Getfield(data,1,",") + Getfield(data,2,",") + Getfield(data,3,",")) 
don't know if it is better or worse, just another way to do it. Getfield seperates a string at the character given and without including it.

If you use format, Extra will round for you

CODE

cust=format((net+pen+fee+arrexvat),"#.##") 

And Skip is right. Be careful when using floating point. You can get some nasty surprises if you don't handle it.

Lakare

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members! Already a Member? Login

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close