INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Jobs

Displaying calulated values without losing the precision

Displaying calulated values without losing the precision

(OP)


Debug.? 12345678.12345678# + 123456.67#

So, how do I display the right sum without losing the precision?

Thank you!

RE: Displaying calulated values without losing the precision

When I do:
Debug.Print 12345678.1234568 + 123456.67


I get:
12469134.79345689

What do you get that is wrong...?

Have fun.

---- Andy

A bus station is where a bus stops. A train station is where a train stops. On my desk, I have a work station.

RE: Displaying calulated values without losing the precision

(OP)



Sorry, I should have mentioned that I was testing it in the Debug window (if you'll notice, the values you posted are different from what I posted).
(even though, I still do not get what you have - your last digit is a 9)

Try:

Debug.? CDbl("12345678.12345678") + CDbl("123456.67")

CODE

I get:    12469134,7934568 
Expected: 12469134,79345678 






RE: Displaying calulated values without losing the precision

try the decimal data type.

CODE

Debug.? CDec("12345678.12345678") + CDec("123456.67") 

-George
Microsoft SQL Server MVP
My Blogs
SQLCop
twitter
"The great things about standards is that there are so many to choose from." - Fortune Cookie Wisdom

RE: Displaying calulated values without losing the precision

(OP)

@gmmastros

I guess I was tring to make it too simple, and mislead things in my second post, using the values as strings. I should have used doubles, as in the OP.

That may be fine, using CDec(), if there was a Decimal type.

But, I would need to always store the input values in strings and for each calculation, use CDec() in order for that to work
(there many more calculations to be done).

Please allow me to clear up my mistake:

CODE -->

Dim x As Double
    Dim y As Double
    Dim z As Double
    
    x = "123456789.123456789" 'Store Input from user
    y = "123456.67"  'Store Input from user
    z = x + y
    
    Debug.Print CDec(z) 

RE: Displaying calulated values without losing the precision

Observe:

debug.Print 1234567.12345678#
debug.Print 12345678.12345678#
debug.Print 123456789.12345678#

15 places?

RE: Displaying calulated values without losing the precision

(OP)


I guess so.


CODE -->

X = CDbl(12345678.12345678)
Debug.? x + x + x + x + x + x + x + x 

The values will be calculated right.


Just the displayed value will be cut off and rounded and cannot be shown to a higher precision.

Thanks

RE: Displaying calulated values without losing the precision

(OP)

I guess I could just create the function in a Visual Basic 2005 dll, using the decimal type (29 significant digits) and call that from VB6, passing the results back as a string.

RE: Displaying calulated values without losing the precision

>Please allow me to clear up my mistake:


Please allow me to give you an alternative:

CODE

    Dim x as Variant
    Dim y As Variant
    Dim z As Variant
    
    x = CDec("123456789.123456789")  'Store Input from user
    y = CDec("123456.67")  'Store Input from user
    z = x + y
    
    Debug.Print z 

This has exactly the same accuracy as your idea of using VB 2005 decimal type

RE: Displaying calulated values without losing the precision

(OP)

Yes, prior to my last post yesterday, I saw that on another site, posted by dilettante: http://www.vbforums.com/showthread.php?654076-RESO... and though the Decimal type is also slow, I decided to go with that, rather than using the variant type.

I guess I was, to begin with, just naive concerning this and have since read up some on the IEEE 754 specification.

My thanks to everyone!



RE: Displaying calulated values without losing the precision

The code there was using Variants as containers for Decimal too. There is no separate Decimal type in VB6, only the Decimal subtyped Variant.

When you omit the type you get a Variant. These do the same thing:

CODE

Dim A
Dim B As Variant 

RE: Displaying calulated values without losing the precision

(OP)

Yes, I realize that.

What I meant with
"and though the Decimal type is also slow, I decided to go with that,"
is to do what I said in my post prior to that: To use the stronger Decimal type variable in a VS 2005 dll and run it through that.

Thank you.

RE: Displaying calulated values without losing the precision

>To use the stronger Decimal type variable in a VS 2005 dll

Stronger? Please explain why you think this.

BTW, VB6/VBA documentation on decimal subtype:

96-bit (12-byte) integer numbers scaled by a variable power of 10. The scaling factor specifies the number of digits to the right of the decimal point; it ranges from 0 through 28. With a scale of 0 (no decimal places), the largest possible value is +/-79,228,162,514,264,337,593,543,950,335 (+/-7.9228162514264337593543950335E+28). With 28 decimal places, the largest value is +/-7.9228162514264337593543950335, and the smallest nonzero value is +/-0.0000000000000000000000000001 (+/-1E-28)

And here's what VB.NET says:

96-bit (12-byte) integer numbers scaled by a variable power of 10. The scaling factor specifies the number of digits to the right of the decimal point; it ranges from 0 through 28. With a scale of 0 (no decimal places), the largest possible value is +/-79,228,162,514,264,337,593,543,950,335 (+/-7.9228162514264337593543950335E+28). With 28 decimal places, the largest value is +/-7.9228162514264337593543950335, and the smallest nonzero value is +/-0.0000000000000000000000000001 (+/-1E-28)

Seriously. These are cut'n'pasted from each set of documentation. Can you spot a difference? Of any sort?

RE: Displaying calulated values without losing the precision

(OP)


I did write "stronger Decimal type variable".

Ok, then: "variable declared explicitly as a Decimal type".

But if taken in context to my previous post to that one, I assumed it would be understood that I was referring to my preference in using a variable declared as a Decimal type over that a Variant type.

So, though I wouldn't have imagined that there would have been a problem with what I wrote, I am sorry that there was.

BTW: Is there an explicit definiton of "Strong typing" or "Weak typing" defined somewhere, as I understand you are referring to? (I didn't find any on the MS site, so I looked in the not-always-reliable Wikipedia).




RE: Displaying calulated values without losing the precision

(OP)
I had searched for "Strong Type" instead of "Strong typing" on the MS Site. So now I have found this: Link

RE: Displaying calulated values without losing the precision

>my preference in using a variable declared as a Decimal type over that a Variant type

But you are not. Not in VB6. The fact that it is typed in VB.NET is neither here nor there.

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Resources

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close