Integer to Double Comparision

[EagleTempest]

VB6

Hello, shouldn't 15=15? My program allows for both Metric and Imperial input but I want the Imperial display shown as 15'-6 3/8". This I can accomplish but I notice if I convert from Imp. to Met to Imp, I sometimes lose one inch.

I made a test module which can be run in a blank project check it out.

Private Sub Form_Load()
  Dim intInches As Integer
  Dim dblInches As Double
  Dim dblDifference As Double
  
  'Convert 24inches to mm and back to inches
  dblInches = 24 * 25.4
  dblInches = dblInches / 25.4
  
  intInches = 24
  
  dblDifference = dblInches - intInches

  Debug.Print dblDifference

  
End Sub

The result is -3.5527136788005E-15.

Obviously I could stick the Round function in but I would think that shouldn't be nessessary.

 

[johnwm]

Floating point representations will - in general - be approximations. That's why you should never do comparisons on Singles or Doubles. If you want precision on non-integer values look into using the Scaled Integer Data Types. There is Currency and Decimal datatypes. VBHelp contains definitions and examples of use

________________________________________________________________
If you want to get the best response to a question, please check out FAQ222-2244 first

'If we're supposed to work in Hex, why have we only got A fingers?'

 

[JeffTullin]

>>>The result is -3.5527136788005E-15.<<<
A very very small number.
This is the different between storing things as integers and string them as doubles.

-0.0000000000000035527136788005 is pretty close to zero, and that is how you should compare doubles.

eg

if abs(dblInches - intInches) < 0.001 then
      'close enough for rock'n'roll
end if

Stick it in a function if you need it a lot.

function AreRoughlyEqual(v1 , v2) as boolean
AreRoughlyEqual = (abs(v1- v2) < 0.001 )
end function

 

[EagleTempest]

Thanks for the help. I forgot about the ABS function.