Why am I getting this value? 1

[snowcold]

I am retrieving the database value with this statement:

Public Function gGetFieldData(ByRef dynToUse As Object, ByVal strFieldName As String, ByVal udeDataType As cpcDataType, Optional ByVal udeDataFormat As cpcDataFormat) As Variant
                            
    Dim varToReturn As Variant
    
  Select Case udeDataType         
   Case cpcdouble
     If Not IsNull(dynToUse.Fields(strFieldName).value)
     Then
        'we have an double so use it
         varToReturn = CDbl(Val(dynToUse.Fields(strFieldName.value))
     Else
         'field not used so default to 0
          varToReturn = CDbl(0)
      End If
      
    Case Else
         'if we add more types
   End Select
    
    'return the data
    gGetFieldData = varToReturn
End Function

then I am using to the returned value here:

dblMinus = dblNominal - gGetFieldData(dynLimits, "Minimum", cpcdouble)

Here is where the issue begins. When I step through it.
dblNominal = 14 and gGetFieldData() is returning 13.8.

My math tells me that 14-13.8 = .2 but the I keep getting a .19999999999999...


The database field is stored as a Number...

Any ideas?

 

[ThatRickGuy]

What database are you running? I have a 3rd party app running on Sybase that I have to pull data from and I am constantly running into things like that.

You can use math.Round(value, digits) to make sure you are getting an appropriate number. using Math.Round(0.199999999999, 8) should return .2.

-Rick

VB.Net Forum forum796 forum855 ASP.NET Forum
I believe in killer coding ninja monkeys.

 

[chrissie1]

the datafield is probably stored as a floating point (double, float,...). So these kinds of things are pretty normal. Try saving it as a decimal if you have that option or something less floating.

Christiaan Baes
Belgium

"My new site" - Me

 

[snowcold]

Thanks for the replies...

ThatRickGuy: It's an Oracle 9.2 Db.

chrissie1: Yes the field is stored as a double...It appears that I cannot save it as a decimal. Something less floating? any suggestions?

 

[ThatRickGuy]

How about a single?

-Rick

VB.Net Forum forum796 forum855 ASP.NET Forum
I believe in killer coding ninja monkeys.

 

[snowcold]

I just checked it out....It appears that single does work.
I need to recheck the different datatypes. I was thinking a single would not work with a value like .#

Thanks,

 

[ThatRickGuy]

The only difference between single and double is the size of the number. A double is 8 bytes (64-bit), a single is 4 bytes (32-bit). That 4 or 8 bytes stores the number, a positive/negative indicator and an indicator that determins where the decimal point should go (that's why it's called a floating point).

-Rick

VB.Net Forum forum796 forum855 ASP.NET Forum
I believe in killer coding ninja monkeys.