Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations bkrike on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

Mutiplication iof two high precision fields still leads to inaccuracie

Status
Not open for further replies.

eo

MIS
Apr 3, 2003
809
SQL Server 2005:

I have two fields which are decimal (38,15). In one data set I have 628 records, in which I multiply FieldA (decimal 38,15) with FieldB (decimal (38,15), to obtain calculated FieldC.

Adding all the 628 FieldC's should deliver a total of 10,000, but delivers a total of 9996.738063

How can I apply a greater level of precision?

EO
Hertfordshire, England
 
eo said:
How can I apply a greater level of precision?
Actually, it sounds like you want it to be less precise, since you seem to want it to round up.

Probably you need to round the individual numbers before you sum them.

Joe Schwarz
Custom Software Developer
 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top