INCORECT SUM
INCORECT SUM
(OP)
MY PROBLEM:
I made a stored procedure which extract the following values from a table
UNITPRICE ITEMQTY
====================== ===========
12.52 16
59.25 6
480.70 1
841.56 10
8786.63 2
and compute amount with the following formula
amount=(unitprice/1.1)* quantity
having defined amount a numeric(15,2) the stored procedure I created returned the following values
AMOUNT
======================
182.05
323.19
437.00
7650.56
15975.69
Using the statement "select sum(amount) from myproc" to get the sum I got
SUM
======================
24568.48
question:
Given the amount extracted why did I arrived with 24568.48 as sum instead of 24568.49 and how will i arrive with the accurate results?THANKS!:>
I made a stored procedure which extract the following values from a table
UNITPRICE ITEMQTY
====================== ===========
12.52 16
59.25 6
480.70 1
841.56 10
8786.63 2
and compute amount with the following formula
amount=(unitprice/1.1)* quantity
having defined amount a numeric(15,2) the stored procedure I created returned the following values
AMOUNT
======================
182.05
323.19
437.00
7650.56
15975.69
Using the statement "select sum(amount) from myproc" to get the sum I got
SUM
======================
24568.48
question:
Given the amount extracted why did I arrived with 24568.48 as sum instead of 24568.49 and how will i arrive with the accurate results?THANKS!:>
Talk To Other Members
RE: INCORECT SUM
Here is what I used:
create table test1
(
unitprice numeric(15,2),
itemqty integer
);
commit;
insert into test1 values (12.52,16);
insert into test1 values (59.25,6);
insert into test1 values (480.70,1);
insert into test1 values (841.56,10);
insert into test1 values (8786.63,2);
commit;
select sum((unitprice/1.1)*itemqty) from test1;
Unless you need 15 digits of precision I'd be inclined to
reduce it to numeric(9,2), at least this will avoid the problem where IB turns it into a real when using a
precision > 9
I may be wrong about this its been a few years since I last read the user manual
I tried it with different precisions 9,2 and 7,2 and got the same result so maybe there is something in the sp that
I am missing.