×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

INCORECT SUM

INCORECT SUM

INCORECT SUM

(OP)
MY PROBLEM:

I made a stored procedure which extract the following values from a table

        UNITPRICE     ITEMQTY
====================== ===========

                 12.52          16
                 59.25           6
                480.70           1
                841.56          10
               8786.63           2

and compute amount with the following formula

amount=(unitprice/1.1)* quantity

having defined amount a numeric(15,2) the stored procedure I created returned the following values

                AMOUNT
======================

                182.05
                323.19
                437.00
               7650.56
              15975.69

Using the statement "select sum(amount) from myproc" to get the sum I got

                   SUM
======================

              24568.48

question:
Given the amount extracted why did I arrived with 24568.48 as sum instead of 24568.49 and how will i arrive with the accurate results?THANKS!:>

RE: INCORECT SUM

Using your figures the amount I get is 24568.504

Here is what I used:

create table test1
(
unitprice numeric(15,2),
itemqty integer
);
commit;

insert into test1 values (12.52,16);
insert into test1 values (59.25,6);
insert into test1 values (480.70,1);
insert into test1 values (841.56,10);
insert into test1 values (8786.63,2);
commit;


select sum((unitprice/1.1)*itemqty) from test1;

Unless you need 15 digits of precision I'd be inclined to
reduce it to numeric(9,2), at least this will avoid the problem where IB turns it into a real when using a
precision > 9

I may be wrong about this its been a few years since I last read the user manual

I tried it with different precisions 9,2 and 7,2 and got the same result so maybe there is something in the sp that
I am missing.

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members! Already a Member? Login

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close