×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
• Talk With Other Members
• Be Notified Of Responses
• Keyword Search
Favorite Forums
• Automated Signatures
• Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

#### Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

# real number decimal prob.?

## real number decimal prob.?

(OP)
In our postscript program we have prices that sum up in a total value. Each price is also printed.

When the total and the price is summed we get a automatic round if the value has more than 9999 and two decimals.

{
show
} bind def

this call produces in our form: 31220.0

If we change the value to this:
this call produces in our form: 3122.01
and that is what we was expecting whith the value 31220.01

The problem is when the values are shown as strings there is only one decimal and when the values are summed the decimal is rounded to ceil, so .01 == .1?

the total_price has incorrect value because of the rounding, i dont know postscript that well and dont know if the error comes when the values are passed and converted to strings or if threre is a limit in the real datatype or mabye ghostscript cant handle the values????

Weve tried to use whole integers and when we sould print them divide with 100 to shift the decimal, but we get the same result as above.

Any remark is welcome.
Best regards
/Olof Lager.

"Some thoughts has a certain sound..."

### RE: real number decimal prob.?

I hate this about PostScript. While you can have numbers accurate to about 8 decimal places, each interpreter can do what it wants with regard to precision when performing calcuations with those numbers.

Meaning, you can start with accurate numbers, but as soon as you want to DO ANYTHING WITH THEM, including "add" and "cvs" or even "==", the accuracy defenestrates (that's Latin for "flies out the window").

Keep in mind that PostScript usually uses numbers to specify tints of color, or positions on a page, where such precision as "0.01" doesn't come into play, or if it did, would be visually insignificant.

The bottom line: do your calculations in the program that produces your PostScript, and only output the final, computed numbers as strings in the PostScript.

Final Note: precision does indeed vary from one PostScript interpreter to another. For example, Acrobat Distiller is horrible and doesn't retain much precision in calculations. PSAlter (a PostScript debugger) does a very good job in retaining precision. GhostScript does not do well.

### RE: real number decimal prob.?

Through testing, it appears that decimals are retained in proportion to the whole number component.

For example, compare:

to:

#### Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

#### Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Close Box

# Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

• Talk To Other Members
• Notification Of Responses To Questions
• Favorite Forums One Click Access
• Keyword Search Of All Posts, And More...

Register now while it's still free!