Rounding practices

[[Jo...]]

I have this conversation about once a month with engineering. Can anyone direct me to the procedure regarding rounding of numbers. The argument is.. if you report to one decimal place you can round off to that place.(when it suits them). Usually the decimal place is done to follow a block tolerance.

If it is profile or position, you used all the tolerance...
I have another part with 12.000 diameter ± .001. Following that logic it could range from 11.9985 to 12.00149 essentially bloating the tolerance range from .002 to .00299, nearly 50%

[[Ri...]]

Oh the rounding discussion. I feel like this gets brought up all the time.

The biggest argument that I hear is that if the tolerance is to 2/3/4 decimal places, that is all you report to because that is what the drawing is conveying to you. Hopefully someone can point me to an ISO or ASME/ANSI standard that confirms this, but all I've ever read is that there is no difference in 0.00100000000000000000000000 and 0.001. I feel like the confusion starts with the block tolerancing.

You are hamstringing yourself if you do not report beyond the 2/3/4 decimal place. The case in point would be your 12 ±0.001 tolerance. If you reported only to the 3rd decimal place, every measurement is only going to show nominal or OOS. Your GR&R would look horrible.

I also hear the argument that the CMM is not accurate the 4th, 5th, 6th, etc decimal place. My counter-argument is that you are confusing accuracy with resolution. They are two different things.


At the end of it all, do whatever your boss says to do, and make sure it's documented.

[[Ri...]]

In addition, there is a rounding function in the result element if you'd like to use that. A lot of extra work if you have a lot of characteristics, but it's there.

[[Jo...]]

Please sign in to view this quote.

That's the problem, an engineer shouldn't dictate metrology. Yes I report to 4 places on the 12.000 dimension, and get instructed to display, and round to 3, because that is the number of places in nominal. It is ridiculous argument, but I haven't found documentation to back of the logical conclusion

[[Ja...]]

ASME Y14.5-2009

2.4 INTERPRETATION OF LIMITS
All limits are absolute. Dimensional limits, regardless
of the number of decimal places, are used as if they were
continued with zeros.

EXAMPLES:
12.2 means 12.20 . . . 0
12.0 means 12.00 . . . 0
12.01 means 12.010 . . . 0

[[Ri...]]

Per ASME Y14.5-2018

[[Je...]]

I have a three word standard answer that seems to work fairly well: Calypso doesn't round.

If they want to round the numbers, they can do so themselves on a separate form.

[[Jo...]]

Please sign in to view this quote.

Oh yeah, I had a condescending exchange regarding truncated vs rounded numbers...

[[Je...]]

Please sign in to view this quote.

😱 Sounds like they truncated his empathy training.

[[Jo...]]

Please sign in to view this quote.

Not really, He thought I was condescending because I stated the definition. I was told my report was misleading because the result was formatted red while the numbers displayed were in tolerance. I considered it misleading to delete the trailing info,.

I'm not a great communicator so I state things twice.
If you say Basic dimensions have no tolerance (they hear wide open)
If you say Basic dimensions have zero tolerance(they hear you contradicted yourself)

[[Ri...]]

"Y14.5 tells us in section 2.4 that all numbers are absolute. In other words, 1 means 1.000000…….forever. Y14.5 also refers us to ANSI/IEEE 268 standard on rounding off. In this standard we are told that you cannot increase the limits of a dimension as a result of rounding "where the original limits may not be violated." Let's say the part thickness shown measures .236. Back in school most of us were told that we could round this to .24. That might be fine in sixth grade, but when measuring parts, it won't pass. Bottom line, the .236 measurement is out of tolerance."

http://www.dimcax.com/gdt_web/august-05.htm

[[Is...]]

If you deliver the data in digital, then you could give the responsibility to round to someone else..

I deliver 4 or more digits in mm to the server and then the users decide if they want to round when they consult the data:

sql.png

[[Mi...]]

For clarity, we report to the engineers at least 10 steps of a tolerance as a standard in house rule, it works for most things.

Your example of 12 +/- 0.001 is only 2 at that resolution (11.999,12.000 and 12.001), so we would definitely report at least the 4th decimal in that case.

If something like a Cap study, GRR, or PPAP is being done we give them the data with full CMM resolution and have them decide how they want to deal with the numbers. It's well understood here that Calypso doesn't round results so we don't have to have that discussion.

[[Da...]]

In ISO 8015:2011 there is a paragraph 5.6 Decimal principle. And it's saying something about infinite number of zeros at the end :)

[[Jo...]]

Thanks to all who contributed. I had this come up again yesterday.

[[Ri...]]

[[Jo...]]

Please sign in to view this quote.

Had this exact argument again today.. You only are required to report to the number of decimal places on print, and can round to that.. I keep getting this discussion, but no one can't show me where that practice is stated.. anyone ? It's hard to argue against , I read it somewhere

I keep seeing people reference the FORMAT section of ASME regarding decimal places on prints, and they infer that also means reporting..

[[Je...]]

Please sign in to view this quote.

.
Send this to the person who is using the phrase "I read it somewhere" to argue their point.


.

.

[[Jo...]]

Unfortunately that response won't work. I'm contesting 6 senior level engineers and plant manager. All who point to ASMEY14.5-2018 5.4 regarding print decimal place formatting being the same as tolerance decimal places. Then construe that to mean I can round to that level.

[[Je...]]

Sorry to hear that, John. That must feel very frustrating.

I would agree with them that both print and tolerance decimal placing is the same, however, reporting of the actual should be at least one place further than the tolerance.

[[Ri...]]

Please sign in to view this quote.

Are you sure they've actually looked at it?

At the end of the day, it's probably not worth arguing. If they want you to round, then round. A lot of work to do though, but it can be done with Result Element.

 round(Number[,NumberDecimalPlaces])

[[Ri...]]

Please sign in to view this quote.

https://portal.zeiss.com/my-voice/softw ... 4f5c4baad8

Please upvote.

[[Jo...]]

Please sign in to view this quote.

See also 5.3.2, that is the format example

[[Ri...]]

Seems like ASME wanted out of this debate.

In 1994 it said this:

Please sign in to view this quote.

In 2009 and beyond they removed the:

Please sign in to view this quote.

[[Da...]]

5.3.2 is speaking of uniformity. It is also saying to ADD zeros as needed. I was taught that the tolerance is the limit, so if it was ±.005 and the actual was .0050005 it has violated the limit.