PDA

View Full Version : Bug: LUPREC affects precision of lookup values



lcamara
2005-06-04, 07:05 PM
While creating a dynamic block with a lookup, I noticed that the values I entered were being rounded down based on LUPREC. This is the expected behavior, except that it wasn't just rounded for display, it was actually rounding the data! Here's how I found that out:

I created a block with some linear parameters. I used a list for the value set (added through the "Add Distance Value" dialog. The values were entered while LUPREC=4 (some values included 1/32 or 1/64 precision). Then when I was entering the values in the lookup table, I used the pull-down to select the values for the table. But, when I was using the lookup pull-down in the block, the block would change to a "custom" lookup configuration for combinations that included values with higher precision. I finally figured out what was happening.

For example, when entering a 1/32 for the linear parameter list (with LUPREC=4), the value being stored in the list was 0.03125, though it was displayed as 0.0313. When I chose the "0.0313" value from the list (in the lookup table dialog), it was actually storing the displayed value (0.0313) instead of the value that was actually stored in the list (0.03125). Then, when I used the reverse lookup on the block, it set the linear parameter to 0.0313, but because I had it constrained to a list, it rounded it to the nearest value (which is 0.03125). That doesn't match what's in the lookup table, so the configuration shows up as "custom".

To fix it, I had to set LUPREC high enough to display the values that I needed (LUPREC=6), then re-entered the problem values in the lookup table. After that, I could set LUPREC back down, and the lookup still worked. So basically, when you're creating your dynamic block's lookup table, you should set LUPREC high enough to display full precision.

According to the definition of LUPREC, it shouldn't have this behavior:


LUPREC
Sets the number of decimal places displayed for all read-only linear units, and for all editable linear units whose precision is less than or equal to the current LUPREC value. For editable linear units whose precision is greater than the current LUPREC value, the true precision is displayed.
Lionel