MathLib.precision returns the maximum number of digits that are possible in a variable value.
eglx.lang
static function precision(value smallint in) returns(int); static function precision(value int in) returns(int); static function precision(value bigint in) returns(int); static function precision(value float in) returns(int); static function precision(value smallfloat in) returns(int); static function precision(value decimal in) returns(int);
After the following code runs, the value of result is 9:
result, myVar INT; result = mathLib.precision(myVar);
In EGL, precision is the total number of digits a variable value, not just the number of decimal places. For example, the precision of an INT value is 9. For floating-point numbers, the precision is the maximum number of digits that the number can represent on the system on which the precisionic is running.
In the case of Java™, EGL uses methods in the Java StrictMath class that are equivalent to the EGL functions. This usage ensures that the runtime behavior is the same for every Java Virtual Machine.
| Target | Issue |
|---|---|
| Java | No issues. |
| JavaScript | No issues. |