Skip to content

Precision

Since method is a numerical property, it has a certain Numerical Precision associated with it with regards to the properties that it calculates, characterizing the degree of numerical approximation.

Example Precision for a Model

If we use Newtonian mechanics as the model, then Precision would be limited by the numerical precision of the number format (eg. float/double) that we use while calculating F = ma.