Precision (computer science)
- Precision (computer science)
In computer science, precision of a numerical quantity is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to precision in mathematics, which describes the number of digits that are used to express a value.
In Java, one of the few programming languages with standardized precision data types, the following precisions are defined for the standard integer numerical types of the language. The ranges given are for signed integer values represented in standard two's complement form.
Rounding error
Precision is often the source of rounding errors in computation. The number of bits used to store a number will often cause some loss of accuracy. An example would be to store "sin(0.1)" in IEEE single precision floating point standard. The error is then often magnified as subsequent computations are made to the data (it can also be reduced).
See also
* Arbitrary-precision arithmetic
* Precision (arithmetic)
* IEEE754 (IEEE floating point standard)
Wikimedia Foundation.
2010.
Look at other dictionaries:
Integer (computer science) — In computer science, an integer is a datum of integral data type, a data type which represents some finite subset of the mathematical integers. Integral data types may be of different sizes and may or may not be allowed to contain negative values … Wikipedia
Abstraction (computer science) — In computer science, abstraction is the process by which data and programs are defined with a representation similar to its pictorial meaning as rooted in the more complex realm of human life and language with their higher need of summarization… … Wikipedia
Scale factor (computer science) — A scale factor is used in computer science when a real world set of numbers needs to be represented on a different scale in order to fit a specific number format. For instance, a 16 bit unsigned integer ( uint16 ) can only hold a value as large… … Wikipedia
Dept. of Computer Science, University of Delhi — संगणक विज्ञान विभाग Established 1981 Students 200 Location New Delhi, India Campus … Wikipedia
Career domains in computer science — The knowledge developed by academic computer science (CS) is applied to various non academic situations to arrive at systems that help humans perform tasks that were either out of their reach for being too complex or the tasks that are repetitive … Wikipedia
Precision — has the following meanings:;Concepts * Accuracy and precision, the degree of mutual agreement among a series of individual measurements or values * Precision (arithmetic), the number of digits with which a value is expressed * Precision… … Wikipedia
Namespace (computer science) — For namespaces in general, see Namespace. A namespace (sometimes also called a name scope) is an abstract container or environment created to hold a logical grouping of unique identifiers or symbols (i.e., names). An identifier defined in a… … Wikipedia
Computer-assisted orthopedic surgery — or computer assisted orthopaedic surgery (sometimes abbreviated as CAOS[1]) is a discipline where computer technology is applied pre , intra and/or post operatively to improve the outcome of orthopedic surgical procedures. CAOS is an active… … Wikipedia
computer — computerlike, adj. /keuhm pyooh teuhr/, n. 1. Also called processor. an electronic device designed to accept data, perform prescribed mathematical and logical operations at high speed, and display the results of these operations. Cf. analog… … Universalium
Computer — For other uses, see Computer (disambiguation). Computer technology redirects here. For the company, see Computer Technology Limited. Computer … Wikipedia
