Smaller than milliseconds?

Dar Scott dsc at swcp.com
Fri Jul 11 15:13:00 EDT 2003


On Friday, July 11, 2003, at 12:54 PM, Ray G. Miller wrote:

> Adding three more zeros yielded:
> 1057948840.14027500152587890625000
> 1057948841.41388595104217529296880
> 1057948842.74860799312591552734380
> 1057948848.56447696685791015625000
> 1057948852.67774403095245361328120
> 1057948979.99965500831604003906250
> 1057948984.09295594692230224609380
> 1057948985.36483705043792724609380
> 1057948998.36404299736022949218750
> 1057948999.02833700180053710937500
                    ^
And right about here (depending on how quoting is handled in your mail 
client and whether you use a fixed-pitch font) is a column that 
consists of only 0's and 9's.  I think that means the real resolution 
is 1 microsecond.  The rest is all binary-to-decimal.  The binary 
floating point can only represent values to about 1/9 of a microsecond. 
  The extra digits are the results of the number not being able to get 
closer than 1/9 microsecond.

Dar Scott




More information about the metacard mailing list