numberFormat question

dfepstein at comcast.net dfepstein at comcast.net
Mon Jun 16 20:13:33 CEST 2014



I am trying to set the numberFormat so that calculation is precise enough for the situation.  Since I don't know ahead of time how many decimal places will be used, I wrote a function that I hoped would adjust things as necessary. 
But when this function is called with parameters m = 1.09131 and n = .0000001 and k = 1, it returns 1.09131 rather than 1.0913101. 
The last couple of lines were added for testing, and the debugger shows that the numberFormat is being set correctly but that the truncated value is put into hold. 
Can anybody see what is going wrong? 



Many thanks. 



David Epstein 



function preciseEnough m,n,k 
   -- return the value m + k*n 
   -- default numberFormat shows up to 6 decimal places 
   -- If m, n, or k  has more than that precision, this function sets the numberFormat with a margin of safety 
   -- before returning the answer 
   put length(m) - offset(".",m) into aPlaces 
   put length(n) - offset(".",n) into bPlaces 
   put length(k) - offset(".",k) into cPlaces 
   put the numberFormat into myString 
   if max(aPlaces,bPlaces,cPlaces) + 4 > length(myString) then 
      get myString & "####" 
      set the numberFormat to it 
   end if 
   put the numberFormat into nf 
   put m + k*n into hold 
   return hold 
end preciseEnough 


More information about the use-livecode mailing list