numberformat question

[jbv at souslelogo.com]

Paul, Sean,

Thank you both for your answers, although it doesn't really solve my 
problem.
The most surprising thing is that javascript runs these calculations 
without
problem :
(-0.0000007347880794884118999 / 1000000000) returns 
-7.347880794884119e-16
(-0.0000007347880794884118999 / 1000000000000) returns 
-7.347880794884119e-19

Best


Le 2021-03-21 07:11, Sean Cole (Pi) via use-livecode a écrit :
> Livecode can handle floating point numbers up to 2^52. But then
> calculations would have to fit within that range in a floating point
> perspective too. So, once you add that ninth zero to the end of your
> divider, it struggles with the calculation and returns, effectively, 
> number
> out of range.
> 
> Fractions in FP binary are tremendously complicated. Calculation in it 
> are
> even more so.
> 
> https://www.h-schmidt.net/FloatConverter/IEEE754.html
> 
> 
> On Sat, 20 Mar 2021 at 20:23, Paul Dupuis via use-livecode <
> use-livecode at lists.runrev.com> wrote:
> 
>> I don't have an answer, other than to say there has been a LOT of
>> discussion over the years about the limits of number precision in
>> LiveCode and a number of enhancement requests and 3rd party libraries
>> that have attempted to provide a framework for high precision math. 
>> You
>> might search the list archives or search the forum on the LC web site
>> for past discussions about number precision.
>> 
>> On 3/20/2021 3:06 PM, jbv via use-livecode wrote:
>> > Hi list,
>> > I am a bit lost :
>> >
>> >    set numberformat to ".#########################"
>> >    put -0.0000007347880794884118999 / 1000
>> > -- returns -0.0000000007347880794884119  which is ok
>> >
>> >    put -0.0000007347880794884118999 / 1000000
>> > -- returns -0.0000000000007347880794884 -> also ok
>> >
>> >    put -0.0000007347880794884118999 / 100000000
>> > -- returns -0.0000000000000073478807949 -> still ok
>> >
>> >    put -0.0000007347880794884118999 / 1000000000
>> > -- returns 0 -> ?????
>> >
>> > Any explanations ?
>> > Thanks in advance.
>> > Best,
>> > jbv
>> >
>> > _______________________________________________
>> > use-livecode mailing list
>> > use-livecode at lists.runrev.com
>> > Please visit this url to subscribe, unsubscribe and manage your
>> > subscription preferences:
>> > http://lists.runrev.com/mailman/listinfo/use-livecode
>> 
>> 
>> _______________________________________________
>> use-livecode mailing list
>> use-livecode at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>