numberformat question

Sean Cole (Pi) sean at pidigital.co.uk
Sun Mar 21 07:11:42 EDT 2021


Livecode can handle floating point numbers up to 2^52. But then
calculations would have to fit within that range in a floating point
perspective too. So, once you add that ninth zero to the end of your
divider, it struggles with the calculation and returns, effectively, number
out of range.

Fractions in FP binary are tremendously complicated. Calculation in it are
even more so.

https://www.h-schmidt.net/FloatConverter/IEEE754.html


On Sat, 20 Mar 2021 at 20:23, Paul Dupuis via use-livecode <
use-livecode at lists.runrev.com> wrote:

> I don't have an answer, other than to say there has been a LOT of
> discussion over the years about the limits of number precision in
> LiveCode and a number of enhancement requests and 3rd party libraries
> that have attempted to provide a framework for high precision math. You
> might search the list archives or search the forum on the LC web site
> for past discussions about number precision.
>
> On 3/20/2021 3:06 PM, jbv via use-livecode wrote:
> > Hi list,
> > I am a bit lost :
> >
> >    set numberformat to ".#########################"
> >    put -0.0000007347880794884118999 / 1000
> > -- returns -0.0000000007347880794884119  which is ok
> >
> >    put -0.0000007347880794884118999 / 1000000
> > -- returns -0.0000000000007347880794884 -> also ok
> >
> >    put -0.0000007347880794884118999 / 100000000
> > -- returns -0.0000000000000073478807949 -> still ok
> >
> >    put -0.0000007347880794884118999 / 1000000000
> > -- returns 0 -> ?????
> >
> > Any explanations ?
> > Thanks in advance.
> > Best,
> > jbv
> >
> > _______________________________________________
> > use-livecode mailing list
> > use-livecode at lists.runrev.com
> > Please visit this url to subscribe, unsubscribe and manage your
> > subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
-- 
Pi Digital



More information about the use-livecode mailing list