String to Number conversion generates unexpected decimals, fix by toPrecision

Revision: 37713

at December 15, 2010 18:52 by godstroke

Initial Code

var a:Number = 0.99;
var s:String = String(a);trace(s); // 0.99
s = String(a/100); trace(s); // 0.009899999999999999 // 8 ?

var p:String = "0.99";
trace(Number(p)/100); // 0.009899999999999999 // 8 ?
trace(Number(p)); // 0.99

trace(Number(String(Number(p)/100))*100); // 0.9900000000000001  // 1 ?


trace(   Number(Number(p).toPrecision(8))/100    ); // 0.009899999999999999

// fix - as I need the first 2 decimals... problem is solved
trace((Number(p)/100).toPrecision(8)) // 0.0099000000

var f:String = (Number(p)/100).toPrecision(8); 

// seems fixed
var fs:Number = Number(f); trace(fs); // 0.0099

// but also
trace(fs*100); // 0.9900000000000001

Initial URL

Initial Description

This seems to be a bug.. or not..
But explains how to fix it anyway.

Initial Title

String to Number conversion generates unexpected decimals, fix by toPrecision

Initial Tags

number

Initial Language

ActionScript 3

Choose a language for easy browsing: