Advanced Search

String to Number conversion generates unexpected decimals, fix by toPrecision

/ Published in: ActionScript 3
Save to your folder(s)
This seems to be a bug.. or not..
But explains how to fix it anyway.

Expand | Embed | Plain Text

Copy this code and paste it in your HTML 
  1. var a:Number = 0.99;
  2. var s:String = String(a);trace(s); // 0.99
  3. s = String(a/100); trace(s); // 0.009899999999999999 // 8 ?
  4.  
  5. var p:String = "0.99";
  6. trace(Number(p)/100); // 0.009899999999999999 // 8 ?
  7. trace(Number(p)); // 0.99
  8.  
  9. trace(Number(String(Number(p)/100))*100); // 0.9900000000000001  // 1 ?
  10.  
  11.  
  12. trace(   Number(Number(p).toPrecision(8))/100    ); // 0.009899999999999999
  13.  
  14. // fix - as I need the first 2 decimals... problem is solved
  15. trace((Number(p)/100).toPrecision(8)) // 0.0099000000
  16.  
  17. var f:String = (Number(p)/100).toPrecision(8); 
  18.  
  19. // seems fixed
  20. var fs:Number = Number(f); trace(fs); // 0.0099
  21.  
  22. // but also
  23. trace(fs*100); // 0.9900000000000001

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.