Weird? Cell shows fractions...

User avatar
ErikJan
BronzeLounger
Posts: 1258
Joined: 03 Feb 2010, 19:59
Location: Terneuzen, the Netherlands

Weird? Cell shows fractions...

Post by ErikJan »

Maybe I'm getting rusty...

With .Offset(CurRow, 8)
.Value = Int(MyRecord.TempI * 10) / 10
End With

When the TempI element is 24.8 I see it in my cell as 24.7999992370605 (even with the extra rounding I added above)... What is this???

User avatar
HansV
Administrator
Posts: 78629
Joined: 16 Jan 2010, 00:14
Status: Microsoft MVP
Location: Wageningen, The Netherlands

Re: Weird? Cell shows fractions...

Post by HansV »

It's a rounding problem. Has Templ been defined as Single? If so, try defining it as Double instead. The greater precision of the Double data type reduces the incidence of rounding errors (but it doesn't eliminate them entirely, computing precision is always finite).
Best wishes,
Hans

User avatar
ErikJan
BronzeLounger
Posts: 1258
Joined: 03 Feb 2010, 19:59
Location: Terneuzen, the Netherlands

Re: Weird? Cell shows fractions...

Post by ErikJan »

So the INT(x*10)/10 doesn't fix that???

User avatar
HansV
Administrator
Posts: 78629
Joined: 16 Jan 2010, 00:14
Status: Microsoft MVP
Location: Wageningen, The Netherlands

Re: Weird? Cell shows fractions...

Post by HansV »

Nope - the Single data type doesn't store the number 24.8 with sufficient accuracy. Single has a precision of about 7 significant digits, while Excel internally uses the Double data type, which has about 15 significant digits. So if you use Single in your VBA code, you're bound to see rounding errors when you store values in cells.
Best wishes,
Hans