[Cuis] Benching Cog (was Re: A few proposed is: additions)

[Phil (list)]

Casey,

On Jun 18, 2012, at 9:54 PM, Casey ransberger wrote:

> Sadly, don't have my machine with me. On Cog, run it two or three times before benching. I can't recall if it's the second or third run that does the trick (I think it's the second,) but Cog doesn't actually start putting JIT'd methods into the poly whosit whatchamacallit cache (PIC) on the first run. It's sort of a stopgap until the work is done to make it smarter about inlining machine code, IIRC.
> 
> The fact that your slowest run was took the same amount of time on the interpreter suggests to me that on your slowest run, the method implementation wasn't JIT'd or cached. Of course I'm a VM noob so don't take my word for it; CC Eliot so he has an opportunity to set me straight (which would be a very nice thing to do!) if I'm wrong here.
> 
> Casey
> 

I can see how reading my posts without seeing the output of the timings could lead you to that conclusion... my response was going on the assumption that you had run the tests on the Squeak VM and were seeing dramatically different (i.e. bad) results, so I followed up with my results on the Squeak VM which probably just muddied the waters.  Cog was quite consistent and was 3x-10x faster than the Squeak VM (it varied by test which is expected).  But there were no surprises comparing one VM vs. the other: Cog is *significantly* faster across the board, so I'm quite happy with Cog's performance :-)  When I ran the same tests on the different VM's, only the absolute performance changed as the relative performance re: old vs. new approach was similar.

To help you see what I'm talking about re: timings, here's the output from a complete run on Cog (the % represents the new vs. old durations which are displayed on the same line).  Each time is the result of 5,000,000 iterations of the test (i.e. they're both pretty fast) comparing old vs. new timings:

Timing tests
=========
ObjectMultiOldIs vs. ObjectMultiNewIs
26% is - ObjectMultiOldIs: 0:00:00:00.664, ObjectMultiNewIs: 0:00:00:00.174
26% isNess - ObjectMultiOldIs: 0:00:00:00.678, ObjectMultiNewIs: 0:00:00:00.182
31% isNess1 - ObjectMultiOldIs: 0:00:00:00.578, ObjectMultiNewIs: 0:00:00:00.18
29% isNess2 - ObjectMultiOldIs: 0:00:00:00.602, ObjectMultiNewIs: 0:00:00:00.178
26% isNess3 - ObjectMultiOldIs: 0:00:00:00.632, ObjectMultiNewIs: 0:00:00:00.17
27% isNessSub - ObjectMultiOldIs: 0:00:00:00.676, ObjectMultiNewIs: 0:00:00:00.184
27% noWay - ObjectMultiOldIs: 0:00:00:00.676, ObjectMultiNewIs: 0:00:00:00.184

ObjectMultiSubOldIs vs. ObjectMultiSubNewIs
25% is - ObjectMultiSubOldIs: 0:00:00:00.696, ObjectMultiSubNewIs: 0:00:00:00.176
26% isNess - ObjectMultiSubOldIs: 0:00:00:00.698, ObjectMultiSubNewIs: 0:00:00:00.184
29% isNess1 - ObjectMultiSubOldIs: 0:00:00:00.616, ObjectMultiSubNewIs: 0:00:00:00.184
28% isNess2 - ObjectMultiSubOldIs: 0:00:00:00.622, ObjectMultiSubNewIs: 0:00:00:00.176
26% isNess3 - ObjectMultiSubOldIs: 0:00:00:00.65, ObjectMultiSubNewIs: 0:00:00:00.172
238% isNessSub - ObjectMultiSubOldIs: 0:00:00:00.072, ObjectMultiSubNewIs: 0:00:00:00.172
26% noWay - ObjectMultiSubOldIs: 0:00:00:00.696, ObjectMultiSubNewIs: 0:00:00:00.184

ObjectNoneOldIs vs. ObjectNoneNewIs
237% is - ObjectNoneOldIs: 0:00:00:00.074, ObjectNoneNewIs: 0:00:00:00.176
242% isNess - ObjectNoneOldIs: 0:00:00:00.076, ObjectNoneNewIs: 0:00:00:00.184
244% isNess1 - ObjectNoneOldIs: 0:00:00:00.076, ObjectNoneNewIs: 0:00:00:00.186
251% isNess2 - ObjectNoneOldIs: 0:00:00:00.074, ObjectNoneNewIs: 0:00:00:00.186
244% isNess3 - ObjectNoneOldIs: 0:00:00:00.076, ObjectNoneNewIs: 0:00:00:00.186
247% isNessSub - ObjectNoneOldIs: 0:00:00:00.076, ObjectNoneNewIs: 0:00:00:00.188
244% noWay - ObjectNoneOldIs: 0:00:00:00.076, ObjectNoneNewIs: 0:00:00:00.186

ObjectRootOldIs vs. ObjectRootNewIs
247% is - ObjectRootOldIs: 0:00:00:00.072, ObjectRootNewIs: 0:00:00:00.178
254% isNess - ObjectRootOldIs: 0:00:00:00.074, ObjectRootNewIs: 0:00:00:00.188
247% isNess1 - ObjectRootOldIs: 0:00:00:00.076, ObjectRootNewIs: 0:00:00:00.188
247% isNess2 - ObjectRootOldIs: 0:00:00:00.076, ObjectRootNewIs: 0:00:00:00.188
254% isNess3 - ObjectRootOldIs: 0:00:00:00.074, ObjectRootNewIs: 0:00:00:00.188
235% isNessSub - ObjectRootOldIs: 0:00:00:00.08, ObjectRootNewIs: 0:00:00:00.188
247% noWay - ObjectRootOldIs: 0:00:00:00.076, ObjectRootNewIs: 0:00:00:00.188

ObjectSingleOldIs vs. ObjectSingleNewIs
204% is - ObjectSingleOldIs: 0:00:00:00.088, ObjectSingleNewIs: 0:00:00:00.18
244% isNess - ObjectSingleOldIs: 0:00:00:00.072, ObjectSingleNewIs: 0:00:00:00.176
208% isNess1 - ObjectSingleOldIs: 0:00:00:00.09, ObjectSingleNewIs: 0:00:00:00.188
211% isNess2 - ObjectSingleOldIs: 0:00:00:00.09, ObjectSingleNewIs: 0:00:00:00.19
208% isNess3 - ObjectSingleOldIs: 0:00:00:00.09, ObjectSingleNewIs: 0:00:00:00.188
204% isNessSub - ObjectSingleOldIs: 0:00:00:00.092, ObjectSingleNewIs: 0:00:00:00.188
208% noWay - ObjectSingleOldIs: 0:00:00:00.09, ObjectSingleNewIs: 0:00:00:00.188

Memory used
==========
111% ObjectMultiOldIs: 92, ObjectMultiNewIs: 103
142% ObjectMultiSubOldIs: 61, ObjectMultiSubNewIs: 87
0% ObjectNoneOldIs: 0, ObjectNoneNewIs: 0
549% ObjectRootOldIs: 61, ObjectRootNewIs: 335
134% ObjectSingleOldIs: 61, ObjectSingleNewIs: 82

Thanks,
Phil