Simulation of GVA and the estimation error of GVA
by DoctorMyBrainHurts on Jul 19, 2010 9:17 PM EDT
8 comments
Sunny Mehta recently suggested that the observed range of goaltender save percentages over the course of a single season could essentially be duplicated by an average goalie and random factors. It dawned on me that, if this were true, that observed goaltender performance metrics ought to similarly be indistinguishable from metrics derived from an average goaltender plus random factors. To explore this, I ran some simulations using a pseudo-random number generator. For these simulations I used an ES save percentage of 0.917, PK save percentage of 0.868, and PP save percentage of 0.914
The range of GVA seen in a single season can be duplicated by an average goaltender and random factors. GVA is a fairly imprecise estimate, with a large standard error and a large confidence interval.
The average NHL team in 2009 faced 2478 shots, 1971 at even strength, 437 on the penalty kill, and 70 on the power play. If a hypothetical starter played 2/3 of his teams minutes, he faced 1314 shots at ES, 291 on the PK, and 47 on the PP. Running 10,000 simulated seasons gives:
GVA
Min. :-45.4900
1st Qu.: -7.4900
Median : 0.5100
Mean : 0.0952
3rd Qu.: 8.5100
Max. : 47.5100
Std. Dev. 11.57677
Thus a 95% CI for GVA is [22.69, -22.69]
Average Goaltender Ryan Miller workload
Miller led the league in GVT. He faced 1690 ES shots, 321 PK shots, and 87 PP shots. 10,000 simulated seasons with an average goaltender facing that workload:
GVA
Min. :-46.8800
1st Qu.: -8.8800
Median : 0.1200
Mean : 0.1159
3rd Qu.: 9.1200
Max. : 48.1200
Std. Dev. 13.10482
Thus a 95% CI for GVA at this workload is [25.69, -25.69]. Miller's GVA was 40.12, which is outside the CI, but within the range generated.
Average Goaltender Vesa Toskala workload
Toskala, on the other hand, had the worst GVT in the league. He faced 540 ES shots, 151 PK shots, and 23 PP shots. 10,000 simulated seasons with an average goaltender facing that workload:
GVA
Min. :-37.3700
1st Qu.: -5.3700
Median : 0.6300
Mean : 0.0421
3rd Qu.: 5.6300
Max. : 30.6300
Std. Dev. 8.026439
Thus a 95% CI for GVA at this workload is [15.73, -15.73]. Toskala's GVA was -21.37, which is outside the CI, but again within the range generated.
Conclusion
The range of GVA seen in a single season can be duplicated by an average goaltender and random factors. Extreme values of GVA do lie outside the 95% CI for average GVA.
As an estimate of goaltender performance, GVA is subject to estimation error. This simulation data suggests that observed GVA is a fairly imprecise estimate, with a large standard error and a large confidence interval.
Acknowledgment
I want to thank Sunny Mehta for his very helpful correspondence.
Addendum
Ryan Miller at Ryan Miller workload
GVA
Min. : 0.1200
1st Qu.: 32.1200
Median : 40.1200
Mean : 39.6262
3rd Qu.: 47.3700
Max. : 83.1200
Std. Dev. 11.79438
95% CI [63.24, 17.00]
Vesa Toskala at Vesa Toskala workload
GVA
Min. :-61.3700
1st Qu.: -27.3700
Median : -20.3700
Mean : -20.0783
3rd Qu.: -14.3700
Max. : 13.6300
Std. Dev. 8.950644
95% CI [-2.54, -37.62]
0 recs |
8 comments
|
Email
Print
Comments
So this means that an average goalie in a good season could replicate single-season numbers from either Miller or Toskala (read: the best and worst goalies in the league)?
(not saying Miller was the best or Toskala is the worst, btw)
Cидни Kросби: Александр Oвечкин, он твой папа теперь
матовая Клими, Михал нуивирт ваш папа теперь
Red Line Station: for Capitals fans who can bear reading something less intelligent than a story at Japers' Rink
Follow me on Twitter!
Or it could mean that the statistical model being used is not a perfect model for modeling goalie performances. Have we witnessed any/many seasons where an otherwise average goalie put up top 5 or bottom 5 goalie numbers?
----------------------------------------------------------------------------------
HockeyAnalysis.com - Taking a Deeper Look at the World of Hockey
by HockeyAnalysis on Jul 20, 2010 11:18 AM EDT up reply actions
Also correct. A model is only a model.
I can’t in general show a particular goalie is better than average in a single season. (Luongo in 2003 comes to mind as an exception) I can show replacement level goalies are worse than average in a single season. As such, I might argue that every season otherwise average goalies populate the top 5 and bottom 5.
by DoctorMyBrainHurts on Jul 20, 2010 12:24 PM EDT up reply actions
What is the consistency of rank ordering between goaltendeders between seasons? If someone is in the top 20% one year, do they tend to stay in that percent next year?
If that were high it would suggest that the current findings are a model specification error. If it were low (as in, the average goalie bounces around randomly in league rank ordering), then it would suggest that there really isn’t much to choose between them (other than the absolute stars and dogs), as your analysis suggests.
There are some metrics for this that might be itneresting, but I’m blowed if I can remember what they are off the top of my head?
Also also correct.
In his original description of VORP, http://www.stathead.com/bbeng/woolner/vorpdescnew.htm, Keith Woolner says "The talent distribution in baseball can be summed up as follows: there are very few “superstar” level players, a somewhat larger number of “average” producers, and a practically unlimited number of “scrubs”. " My data so far suggests that, for NHL goalies, that is also true, and there may be no superstars who are clearly better than the average group.
Do top goalies stay near the top? Somewhat. It’s not totally random one season to the next, but it’s not as consistent as I thought it was when I started digging.
Grant
by DoctorMyBrainHurts on Jul 20, 2010 10:27 PM EDT up reply actions
That is correct. The probability is low (<5%) but not zero.
I agree that they are not necessarily the best or worst. They had the highest and lowest GVT (and GVA). That’s why I picked them.
Sunny’s simulation data convinced me this had to be the case.
http://www.behindthenethockey.com/2010/4/13/1419530/new-jersey-devils-vs-philadelphia
scroll down to the comments and look for “A Study Of One Season”
Grant
by DoctorMyBrainHurts on Jul 20, 2010 9:01 AM EDT reply actions
I don’t quite understand your conclusion. You analyze Ryan Miller, find that he’s over 3 standard deviations above average, and conclude that all goaltenders are average just because he didn’t exceed the best of your 10000 simulations?
Your math is completely correct, but the conclusion seems wrong to me. We didn’t have 10000 goaltenders last year. We had ~30. As such, you should compare your tops and bottoms to the best of ~30 goalies, not 10000.
No, Ryan Miller fell outside the 95% CI for average. As such, I would conclude he was better than average in 2009-2010. Frankly so were Vokoun, Nabokov, Howard, Rask, Bryzgalov, and Halak. Toskala was below the CI. However, there were 84 goalies in 2009-2010. We would expect 5% of them, or about 4, to fall outside the CI. 8 did. That’s in the ballpark.
Now, is my conclusion that Miller is better than average really correct? Maybe. Certainly his GVA is a very low probability event under the null hypothesis. But, even under the null, where all goalies really are the same, I can get outcomes that are as extreme as his GVA.
In Miller’s specific case, his GVA on the PK was +16.37. His ES and PP GVA are in the CI’s for ES and PP. Does he (or any goalie) deserve the credit/blame for PK goals? I’ll get into that more in a subsequent post.
by DoctorMyBrainHurts on Jul 21, 2010 8:41 AM EDT reply actions
Comments For This Post Are Closed
Secondary Sidebar
FanPosts
Community blog posts and discussion.
Certain photos copyright © 2010 by Associated Press or Getty Images. Any commercial use or distribution without the express written consent of Associated Press and Getty Images is strictly prohibited.
