clock menu more-arrow no yes

Filed under:

Assessing Scouting Accuracy For The Last 20 Years (Part VIII)

How have scouting services fared while assessing prospect abilities? Is there any alignment between prospect rankings and professional career outcomes? We take a look at these and other things thanks to the RSCI database of the last 20 years.

UNLV v San Diego State Photo by Ethan Miller/Getty Images

Welcome back to the exploratory series on recruiting rankings gathered by RSCI!

If you missed the prior parts, fear nothing. Here is a quick recap of the introduction so you don’t feel lost.

What I’m aiming with this series is to get a better sense of how different scouting services have fared in terms of ranking the Top-100 prospect of each of the last 20 classes of hoopers around the nation coming out of high school. For that, I’m using a database based on data from the Recruiting Services Consensus Index (RSCI), which gathered information from multiple scouts and generated composite rankings of recruits for each year. On top of that, using information coming from Basketball-Reference.com, I have access to the Win Shares produced by each player that eventually reached the NBA.

Just armed with that information we can quickly see how rankings and production align, and even generate some metrics such as the simple “Prospect Value (PV)” metric I introduced in the first chapter. It just puts together the ranking of prospects and their production in order to find high-ranked busts or low-ranked diamonds. The higher the PV of a player, the more underrated he was back in the day by scouts.

With all that said, let’s keep surfing the RSCI recruiting data to answer more and more questions!


Ultimately, when it comes down to RSCI rank vs. NBA production, should we really trust the scouts’ assessments?

This is the eight part of the series about the RSCI rankings. After solving a bunch of questions, I have realized I have not yet shown proper conclusions about how good or bad scouts and talent evaluators from different platforms have truly fared.

I had the chance to exchange a few words with the man behind the RSCI tables. Without entering into many details, I can paraphrase something he shared with me without much trouble. Asked about potential questions to solve, he told me that the most repeated questions from people coming to him were always something like this: “Yeah, but what do all these numbers really translate into? Do rankings even matter?”

So here we are today, trying to tackle the holy grail of recruiting, evaluating and assessing basketball prospects’ abilities and rankings.

The easiest way to get a first view of how good RSCI ranks align with NBA production at the professional level of the sport is to use Win Shares and the actual rank of each prospect. With the data set I’m using this is fairly simple and correct, as each class contains the same amount of ranked players (100) and all of those that have made it to the NBA have produced a WS value over their careers.

In order to work with similar prospect-to-prospect values, so career lengths are not factored, I’ll be using WS/Yr instead of raw WS. That way, long-tenured players and one-year-performers may have the same WS/Yr, while the WS of the first could be massive when compared to the latter one.

I’ll also use the WS/Yr values of the prospects at each rank position added together, instead of an average. That prevents a case in which, say, only one player at the 32nd slot made it to the NBA in the past 20 years, yet he was a superstar. He would make for a false high average for the 32nd spot (only one player after all), while others at say the 1st rank position may have an average WS/Yr lower on average because of some hard flops.

Here is the relation between WS/Yr and RSCI Rank:

Turns out, there is a relation. And yes, it is the expected one.

This is not groundbreaking news, though. Scouts and talent evaluators are paid workers and their salaries depend on how good they are at assessing players’ abilities from the earliest of ages. When prospects leave their high schools to make it to the collegiate ranks (or the pros), the most they know about them, the better.

Colleges, more than ever working under the looming possibility of prospects becoming one-and-dones, need to get the best-current-ability players available around the nation just in case they leave after only one year, so they maximize what they put on the court. The same goes for NBA teams. The better the player already is leaving HS, the sooner he would probably become a productive pro.

As can be seen in the chart, the line presents a steady downwards trend from prospects ranked No. 1 to those ranked No. 100. Actually, the R-Squared relation between ranking and WS/Yr per rank slot is up to 0.406 (where a 1 means there is a total relation between both variables). That is such a high correlation, and should be praised.

Here are the values for the top-ranked prospects in terms of WS/Yr added:

  • No. 1 Prospects: 88.6 WS/Yr
  • No. 2 Prospects: 70.75 WS/Yr
  • No. 3 Prospects: 52.37 WS/Yr
  • No. 4 Prospects: 37.71 WS/Yr
  • No. 5 Prospects: 63.99 WS/Yr

Barring the fluke at the fourth position, those numbers are staggering. Out of Top-10 ranked prospects only those at the No. 16 spot get even closer to them and that is because of (mostly) Blake Griffin and Joel Embiid were ranked in that spot in their respective HS classes.

So if you were looking for the ultimate answer, that of if the rankings matter or not: yes, indeed they do.

How does the RSCI-NBA relation translate on individual cases?

Now that we know that there is a relation between the way players are seeing coming out of HS and how they produce in the NBA, we can try and do some fun stuff trying to get to more insights.

First of all, keeping with the relation between the players’ ranking and their production in the NBA (as per WS/Yr) we can plot each individual player and see where they fall. I have filtered out players with less than five years of experience to avoid some outliers.

Here is how things look for those that made the cut (315 players total):

Some takeaways can be gathered just taking a quick glance:

  • The majority of marks are on the left side of the plot, mostly in the first quarter. That means the majority of players ranked by RSCI that have played 5+ years of pro ball have ranked in the Top-25 of their respective classes.
  • Unfilled marks highlight players to not play college basketball. Practically all of those fell inside the Top-10 players of their classes. This is normal as if a kid is not deemed one of the best of his class he’s probably not going to be good enough to jump straight to the professional level.
  • The bigger the circle, the longer the professional career of the player. Again, the bigger circles are mostly on the left side of the chart and also on the upper half of it, given that well-ranked prospects tend to perform better once they reach the NBA.

I remember you talked about some odd-defying cases earlier in the series. Can we plot the players in a way that showcases this information?

Sure, sure we can.

I have introduced some made-up metrics in past entries of the series. One of those tried to measure how a prospect defied the odds and became a better, worse or average player once he reached the NBA after being assigned a place in the RSCI rank of his class.

In order to do that I calculated something simple that I named “Player Value” taking the player’s rank and his WS/Yr production once in the NBA. This is meant to produce a value that would be higher the more a prospect has surpassed expectations.

For example, LeBron James was ranked No. 1 in 2003 and has the highest WS/Yr of the data set (14.2), something we could expect from such a highly ranked prospect. A little lower in the ranks is Chris Paul, who was ranked No. 6 in 2003 yet has the second-best WS/Yr (12.3). That makes Paul’s Player Value 73.5 (multiplying his rank for his WS/Yr) to James’ 14.2, showing how even while ranked lower he still achieved a very good production at the NBA level. In order to have a better idea of where things stand, I then calculated a PV/Yr value for each player given their experience in the league, so everything gets put in context for long-tenured players and also for young players.

Again keeping the field limited to players with 5+ years of experience in the Association, here is how things look:

There are multiple things worth commenting:

  • Almost no prep-to-pros prospects surpassed their expected outcomes. We already know those players normally come to the league expected to be so good from early on, but only the truly best prospects can sustain such a level (think of LeBron James or Dwight Howard). To some extent, the only “successful” cases were those of Enes Kanter, Andrew Bynum, Andray Blatche, and Dorrell Wright.
  • No wonder three players (Kawhi Leonard, Klay Thompson, and Draymond Green) of those with a PV/Yr were part of the 2019 NBA Finals, and another one (Khris Middleton) featured on the best regular season team in 2019. Leonard is the highest-ranked of the bunch only at No. 48 yet he has a WS/Yr production of 8.23. Only 8 more players have a value equal or higher, and none of them was ranked out of the Top-17 on his class.
  • We all know about Draymond Green nowadays, but still, all hail the king at surpassing expectations. Green spent his high school days in Saginaw (MI), and is the only prospect to make it from there to the NBA, even though he was only the 95th-best player of his class. He had to improve his game at Michigan State for a full four-season collegiate career to convince teams he was valuable. Golden State picked him in the second round (35th pick overall). Now, he has three titles and three All-Star appearances to his name and has become a DPOY. Talk about defying the odds.

Focusing only on Top-10 prospects, there are wild variations.

  • The higher the rank, the bunched the pack, which is logical. The more we expect from a kid, the more probable they are to actually be as good as we expect, as scouting aligns with performance. Young LeBron (No. 1) was expected to become an all-time legend, and he checked. Young Chris Paul (No. 6) was expected to be great and became a HoF playing a little bit over the expectations. Jared Jeffries (No. 9) was expected to have good pro production, yet he fell short of the level where evaluators expected him to play at.
  • The case of Steven Adams is that of the Top-10 player to be the most separated from the second-best of his same ranked spot inside the Top-10. He’s got a PV/Yr of 5.3 while the second best is that of Ty Lawson at 3.2, which makes for a 2.1 point difference.
  • No longer part of the NBA, Marquis Teague performed way below what we would expect from a No. 7 nationally-ranked prospect, posting minus-0.15 WS/Yr and a PV/Yr of minus-0.7
  • Although it is still early, Anthony Davis is truly playing above his head while being a No. 1 prospect in his class. Yes, he only has seven years of experience to the 16 of James or the 15 of Dwight Howard, and things will probably stabilize a bit, but he’s definitely on the path to become one of the best prospects/pros ever.

That’s it for today. At the end of the day, the aim of this entry was to once for all settle the main debate under the usefulness of prospects rankings and whether they translate or not into professional production.

And the answer, as couldn’t be otherwise, it is a resounding yes. So keep an eye on prospect big boards from day one because they mostly tell the truth, but also have some fun with them as you can always find the next big thing buried down the list.