Assessing Scouting Accuracy For The Last 20 Years (Part I)

Each year, thousands of kids finish their high school years and make the jump to the university ranks. Hundreds of those kids happen to play sports. Some of them have high expectations and dream of becoming professionals and reaching the highest of levels at their sport of preference. Reality though, says that only a handful of them achieve those goals.

Basketball is not the largest of talent-producing sports around the nation. The scholarships handed out each year by NCAA colleges to hoopers are just a few given the limited roster slots in comparison to, let’s say, american football. That makes scouting rankings way important in assessing prospect talent and where to look at.

Speaking of rankings, they have been there forever. Scouts around the United States and overseas have assessed kids’ talents for years and years already. Good for us, there are sites keeping track of those rankings on a yearly basis to let us compare how each service valued players and also to offer an average ranking for each of them.

One of those places is the Recruiting Services Consensus Index (RSCI). It’s telling how you pronounce it: “Risky”. The creators of the platform say that “Given the relative subjectivity of the underlying rankings, this seems to be an appropriate pronunciation.” Not that they are wrong.

Thanks to RSCI and Basketball-Reference.com, it is easy to get a straightforward path to knowing how good on average players were ranked by scouting services back in the day. Every RSCI ranking from 1998 to 2018 can be found on B-Ref website along with information regarding the college each player attended, the year/round/pick/team he was drafted at and by, the years at which he entered the NBA and lastly played a game, and the Win Shares he produced over his career. Keep in mind, then, that by “years pro” I’m referring to the span of years that goes from the first in which the player saw an NBA court and the last one in which he did so, and not the true number of seasons he played in the league. Sadly, that data is not available so we can only guess, although most players complete their runs without long interruptions.

With all of that in place, it is easy to calculate a quick metric that shows the WS produced by the player per season. Just take the whole WS production and divide it by the years he was in the Association. It is not the most complicated and thought of and reliable thing, but it’s enough to have a picture of how well a player performed over his career. It also provides a way to compare players from different classes quickly.

In this series, I will take a look at RSCI rankings and break them down to see how scouts fared while assessing prospect’s talent over the past 20 years, that is, from 1998 to 2017 (we skip the 2018 class as they have yet to go through the NBA draft). In order to do it, I will answer a bunch of questions in each installment of the series, so it becomes a little clearer how scouting has worked lately and how accurate experts have been to try and get some insight from the data.

---

How many players are in the RSCI rankings (1998 to 2017) and how many of them made the NBA?

There are 2001 players in the data set. Every class contains the Top-100 players of that year except that of 2016, which has 101 prospects to its name with two players (Abdul Ado and Myles Powell) tied at the 100th position.

Of the 2001 players, 686 played at least a game in the NBA. Of those 686, 540 entered the league through the draft. Of those drafted, 348 heard their name in the first round while 192 did in the second. The remaining 146 players made it to the pros as undrafted free agents.

Given the volatility of ranking prospects, the fact that experts were correct in putting 686 prospects out of 2001 (around 35%) among the Top-100 of their respective classes is not that bad an outcome. Even with that, though, the low number shows how hard is to assess ability at young ages and how it is very hard to project pro careers at those early stages.

---

How long, on average, does the RSCI-ranked prospect last in the NBA?

If we know something, it is that NBA teams are in for the kill. Pro franchises don’t want to waste draft picks, roster spots, and much less a bunch of millions of dollars. That is why trying to know how long a player’s career will last is of high importance for them.

Given a Top-100 ranking of players the most obvious is that probably most of them would go on to have long professional careers. We’re talking about the cream of the crop, so that should be the case, right? Wrong.

The average NBA career lasts around five seasons. If we only account for players ranked by RSCI from 1998 to 2014 and that played their first professional game in 2014 or earlier (so we don’t count players that haven’t had the chance of playing five seasons yet), we get 454 players. Of the 454, their average pro-playing years sit at 7 years, two above average. Fair number considering we’re talking about the most coveted prospects of their classes.

The problem with that average is that long-tenured players such as Jamal Crawford (19-years pro) and Tyson Chandler (18) make it look much better than what it really is. Here is a more realistic and comprehensive breakdown:

As can be seen, more than half the players don’t reach the “average” seven-season threshold and leave the NBA before reaching that point.

---

Who are the best WS/Yr players in RSCI rankings?

The most important thing about a player is not where he ranks as a prospect, but rather the production he gives his team (or teams) over his career. It doesn’t matter if he was deemed a No. 1 in his class, or if he flew under the radar to ultimately became a superstar. If that last statement checks, the former one renders insignificant.

Without knowing the names on the list you probably have some of them popping in your head. Even if you haven’t followed the recruiting trail, you have heard about coveted recruits back when they were exiting their HS tenures.

LeBron James was getting Sports Illustrated headlines before finishing his prep days. Same with Dwight Howard or Amar’e Stoudemire. They all jumped straight from prep-to-pros. And it is not that kids like Kevin Durant or Anthony Davis wouldn’t have done so too if allowed by the NBA.

All of those players are among the Top-20 when ranked by WS/Yr. Here is the Top-10:

Unsurprisingly, top players were also top prospects. Every one of those in the chart was among the Top-17 prospects in their respective classes, with James Harden being the lowest ranked at No. 17 in 2007, followed by Blake Griffin (2007 No. 16) and LaMarcus Aldridge (2004 No. 12).

More interesting cases can be found buried under those. These five instantly called for my attention:

• 2009 No. 48 // Kawhi Leonard // 8.2
• 2004 No. 47 // Al Horford // 7.1
• 2003 No. 66 // Paul Millsap // 6.8
• 2002 No. 48 // Deron Williams // 6.4
• 2008 No. 95 // Draymond Green // 6.2

As you can see, scouting and ranking prospects is a bit of a lottery. Normally, the highest-ranked kids will go on and fulfill the expectations, but every year there are a bunch of players that will perform way over where they were expected to.

---

Who are some players scouts completely misjudged?

We can treat misjudged players’ abilities as being over or underrated. Let’s try and look for names that fit each case, given that we know how things turned out after the rankings were published.

In order to make this a little more objective than just cherry picking names myself, I calculated a fairly simple metric that accounts for both WS/Yr and the position in the ranking each player was assigned in his class. This way, we would know how over or under the expectations each and every one of the players ranked ultimately performed once they got to the professional ranks.

Of course, LeBron James tops the WS/Yr rank, but it was somewhat expected that he performed at such a level being the No. 1 prospect in 2003. Same goes for Anthony Davis or Ben Simmons, for example. And it works similarly the other way around. Nobody expected Anthony Morrow (No. 91) or Greivis Vasquez (No. 93) to produce much, and indeed they didn’t.

We’re looking for outliers. Low-ranked prospects with great WS/Yr production, and high-ranked prospects that ultimately flopped. The value I’m using to asses this comes from this simple formula: (WS/Yr) * Rank. Let’s call it Prospect Value, or PV. The bigger the number, the better the outcome with relation to the prospect’s rank.

Here are the Top-10 players in ProspectValue with at least 5-yrs of experience:

The highest-ranked player of the field was Al Horford at No. 47. All of them have produced at least 4.4 WS/Yr while in the NBA except Emeka Okafor, who had a WS/Yr production of 3.2 but was ranked No.99 in 2001.

And here are those making the Bottom-10 (at least 5-yrs pro):

There are some truly recognizable names in there. DaJuan Wagner’s (2001 No. 3) career was derailed and after being quite a heralded prospect he never amounted to a lot in the NBA. Donnell Harvey (1999 No. 1) is the best-ranked player of the list with the lowest PV of all. Other No. 1s are Greg Oden (though his career was cut short due to injuries), Josh McRoberts, Shabazz Muhammad, Andrew Wiggins and Austin Rivers.

It can’t be said that those players have not produced at the professional ranks, but the expectations were much higher once they left high school than what they ultimately got to achieve in the NBA. Good for some of them, there is still time to prolong their careers and improve their numbers.

---

That’s it for the first installment of the series.

For this article, I’ve just gone through some basic questions regarding the data set of top recruits ranked since 1998, but keep an eye on this space to keep getting answers to some pressing questions about scouting and recruiting.

We’ve only scratched the surface of the incredible world or prospect analysis and there is much more to come during the following days and weeks. And you surely don’t want to miss it.