Why Are CFA Pass Rates Declining?

08/09/2012

Credit: xkcd.com

So June 2012 CFA season is fully over! Congratulations to those who passed, and to those who didn't make it this time - come back fighting and you'll get it in the end!

Every year while waiting for the results, candidates speculate on the pass rate. And every year, most candidates expect it to be lower than the past years.

And for good reason - the pass rates have been declining since... well, since the start, really. There is a general consensus within candidates and charterholders alike that the exams have been getting harder, and that's probably a significant factor. I did some quick & dirty analysis this week & would like to offer an additional possibility:

The increase in CFA candidates is a significant factor in declining pass rates.

The CFA Institute publishes on their website all the pass rates for the CFA exams since its inception in June 1963. Here's the data for all 3 levels, presented in interactive-graphical glory. Drag the markers in the blue timeline at the bottom of the chart to zoom in and out, or alternatively scroll.

As you can clearly see, the pass rates have taken a swing downwards from above 80% in 1963 to 40%-50% today.

Comparing this with candidate volume, it seems that a possible significant factor is simply that more candidates are participating. Below is the number of candidates that took the exam over the same timeline. (Note: The dip in 2003 for Level I candidates is due to the introduction of the December exam)

Although pass rates have been declining somewhat before that, the large declines in pass rates happen after the early 90's, which coincides with an exponential increase in candidates. So a highly plausible factor is that average candidate quality has been lowered by the increasing interest and candidate volume.

Taking it a step further, below is the scatter chart of candidate numbers to pass rates year by year, and the corresponding R-squared values:

As you can see, there's some good correlation evidence there. Not perfect, but obviously declining pass rates can't be purely down to increasing candidate volume.

So while the increasing difficulty of the exams may well be a factor, it may simply be a function of the large candidate pool and decreasing average performance. I suppose this analysis doesn't necessarily disprove that pass rates are causing the candidate increase - that the CFA Institute may be running a sinister conspiracy to lower the pass rates to encourage multiple retakes - hence increasing the candidate volume... but I doubt it.

What do you think - do you have your own theories & explanations? New harder topics being introduced, policies, marking systems? Let me know in the comments below!

I've put up the charts in a permanent page - Historical Data under 'Analyze Results' - feel free to browse and suggest more charts to include in that section!

Other trending posts:

Comments

Matty J

08/09/2012 9:43pm

Lovely piece of analysis, especially the scatter chart which takes it a step further than the standard data available on the CFA website. The R-squared numbers suggest there is a definite correlation here. What would be interesting (though I think impossible to calculate) would be to adjust the figures to isolate a) first time takers at each level; and b) average number of attempts at each level.

Great read though!

Ash

08/09/2012 10:15pm

A very good read!
I have a question though.. In the first graph, what does the blue line under the Level I, II & III lines represent? I initially thought that THAT was the volume line?

x link

08/09/2012 10:29pm

Hi Ash! The blue graph is actually to represent the timeline - there are markers on either side that you can drag to zoom in and out. As for what the actual line represents, it's the Level III line.

Matty J, unfortunately CFA Institute doesn't publish first time takers or retaker details, but they have mentioned that it takes an average of 4 years for the average charterholder to work through all 3 levels, which implies an average of 2 retakes...

Chris Sarda link

08/10/2012 4:16am

Well I sure hope this is the case rather than the test getting more difficult. If it's just that more people are taking it then I'm not as afraid.

Alex Jenkins

08/10/2012 6:54pm

I like the chart and I understand why you came to that conclusion... but I think the answer is fall more simple. Just take a look at the 2007 Level 3 CFA Exam compared to the last two years. That's a four year difference yet the exam was easier by leaps and bounds.

But I do think the larger number of candidates plays a part. I think that as the number of candidates increased, CFAI realized they had to either increase the difficulty level or lower the minimum passing score. Otherwise there would be too many charterholders and it would cheapen the designation (thus making CFA and CFAI irrelevant). Just my 2 cents.

FlyingAfrican

08/11/2012 4:53pm

As Alex mentioned it is some combination of both.

It's much like the Chartered Accountancy exam in the commonwealth.

My professor was saying when he did his CA, there were 2 accounting standards to worry about, now there are in excess of 30. # of candidates have also increased, so pass rates have been trending down too.

Makes you wonder, if in 50 years, accounting and finance is so complicated that you can only write for one specialised topic.

Mace Raydar

01/16/2013 2:11pm

What is so ironic about the CFA is that the final decision that gets made on the minimum passing score is made by the board of directors. Why would this be ironic? Because they are very very highly compensated based on the income the CFA takes in. If you think they haven't calculated the percentage increase or decrease in CFA revenue for every percentage increase or decrease in the MPS then you're crazy. They know the retake rates, they know they stand to lose a lot of money if they raise the MPS. This model of ethical superiority is such a scam to cover up their complete lack of governance. Why do you think it is so non transparent? Compare it to the practices of every other major certification exam out there and it becomes obvious that there are fradulant activities going on here. There is no way the pass rate should be higher for the Chemical Engineer exams or the MCAT. CFA is a very clever scam indeed.

Mac Gensen

01/24/2013 3:38am

Seriously I am always surprised that no one ever brings up the serious governance issues with the grading of this test. All three elements of the fraud triangle are clearly there. The ones you stand to make money off of the MPS get to pick it... really? They don't even decide how to score the questions until they see a statistical breakout of which answer everyone chose.

Opportunities (incentives): this one is obvious $$$$$ massive incentive for them to continuously lower the MPS. I'm talking hundreds of millions a year with almost no overhead, plus member fees! This test is seriously profitable.

Pressures: they have to keep it elite. It's like a fraternity that brutally hazes their pledges. The same was done to them so they do it to the next generation because "we just can't let anybody in. right?" By continuously lowering the MPS it makes their own accomplishments (the CFA designation) seem better and better because the test keeps getting harder and harder. Even though the pass rate was much higher when they took it... has finance really gotten that much harder over the years? I think the grader is the one getting harder.

Rationalization: ETHICS, ETHICS, ETHICS!!! We are the highest ethical establishment in the land! Our moral superiority is renowned, we are so careful to grade this test especially ethical, see all the test monitors, all the cops, this test is so fair and unbiased, look at all these rules.

Read the test grading procedures it's all there. They have all CFAs grading the exam, no outside consulting supervising whatsoever. Wow what great governance, ok so it gets better, each group of CFAs have a CFAI staff member (on the payroll) to oversee and "guide" their decision making process. Ultimately the CFAI member has to bless whatever decision they make about the question. Oh and each question doesn't have a right answer until they get to see how many people picked each answer, convenient.

Is it really all that shocking? Bankers and finance sharks would never create a clever way to get money out of people... these people aren't stupid, they know the pass rates, they did the math, lowering the MPS = raising the revenues.

Is it really all that shocking?

X link

01/24/2013 12:25pm

Hi Mace Raydar and Mac Gensen

I do take your points. Without taking sides, I'm going to try and comment on this. I certainly do not speak for CFA Institute nor are affiliated with them, but I have some experience and knowledge of exam practices and will air my thoughts. I was a frustrated candidate once too.

It is true that from an outside view, the CFA Institute can be seen to have an incentive to continue to fail candidates. I've had the opportunity to discuss this at length with some of the people in CFA societies as well as CFA Institute, and here are my opinions:

1. Drop-off rates. A decrease in pass rate will almost certainly follow with a decrease in take-up rates and increase in drop-off rates. This will actually harm exam fee revenue in the long term, so it's hard to imagine that lowering CFA pass rates is a key part of CFAI's revenue strategy.
2. Increase in variety of candidates. In the recent years, the CFA qualification has seen a lot of participation not just from asset managers, but now traders, investment bankers and even support staff are taking the exam. As the above post discusses, this will naturally lower the pass rate. The CFA Institute is trying to address this by diversifying the format of the exams, hence their pilot Claritas program which is aimed at support staff.
3. Subscription fees. Remember that CFA Institute continues to collect charter renewal fees - currently $275 a year. Although this is much less compared to exam registration, remember that this is at almost no effort or cost to CFA Institute. Thus to maximize profit, CFA Institute should pass as many people as possible, thus converting them into a steady revenue stream without having to worry about setting questions or organizing exams.
4. Corporate Governance. It's true that CFA Institute makes money from administering exams and distributing material. It's quite a lot of money at that. CFA Institute however has to report to quite a few regulatory bodies that audit all their processes annually, and these are no routine checks. The board that determines the pass rates are most likely uncompensated by CFA Institute and have no incentive to increase their revenues.
5. Industry scrutiny. The CFA charter is currently seen as the gold standard as far as investment and asset management is concerned. Above all, CFA Institute will regard keeping that status of utmost importance. There are many, many other rival qualifications that would love to take that place, and pass rate manipulation seems like a pretty risky strategy: gambling the incumbent status of the charter for the sake of a few extra percentage points in revenue for a few years.

I do think that a lot of the frustration is due to the opaqueness of the results (hence our Analyze Results section to try and compensate for this somehow), but in the end reasoning on a macro level, pass rate manipulation makes little sense when you consider CFA Institute's other options. There are just many, many easier ways for CFA Institute to increase their revenue streams without having to resort to the extremely risky behavior of manipulating pass rates.

Hope this helps. More than happy to continue the discussion here: http://www.300hours.com/qa-board.html#/

Comments are closed.