BattleActs Analytics

Note for Exam 6-US beta-testers: This article relates to statistics from the first 3 sittings of Exam 6-Canada for BattleActs users. Since Spring 2019 is the first sitting for Exam 6-US, we don't yet have any US data. For that reason, I've simply copied the corresponding article from the Canadian version of BattleActs. The exam has a similar format so the general points I've made below should apply to both exams. -Graham

Introduction

The BattleActs system tracks basic information about users. We do this to create a profile of a successful candidate. You can then adjust your study strategy to align more closely with this optimal profile.

Our original hypothesis was that passing would be strongly correlated with BRQ (Battle-Readiness Quotient). There is a definite correlation, but so far not quite as strong as we would like. Too many confounding variables? This is discussed below. (Things in the real world never work out as well as in a textbook!)

Executive Summary

The analytics from the 2018.Fall exam sitting are distorted because of the low pass rate of 31%. The average raw pass rate on the previous 5 sittings was 44%. Given that 155 candidates sat for the 2018.Fall exam, the low pass rates translates in roughly 20 fewer candidates passing than under normal conditions. That means many candidates who scored 5 in 2018.Fall would normally have passed. In the examiner's report, the CAS claimed that the candidate pool was weaker. I can't verify that independently, but I think such an assessment is more subjective than the CAS would like to believe - unless you give the exact same exam to each cohort of candidates. Of course, that isn't possible because the exams are published.

My opinion is that the 2018.Fall exam was longer and harder than other recent exams for 3 reasons:

• there were more questions: 30
• there were detailed questions on low-ranked readings: #17 from CAS.ATRA and #28 from CIA.Valn
• there was a very difficult and time-consuming calculation question: #16 from CCIR.ARinstr (This is one of the infamous Alphabet City questions.)

Indeed these 3 questions, #17, #28, #16, were the worst done questions on the exam according to the detailed score reports from unsuccessful candidates. (If you're lucky enough never to have failed a high-level exam, these score reports don't give you exact point-values but if you have enough of them you can infer the average scores on each question.)

2018.Fall Avg Scores - All Candidates (Exam 6-Canada) (click to download then open in Excel)

Anyway, incorporating the 2018.Fall results, here is the current ranking of variables with respect to probability of passing:

category	variable	importance	relative predictive strength 1
BRQ	BRQ on the day before the exam	moderate	71
logins	# of days with at least 1 login to BattleActs	moderate	57
forum	# of posts to the BattleActs forum	moderate	50
logins	total # of logins to the system	moderate	50

¹ This ranking doesn't have a specific interpretation other than to represent the relative strength from 0-100 in predicting a pass. It is not a correlation coefficient. (The range of 50-80 is considered moderate.)

A few other observations that may be helpful:

• Returning BattleActs users had a better chance of passing than first-time users. This is likely because returning users have had at least 6 months of extra study time, depending on how many attempts they've had. The CAS doesn't publish statistics on pass rates versus number of attempts. I know there are candidates who pass the first time but I have no idea what proportion of the total they represent. I also know people who have had 6+ attempts before passing. I would be very interested in knowing the average number of attempts required to pass. This seems to be a well-guarded secret!

• Returning users who requested a personalized score report analysis also had a better chance of passing than those returning candidates who did not make this request. (As a reminder, I assist unsuccessful candidates in analyzing their score report. The purpose is to help direct their study for the next sitting.)

• Registration date does not appear to have much effect on probability of passing. The statistics on this were all over the map. The only concrete facts I can provide are that about 1/2 of BattleActs users sign up 4 months before the exam. The other half (roughly) are split between 3 months and 5 months prior. Then there are a handful who sign up either earlier or later. There were passing candidates who registered in September for the October sitting but they must have been studying prior to that. Not many people could start from scratch 2 months before the exam and pass.

Regarding BRQ scores:

• The BRQ scores of successful candidates have varied from 0 percent (yes 0%) to the high 90s. We discovered in our original beta-test that some people just don't like doing BattleCards (our word for flash cards). They study from the wiki, examiner's reports, and source readings. You have to decide for yourself which study strategy best suits your learning style. There are many ways to do it.

• Having said that, however, candidates with a higher BRQ did have a better chance of passing. Not only that, most of the unsuccessful candidates for whom we provided a score report analysis (meaning we know their score) and who also had high BRQs almost always scored 5. As noted above, most of those candidates would have passed if the pass rate had been closer to the historical average.

• To summarize, if you like doing BattleCards (both the "memory" cards and the "practice templates" for calculations), they can really boost your chance of passing. But remember you have to do them honestly and consistently. More on this below. You should aim for at least 80%, preferably 90%.

Data Quality

Obviously the quality of these results depends on the quality of the data. At this early stage in the development of the BattleActs system, our data has the following shortcomings:

1. Our data is still relatively thin. (We have data only for the beta-test for 2017.Fall, and 2 live versions: 2018.Spring & 2018.Fall.)
2. We generally don't know whether a candidate had prior attempts at Exam 6. This would be a confounding variable.
3. There is an element of luck in passing this exam. Many candidates have scores that are very, very close to the pass mark, and just a tiny bit of luck could have lifted them from a fail to a pass. This element of luck can skew our data.

As we develop the system and gather data from subsequent sittings, the quality of data will improve.

Detailed Comments

BRQ

The most significant point that deserves elaboration is the relative importance of the BRQ (Battle-Readiness Quotient). The intent of the BRQ is to provide a continuous assessment of your readiness to take the exam. We were hoping to find a strong correlation between BRQ and probability of passing, but the correlation is only moderate. Here are a few possible explanations:

• Most of the BattleCards are self-scored, so you have to be brutally honest. It's very tempting to always click "correct" because there is instantaneous positive feedback! You see the +27 or whatever in the navigation bar and your BRQ goes up by that amount. Now, you guys are all smart and I don't think you're being intentionally dishonest, but there is skill involved in using the BattleActs system effectively. More on this later in the article.
• Some candidates have told us they don't like flash cards. They do virtually all of their studying using the wiki articles and the examiner's reports. We've had successful candidates with BRQs of under 20%. (Note that we are continually revising and improving the BattleCards, and we're noticing that people are generally using them more and more with each subsequent sitting.)

In any event, we hope that through better support in using BattleActs effectively, the correlation between BRQ and passing will become stronger. (But it will never be perfect because there will always be external influences that we can neither measure nor control.)

Forum & Logins

Logins and forum posts seem to be part of the profile of successful candidates.

• If you're really engaged in the material, you'll likely have questions. And if you have questions, you can get them answered on the forum. That's the connection. Even if you don't post to the forum, you can glean information from other people's posts. If you aren't at looking at the forum from time to time, you're missing out on a key element in preparing for the exam.
• The correlation between passing and [# of days with at least 1 login] is pretty obvious. Interestingly, it is a better predictor than [total # of logins]. In a very rough sense, a high value for [total # of logins] may indicate a tendency to cram (not good!) whereas [# of days with at least 1 login] may indicate consistency. In other words, a certain number of study hours spread over a longer period is better than the same number of hours over a shorter period.

The positive effect of spreading your study hours over a greater number is days is more pronounced when you're learning a lot of material. If you only had to learn 10 facts, you could cram that into your brain the night before. But for Exam 6, you have to learn something like 1500 facts. That requires a disciplined, consistent, and long-term approach.

Registration Date

Candidates who registered at least 4 months prior to the exam have a small advantage. At the same time, there were a handful of passing candidates with registration dates much closer to the exam. It seems plausible that these late-registering passing candidates may have had prior attempts because there aren't many people who could start from scratch and pass with only 6 weeks of studying. Anyway, a good rule of thumb is:

• Register by the end of January for the Spring exam.
• Register by the end of July for the Fall exam.

This gives you 3 months to study, and if you study effectively, you can give yourself a good chance to pass. (Of course, giving yourself 4 or 5 months provides a margin for error, just in case you get sick or get busy at work.)

Common Reasons for Not Passing

For returning BattleActs users, we provide a free analysis of grade reports. From those grade reports, we've identified common reasons for not passing. (It may be helpful to keep the Spring 2018 BattleTable open in a separate tab for easy reference.) In order of importance, the reasons for not passing relate to the following areas:

1. Memorization of facts
2. Time management during the exam
3. Bloom’s Taxonomy
4. Calculations

We'll deal with each item in turn and offer suggestions on how to improve.

Memorization

This is the number one reason people fail Exam 6.

• If you have to memorize 10 facts, you can probably do it in 10 minutes. But if you have to memorize 20 facts, it will take more than twice as long, maybe 25 minutes. Now extrapolate that to Exam 6 where you have to memorize something like 1500 facts. To learn these facts reliably, you need to go over each one dozens of times, and you have to do it carefully. It's very easy to "review" a BattleCard too quickly and fool yourself into thinking you know the answer when you really don't.
• This cannot be stressed enough! Don't fall into the trap of doing the BattleCards too quickly. There's a difference between merely recognizing a question and truly thinking through the answer. A good habit is to write the answer on a piece of paper. That takes way longer but it's a truer test of whether you know it versus just saying to yourself, "Yup, I've done this a million times. I totally know it!" After writing out 10 or 20 BattleCard answers in full, you can switch to saying the answer out loud, or train yourself to recite it mentally. But even if you do it mentally, each BattleCard could take anywhere from 10 seconds to 30 or 40 seconds to do properly.

BattleActs was designed primarily to help you memorize the facts. If you've used BattleActs, you already know that the papers are ranked by how often they've appeared on prior exams. This is the basis of the Suggested Study Schedule. And your BRQ is designed to indicate how well you're progressing through the material. Let the BRQ do its job!

• If in doubt, mark a BattleCard wrong. Then use Level 3: Custom Battles to select only those wrong BattleCards from a particular paper. (Do this using Optional Setting B and selecting "red" for correctness status.)
• Then go through those BattleCards carefully, as described above, but don't mark them "correct" unless you've gone through them several times over the course of a few days.

It takes much longer than you think to memorize the facts reliably.

Another trick is to use the Shuffle button on the BattleCard page. Once you've done a set of BattleCards several times, you'll solidify your knowledge if you do them in a different order every so often. An established principle of learning is that you have to practice in different ways. If you play a musical instrument, you might know that practicing a piece at different tempos and/or using different rhythms will enhance your playing proficiency, versus playing it at the same tempo in the same way every time.

Time Management

Running out of time is a heart-breaking way to fail an exam. If you're well-prepared, then time management will be a bigger issue because you'll know how to answer most (or all) of the questions. You have to make sure you finish, even if that means writing less than you want to for a particular question. If you have the time, it's better to write more than less, but not at the expense of missing a question you know how to do.

There is a detailed discussion of time management in the wiki article On BattleDay. If you don't like my method, invent your own. But you have to take time management seriously! Don't wing it on exam day!

Bloom's Taxonomy

Examples of Bloom's Taxonomy questions from 2018.Fall are listed below. (See 2018.Fall BattleTable.) This is something that's hard to teach because being good at answering these types of questions improves mainly through experience. Very briefly, this is what these questions are about:

#14c: create a government solution to terrorism insurance other than TRIA

#18c: identify risk-limiting features in a contract

#20c: identify a possible adverse scenario consistent with the given data

#22ac: stress-testing program for overland flooding

#23: where did the Appointed Actuary go wrong in their use of ORSA and setting internal targets

#26c: concept questions on MCT and premium liabilities

These questions are based on facts, but facts alone aren't enough to give a complete answer. The best advice here is to study the answers in the examiner's reports for these types of questions, and just try to absorb how these problems are done.

One concrete piece of advice, however, is to first try to recognize which facts may be relevant to the answer. As an example, consider question #23 on the actuary's actions in setting an internal target for that company. You might recall that setting internal targets relates to ORSA (see OSFI.ORSA) so you can start by jotting down what you know about ORSA. Even if you don't quite know the answer to the question, writing something down will get you started, and may even get you some points! Be imaginative. For a question like #14c, where you have to suggest an alternative government program, there are likely many variations of correct answers.

Side note: There's a great math book by the famous Hungarian mathematician George Polya called How to Solve It. It's all about heuristics for solving problems. He doesn't use the term Bloom's Taxonomy, but that's essentially what it's about. You have to put ideas together in an imaginative way. Look it up sometime.

Calculations

This is the last item on the list and it is already an area of strength for actuaries. The only piece of advice I have is to practice more. BattleActs has practice templates for many of the common types of calculation problems on the exam. Once you're good at the practice templates, make sure you can do all the recent exam problems. But it's not enough to do a problem once or twice. I've found that to really get it into my head, I have to do each problem many times over the course of my study. Going back to the music analogy, you know that to learn a piece of music you have to practice it daily (and carefully) over a period of weeks (or months!)

Closing Thoughts

The single best piece of advice I can give you is to find a way to stay engaged. That will support consistent and effective studying. Try not to study when you're tired because that's usually a waste of time. Take nap instead. And don't be too hard on yourself. If you start early and follow a plan, you'll be in good shape. You've made it this far in the exam process. You will get through it!

Slay the beast. :-)