BattleActs Analytics

Introduction

The BattleActs system tracks basic information about users. We do this to create a profile of a successful candidate. You can then adjust your study strategy to align more closely with this optimal profile.

Our original hypothesis was that passing would be strongly correlated with BRQ (Battle-Readiness Quotient). So far, we have found a moderate correlation. Too many confounding variables? This is discussed below. (Things in the real world never work out as well as in a textbook!)

Executive Summary

Click the link below for a tabulation of questions by exam as well as the estimated average score for each question across all candidates. I always find it interesting to see both the high-scoring and low-scoring questions.

Average Score by Question - All Candidates

Here is the current ranking of variables with respect to probability of passing:

category	variable	relative predictive strength
BRQ	BRQ on the day before the exam	moderately high
logins	# of days with at least 1 login to BattleActs	moderate
logins	total # of logins to the system	moderate
forum	# of posts to the BattleActs forum	low/moderate

A few other observations that may be helpful:

• Returning BattleActs users had a better chance of passing than first-time users. This is likely because returning users have had at least 6 months of extra study time, depending on how many attempts they've had. The CAS doesn't publish statistics on pass rates versus number of attempts. I know there are candidates who pass the first time but I have no idea what proportion of the total they represent. I also know people who have had 6+ attempts before passing. I would be very interested in knowing the average number of attempts required to pass. This seems to be a well-guarded secret!

• Returning users who requested a personalized score report analysis also had a better chance of passing than those returning candidates who did not make this request. (I had provided this service up until the 2019-Fall exam because the score reports provided the candidate with their score on each question and their performance against the average on each question. The CAS also published the exams and provided detailed answers and commentary. Since the CAS no longer publishes exams and the current version of the score reports provide so little detail, it is no longer possible to provide assistance using the score reports.)

• Registration date does not appear to have much effect on probability of passing. The statistics on this were all over the map. The only concrete facts I can provide are that about 1/2 of BattleActs users sign up 4 months before the exam. The other half (roughly) are split between 3 months and 5 months prior. Then there are a handful who sign up either earlier or later. There were passing candidates who registered in September for the October sitting but they must have been studying prior to that. Not many people could start from scratch 2 months before the exam and pass.

Regarding BRQ scores:

• The BRQ scores of successful candidates have varied from 0 percent (yes 0%) to the high 90s. We discovered in our original beta-test that some people just don't like doing BattleCards (our word for flash cards). They study from the wiki, examiner's reports, and source readings. You have to decide for yourself which study strategy best suits your learning style. There are many ways to do it.

• Having said that, however, candidates with a higher BRQ did have a better chance of passing. Not only that, most of the unsuccessful candidates for whom we provided a score report analysis (meaning we know their score) and who also had high BRQs almost always scored 5.

• To summarize, if you like doing BattleCards (both the "memory" cards and the "practice templates" for calculations), they can really boost your chance of passing. But remember you have to do them honestly and consistently. More on this below. You should aim for at least 80%, preferably 90%.

Data Quality

Obviously the quality of these results depends on the quality of the data.

1. We generally don't know whether a candidate had prior attempts at Exam 6. This would be a confounding variable.
2. There is an element of luck in passing this exam. Many candidates have scores that are very, very close to the pass mark, and just a tiny bit of luck could have lifted them from a fail to a pass. This element of luck can skew our data.
3. The length and difficulty of the exam varies from sitting to sitting and 1 or 2 unusual exam questions can skew the scores and results.
4. New readings are introduced to the syllabus regularly and there can be a wide distribution of scores for questions on new readings. It's normally takes a few sittings to know what the exam committee will emphasize from each reading.

But even with these random variables in play, our conclusions should be broadly correct and helpful to guide your studying.

Detailed Comments

BRQ

The most significant point that deserves elaboration is the relative importance of the BRQ (Battle-Readiness Quotient). The intent of the BRQ is to provide a continuous assessment of your readiness to take the exam. We were hoping to find a strong correlation between BRQ and probability of passing, but the correlation is only moderate. Here are a few possible explanations:

• Most of the BattleCards are self-scored, so you have to be brutally honest with yourself. It's very tempting to always click "correct" because there is instantaneous positive feedback! You see the +27 or whatever in the navigation bar and your BRQ goes up by that amount. There's a skill involved in using the BattleActs system effectively – more on this later in the article.
• Some candidates have told us they don't like flash cards. They do virtually all of their studying using the wiki articles and the examiner's reports and we've had successful candidates with BRQs of under 20%. (Note that we are continually revising and improving the BattleCards, and we're noticing that people are generally using them more and more with each subsequent sitting.)

In any event, we hope that through better support in using BattleActs effectively, the correlation between BRQ and passing will become stronger. (But it will never be perfect because there will always be external influences that we can neither measure nor control.)

Forum & Logins

Logins and forum posts seem to be part of the profile of successful candidates.

• If you're really engaged in the material, you'll likely have questions. And if you have questions, you can get them answered on the forum. That's the connection. Even if you don't post to the forum, you can glean information from other people's posts. If you aren't at looking at the forum from time to time, you're missing out on a key element in preparing for the exam.
• The correlation between passing and [# of days with at least 1 login] is pretty obvious. Interestingly, it is a better predictor than [total # of logins]. In a very rough sense, a high value for [total # of logins] may indicate a tendency to cram (not good!) whereas [# of days with at least 1 login] may indicate consistency. In other words, a certain number of study hours spread over a longer period is better than the same number of hours over a shorter period.

The positive effect of spreading your study hours over a greater number of days is more pronounced when you're learning a lot of material. If you only had to learn 10 facts, you could cram that into your brain the night before. But for Exam 6, you have to learn something like 1500 facts. That requires a disciplined, consistent, and long-term approach.

BattleActs Purchase Date

Candidates who purchased at least 4 months prior to the exam have a small advantage. At the same time, there were a handful of passing candidates with registration dates much closer to the exam. It seems plausible that these late-purchasing passing candidates may have had prior attempts because there aren't many people who could start from scratch and pass with only 6 weeks of studying. Anyway, a good rule of thumb is:

• Purchase by the end of January for the Spring exam.
• Purchase by the end of July for the Fall exam.

This gives you 3 months to study, and if you study effectively, you can give yourself a good chance to pass. (Of course, giving yourself 4 or 5 months provides a margin for error, just in case you get sick or get busy at work.)

Common Reasons for Not Passing

Common reasons for not passing relate to the following areas:

1. Memorization of facts
2. Time management during the exam
3. Bloom’s Taxonomy
4. Calculations

We'll deal with each item in turn and offer suggestions on how to improve.

Memorization

This is the number one reason people fail Exam 6.

• If you have to memorize 10 facts, you can probably do it in 10 minutes. But if you have to memorize 20 facts, it will take more than twice as long, maybe 25 minutes. Now extrapolate that to Exam 6 where you have to memorize something like 1500 facts. To learn these facts reliably, you need to go over each one dozens of times, and you have to do it carefully. It's very easy to "review" a BattleCard too quickly and fool yourself into thinking you know the answer when you really don't.
• This cannot be stressed enough! Don't fall into the trap of doing the BattleCards too quickly. There's a difference between merely recognizing a question and truly thinking through the answer. A good habit is to write the answer on a piece of paper. That takes way longer but it's a truer test of whether you know it versus just saying to yourself, "Yup, I've done this a million times. I totally know it!" After writing out 10 or 20 BattleCard answers in full, you can switch to saying the answer out loud, or train yourself to recite it mentally. But even if you do it mentally, each BattleCard could take anywhere from 10 seconds to 30 or 40 seconds to do properly.

BattleActs was designed primarily to help you memorize the facts. If you've used BattleActs, you already know that the papers are ranked by how often they've appeared on prior exams. This is the basis of the suggested study in Getting the Most out of BattleActs. And your BRQ is designed to indicate how well you're progressing through the material. Let the BRQ do its job!

• If in doubt, mark a BattleCard wrong. Then use Custom Battles to select only those wrong BattleCards from a particular paper. (Do this using Optional Setting B and selecting "red" for correctness status.)
• Then go through those BattleCards carefully, as described above, but don't mark them "correct" unless you've gone through them several times over the course of a few days.

It takes much longer than you think to memorize the facts reliably.

Another trick is to use the Shuffle button on the BattleCard page. Once you've done a set of BattleCards several times, you'll solidify your knowledge if you do them in a different order every so often. An established principle of learning is that you have to practice in different ways. If you play a musical instrument, you might know that practicing a piece at different tempos and/or using different rhythms will enhance your playing proficiency, versus playing it at the same tempo in the same way every time.

Time Management

Running out of time is a heart-breaking way to fail an exam. If you're well-prepared, then time management will be a bigger issue because you'll know how to answer most (or all) of the questions. You have to make sure you finish, even if that means writing less than you want to for a particular question. If you have the time, it's better to write more than less, but not at the expense of missing a question you know how to do.

There is a detailed discussion of time management in the wiki article On BattleDay. If you don't like my method, invent your own. But you have to take time management seriously! Don't wing it on exam day!

Bloom's Taxonomy

This is something that's hard to teach because being good at answering these types of questions improves mainly through experience. Bloom's Taxonomy questions are based on facts, but facts alone aren't enough to give a complete answer. The best advice here is to study the answers in the examiner's reports, and just try to absorb how these problems are done.

One concrete piece of advice, however, is to first try to recognize which facts may be relevant to the answer. Even if at first you don't know what to do, writing something (anything!) will get your brain moving in the right direction. There are likely many variations of correct answers so don't edit yourself. Be imaginative!

Side note: There's a great math book by the famous Hungarian mathematician George Polya called How to Solve It. It's all about heuristics for solving problems. He doesn't use the term Bloom's Taxonomy, but that's essentially what it's about. You have to put ideas together in an imaginative way. Look it up sometime.

Calculations

This is the last item on the list and it is already an area of strength for actuaries. The only piece of advice I have is to practice more. BattleActs has practice templates for many of the common types of calculation problems on the exam. Once you're good at the practice templates, make sure you can do all the recent exam problems. But it's not enough to do a problem once or twice. I've found that to really get it into my head, I have to do each problem many times over the course of my study. Going back to the music analogy, you know that to learn a piece of music you have to practice it daily (and carefully) over a period of weeks (or months!)

Closing Thoughts

The single best piece of advice I can give you is to find a way to stay engaged. That will support consistent and effective studying. Try not to study when you're tired because that's usually a waste of time. Take a nap instead. And don't be too hard on yourself. If you start early and follow a plan, you'll be in good shape. You've made it this far in the exam process. You will get through it!

Slay the beast. :-)