BattleActs Analytics

Note for Exam 6-US beta-testers: This article relates to statistics from the first 2 sittings of Exam 6-Canada for BattleActs users. Since Spring 2019 is the first sitting for Exam 6-US, we don't yet have any US data. For that reason, I've simply copied the corresponding article from the Canadian version of BattleActs. The exam has a similar format so the general points I've made below should apply to both exams. -Graham

Introduction

For Fall 2017 and Spring 2018, candidates using BattleActs had a significantly higher pass rate than the overall candidate pool. For Fall 2018 however, almost all Exam 6-Canada candidates are using BattleActs. That means that going forward, there will be little difference between the BattleActs pass rate and the overall pass rate.

The BattleActs system tracks basic information about users. We do this to create a profile of a successful candidate. You can then adjust your study strategy to align more closely with this optimal profile.

Our original hypothesis was that passing would be strongly correlated with BRQ (Battle-Readiness Quotient), but the correlation has not been as strong as expected. This is discussed below. (Things in the real world never work out as well as in a textbook!)

Executive Summary

Here is the current ranking of variables with respect to probability of passing:

category	variable	importance
forum	# of posts to the BattleActs forum	high
logins	# of days with at least 1 login to BattleActs	high
logins	total # of logins to the system	moderate
BRQ	BRQ on the day before the exam	moderate
registration date	date of BattleActs registration	low

Successful candidates have a higher level of engagement. We used [# of posts to forum] as a proxy for strength of engagement, and [# of days with at least 1 login] as a proxy for frequency of engagement.

• Of the approximately 40 candidates who used BattleActs for Spring 2018, 10 posted to the forum at least once. Of those 10, 80% passed the exam (although many of our passing candidates did not post at all.)
• Of the candidates who signed in to BattleActs on 86 or more different days, 70% passed. (roughly 3 months of daily study)
• Of the candidates who signed in to BattleActs on 63 or more different days, 60% passed. (roughly 2 months of daily study)

So, keeping in mind that our data is thin after only 2 sittings, a profile of a successful candidate is one who logs in consistently and posts at least occasionally to the forum.

Data Quality

Obviously the quality of these results depends on the quality of the data. At this early stage in the development of the BattleActs system, our data has the following shortcomings:

1. Our data is still relatively thin. (We have data only for the beta-test for Fall 2017, and for the live version for Spring 2018.)
2. We generally don't know whether a candidate had prior attempts at Exam 6. This would be a confounding variable.
3. There is an element of luck in passing this exam. Many candidates have scores that are very, very close to the pass mark, and just a tiny bit of luck could have lifted them from a fail to a pass. This element of luck can skew our data.

As we develop the system and gain more users, the quality of data will improve.

Detailed Comments

BRQ

The most significant point that deserves elaboration is the relative importance of the BRQ (Battle-Readiness Quotient). The intent of the BRQ is to provide a continuous assessment of your readiness to take the exam. We were hoping to find a strong correlation between BRQ and probability of passing, but the correlation is only moderate. Here are a few possible explanations:

• Most of the BattleCards are self-scored, so you have to be brutally honest. It's very tempting to always click "correct" because there is instantaneous positive feedback! You see the +27 or whatever in the navigation bar and your BRQ goes up by that amount. Now, you guys are all smart and I don't think you're being intentionally dishonest, but there is skill involved in using the BattleActs system effectively. More on this later in the article.
• Some candidates have told us they don't like flash cards. They do virtually all of their studying using the wiki articles and the examiner's reports. We had one successful candidate who had a BRQ of only 15%. (Note that we are continually revising and improving the BattleCards, and we're noticing that people are generally using them more than during the original beta-test.)

In any event, we hope that through better support in using BattleActs effectively, the correlation between BRQ and passing will become stronger. (But it will never be perfect because there will always be external influences that we can neither measure nor control.)

Forum & Logins

As stated above, the most significant predictors in our data are forum posts and consistency of logins.

• If you're really engaged in the material, you'll likely have questions. And if you have questions, you can get them answered on the forum. That's the connection. Even if you don't post to the forum, you can glean information from other people's posts. If you aren't at least registered for the forum, you're missing out on a key element in preparing for the exam.
• The correlation between passing and [# of days with at least 1 login] is pretty obvious. Interestingly, it is a better predictor than [total # of logins]. In a very rough sense, a high value for [total # of logins] may indicate a tendency to cram (not good!) whereas [# of days with at least 1 login] may indicate consistency. In other words, a certain number of study hours spread over a longer period is better than the same number of hours over a shorter period.

The positive effect of spreading your study hours over a greater number is days is more pronounced when you're learning a lot of material. If you only had to learn 10 facts, you could cram that into your brain the night before. But for Exam 6, you have to learn something like 1500 facts. That requires a disciplined, consistent, and long-term approach.

Registration Date

Candidates who registered in January for the Spring 2018 exam seemed to have a small advantage in terms of passing. At the same time, there were a handful of passing candidates with March registration dates. It seems plausible that these late-registering passing candidates may have had prior attempts because there aren't many people who could start from scratch and pass with only 6 weeks of studying. Anyway, a good rule of thumb is:

• Register by the end of January for the Spring exam.
• Register by the end of July for the Fall exam.

This gives you 3 months to study, and if you study effectively, you can give yourself a good chance to pass. (Of course, giving yourself 4 or 5 months provides a margin for error, just in case you get sick or get busy at work.)

Common Reasons for Not Passing

For returning BattleActs users, we provide a free analysis of grade reports. From those grade reports, we've identified common reasons for not passing. (It may be helpful to keep the Spring 2018 BattleTable open in a separate tab for easy reference.) In order of importance, the reasons for not passing relate to the following areas:

1. Memorization of facts
2. Time management during the exam
3. Bloom’s Taxonomy
4. Calculations

We'll deal with each item in turn and offer suggestions on how to improve.

Memorization

This is the number one reason people fail Exam 6.

• If you have to memorize 10 facts, you can probably do it in 10 minutes. But if you have to memorize 20 facts, it will take more than twice as long, maybe 25 minutes. Now extrapolate that to Exam 6 where you have to memorize something like 1500 facts. To learn these facts reliably, you need to go over each one dozens of times, and you have to do it carefully. It's very easy to "review" a BattleCard too quickly and fool yourself into thinking you know the answer when you really don't.
• This cannot be stressed enough! Don't fall into the trap of doing the BattleCards too quickly. There's a difference between merely recognizing a question and truly thinking through the answer. A good habit is to write the answer on a piece of paper. That takes way longer but it's a truer test of whether you know it versus just saying to yourself, "Yup, I've done this a million times. I totally know it!" After writing out 10 or 20 BattleCard answers in full, you can switch to saying the answer out loud, or train yourself to recite it mentally. But even if you do it mentally, each BattleCard could take anywhere from 10 seconds to 30 or 40 seconds to do properly.

BattleActs was designed primarily to help you memorize the facts. If you've used BattleActs, you already know that the papers are ranked by how often they've appeared on prior exams. This is the basis of the Study Schedule. And your BRQ is designed to indicate how well you're progressing through the material. Let the BRQ do its job!

• If in doubt, mark a BattleCard wrong. Then use Level 3: Custom Battles to select only those wrong BattleCards from a particular paper. (Do this using Optional Setting B and selecting "red" for correctness status.)
• Then go through those BattleCards carefully, as described above, but don't mark them "correct" unless you've gone through them several times over the course of a few days.

It takes much longer than you think to memorize the facts reliably.

Another trick is to use the Shuffle button on the BattleCard page. Once you've done a set of BattleCards several times, you'll solidify your knowledge if you do them in a different order every so often. An established principle of learning is that you have to practice in different ways. If you play a musical instrument, you might know that practicing a piece at different tempos and/or using different rhythms will enhance your playing proficiency, versus playing it at the same tempo in the same way every time.

Time Management

Running out of time is a heart-breaking way to fail an exam. If you're well-prepared, then time management will be a bigger issue because you'll know how to answer most (or all) of the questions. You have to make sure you finish, even if that means writing less than you want to for a particular question. If you have the time, it's better to write more than less, but not at the expense of missing a question you know how to do.

There is a detailed discussion of time management in the wiki article On BattleDay. If you don't like my method, invent your own. But you have to take time management seriously! Don't wing it on exam day!

Bloom's Taxonomy

Examples of Bloom's Taxonomy questions from Spring 2018 are #3, #9, #11, #24. (See Exam Summaries.) This is something that's hard to teach because being good at answering these types of questions improves mainly through experience. Very briefly, this is what these questions are about:

#3: analyze a scenario related to credit scoring

#9: create a flood insurance program

#11: analyze a scenario for potential insolvency

#24: analyze a given DCAT scenario and evaluate the actions of the actuary

These questions are based on facts, but facts alone aren't enough to give a complete answer. The best advice here is to study the answers in the examiner's reports for these types of questions, and just try to absorb how these problems are done.

One concrete piece of advice, however, is to first try to recognize which facts may be relevant to the answer. For example in question #3 on credit scoring, you might recall that privacy and customer consent are important concepts. Write that down! Then think about how those concepts might relate to the specific question that was asked. Be imaginative. There are many variations of correct answers for these kinds of questions.

Side note: There's a great math book by the famous Hungarian mathematician George Polya called How to Solve It. It's all about heuristics for solving problems. He doesn't use the term Bloom's Taxonomy, but that's essentially what it's about. You have to put ideas together in an imaginative way. Look it up sometime.

Calculations

This is the last item on the list and it is already an area of strength for actuaries. The only piece of advice I have is to practice more. BattleActs has practice templates for many of the common types of calculation problems on the exam. Once you're good at the practice templates, make sure you can do all the recent exam problems. But it's not enough to do a problem once or twice. I've found that to really get it into my head, I have to do each problem many times over the course of my study. Going back to the music analogy, you know that to learn a piece of music you have to practice it daily (and carefully) over a period of weeks (or months!)

Closing Thoughts

The single best piece of advice I can give you is to find a way to stay engaged. That will support consistent and effective studying. Try not to study when you're tired because that's usually a waste of time. Take nap instead. And don't be too hard on yourself. If you start early and follow a plan, you'll be in good shape. You've made it this far in the exam process. You will get through it!

Slay the beast. :-)