If you have ever taken a quiz entitled “How to tell if your boyfriend is cheating on you” or “Answer these five questions to learn to see if you’re a good friend,” then you’ve probably taken something akin to what we call a “Cosmo test.” Your answers to just a few questions will result in a detailed analysis of your personality, along with a lengthy set of advice on how to improve your life (or a massive pat on the back if you score high). Cosmo tests are fun, entertaining, and engaging: they lend themselves to having results posted on social media feeds and to being talked about with friends. If you’re trying to kill a few minutes or just curious about your results, then these tests are perfectly designed.
If you’re hoping for entertainment, then the science of test development doesn’t matter much. The trouble comes when you make decisions based on less-than-scientific test results. In financial services, many consumers and advisors aren’t aware of the difference between Cosmo and “real” tests. The prominent examples in financial services are less-than-scientific versions of investor risk tolerance assessments (for example, see this discussion on Bogleheads). These include homegrown tests that lack research or theory, or tests without any documentation as to their validity. Many of these types of tests ultimately connect your responses to just a few questions to a suggested investment portfolio.
Why does science matter when we’re talking about tests? Let’s look at an example outside of financial services first:
Imagine you’re a senior in high school, and your counselor asks you to take a test that will help determine what career you should pursue after graduation. Your results indicate that you should be a nurse. You’re a little unsure of the results (you don’t like blood or sick people), but armed with your test results, you began looking into how to get a nursing degree, reading about nursing, interning in a doctor’s office, and volunteering with the Red Cross. It doesn’t feel like a good fit, but the test said you should be a nurse, right? Imagine that a few months later, and several hours devoted to this path, you get an email from your counselor, letting you know that the test you took was inaccurate. And there’s more: using a new, more scientific test, it turns out you are actually best suited to be an accountant.
In our student example above, only time was lost (and perhaps, a bit of trust in counselors). Here’s a different, and potentially most costly case, from wealth management:
Each day, you get a 10:00 am alert as to how the market is performing. This morning’s alert lets you know that the market is up 3% today. Just then, your advisor sends you an email asking you to complete a risk tolerance assessment. The assessment asks you questions about what you would do in the future related to losing and gaining money—essentially income-gamble questions to uncover your preference for risk. Based on the results your preference for risk is “high,” so the advisor places you in a portfolio of 95% stocks and 5% bonds. All is well.
The next day, your 10:00 am alert tells you that the market is down 5%. You call your advisor and ask to sell everything. Your advisor does just that. However, perplexed by your request (recall the test linked your risk tolerance results to an allocation of 95% stocks and 5% bonds), your advisor asks you to retake the risk tolerance test. You retake the test. But this time your preference for risk registers “low.” So your advisor now places you in a portfolio that is 55% stocks and 45% bonds.
On the third day, the market is up 4% at 10:00 am. You want to buy, buy, buy. Once again your advisor honors your request, but is still somewhat confused about the results of the “risk tolerance” assessment. She then asks you to take it a third time. This time, you’re “high” again: 90% stocks, 5% bonds.
What was the test really measuring? Your relatively stable overall risk tolerance, or a more temporary notion of risk preference? Or maybe yet a shifting attitude toward risk based on the market? How can your advisor put you in a portfolio based on the results if the results are always changing?
The science of measuring personal characteristics or psychological constructs is called psychometrics. This field provides specific methods for creating accurate and appropriate tests that can help uncover our characteristics, including our personality, attitudes, values, and other competencies. This type of insight can help us make decisions about where we go to college, the careers we pursue, who we hire, how we develop leaders, and where we should focus self-improvement efforts. In the world of investing, scientifically created assessments can be valid and reliable measures of psychological risk tolerance—one piece of the puzzle when it comes to determining how we should be invested. These types of tests can help us understand our characteristics today, and also potentially help us improve upon competencies related to investing.
So what’s the science behind designing tests, and what makes a good test? There are a couple of key features of scientific assessments that are important to know when you think about how those results might impact the choices you make in the future. We created an infographic on what makes a good test (you can download it here), but the basics are listed below:
- Reliable: The first is that the assessment is reliable. Each time you take a personality assessment as an example, you should get the same scores on that test each time. It is reliable in the sense that each time you take the test, if it’s an accurate measurement, you’re going to get the same scores. Your scores aren’t changing because of the weather, or the markets, or some other factor. Think about the SAT as an example. While your test scores could increase (especially with tutoring or taking classes), generally, you’re getting just about the same score each time you take it. In our investing example above, the test was not reliable in the sense that over the course of three days, our investor had three very different scores.
- Valid: The second general requirement for a good test is that it is valid. In order for a test to be valid, it first has to be reliable. Validity refers to the test’s ability to measure what it says it is measuring. In other words, if I’m trying to measure your conscientiousness, I should make sure that the test I use has research backing up its ability to measure conscientiousness. When it comes to psychological risk tolerance assessments, there should be some documentation for the test you are taking that relates its results to other, similar types of assessments.
- Backed by research: Related to the points above, psychometrics relies on having solid research and data behind the test. Background research should serve as a starting point for test development (for example, using a well-researched model of personality for a personality test), and data collection and analysis should provide the verification that the test is, in fact, reliable and valid.
If you are using or taking an assessment, evaluate how the assessment was developed and how well it does what it says it’s doing. If it seems too game-like, provides results that appear way off (nurse versus accountant), or gives you a different conclusion based on the weather (or the markets), you may want to reconsider how useful the results are for any type of decision-making or self-improvement efforts.