Misleading Career "Assessments"

As you are aware, there has been a profusion of new career tests, quizzes, sorters, finders, and profilers on the Internet. In addition, many companies are selling educational or career guidance website systems that use such tests. As a professional, you have probably asked yourself, "Will these measures help me in my work? Can I recommend them to those with whom I work? Are they valid measures?" Unfortunately, the answer in most cases is "No." In fact, most of these new measures are probably harmful.

An Example: The O*NET Interest Profiler (1999). It is offered for free on the Internet by the U.S. Department of Labor. It is also included in web-based educational/career services offered to schools by companies and state agencies sold separately as a paper-pencil career test. It is designed to assess Holland's six types. According to its research, it is a reliable measure. But, unfortunately, it is an invalid measure of Holland's personality types.

The User's Guide, for example, on page 43 reports that there is "gap between the Realistic and Conventional Interest Areas"; the results from these two scales do not fit the Holland Model. In addition, their research shows that the Enterprising and Social scales are flawed. For example, those who are primarily Enterprising are often classified as some other personality type – like Investigative or Artistic. Similarly, a large number of individuals classified as primarily Social are misclassified as primarily Conventional, Enterprising, or Investigative. Their research shows that the computerized version of the Interest Profiler found on the Internet has similar problems. 1, 2, 3, 4

Another form of the IP, the Interest Profiler Short Form, was published in 2010 that the authors say overcomes the problems of the IP Long Form.

Unfortunately, there are no studies published in scientific journals regarding the long or short form of the IP test. When this is done, you know there has been a "blind," peer review by experts in the field. This is expected for any respected test.

It's common sense. You want to say, "Yes," with confidence to the question, "Does this test truly measure the Holland personality types?"

Harmful Effects: How harmful is an invalid career test? Here are several scenarios that illustrate their damaging effects:

Miscategorized

  1. Imagine that:
    1. There are a number of people you are working with who like to lead, persuade and sell, what Holland would call the Enterprising type; and
    2. You have them take the Interest Profiler and their highest score is Artistic – in effect, telling them that they are not interested in Enterprising activities but in doing creative activities like art, drama, or creative writing.

      How does this invalid result affect them? Confusion? Disbelief? How will it affect your efforts in working with them?
  2. Similarly, imagine that you are working with people you know who love helping people (Social). On the Profiler their highest score is for the Investigative type, misinforming them that their primary interest is in solving math and science problems.

    How will this affect their views of themselves? Of the counseling process?

    Of the value of career assessments?

Mismatched

  1. Even more important, imagine the effect of "matching" people with training programs or majors that do not fit their personality? For example, suggesting to persons who are primarily Investigative that they major in Social majors like counseling rather than majors like chemistry or biology?

    Will this discourage further exploration? Encourage entry into an unsuitable major? Will it cause frustration? Confusion? Discouragement? Result in poor academic performance? Post-decisional regret?
  2. What is the effect of not encouraging students to explore academic programs in which they are likely to enjoy and succeed?
  3. What is the impact of encouraging people to consider careers that do not fit them? Will it discourage further career exploration? Encourage them to enter careers in which they will be dissatisfied and perform poorly?

These scenarios show the harm that invalid career tests can cause. Of course, no one should make a career choice based primarily on the results of a career test. But with career tests playing such a central role in computerized career guidance systems and career counseling practice, it is vital that valid career measures be used.

Another invalid example: Career Clusters Interest Survey (CCIS). For years it has been widely promoted by states and the federal government to assess students' interests and guide them in choosing one of the U.S. Department of Education's 16 Career Clusters (and Pathways) -- a program of study to pursue in high school and college.

A recent study of its validity, the first and only one done, shows that it does not measure interests in the Data/Idea area -- four of the six Holland personality types: Enterprising, Conventional, Artistic, and Investigative. (Prime & Tracey, 2010).

In other words, thousands of students (and their parents) are being told their interests are something they are not, and are directed toward programs of study and college majors that do not fit their personality.

Some recommend using "informal assessments" like the CCIS "just for exploration". This is unsound even in the middle school years -- when students' RIASEC interests are fairly stable (Tracey, Robbins & Hofsess, 2005), and they and their parents are beginning to make serious decisions about future schooling.

Invalid Career Measures a Serious Problem
Daily, tens of thousands of people use the Internet for career assistance. Most will take a career assessment, and they have many to choose from. "Career test" on the Google search engine results in more than 8 million hits.

There are also numerous Internet-based educational career guidance systems being sold to schools and other organizations that offer career measures.

But are these web-based career measures valid? Without research, it is impossible to know. Rarely do they offer a professional manual that reports the results of such studies. Knowing that (a) it takes years and substantial funds to develop a valid career measure, and (b) that most of these measures have appeared recently -- it is likely that most of them are invalid and harmful. Some make false claims and several are advertised in professional counseling publications as being "valid and reliable".

Click here to download an article on this issue that appeared in NCDA's Career Developments.

Professional career tests help people,

  • Learn more about themselves,
  • Identify promising careers,
  • Encourage career exploration, and
  • Become informed about the occupations in which they are interested.

Recommendations

  1. "Counselors [shall] carefully consider the validity, reliability, psychometric limitations, and appropriateness of instruments when selecting tests for use in a given situation or with a particular client." -- American Counseling Association's Code of Ethics and Standards of Practice
  2. Ask for a manual and study it closely. Does it contain research studies published in respected professional journals? Does it meet the standards of validity and reliability expected in your profession.
  3. Beware of career measures that are called something else, like career quiz, profiler, sorter, or finder. If individuals are being asked to assess their values, interests, or personality to match with occupations – it is a career test, and it should meet professional standards for its validity and reliability. An example of this is the Office of Science Education's Career Finder.
  4. Ask yourself, "If the tool being used is not a career test, why is the website using it? Is there evidence to show that it is superior to a good career test?"
  5. Check carefully any career assessment to which you link from your website. The public does not have the benefit of your training, knowledge, and experience to judge the merits of career assessments. They depend on your professional judgment.
  6. Be wary of testimonials and links from the websites of professional groups. Likewise, be careful when considering endorsements or certifications from industry associations. The standards for one prominent group, for example, the Association of Computer Based Systems for Career Information, are often vague. The endorsement depends on a loose self-assessment of compliance by the company or organization. The only way you will know if a career test is valid is to study the professional manual.

The use of invalid career tests on the Internet is a serious problem. Several articles have recently appeared in publications of the National Career Development Association, American Counseling Association, and the American School Counselors Association. Raise this issue with the leaders in your profession. Send them the URL for this article.

 

Disclosure: We license the content of the Career Key to educational companies offering web-based services.  We believe that the points in this article are fair and objective, but you will want to decide this for yourself.

  1. Lewis, P, & Rivkin, D. (1999). O*Net Interest Profiler. Raleigh, NC: National Center for O*NET Development.
  2. Rounds, J., Smith, T., Hubert, L., Lewis, P., & Rivkin, D. (1999). O*Net Interest Profiler: Reliability, validity, and self-scoring. Raleigh, NC: National Center for O*NET Development.
  3. Rounds, R., Mazzeo, S. E., Smith, T. J., & Hubert, L. (1999) . O*Net Interest Profiler: Reliability, validity, and comparability. Raleigh, NC: National Center for O*NET Development.
  4. U.S. Department of Labor. (2000). O*Net Interest Profiler, User's Guide. Washington, DC: U.S. Government Printing Office.