Tag Archives: Rasch Measurement

Australian CAT for Kids

The Australian State of Victoria’s standard educational assessments include computer-adaptive tests (CATs), reports their new, free manual on report interpretation.  I was pleased to discover that the Victorian Curriculum and Assessment Authority uses the most modern form of human assessment to help children of all ages learn.

In particular, it is noteworthy that their easy-to-read manual includes an understanding of Rasch Measurement.  It notes the specific locations where there are items that are out of scope for a given assessment.  In these places, the child is mismatched with the test – the questions are either too hard or too easy to produce a trustworthy metric.

I’m hopeful that Australia’s educational leadership rubs off on more schools around the world.

, , , , , , , , , , ,

Advertisements

Leader Due Diligence – Psychometrics as Part of Financial Transparency?

With the Bernie Madoff being the latest in a series of massive financial frauds caused by leaders who misrepresented themselves, the time may have come to broaden the financial world’s definition of “transparency”.  I’d like to offer a broader view to include publicly reported reports on leadership knowledge, skills, abilities, traits, values and interests.  Would you have invested in Madoff’s Ponzie scheme if you had previously reviewed a report from a trusted authority on leadership assessment that noted he is low on conscientiousness and prudence?  How would a board view this same report on a founder-CEO?

How well do you know your leaders?

How well do you know your leaders?

Poor leadership is common, but leaders rarely fail in such a public way.  In one study of nearly 400 Fortunte 1000 companies, 47% of executives and managers rated their company’s overall leadership as fair or poor; and only 8% rated it as excellent (Csoka, 1998).  Personality traits predict both performance and ineffective leadership.  For example, conscientiousness is one of the “Big 5” factors of normal personality that has been shown to consistently predict both job performance and dishonest behavior in the worklpace. Former professors of mine, Robert and Joyce Hogan have written extensively about this area, and have authored some of the better classical test theory instruments for normal personality, the “dark side” or disfunctional leadership, and leader motives, values and preferences.  None of these sorts of assesments are typically used systematically to plan CEO development in private by the board.  And it is entirely unheard of for these reports to be shared publicly with prospective customers, partners and shareholders.  Perhaps we should reconsider making these transparent, systematically, given the risk and lack of confidence in markets of late?   The free paper I drafted, “The Three Stooges of Operational Risk: Advances in Leadership Due Diligence and Rasch Measurement” proposes a way of improving our leadership assessments.  If desired, they could be used for this transparency purpose.   I welcome your feedback.

Special thanks to Alexei M for inspiring this idea.

References

Csoka, L. S. (1998).  Bridging the Leadership Gap.  New York: Conference Board.

Hogan, R., Curphy, G., & Hogan, J. (1994).  What We Know About Leadership: Effectiveness & Personality.  American Psychologist 49(6), 493-504.

Robie, C., Brown, D., & Bly, P. (2008, March).  Relationship Between Major Personality Traits and Managerial Performance: Moderating Effects of Derailing Traits.  International Journal of Management, 25(1), 131-139.

Electronic Health Information Systems Natural For Rasch Computer Adaptive Testing

I was pleased to discover a new instrument in the Journal, Psychiatry that uses Rasch Measurement to better assess Catatonia.  Catatnonia is a psychiatric condition that groups a number of pathologies ranging from bipolar disorder to drug abuse and schizophrenia.  The authors use a partial-credit Rasch approach that allows for ratings along a behaviorally-anchored rating scale. An example of one of the scale’s partial-credit items appears here:

This research did not benefit from the Electronic Health Record (EHR) or Electronic Health Information Systems that are universally starting to be used in healthcare.  It seems inevitable that as EHRs become ubiquitous in healthcare, that computer-adaptive Rasch scales like the author’s new KANNER scale will be come commonplace.  Whether for patient pain measurement and consequent medication, or psychopathology, or occupational therapy/science, Rasch Measurement has a long history of successful use in these areas – but without the benefit of seamless integration in a CAT in your health record.  I foresee CAT-based assessments being combined with Statistical Process Control (SPC) and Fuzzy Logic to better track patient conditions over time.  In hospital settings, I suspect this will happen first as patients with acute and potentially fatal ailments already are tracked continuously.  But non-acute patients, through wireless devices, could easily benefit from these approaches.  This is particularly true if medicine were to use one of the patents I authored while at Motorola, still pending, that combines computer-adaptive assessment and wireless devices.

This is a good example of the power of interdisciplinary innovations combining for something significantly better.  Rasch Measurement comes from Psychometrics; Statistical Process Control was developed by Industrial/Systems Engineers and Statisticians; Wireless technology comes from Computer Scientists; and the content comes from the various health sciences including medicine, nursing, physical therapy and occupational therapy.

Utah Leadership Supports Computer-Adaptive Testing In Spite of “No Bureaucrat Left Behind” Act

Nine schools in Utah have found the benefits of Computer-Adaptive Testing to trump older methods.  Adaptive tests change to match a student’s skill level, avoiding wasted time and effort on questions that are far below or above their proficiency level.  They’re also at least 20% shorter.  This allows for periodic reassessment, and personalized focus on the specific curricular areas a learner needs to work on.  Each student is treated as a special, unique person.

But the US Federal Government’s Department of Education is behind the times, and making it difficult for Utah to use the modern psychometric methods, according to Utah’s Daily Herald.  The “No Child Left Behind Act” requires outdated, non-adaptive methods to be used in addition to the modern approaches.  While on the surface, the DoE’s request for peer review is something that is good, in applied settings, it’s rarely used.  The instruments I’ve developed would certainly pass the scrutiny of my peers, and the feedback they give is useful.  But these extra steps are typically unnecessary to ensure that instrumentation is useful, as long as professionals develop the Computer-Adaptive Tests.  It’s downright destructive to children for the federal government to force Utah to use outdated, longer, and less precise measures of learning.  While I presume those favored by Washington are “peer reviewed”, I suspect that the review committee is selected by those who are friends of politicians, and are likely unskilled in the recent developments in computer-adaptive measurement.

Fortunately, Utah appears to have visionary, contemporary leadership about steadfastly supporting good measurement to help children learn.  The Utah Legislature, the State School Board and Governor all approved the plan to continue to use it – and the Feds require the outdated assessments to be used as well.  This is both a hassle, unnecessary cost, and an opportunity cost – the children could have been spending the time they’ll take on the DoE tests on learning something new.  Are you a visionary leader like the folks in Utah?  More by The Scientific Leader on Computer-Adaptive Measurement, applied to organizations and business is free here.

, , , , , , , , , , , ,

Clash of the Psychometric Titans: Rasch and IRT

Does the approach you take to measuring your customers, employees or patients really matter?  If bad decisions are costly, then yes it really does.  As an Industrial/Organizational Psychologist, I was taught about the science of human measurement.  This included historical treatment of “true score” or classical test theory and also Item Response Theory (IRT).   Classical approaches are slow to create, and require comparisons with others (“norms”) to make sense of them.  But when you measure temperature with a thermometer, do you need to know the distribution of other thermometers in the area for it to make sense?  No – physical and biological sciences have a long history of successful measurement prior to Social Sciences pseudo-measurement approach. 

IRT is relatively better than Classical Test Theory, however, it violates some of the physical science axioms for measurement, is very complex, requires large sample sizes, and produces weird results that are nearly impossible to explain to the untrained.

Unfortunately, I/O Psychologists like me don’t get training in Rasch Measurement, other than believing that it’s the same as the simplest form of IRT.  This isn’t accurate – rather, these are two competing paradigms, and 12 years past my Ph.D., I’ve decided to dedicate myself to Rasch Measurement for both practical and scientific reasons. 

Practical Benefits of Rasch:

  • Smaller, unrepresentative samples are sufficiently useful
  • Accuracy and precision have same meaning in psychology as in physics and biology.  This makes it easier to communicate with physical and biological-science colleagues
  • People and items are on the same “ruler”.  For development, this is extremely useful to help focus learning on areas that are “just right” and not too hard or easy.
  • Raw scores are sufficient.  When communicating the results of a test, quiz, or assessment, this is essential.  With IRT, you can have a lower raw score but have a higher result – good luck explaining that to parents and juries.

After taking two terrific classes from Mike Linacre, I changed my mind and switched all my practice and science to Rasch.  Statistics.com still has the classes, both in the basics, advanced methods and the extremely powerful Many-Facet Rasch Measurement that he invented.   While the class format is a bit awkward and highly self paced, Linacre is an enthusiastic and responsive teacher.  Highly recommended.

, , , , , , , ,

India’s Harvard to Use Computer Adaptive Testing

The premier management schools in India, the Indian Institutes of Management (IIMs) are migrating their common admission test to a computer adaptive test, using the same technologies deployed in the US to certify physicians, and offered by The Scientific Leader.  The Business Standard reports that IIM is seeking better information on the performance students, hence their desire to migrate to the most sophisticated form of human assessment available in the field of psychometrics.  IIM will use their CAT system to assess over 250,000 students for only 1,800 seats at IIM and other business schools.  With those large numbers taking the test, IIM will also benefit from adaptive testing’s improved question security – as every person gets a personalized test, and it’s rare that two people get the same questions with adaptive measurement.  CATs are unrelated to any sort of measurement of felines.

Are you tired of tedious tests?  Worried about the security of your high-stakes tests?  I encourage you to check out The Scientific Leader’s free whitepaper on the applications of Computer Adaptive Testing to leadership assessment.

, , , , , , , , , , ,

Improved Leadership Due Diligence White Paper

Enron's Ken Lay

Key Lay of Enron

I’m very grateful to all the helpful words of encouragement about my free whitepaper, “The Three Stooges of Operational Risk: Advances in Leadership Due Diligence and Rasch Measurement“.  I made numerous additions to the white paper including:

a) Detailed treatment of the evidence-based definition of leadership

b) Virtual Reality-based Assessment

c) Six Sigma / Industrial Engineering-inspired improvements to detecting faking

and much more.  I would greatly appreciate more feedback.  Is the business case compelling to you?  How much do traditional methods hurt your ability to assess and mitigate executive risks?

Technorati Tags: , , , , , , , , , ,