Tag Archives: Computer Adaptive Testing

Australian CAT for Kids

The Australian State of Victoria’s standard educational assessments include computer-adaptive tests (CATs), reports their new, free manual on report interpretation.  I was pleased to discover that the Victorian Curriculum and Assessment Authority uses the most modern form of human assessment to help children of all ages learn.

In particular, it is noteworthy that their easy-to-read manual includes an understanding of Rasch Measurement.  It notes the specific locations where there are items that are out of scope for a given assessment.  In these places, the child is mismatched with the test – the questions are either too hard or too easy to produce a trustworthy metric.

I’m hopeful that Australia’s educational leadership rubs off on more schools around the world.

, , , , , , , , , , ,

CAT for Senior Citizens

Professor Jette

Can Computer-Adaptive Testing Help Senior Citizens?  Research from Boston University suggests it can.  Professor Alan Jette, Director of the Boston University Health & Disability Research Institute published a recent paper examining disability assessments in traditional and computer forms.  With data from 671 older adults residing in care facilities, CAT compared favorably with fixed-form scales, even for a version of the CAT with only 10 questions.  In his study, each CAT was administered in less than three minutes, and were highly correlated with the original instrument.

His research strongly suggests that in situations where time is a scarce resource, and measurement fidelity is still important, that Computer-Adaptive Measurement approaches are often more useful than others.

, , , , , , ,

Leader Due Diligence – Psychometrics as Part of Financial Transparency?

With the Bernie Madoff being the latest in a series of massive financial frauds caused by leaders who misrepresented themselves, the time may have come to broaden the financial world’s definition of “transparency”.  I’d like to offer a broader view to include publicly reported reports on leadership knowledge, skills, abilities, traits, values and interests.  Would you have invested in Madoff’s Ponzie scheme if you had previously reviewed a report from a trusted authority on leadership assessment that noted he is low on conscientiousness and prudence?  How would a board view this same report on a founder-CEO?

How well do you know your leaders?

How well do you know your leaders?

Poor leadership is common, but leaders rarely fail in such a public way.  In one study of nearly 400 Fortunte 1000 companies, 47% of executives and managers rated their company’s overall leadership as fair or poor; and only 8% rated it as excellent (Csoka, 1998).  Personality traits predict both performance and ineffective leadership.  For example, conscientiousness is one of the “Big 5” factors of normal personality that has been shown to consistently predict both job performance and dishonest behavior in the worklpace. Former professors of mine, Robert and Joyce Hogan have written extensively about this area, and have authored some of the better classical test theory instruments for normal personality, the “dark side” or disfunctional leadership, and leader motives, values and preferences.  None of these sorts of assesments are typically used systematically to plan CEO development in private by the board.  And it is entirely unheard of for these reports to be shared publicly with prospective customers, partners and shareholders.  Perhaps we should reconsider making these transparent, systematically, given the risk and lack of confidence in markets of late?   The free paper I drafted, “The Three Stooges of Operational Risk: Advances in Leadership Due Diligence and Rasch Measurement” proposes a way of improving our leadership assessments.  If desired, they could be used for this transparency purpose.   I welcome your feedback.

Special thanks to Alexei M for inspiring this idea.

References

Csoka, L. S. (1998).  Bridging the Leadership Gap.  New York: Conference Board.

Hogan, R., Curphy, G., & Hogan, J. (1994).  What We Know About Leadership: Effectiveness & Personality.  American Psychologist 49(6), 493-504.

Robie, C., Brown, D., & Bly, P. (2008, March).  Relationship Between Major Personality Traits and Managerial Performance: Moderating Effects of Derailing Traits.  International Journal of Management, 25(1), 131-139.

Utah Leadership Supports Computer-Adaptive Testing In Spite of “No Bureaucrat Left Behind” Act

Nine schools in Utah have found the benefits of Computer-Adaptive Testing to trump older methods.  Adaptive tests change to match a student’s skill level, avoiding wasted time and effort on questions that are far below or above their proficiency level.  They’re also at least 20% shorter.  This allows for periodic reassessment, and personalized focus on the specific curricular areas a learner needs to work on.  Each student is treated as a special, unique person.

But the US Federal Government’s Department of Education is behind the times, and making it difficult for Utah to use the modern psychometric methods, according to Utah’s Daily Herald.  The “No Child Left Behind Act” requires outdated, non-adaptive methods to be used in addition to the modern approaches.  While on the surface, the DoE’s request for peer review is something that is good, in applied settings, it’s rarely used.  The instruments I’ve developed would certainly pass the scrutiny of my peers, and the feedback they give is useful.  But these extra steps are typically unnecessary to ensure that instrumentation is useful, as long as professionals develop the Computer-Adaptive Tests.  It’s downright destructive to children for the federal government to force Utah to use outdated, longer, and less precise measures of learning.  While I presume those favored by Washington are “peer reviewed”, I suspect that the review committee is selected by those who are friends of politicians, and are likely unskilled in the recent developments in computer-adaptive measurement.

Fortunately, Utah appears to have visionary, contemporary leadership about steadfastly supporting good measurement to help children learn.  The Utah Legislature, the State School Board and Governor all approved the plan to continue to use it – and the Feds require the outdated assessments to be used as well.  This is both a hassle, unnecessary cost, and an opportunity cost – the children could have been spending the time they’ll take on the DoE tests on learning something new.  Are you a visionary leader like the folks in Utah?  More by The Scientific Leader on Computer-Adaptive Measurement, applied to organizations and business is free here.

, , , , , , , , , , , ,

American and European Human Capital Options

A derivative is a financial instrument that gets it’s value from something else.  One special type is an “option” that gives the owner a right, but not the obligation, to buy or sell an asset.  Human resource, or personnel, employment is a special case of Real Options applied to people. In our case, though, we own ourselves and our labor, but for the right price we’re willing to lease it out to employers and clients for a period of time. 

In the case of employment, hiring an employee amounts to buying (call option) the right to utilize labor for 8 hours a day.  In some cases, the terms of an employment contract are extremely limited, such as contracts for professional athletes and union members.  These amount to a European option, such as the case where the owner of a baseball team has the right to release an athlete, on but not before a certain date.  The more flexible type of employment arrangement is the American option variety.  With an American option, the owner can exercise their right (e.g. liquidate the asset, or fire the employee) without respect to a specific date.  In option terms, letting a person go, or not hiring a consultant for another project is called a “put option”.

But with human assets, there are numerous other options employers can choose to take – including redeploying people to work on new projects, in new departments and in different jobs.  The more flexible a person in having skills and motivation to work in areas that are profitable to the firm, the more valuable the person. 

When mixed with psychometrics such as The Scientific Leader’s Computer-Adaptive Measurement(TM) approach, human capital can be valued the same way as other uncertain financial assets – using Managerial Real Options.  Have you valued your human capital the same way as professional financiers?  Do you know where your human capital is best deployed in the portfolio of job tasks and projects in your firm?  How flexible and adaptable is your workforce – to be able to redeploy them to new work as customers, markets, and economic crises unfold?  Have you considered the option of investing in growing the skills of your employees to increase their value and reduce your risk, through training and development?

, , , , , , , , , , , , , , , , ,

Market Yodels for New Yahoo CEO

Yahoo’s stock rose 17% today, after CEO and co-founder Jerry Yang announced his decision to step down from the top post., according to Bloomberg.

Yang was a leader who co-founded Yahoo, which in its’ hayday was one of the premier new economy internet companies.  But in the last decade, Yahoo has not maintained it’s innovative streak, and lost search engine share to Google and Microsoft.   Yang’s brief 17-month tenure was marked by failed mergers and partnerships that caused the stock price to plunge, reports CNN.  The New York Times reports that one of his highest profile failures was rejecting a $44B acquisition offer from Microsoft last spring.  Could leadership science have predicted this with Yang at the lead?

Perhaps.  The “Full Range” leadership model by Bass & Avolio is supported by 20+ years of research showing that charismatic, transformational leaders outperform their counterparts.  Their work strongly shows that followers walk over hot coals for leaders who connect with their people as individuals, provide intellectual stimulation and walk their talk.  According to CNET, Yang’s early November performance at a Web 2.0 summit was described as a “train wreck”, “self-delusional” and “making a mockery of the vaunted company he helped create”.  They argue that Yang can’t sell his vision inside or outside the company, that

 “He lacks the out-sized personality and charisma that is needed to inspire confidence in battles for the soul of a company.  He has to convince employees, shareholders, customers, and partners
that no matter how difficult the situation, he can lead Yahoo to the
promised land. Think reality-distortion field Steve Jobs, no-software
Marc Benioff, dancing bear Steve Ballmer, the disarming Howard
Stringer, the professorial Eric Schmidt, or the preacher John Chambers.
Bill Gates doesn’t have the most charismatic or endearing personality,
but he manages to control interviews, delivering the messages he wants.

Because there is substantial evidence that the sorts of charismatic leadership factors predict leadership performance, or in Yahoo’s case, non-performance, The Scientific Leader’s computer-adaptive assessments of leadership are based on Bass & Avolio’s work.  I invite you to make your own judgment of Mr. Yang by reviewing this link of one of his last interviews before stepping down.  Is this interview a trainwreck?  Is he charismatic to you?

Technorati Tags: , , , , , , , , ,

Improved Leadership Due Diligence White Paper

Enron's Ken Lay

Key Lay of Enron

I’m very grateful to all the helpful words of encouragement about my free whitepaper, “The Three Stooges of Operational Risk: Advances in Leadership Due Diligence and Rasch Measurement“.  I made numerous additions to the white paper including:

a) Detailed treatment of the evidence-based definition of leadership

b) Virtual Reality-based Assessment

c) Six Sigma / Industrial Engineering-inspired improvements to detecting faking

and much more.  I would greatly appreciate more feedback.  Is the business case compelling to you?  How much do traditional methods hurt your ability to assess and mitigate executive risks?

Technorati Tags: , , , , , , , , , ,