Photographs As Items for Assessment – Free Example

Photographs of people in activity is a promising newer area for development of business-relevant assessments that has been in use for years in healthcare.  Originally developed in the Netherlands to help patients suffering from fear of pain when moving the body (kinesiophobia), the University of Maastricht’s website has details on citations and free compressed (zipped file) short version of the main test.

Clearly, this same approach could be used to develop more engaging employee and organizational assessments that may be difficult to fake, have better face validity, and more workplace fidelity than other types of items.  Further, with cheap and even free video sites, video items could also play a bigger role in future assessments.

Consider these possible fruitful examples.

a) Vocational Interest Assessment

Vocational interest tests help people identify career paths for which their interests, values, and aptitudes are particularly suited.  But most all are purely text-based.  What if each career alternative had photographs of the tasks in each job or job family, with video vignettes of major tasks?  Perhaps this could be a fun way to assess what activities and careers would ultimately help the person realize their goals.  Take another look at the picture at the top of this article.  It’s an actual picture from PHODA’s assessment, but couldn’t it represent the task of lifting articles out of a trunk for the job of a taxicab driver?

b) Employee Selection

Cognitive and knowledge-based tests are often used to select new employees, but not nearly as often or instead of the ubiquitous job interview.  What if good instruments could be developed, perhaps with a combination of item types, to include pictures?  People could rate pictures like this one on the degree to which it looks similar to their desk – would you expect highly conscientious people to endorse this picture?

I would guess that highly conscientious and prudent people would be unlikely to indicate that this picture reflects their own office.  Sales Convention pictures would be good for the high-end of extraversion; Police taking down violent offenders for low levels of agreeableness.  The potential for pre-hire selection, especially using to add to Computer-Adaptive Testing item banks is tremendous.

c) Culture & Climate

Static pictures may be difficult to identify that reflect various organizational cultural differences, but videos could certainly be used to assess these. 

Limitations
As optimistic as I am about the potential for picture-based items to take a larger role in organizational assessment, I recognize there are also downsides.  First, while digital cameras are cheap, actors may not be.  If you can find existing workplaces where you can take these pictures, it may help you avoid hiring actors for static pictures, but perhaps not for videos that could really suffer with amateur actors.

Second, one New Zeland user of the PHODA complains that if the photographs are context-specific, they can loose value in other contexts.  I remember once when I worked for AT&T Microelectronics, we hired Wally Borman to redo his 1970’s era rater training videos because while the content was good, the actors wore sideburns, bell-bottoms and leisure suits.  This was never going to be very persuasive as “cutting edge” to managers in a bleeding-edge semiconductor factory (computer chips).

Do you see the same potential for photograph-based items as The Scientific Leader?

Assessment, Test, Quiz, Survey, Questionnaire, Psychometrics, Photograph, Photo, Video, YouTube, Computer Adaptive Test, CAT, Measurement, Rasch, IRT, Item Response Theory, Item Bank, Vocational Interest, Selection, Culture, Climate, Free, Freebie

Leader Due Diligence – Psychometrics as Part of Financial Transparency?

With the Bernie Madoff being the latest in a series of massive financial frauds caused by leaders who misrepresented themselves, the time may have come to broaden the financial world’s definition of “transparency”.  I’d like to offer a broader view to include publicly reported reports on leadership knowledge, skills, abilities, traits, values and interests.  Would you have invested in Madoff’s Ponzie scheme if you had previously reviewed a report from a trusted authority on leadership assessment that noted he is low on conscientiousness and prudence?  How would a board view this same report on a founder-CEO?

How well do you know your leaders?

Poor leadership is common, but leaders rarely fail in such a public way.  In one study of nearly 400 Fortunte 1000 companies, 47% of executives and managers rated their company’s overall leadership as fair or poor; and only 8% rated it as excellent (Csoka, 1998).  Personality traits predict both performance and ineffective leadership.  For example, conscientiousness is one of the “Big 5” factors of normal personality that has been shown to consistently predict both job performance and dishonest behavior in the worklpace. Former professors of mine, Robert and Joyce Hogan have written extensively about this area, and have authored some of the better classical test theory instruments for normal personality, the “dark side” or disfunctional leadership, and leader motives, values and preferences.  None of these sorts of assesments are typically used systematically to plan CEO development in private by the board.  And it is entirely unheard of for these reports to be shared publicly with prospective customers, partners and shareholders.  Perhaps we should reconsider making these transparent, systematically, given the risk and lack of confidence in markets of late?   The free paper I drafted, “The Three Stooges of Operational Risk: Advances in Leadership Due Diligence and Rasch Measurement” proposes a way of improving our leadership assessments.  If desired, they could be used for this transparency purpose.   I welcome your feedback.

Special thanks to Alexei M for inspiring this idea.

References

Csoka, L. S. (1998).  Bridging the Leadership Gap.  New York: Conference Board.

Hogan, R., Curphy, G., & Hogan, J. (1994).  What We Know About Leadership: Effectiveness & Personality.  American Psychologist 49(6), 493-504.

Robie, C., Brown, D., & Bly, P. (2008, March).  Relationship Between Major Personality Traits and Managerial Performance: Moderating Effects of Derailing Traits.  International Journal of Management, 25(1), 131-139.

Madoff Destroys $50 Billion with “Giant Ponzie Scheme”

Bernie Madoff is the latest in the series of senior executives to destroy value, this time with an apparent $50 billion dollar fraud, according to the Financial Times.  Madoff, a former Chairman of the NASDAQ stock market, on thursday admitted to his employees including his two sons that his operations were “all just one big lie” and “basically, a giant Ponzi scheme”.  The alleged fraud is the largest ever investor fraud ever blamed on a single individual.

Previously, I had written about the “Three Stooges of Operational Risk“, where I detailed senior executive destruction from Key Lay of Enron, Bernie Evers of Worldcom and most recently, Dick Fuld‘s follies with Lehman Brothers.  In two of those three I noted the dishonesty and fraud that accounted for their downfall similar to Madoff.  But unlike Madoff, they were less candid about thair fraud.  After Madoff’s brazen alleged admission, is there any uncertainty that leadership due dilligence is a critical part of the selection process of hiring senior executives?  Could it be any more clear that the pre-hire assessment procedure is a non-trivial subset of Enterprise Risk Management?

In fairness, these Industrial Organizational Psychology methods have their limitations.  No forecast could ever be perfect, or and even the best assessment procedures only account for 30-60% of the variance in job performance.  But it’s relatively rare that factors such as conscientiousness are used to screen executives – and conscientiousness highly predicts dishonest, and imprudent behavior in the workplace like that of Madoff.  With new methods from Rasch Measurement, Computer-Adaptive Testing, and an innovation from the Scientific Leader, “Inverted Computer Adaptive Testing” using Virtual Realtity, it’s increasingly difficult for people to fake or misrepresent themselves on these assessments. 

How much risk are you accepting when you use standard interviews to hire your employees?

Madoff, Ponzie, Fraud, NASDAQ, Selection, Pre-hire, Assessment, Psychometrics, Rasch, Item Response Theory, IRT, personnel, HR, Due Diligence

Utah Leadership Supports Computer-Adaptive Testing In Spite of “No Bureaucrat Left Behind” Act

Nine schools in Utah have found the benefits of Computer-Adaptive Testing to trump older methods.  Adaptive tests change to match a student’s skill level, avoiding wasted time and effort on questions that are far below or above their proficiency level.  They’re also at least 20% shorter.  This allows for periodic reassessment, and personalized focus on the specific curricular areas a learner needs to work on.  Each student is treated as a special, unique person.

But the US Federal Government’s Department of Education is behind the times, and making it difficult for Utah to use the modern psychometric methods, according to Utah’s Daily Herald.  The “No Child Left Behind Act” requires outdated, non-adaptive methods to be used in addition to the modern approaches.  While on the surface, the DoE’s request for peer review is something that is good, in applied settings, it’s rarely used.  The instruments I’ve developed would certainly pass the scrutiny of my peers, and the feedback they give is useful.  But these extra steps are typically unnecessary to ensure that instrumentation is useful, as long as professionals develop the Computer-Adaptive Tests.  It’s downright destructive to children for the federal government to force Utah to use outdated, longer, and less precise measures of learning.  While I presume those favored by Washington are “peer reviewed”, I suspect that the review committee is selected by those who are friends of politicians, and are likely unskilled in the recent developments in computer-adaptive measurement.

Fortunately, Utah appears to have visionary, contemporary leadership about steadfastly supporting good measurement to help children learn.  The Utah Legislature, the State School Board and Governor all approved the plan to continue to use it – and the Feds require the outdated assessments to be used as well.  This is both a hassle, unnecessary cost, and an opportunity cost – the children could have been spending the time they’ll take on the DoE tests on learning something new.  Are you a visionary leader like the folks in Utah?  More by The Scientific Leader on Computer-Adaptive Measurement, applied to organizations and business is free here.

Leadership, Computer Adaptive Testing, CAT, Government, Bureaucrat, Learning, Training, Education, Child, No Child Left Behind, Psychometrics, IRT, Rasch Measurement

Clash of the Psychometric Titans: Rasch and IRT

Does the approach you take to measuring your customers, employees or patients really matter?  If bad decisions are costly, then yes it really does.  As an Industrial/Organizational Psychologist, I was taught about the science of human measurement.  This included historical treatment of “true score” or classical test theory and also Item Response Theory (IRT).   Classical approaches are slow to create, and require comparisons with others (“norms”) to make sense of them.  But when you measure temperature with a thermometer, do you need to know the distribution of other thermometers in the area for it to make sense?  No – physical and biological sciences have a long history of successful measurement prior to Social Sciences pseudo-measurement approach. 

IRT is relatively better than Classical Test Theory, however, it violates some of the physical science axioms for measurement, is very complex, requires large sample sizes, and produces weird results that are nearly impossible to explain to the untrained.

Unfortunately, I/O Psychologists like me don’t get training in Rasch Measurement, other than believing that it’s the same as the simplest form of IRT.  This isn’t accurate – rather, these are two competing paradigms, and 12 years past my Ph.D., I’ve decided to dedicate myself to Rasch Measurement for both practical and scientific reasons. 

Practical Benefits of Rasch:

• Smaller, unrepresentative samples are sufficiently useful
• Accuracy and precision have same meaning in psychology as in physics and biology.  This makes it easier to communicate with physical and biological-science colleagues
• People and items are on the same “ruler”.  For development, this is extremely useful to help focus learning on areas that are “just right” and not too hard or easy.
• Raw scores are sufficient.  When communicating the results of a test, quiz, or assessment, this is essential.  With IRT, you can have a lower raw score but have a higher result – good luck explaining that to parents and juries.

After taking two terrific classes from Mike Linacre, I changed my mind and switched all my practice and science to Rasch.  Statistics.com still has the classes, both in the basics, advanced methods and the extremely powerful Many-Facet Rasch Measurement that he invented.   While the class format is a bit awkward and highly self paced, Linacre is an enthusiastic and responsive teacher.  Highly recommended.

Psychometrics, Item Response Theory, IRT, Rasch Measurement, Linacre, Statistics, Facets, Practical, Practitioner

Pandora’s Box Worth Opening: Example of Future Rasch Measurement

I’m really amazed at the combination of the Music Genome Project and my iPhone.  Pandora Radio is a free service that allows you to stream customized music “stations” to your computer or your iPhone.  Today, I was able to stream it to my iPhone and pipe it into my car through the Auxiliary plug.  By selecting a specific song or genre, it is able to use its’ information on similar other songs and create an unlimited series of songs similar to the one you like.  Better still, you can have it mix across channels. And any new channels you create on either your iPhone or computer are automatically available on all your devices.

As each song plays, you’re able to give it a “thumbs up” or “thumbs down” to indicate whether or not you like the song.  Favorable and unfavorable ratings help the system adjust future selections to be in the same general domain as the original song, but will be more similar to those you liked.  These binary or dichotomous selections can be considered the most basic form of data useful for productive Rasch Measurement.  It’s an excellent example of why the Rasch model has so much potential in the Web 2.0 world where people are constantly sharing and rating things on Facebook, LinkedIn, Amazon.com and other social websites.

The current version is useful, but appears to be uninformed by psychometrics.  It is based on ratings of over 400 attributes to describe songs, and an engineering algorithm to organize them.  Each song is represented by a list of attributes.  Wikipedia’s entry describes relatively atheoretical approach used to parse the attributes into “genes”, and then recommend similar alternatives to create a “channel”.  The firm has several patents, both for the algorithm, playlist technology and interface on the technology.

John Michael Linacre

But apparently 10% of songs are rated by more than one “technician” to try to improve reliability.  From the description on Wikipedia, it appears the firm isn’t using it’s data to the fullest.  Pandora has over 2 million users, as of December 2, 2008.  First, even if there were only 500 or so ratings of each song, or genre, the users “thumbs up” and “thumbs down” would be excellent fodder to analyze with Rasch Measurement software that could further improve both song and listener measurement, on the same “ruler” of interest.  Second, whether technicians or listeners, some people are likely to be easy thumbsters and others hard.  This leniency and severity bias in ratings can be adjusted with Mike Linacre’s Many Facet Rasch Model (MFRM) method.  Linacre also happens to write the two most popular Rasch analysis programs, Winsteps and Facets.  Highly robust to missing data, this approach could really help Pandora improve assessment of both the songs, Genres, and listeners simultaneously – especially with a form of Computer-Adaptive Testing.

While Pandora Radio is a fantastic overall offerring today, my only other suggestion for improvement is to allow it to be used outside the USA.  When I was in India a few months ago, I learned the hard way that Pandora’s geolocation software could detect that my IP address was abroad, and refused to send me my normal stations, until I returned to the states.  I didn’t try US-based proxies to try to still connect, but hopefully, both Rasch Measurement and Global Licensing will be future features they’ll be adding in the future.

Music, Rasch, Measurement, Web 2.0, Psychometrics, iPhone, iPod, Music Genome, ratings, customer, algorithm, MFRM, Linacre, Winsteps, Facets, geolocation

India’s Harvard to Use Computer Adaptive Testing

The premier management schools in India, the Indian Institutes of Management (IIMs) are migrating their common admission test to a computer adaptive test, using the same technologies deployed in the US to certify physicians, and offered by The Scientific Leader.  The Business Standard reports that IIM is seeking better information on the performance students, hence their desire to migrate to the most sophisticated form of human assessment available in the field of psychometrics.  IIM will use their CAT system to assess over 250,000 students for only 1,800 seats at IIM and other business schools.  With those large numbers taking the test, IIM will also benefit from adaptive testing’s improved question security – as every person gets a personalized test, and it’s rare that two people get the same questions with adaptive measurement.  CATs are unrelated to any sort of measurement of felines.

Are you tired of tedious tests?  Worried about the security of your high-stakes tests?  I encourage you to check out The Scientific Leader’s free whitepaper on the applications of Computer Adaptive Testing to leadership assessment.

India, IIM, Assessment, Test, Admission, CAT, Computer Adaptive Test, Psychometrics, Measurement, Rasch, IRT, Item Response Theory

American and European Human Capital Options

A derivative is a financial instrument that gets it’s value from something else.  One special type is an “option” that gives the owner a right, but not the obligation, to buy or sell an asset.  Human resource, or personnel, employment is a special case of Real Options applied to people. In our case, though, we own ourselves and our labor, but for the right price we’re willing to lease it out to employers and clients for a period of time. 

In the case of employment, hiring an employee amounts to buying (call option) the right to utilize labor for 8 hours a day.  In some cases, the terms of an employment contract are extremely limited, such as contracts for professional athletes and union members.  These amount to a European option, such as the case where the owner of a baseball team has the right to release an athlete, on but not before a certain date.  The more flexible type of employment arrangement is the American option variety.  With an American option, the owner can exercise their right (e.g. liquidate the asset, or fire the employee) without respect to a specific date.  In option terms, letting a person go, or not hiring a consultant for another project is called a “put option”.

But with human assets, there are numerous other options employers can choose to take – including redeploying people to work on new projects, in new departments and in different jobs.  The more flexible a person in having skills and motivation to work in areas that are profitable to the firm, the more valuable the person. 

When mixed with psychometrics such as The Scientific Leader’s Computer-Adaptive Measurement(TM) approach, human capital can be valued the same way as other uncertain financial assets – using Managerial Real Options.  Have you valued your human capital the same way as professional financiers?  Do you know where your human capital is best deployed in the portfolio of job tasks and projects in your firm?  How flexible and adaptable is your workforce – to be able to redeploy them to new work as customers, markets, and economic crises unfold?  Have you considered the option of investing in growing the skills of your employees to increase their value and reduce your risk, through training and development?

Real Option, Management, Business, Human Capital, Skills, Valuation, Risk, Enterprise Risk Management, Psychometrics, Computer Adaptive Testing, CAT, Computer Adaptive Measurement, Rasch, Training, Learning, Selection, European, American

Market Yodels for New Yahoo CEO

Yahoo’s stock rose 17% today, after CEO and co-founder Jerry Yang announced his decision to step down from the top post., according to Bloomberg.

Yang was a leader who co-founded Yahoo, which in its’ hayday was one of the premier new economy internet companies.  But in the last decade, Yahoo has not maintained it’s innovative streak, and lost search engine share to Google and Microsoft.   Yang’s brief 17-month tenure was marked by failed mergers and partnerships that caused the stock price to plunge, reports CNN.  The New York Times reports that one of his highest profile failures was rejecting a $44B acquisition offer from Microsoft last spring.  Could leadership science have predicted this with Yang at the lead?

Perhaps.  The “Full Range” leadership model by Bass & Avolio is supported by 20+ years of research showing that charismatic, transformational leaders outperform their counterparts.  Their work strongly shows that followers walk over hot coals for leaders who connect with their people as individuals, provide intellectual stimulation and walk their talk.  According to CNET, Yang’s early November performance at a Web 2.0 summit was described as a “train wreck”, “self-delusional” and “making a mockery of the vaunted company he helped create”.  They argue that Yang can’t sell his vision inside or outside the company, that

 “He lacks the out-sized personality and charisma that is needed to inspire confidence in battles for the soul of a company.  He has to convince employees, shareholders, customers, and partners
that no matter how difficult the situation, he can lead Yahoo to the
promised land. Think reality-distortion field Steve Jobs, no-software
Marc Benioff, dancing bear Steve Ballmer, the disarming Howard
Stringer, the professorial Eric Schmidt, or the preacher John Chambers.
Bill Gates doesn’t have the most charismatic or endearing personality,
but he manages to control interviews, delivering the messages he wants.“

Because there is substantial evidence that the sorts of charismatic leadership factors predict leadership performance, or in Yahoo’s case, non-performance, The Scientific Leader’s computer-adaptive assessments of leadership are based on Bass & Avolio’s work.  I invite you to make your own judgment of Mr. Yang by reviewing this link of one of his last interviews before stepping down.  Is this interview a trainwreck?  Is he charismatic to you?

Technorati Tags: Leadership, Business, Yang, CEO, Yahoo, Assessment, Transformation, Charisma, Psychometric, Computer-Adaptive Testing

Free Whitepaper: Three Stooges of Operational Risk

 

The Real Three Stooges

Who are the Three Stooges of the recent Financial Meltdowns?  How do you detect and mitigate leadership risk?  The Scientific Leader is pleased to present a free whitepaper that gives an overview of modern methods of Enterprise Risk Management, combined with Industrial/Organizational Psychology to detect and mitigate Leadership, Culture and Employee Risks. Are you using science to manage your talent pools?