Tag Archives: Measurement

Photographs As Items for Assessment – Free Example

Photographs of people in activity is a promising newer area for development of business-relevant assessments that has been in use for years in healthcare.  Originally developed in the Netherlands to help patients suffering from fear of pain when moving the body (kinesiophobia), the University of Maastricht’s website has details on citations and free compressed (zipped file) short version of the main test.

Clearly, this same approach could be used to develop more engaging employee and organizational assessments that may be difficult to fake, have better face validity, and more workplace fidelity than other types of items.  Further, with cheap and even free video sites, video items could also play a bigger role in future assessments.

Consider these possible fruitful examples.

a) Vocational Interest Assessment

Vocational interest tests help people identify career paths for which their interests, values, and aptitudes are particularly suited.  But most all are purely text-based.  What if each career alternative had photographs of the tasks in each job or job family, with video vignettes of major tasks?  Perhaps this could be a fun way to assess what activities and careers would ultimately help the person realize their goals.  Take another look at the picture at the top of this article.  It’s an actual picture from PHODA’s assessment, but couldn’t it represent the task of lifting articles out of a trunk for the job of a taxicab driver?

b) Employee Selection

Cognitive and knowledge-based tests are often used to select new employees, but not nearly as often or instead of the ubiquitous job interview.  What if good instruments could be developed, perhaps with a combination of item types, to include pictures?  People could rate pictures like this one on the degree to which it looks similar to their desk – would you expect highly conscientious people to endorse this picture?

I would guess that highly conscientious and prudent people would be unlikely to indicate that this picture reflects their own office.  Sales Convention pictures would be good for the high-end of extraversion; Police taking down violent offenders for low levels of agreeableness.  The potential for pre-hire selection, especially using to add to Computer-Adaptive Testing item banks is tremendous.

c) Culture & Climate

Static pictures may be difficult to identify that reflect various organizational cultural differences, but videos could certainly be used to assess these. 

As optimistic as I am about the potential for picture-based items to take a larger role in organizational assessment, I recognize there are also downsides.  First, while digital cameras are cheap, actors may not be.  If you can find existing workplaces where you can take these pictures, it may help you avoid hiring actors for static pictures, but perhaps not for videos that could really suffer with amateur actors.

Second, one New Zeland user of the PHODA complains that if the photographs are context-specific, they can loose value in other contexts.  I remember once when I worked for AT&T Microelectronics, we hired Wally Borman to redo his 1970’s era rater training videos because while the content was good, the actors wore sideburns, bell-bottoms and leisure suits.  This was never going to be very persuasive as “cutting edge” to managers in a bleeding-edge semiconductor factory (computer chips).

Do you see the same potential for photograph-based items as The Scientific Leader?

, , , , , , , , , , , , , , , , , , , , , ,

Pandora’s Box Worth Opening: Example of Future Rasch Measurement

I’m really amazed at the combination of the Music Genome Project and my iPhone.  Pandora Radio is a free service that allows you to stream customized music “stations” to your computer or your iPhone.  Today, I was able to stream it to my iPhone and pipe it into my car through the Auxiliary plug.  By selecting a specific song or genre, it is able to use its’ information on similar other songs and create an unlimited series of songs similar to the one you like.  Better still, you can have it mix across channels. And any new channels you create on either your iPhone or computer are automatically available on all your devices.

As each song plays, you’re able to give it a “thumbs up” or “thumbs down” to indicate whether or not you like the song.  Favorable and unfavorable ratings help the system adjust future selections to be in the same general domain as the original song, but will be more similar to those you liked.  These binary or dichotomous selections can be considered the most basic form of data useful for productive Rasch Measurement.  It’s an excellent example of why the Rasch model has so much potential in the Web 2.0 world where people are constantly sharing and rating things on Facebook, LinkedIn, Amazon.com and other social websites.

The current version is useful, but appears to be uninformed by psychometrics.  It is based on ratings of over 400 attributes to describe songs, and an engineering algorithm to organize them.  Each song is represented by a list of attributes.  Wikipedia’s entry describes relatively atheoretical approach used to parse the attributes into “genes”, and then recommend similar alternatives to create a “channel”.  The firm has several patents, both for the algorithm, playlist technology and interface on the technology.

J. "Mike" Linacre

John Michael Linacre

But apparently 10% of songs are rated by more than one “technician” to try to improve reliability.  From the description on Wikipedia, it appears the firm isn’t using it’s data to the fullest.  Pandora has over 2 million users, as of December 2, 2008.  First, even if there were only 500 or so ratings of each song, or genre, the users “thumbs up” and “thumbs down” would be excellent fodder to analyze with Rasch Measurement software that could further improve both song and listener measurement, on the same “ruler” of interest.  Second, whether technicians or listeners, some people are likely to be easy thumbsters and others hard.  This leniency and severity bias in ratings can be adjusted with Mike Linacre’s Many Facet Rasch Model (MFRM) method.  Linacre also happens to write the two most popular Rasch analysis programs, Winsteps and Facets.  Highly robust to missing data, this approach could really help Pandora improve assessment of both the songs, Genres, and listeners simultaneously – especially with a form of Computer-Adaptive Testing.

While Pandora Radio is a fantastic overall offerring today, my only other suggestion for improvement is to allow it to be used outside the USA.  When I was in India a few months ago, I learned the hard way that Pandora’s geolocation software could detect that my IP address was abroad, and refused to send me my normal stations, until I returned to the states.  I didn’t try US-based proxies to try to still connect, but hopefully, both Rasch Measurement and Global Licensing will be future features they’ll be adding in the future.

, , , , , , , , , , , , , , ,