↓ Skip to main content

Examining the effects of gaming and guessing on script concordance test scores

Overview of attention for article published in Perspectives on Medical Education, June 2018
Altmetric Badge

Mentioned by

facebook
1 Facebook page

Citations

dimensions_citation
4 Dimensions

Readers on

mendeley
16 Mendeley
Title
Examining the effects of gaming and guessing on script concordance test scores
Published in
Perspectives on Medical Education, June 2018
DOI 10.1007/s40037-018-0435-8
Pubmed ID
Authors

Stuart Lubarsky, Valérie Dory, Sarkis Meterissian, Carole Lambert, Robert Gagnon

Abstract

In a script concordance test (SCT), examinees are asked to judge the effect of a new piece of clinical information on a proposed hypothesis. Answers are collected using a Likert-type scale (ranging from -2 to +2, with '0' indicating no effect), and compared with those of a reference panel of 'experts'. It has been argued, however, that SCT may be susceptible to the influences of gaming and guesswork. This study aims to address some of the mounting concern over the response process validity of SCT scores. Using published datasets from three independent SCTs, we investigated examinee response patterns, and computed the score a hypothetical examinee would obtain on each of the tests if he 1) guessed random answers and 2) deliberately answered '0' on all test items. A simulated random guessing strategy led to scores 2 SDs below mean scores of actual respondents (Z-scores -3.6 to -2.1). A simulated 'all-0' strategy led to scores at least 1 SD above those obtained by random guessing (Z-scores -2.2 to -0.7). In one dataset, stepwise exclusion of items with modal panel response '0' to fewer than 10% of the total number of test items yielded hypothetical scores 2 SDs below mean scores of actual respondents. Random guessing was not an advantageous response strategy. An 'all-0' response strategy, however, demonstrated evidence of artificial score inflation. Our findings pose a significant threat to the SCT's validity argument. 'Testwiseness' is a potential hazard to all testing formats, and appropriate countermeasures must be established. We propose an approach that might be used to mitigate a potentially real and troubling phenomenon in script concordance testing. The impact of this approach on the content validity of SCTs merits further discussion.

Mendeley readers

The data shown below were compiled from readership statistics for 16 Mendeley readers of this research output. Click here to see the associated Mendeley record.

Geographical breakdown

Country Count As %
Unknown 16 100%

Demographic breakdown

Readers by professional status Count As %
Student > Master 4 25%
Professor 2 13%
Student > Postgraduate 2 13%
Other 1 6%
Lecturer 1 6%
Other 6 38%
Readers by discipline Count As %
Medicine and Dentistry 6 38%
Unspecified 3 19%
Computer Science 2 13%
Materials Science 2 13%
Physics and Astronomy 1 6%
Other 2 13%

Attention Score in Context

This research output has an Altmetric Attention Score of 1. This is our high-level measure of the quality and quantity of online attention that it has received. This Attention Score, as well as the ranking and number of research outputs shown below, was calculated when the research output was last mentioned on 16 June 2018.
All research outputs
#11,632,215
of 13,092,437 outputs
Outputs from Perspectives on Medical Education
#344
of 363 outputs
Outputs of similar age
#235,205
of 270,351 outputs
Outputs of similar age from Perspectives on Medical Education
#23
of 23 outputs
Altmetric has tracked 13,092,437 research outputs across all sources so far. This one is in the 1st percentile – i.e., 1% of other outputs scored the same or lower than it.
So far Altmetric has tracked 363 research outputs from this source. They typically receive more attention than average, with a mean Attention Score of 9.4. This one is in the 1st percentile – i.e., 1% of its peers scored the same or lower than it.
Older research outputs will score higher simply because they've had more time to accumulate mentions. To account for age we can compare this Altmetric Attention Score to the 270,351 tracked outputs that were published within six weeks on either side of this one in any source. This one is in the 1st percentile – i.e., 1% of its contemporaries scored the same or lower than it.
We're also able to compare this research output to 23 others from the same source and published within six weeks on either side of this one. This one is in the 1st percentile – i.e., 1% of its contemporaries scored the same or lower than it.