Monday, September 19, 2016

Peeking outside of the Wechsler Box

A terrific post from Dr Bridget Regan today. Enjoy!


Like most of us, I have been happily using my Wechsler WAIS and WMS as the backbone of my assessments throughout the 15 years I have been working as a neuropsychologist.  If ever in doubt, administer more subtests and gain more indices with increased psychometric rigor!  I’ve always been reassured that these tools and our face to face approach is the most reliable and most studied in brain disordered groups.  The most attention I have given to technology and neuropsychology was to trial the computer based versions of our existing scales and the vague thought that if possible I would like to be able to administer some tests online or via computer in my practice instead of administering all of them face to face.  However, there have always been some residual concerns about the reliability and validity of self-administered cognitive assessment tools and the challenges involved of keeping test takers motivated.
At the 2016 APS Congress, however, I became aware of some new approaches and technologies that have the potential to substantially improve the quality of our assessments (including online and self-administered) and it made me realise it is now well and truly time to start taking proper notice of what is happening outside of the Wechsler comfort zone! (and to keep reading this blog).  I learnt about these approaches in a conversation with a research psychologist who works for the military (Eugene Aidman also an Adjunct Ass Prof at Sydney Uni), in a talk by Jason Mattingly on the neuroscience of learning, and in the sports psychology seminar on neuro-technologies. 
New approaches include:
1)     CAT – Computerised Adaptive Testing.  This involves an approach to test development in which a very large number of test items are generated and for each item the level of difficulty is measured.  When a person is undertaking the test the computer decides which item to give next based on how well they have answered the previous items (as I understand it using something called Bayesian statistical inference).  CAT has the potential to improve the test experience (with items that match skill level) as well as to reduce the length of the assessment with better targeted items and to better test limits for high and low scorers.  There already seem to be a number of well researched and validated tools in the educational context (e.g., ACCUPLACER).  I’ve searched around the internet and found a few references in plain language such as the following article which describes CAT in educational settings. http://www.edweek.org/dd/articles/2012/10/17/01adaptive.h06.html which I found at the International Association of Computerised Adaptive Testing (IACAT) http://iacat.org/.
I only been able to find a few references to the use of this approach in the development of neuropsychological tests (one using matrix reasoning) but although it might be more challenging to develop than educational type assessments I can imagine it being done– does anyone know of any other new developments in the use of CAT for neuropsych/ cognitive assessment?

2)     Game based assessment approaches.  In order to improve the motivation and interest in undertaking computer based assessments some researchers are now attempting to embed psychometric assessment within a computer game format.  The Educational Testing Service (a large non for profit organisation in the US) has been at the forefront of test development and theory for many years has recently developed new protocols for this approach.  Here is an example of one of their developments http://news.ets.org/news/ets-research-behind-glasslabs-launch-of-mars-based-grade-school-game. Eugene Aidman has also previously pioneered embedded game based assessments see: Aidman, E. V., & Shmelyov, A. G. (2002). Mimics: A symbolic conflict/cooperation simulation program, with embedded protocol recording and automatic psychometric assessment. Behavior research methods, instruments, & computers, 34(1), 83-89. (See also https://www.researchgate.net/profile/Eugene_Aidman/publication/27255397_Performance-based_measurement_of_aggression_in_a_computer_game-embedded_simulation/links/0046353aa9bc29c66c000000.pdf). 

3)     The use of neurophysiological measures such as EEG and eye movements in combination with cognitive assessment to increase their sensitivity to brain functioning.  My understanding was that in neuropsychology (at least from an assessment viewpoint) this type of approach is still largely within the research domain (e.g., Jason’s learning research), however, the sports psychologists at the congress seemed to be using a huge number of physiological measures in their assessment and treatment of athletes– including EEG.   A google search led me to a start-up company in the US called Neurotrack (who have 6.5 million in seed funding) and have a tool that combines a visual processing task with eye movement tracking (assessed via webcam) which is apparently very sensitive to preclinical Alzheimer’s disease.  See: www.neurotrack.com.
Apart from learning about these new technology based assessment techniques, of course we must not forget that modern cognitive neuropsychology researchers also keep gradually reworking and improving on some of our traditional measures.  At the congress there was a key note lecture from Prof David Babcock from UWA that outlined how his program of research has helped to update and improve on the use of the embedded figures test (and its sensitivity to autistic spectrum disorder) with more targeted and less noisy protocols for assessing local feature detection versus global sensory integration.  They have also developed an ipad testing kit.
Finally, if you are like me and wondering how on earth we are going to keep up with and integrate all of these developments (and still provide an affordable assessment!), I found this handy (2013) review of technology in neuropsychology on computer based assessment and virtual reality assessments.  Somewhat reassuringly it suggests there is still quite a bit of work to do in the application to brain disordered groups: http://www.tandfonline.com/doi/full/10.1080/13854046.2013.834971
I guess that none of these developments are going to radically disrupt the basic (Wechsler) neuropsychological assessment approach for now…..

No comments:

Post a Comment