I wondered if anyone would like to read and comment on the essay I've just posted on my blog.
I'll reproduce it below for convenience but don't feel obliged. Only read it if you are interested.
quote:
Surrogate outcomes in education
I have recommended several times Ben Goldacre's book, Bad Science. It's a particular favourite of mine, dealing, as it does, with the ways that people misrepresent, misuse and generally misunderstand science. A section I was re-reading only yesterday deals with the tricks that are used when publishing the results of drugs trials. It's interesting in itself but one of them struck a chord with me as having a wider application, specifically an application in my own area - education. I'll tie the threads up later but first I need to make a couple of detours, one to explain what he was talking about in medical terms and one to deal with the rather more vexed question of what education is actually for. The specific "trick" that I was reading about was measuring surrogate outcomes. When you are testing a drug you have, one presumes, a reason for doing so. There is something that you expect that drug to do. It might be to prevent heart attacks, to alleviate pain, to treat cancer. You therefore design a test, hopefully a properly thought out randomised double-blind trial, and see if patients on the new drug show the required result without too many undesired side effects. But not all drugs trials do this. Some measure surrogate outcomes. For example a drug to prevent strokes might use reduction in blood pressure as a surrogate outcome. It isn't actually checking how many patients have a reduced stroke risk it's checking how many patients have reduced blood pressure and extrapolating that this equates to a reduced stroke risk. It's measuring something that is easily and quickly measurable and then using that to say that the drug has achieved it's desired result. Whether this is a good or a bad thing you can find out by reading Ben's book. The relevence it has to teaching I'll come to later. OK. Detour number two. What is education for? I'd argue that education is to equip people with skills or knowledge that they don't already have so that they can lead a happier, more fulfilled and possibly (though not necessarilly) more productive life. This isn't actually what the Government would like education to be for. The Government would like education to have one aim - to get people out of education, off benefits and into work. The "productive" bit above is the only one they are really interested in. The "happier" and "more fulfilled" bits don't really concern them. Because it's, ultimately, the Government who pay for it they call the shots and schools and colleges toe the line and make education about preparation for work. Now, as I've ranted before, the Government is also obsessed with targets and measurability. This might be viewed as a good thing. Evidence based education surely is as valid a concept as evidence based medicine. After all if we are trying to educate people, for whichever set of reasons, it would be useful to have some evidence that the way we are doing it works. Targets and measurability can be viewed as tools to achieve this aim. So, how do we measure whether the way we are doing it works? We use exams, and a finer example of a surrogate outcome would be hard to find. An exam doesn't test whether the person taking it has acquired the skills or knowledge they set out to get, it tests whether they have acquired the skill of passing an exam. This is especially true in the "soft" subjects, such as languages - subjects where what is being learned isn't simply a list of facts or procedures. I have only this morning been devising some materials for my class to prepare them for their forthcoming reading exam. The materials are very carefully tailored to include practice at exactly the kinds of question that I know will be on the paper and these aren't language questions at all. They are things that we would never ask in real life unless we were, perhaps, doing a course in semantics or semiotics. What is the purpose of this text? Who is the intended audience for this text? How would you describe the language in this text? I will dilligently teach them how to answer all these and more, but I know and they know that what I am teaching them has little if anything to do with the basic skills of speaking, reading and writing that they need. I am teaching them how to pass the exam. Passing the exam is the measure of my success and theirs, but it is a surrogate outcome. Of course that's specific to the design of the exam that we use in our college and different exams may correlate more - or less - closely with the actual language skill in question but whether they do or don't correlate is largely irrelevent. A surrogate outcome is NOT measuring the actual desired objective, it's measuring something else and then claiming that because this is positive then it follows that the real outcome must also be positive. It doesn't follow. Exams test exam passing ability. They test how well the students have been coached to produce the answers that the exam board expects. And we need to accept that this is the case and, if we can, find a better way of doing things, or at least a better exam design, or any decisions and policy we make based on our results will be fundamentally flawed.
"No man but a blockhead ever wrote except for money." Samuel Johnson.
Interesting post. As for the first question, I haven't seen the term "surrogate" outcomes in medical or drug research, and I've read a lot of studies in my day. I do think our FDA has regulations about what the researchers must study before the drugs are approved...and there is no point in the research if the drugs aren't approved. I know, for sure, touting a drug for decreasing strokes, and then having it decrease BP, wouldn't be acceptable. They'd have to conduct longitudinal studies on its effectiveness to decrease strokes.
However, for the next two points (what is education for and don't exams just measure surrogate outcomes), I agree with you. What would you suggest real outcomes would be in your field? I am in the midst of this now. We are designing a transition to practice program and want to measure its effects on patient safety. Other studies have measured nurse satisfaction, retention, etc., but never actual patient outcomes. I can tell you, it isn't easy. How do you know the patient's medication error, for example, is because the nurse was new or for some other reason? If on the other hand errors decrease, how do you know it's because of the new nurse being adequately transitioned...or because of something else that might have gone on in the hugely complex medical center environment. We're trying to measure actual outcomes, but it's not easy.
I haven't seen the term "surrogate" outcomes in medical or drug research, and I've read a lot of studies in my day.
Please note that I only said that I hadn't seen the term used...not that Bob was wrong about it. Clearly from the posts he isn't wrong. Still...with all the studies I've read during my research career, I haven't seen the word used for outcome studies. Right now I am involved with 2 national, multi-site studies of outcomes (one with simulation and another with transition programs), and each is looking specifically at safety and quality outcomes. For each we have a national panel of renowned reearchers onboard advising us, and the term has not been used. That's all I am saying.
We do worry about using the right outcome measures, though. That obviously is very important. For example, the nurse's "report of errors" is much less reliable than actually looking at the number of incident reports. And...once you get into the latter, legal issues arise. So you have to get creative. I have been wracking my brain to see if there is a similar term (to "surrogate") that I've seen used, but I can't come up with one.
Exams have many failings - but one great strength. They are a very cheap and quick way of assessing knowledge.
You will remember all too well the National Vocational Qualifications (NVQ) fiasco that originally suggested that all Youth Training Scheme (YTS) participants should be tested by observation or other direct assessment of actual working practices - examinations were old hat and out. I was closely involved in the early development of NVQs for the travel industry and the whole idea of non-examined assessment was quickly (but very quietly) dropped when it was discovered that the only way to assess without the assessment costing an absolute fortune, was simply to sign off the trainees' logbooks. And the higher the level of trainee, the more ridiculous the idea became. At level 4 (the management level) it was quite impossible. Who was going to be able to shadow a manager for days to see how well he or she dealt with some kind of staff crisis? So the box was simply ticked and the whole paraphenalia of range statements, validity and all the other NVQ jargon was simply ignored.
Candidates who pass examinations may know more about passing an examination that they do about the subject in some cases, but they do know something. Many of the early YTS participants knew nothing at all about many aspects of their subject - even though they had a beautifully completed logbook.
The travel management qualifications with which I am currently involved are examined and I believe that the sampling I do by means of question selection is rigorous enough that nobody will pass unless they have a reasonable measure of knowledge of the topic.
Richard English
Posts: 8038 | Location: Partridge Green, West Sussex, UK
Yes, I too agree that exams have many failings but at present there is no practical alternative. For several qualifications some form of teacher assessment might be possible, but that brings up the question as to how such a scheme should be administered, in particular to achieve some sort of uniformity across providers and to prevent abuse.
Build a man a fire and he's warm for a day. Set a man on fire and he's warm for the rest of his life.
Last night I watched a Harry Potter movie which seemed to lampoon current British education. I don't know which in the series it was, but it seemed to lampoon taking tests as opposed to learning anything.
It is no measure of health to be well adjusted to a profoundly sick society. -J. Krishnamurti
