NZ Skeptics Articles

Are kids leaving High School with good critical thinking skills?

Mark FletcherJessica Macfarlane - 1 November 2018

Highlights and reflections on Mark Fletcher’s Survey: “The Critical Thinking Skills of Curriculum Level 8 Students” By Editor

Mark is an experienced History teacher. As a skeptic himself and a subscriber to this journal, he got in touch to share the findings from a research project he conducted recently with his NCEA level 3 students at Te Aho o Te Kura Pounamu, and we were very excited to read it. This research would shed light on whether kids are leaving school with the skills to think critically – essential skills for a good skeptic. ‘How to think’ not ‘What to think’.

In the end 73 students completed the ‘Critical Thinking Terminology’ survey, and 71 completed the ‘Critical Thinking Skills’ survey. Reading the long answers from the students, it’s clear that there are some bright minds in the group. Mark was clearly surprised by the quality of the answers, but also noticed areas where there seems to be much room for improvement in the group. Through this experience Mark also found areas where he could also improve the questions to some extent to elicit a clearer understanding of the students’ thinking skills next time.

Given all that, it is very clear from the comments throughout that Mark cares about his students and is worried about the sophistication and volume of disinformation out there. In our crazy days where young people spend so much time on Twitter, Instagram & Youtube, all of which are known to promote and re-share click bait even if it’s abhorrent trash and outright lies, this work seems absolutely on point.

Here is the list of questions presented to the students. Before reading the answers from the students and Mark’s review of them, how would you answer these?

  1. Explain what it meant by ‘critical thinking’?

  2. List some important skills of critical thinkers.

  3. Why is it important for students to be good critical thinkers?

  4. What would it take for a critical thinker to change their views on a topic?

  5. Explain the difference between ‘information’ and ‘disinformation.’

  6. When evaluating the reliability of information, what is meant by ‘peer review’?

  7. When evaluating the reliability of information, what is meant by the ‘credentials’ of the author?

  8. When evaluating the reliability of information, what is meant by ‘corroborative evidence’?

  9. When evaluating the reliability of information, what is meant by ‘confirmation bias’?

  10. What is your understanding of the term ‘conspiracy theory’?

And a selection of Mark’s comments and student responses:

(E=editor, M=Mark, S=student)

Q1: Explain what it meant by ‘critical thinking’?

M: Of the 71 responses, only 2 students had referred to the challenging and questioning of assumptions, although 20 students had referred to the evaluation or analysis of information (or an issue/situation), and 14 students referred to analysing and/or evaluating and then drawing conclusions. Many other students expressed the idea that critical thinking was about thinking deeply, perceptively, or outside the box. My feeling was that most students had a basic understanding of ‘critical thinking’ but many offered explanations that were too general. Below is an example of a strong student response that implies the challenging of assumptions:

S: To be able to think deeper - to analyze a text or piece of media, or even the world around you in general, without being superficial or simply accepting everything at face value.

M: Other solid responses were also given that emphasised analysis and evaluation:

S: Multi-faceted thinking, turning a problem over and over in your mind and seeing from different perspectives to come to a solution. Analyzing and being able to explain, discuss and evaluate the object/idea.

Q2. List some important skills of critical thinkers.

E: To be critical thinkers you need to challenge and question assumptions. Mark explains that in order to do that:

M: students need to be able to do the following with the data or information or situation that they are studying: analyse, evaluate, compare and contrast, interpret, infer, draw conclusions, apply the findings, make decisions, and problem solve. All of these skills, it seems to me, are important components of critical thinking and I was most interested to see how many of the students would refer to them in their responses. A count gave me 33 responses that mentioned the skills above which is not quite a half of the total and a little disappointing. Too many students gave vague responses, such as:

S: think rationally common sense or Wide imagination Eager attitude Start to the point or Being able to think above what’s already been thought of

M: Better responses made direct reference to critical thinking skills:

S: The important skills that we need in order to be able to think critically or think deeply are varied and include observation, analysis, interpretation, reflection, evaluation, inference, explanation, problem solving, and decision making.”

Q3: Why is it important for students to be good critical thinkers?

M: I considered this to be one of the more interesting questions and hoped that students would refer to qualities such as being better informed, increasing their understanding, making wise decisions and shaping actions. Some students certainly responded this way:

S: There are many benefits such as better control of your own learning and empathy for other points of view.

M: Some students also offered additional ideas that were particularly valid, such as being able think independently, to identify connections between ideas, to make the world a better place, to be innovative and curious, and to better enable them to interact with the environment and fellow humans:

S: To make the world a better place, we cannot simply be sheep that follow whatever is said by those on media outlets. We have to think for ourselves, and to do that we need to think critically - not all things are good and positive that happens in parts of the world, and not everything that happens in others are terrible. We have to ask ourselves - why are they telling us this, what message are they trying to bring across?

S: It allows them to think outside their internal biases and form their own opinions based on their experiences and available evidence. It enables them to have an open mind and consider things from every angle rather than just one. This can help with discovering a way to do something or what something could do. It allows students to be innovative and curious.

Q4. What would it take for a critical thinker to change their views on a topic?

M: I expected that students would refer to the critical thinker’s exposure to new evidence or new arguments, to their growing understanding of an issue, or an increased ability to think critically. My expectations were correct as the following examples show:

S: Thinking about both sides of the view, evaluating the different facets of the topic, interpreting evidences and experiences, coming up with new and unexpected solutions.

S: New information that negates their view.

S: A compelling argument supported by facts

M: Overall, I was impressed with the students’ responses to this issue. Most appeared to have considered the question thoughtfully and offered valid and logical answers.

Q5. Explain the difference between ‘information’ and ‘disinformation.’

M: Of the 73 students who answered this question showed that 46 (about 63%) understood that disinformation involved deception. This was lower than I expected and of concern: I think that all senior students need to be well aware of online disinformation. However, it is quite possible that many of the 27 (37%) who could not accurately explain ‘disinformation’ may be aware of it but know it by another name, such as ‘propaganda’ or ‘fake data’.

Q6. When evaluating the reliability of information, what is meant by ‘peer review’?

M: Peer review refers to the process the work of one expert being scrutinised by other experts in the same field. It is a very significant component of the scientific method and for determining the reliability of information. It is a process that I always emphasise to my senior History students when they undertake research. Of the 71 students who responded, 48 (67%) were able to give a clear explanation of what peer review is. Some students were confused and defined a ‘peer’ as someone in the same class or age group.

Q7. When evaluating the reliability of information, what is meant by the ‘credentials’ of the author?

M: Of the 72 students who responded to this question, 59 (82%) were able to explain what ‘credentials’ were although some offered quite simple answers, such as ‘their qualifications’. Better answers…explained why the author’s credentials are important.

Q8. When evaluating the reliability of information, what is meant by ‘corroborative evidence’?

M: My expectation was that this would be an easy question for the students. The process of looking for corroborative evidence, or cross-checking, seemed to me to be an obvious method of checking the reliability of a piece of information (my assumption). However, I was quite surprised at the results: of the 67 students who responded to this survey, only 46 (67%) were able to define the term ‘corroborative evidence’ accurately, which was a much lower percentage than I expected. Many students simply responded ‘don’t know’ or ‘not sure’.

Q9. When evaluating the reliability of information, what is meant by ‘confirmation bias’?

M: Of the 68 students who responded to this question, 40 (59%) expressed some understanding of the concept. It was difficult to conduct the count as some students gave answers that were on the cusp of an understanding. Some of the better answers were:

S: Confirmation bias is to only seek out information confirming preexisting notions, therefore, potentially overlooking useful facts relating to the reliability of the information.

S: Confirmation bias, the tendency to process information by looking for, or interpreting, information that is consistent with one’s existing beliefs. This biased approach to decision making is largely unintentional and often results in ignoring inconsistent information. Existing beliefs can include one’s expectations in a given situation and predictions about a particular outcome.

M: However, many of the answers were very marginal:

S: to get information on the topic that supports your views.

Q10. What is your understanding of the term ‘conspiracy theory’?

M: of the 71 responses I received, 58 (82%) of the students had offered at least a basic understanding of what a conspiracy theory was - which was a higher percentage than I expected. Many students commented on a conspiracy theory being contrary to the prevailing view and that many CTs put governments in a sinister role. In addition, many students commented on CTs being deceptive, that covert organisations were involved and that they offered explanations of events that had been unexplained. Some typical student answers to this question are below:

S: A conspiracy or non factual idea or theory relating to a specific topic or idea an, an explanation of an event or situation that invokes an unwarranted conspiracy, generally one involving an illegal or harmful act carried out by government or other powerful actors.

Conspiracy Theories

It was the response to a history project where a very bright student had suggested they write about 9/11 that spurred Mark on to come up with this survey. The student had explained that they wanted to write about the event as they had already done some research and found the area very interesting, but it was clear to Mark from the examples given that most of the research done so far had involved consuming conspiracy theorist’s views, with not enough time spent on checking for credentials, corroborating evidence, references and so on.

So, along with the critical thinking questions above, an anonymous survey asking for responses to 10 conspiracy theories was to year 8 students to gauge how many had skeptical views. Students were also invited to explain why they held those views.

I think you will agree the results are really interesting, and I for one was surprised at how many students thought aliens might exist or weren’t sure about whether Big Pharma was keeping people sick to make money. Mark’s view after reading the responses was that it would be appropriate to choose ‘unsure’ if you didn’t have enough evidence to make an informed decision, but he also clarifies that this could be made up of people that either do, or do not have knowledge about the particular CT

Focusing on the Big Pharma CT (which I assume will be of particular interest to our readers), here are some student responses:

S: I hold this view as a lot of claims are contradicting one another in the world of medicine / science / pharmaceuticals. Supposedly, medicine and science has come a long way , and the advances are almost sci-fi. It seems quite likely that cures for the most economically burdensome diseases are being kept under wraps in order to keep Big Pharma afloat with less-effective and costly pharmaceutical concoctions.

S: I believe there is a cure for cancer and they’re not disclosing with anyone and keeping it to themselves as a population control as it is getting increasingly bigger in the world today

S: I would lean towards disagreement on this one since the people working in these big pharmaceutical business must have at least someone who will report them if this was the case. Having large costs in medicine is different from actually not giving the cure to anyone.

I decided to examine the data Mark came up with in another way, by ranking it. Ranking the conspiracy theories (CTs) by the ones with the most ‘agree’ responses to the least, we get an interesting picture of which ones the students are more likely to believe or be skeptical of.

The CT with the most support was the one about a group being behind the Kennedy Assassination, and then followed closely by the CT that aliens really exist. Why would that be? Could there be a bigger volume of misinformation out there for those CTs? Or perhaps misinformation presented in a particularly slick way swayed their thinking? Oliver Stone’s JFK from 1991 was very much in the CT camp and elicited many editorials lambasting its lack of adherence to the facts when it was released. The controversy may have helped push the movie in the box office, as it did end up being a hit. With such a big reach, it isn’t surprising that particular view has been retold till this day.

At the 3rd position in the rankings Big Pharma being responsible for making people sick to improve profits, Mark cites Ben Goldacre’s book ‘Bad Pharma’ published in 2012 as something that may have contributed to the feelings of distrust that underpin many of the CTs around this area.

M: I am aware of several CTs concerning ‘Big Pharma’ and there is a degree of paranoia in many of them: that ‘Big Pharma’ is plotting against their customers. Some much more credible criticism of Big Pharma has also come from such sources as Ben Goldacre’s book ‘Bad Pharma’ (25). In this book, Goldacre is critical of Big Pharma for manipulating data and misleading their customers on a range of issues. On the other hand, Big Pharma produces a range of products, including antibiotics, vaccines and anaesthetics which most of us have used at some time and which undoubtedly increase human longevity.

I couldn’t agree more with Mark on this point. It is a fact that today largely due to vaccinations and hygiene the human race is healthier than it’s ever been and yet supplement companies continue to extract the millions from the worried well. Also to be clear, Ben Goldacre should not be made out to be a villain to skeptics. He is a skeptic himself and has written extensively on science, critical thinking and pseudoscience, having also written the book “Bad Science” (Mark Fletcher very much enjoyed this!) where he aimed to make people aware of the tricks used to sell alternative medicines and detox products.

I’ll leave you with some key points from Jan-Willem von Prooijen who was interviewed on RNZ Sunday on 2nd September this year. He is a professor of psychology at Vrije University in Amsterdam and an expert on conspiracy theories:

https://www.radionz.co.nz/national/programmes/sunday/audio/2018660736/what-s-going-on-with-qanon

If you would like to read Mark’s paper “Critical Thinking Skills of Curriculum Level 8 Students”, contact editor@skeptics.nz and we’ll put you in touch.