Obi-Wan Kenobi and the challenge of confirmation bias

The first movie I remember seeing at the drive-in with my parents was Star Wars (Episode IV). I was six and was instantly hooked. Luke Skywalker was my hero. Darth Vader was the villain. Why? Because he killed Luke’s father, Anakin.

Well, as it turns out, not exactly. At the end of The Empire Strikes Back, Luke and I learned this horrible truth:

 

 

“Darth Vader is Luke’s father!” I cried. I was nine by then, and my little mind reeled at this information. Obi-Wan had lied! Good guys like Obi-Wan didn’t lie!

When Return of the Jedi was released, I was 12, and I was ready for Luke to give Obi-Wan a piece of his mind. Obi-Wan’s response blew my mind once more.

Obi-Wan: [w]hat I told you was true… from a certain point of view.

Luke: A certain point of view?

Obi-Wan: Luke, you’re going to find that many of the truths we cling to depend greatly on our own point of view.

Obi-Wan was talking about confirmation bias or myside bias.

 

 

Confirmation bias is how our personal experience and beliefs affect not only how we understand and analyze events, but even how we ask the questions that form our hypotheses. Confirmation bias reflects a tendency for people to confirm their beliefs rather than disprove them. Research suggests that the more polarizing an event is, the stronger our confirmation bias is likely to be.  Moreover, such bias deeply affects our memories of events. In Obi-Wan’s case, he remembered Anakin as he wanted to remember him, not how he actually was.

 

 

Likewise, if I go undercover to discover the truth about the 2002 Gujarat riots, and my hypothesis is that Narendra Modi, the then Chief Minister and now Prime Minister, is responsible, then I will find that he is responsible. That’s the thrust of Rana Ayyub’s argument in Gujarat Files.

I’m not an expert on the 2002 Gujarat riots, and I don’t pretend to be. I will leave it to people far wiser and more knowledgeable than me to assess the fact or fiction of Ayyub’s narrative. But, here’s what I do know: that Rana Ayyub went to extraordinary lengths to get her story. Going undercover is a dangerous and difficult experience. Ayyub is passionate, dedicated, and courageous. I admire those qualities. Ayyub seems truly interested not in TRPs or papers sold, but in discovering the truth of what happened during the 2002 Gujarat riots and in the encounters that followed.

But the question remains: whose truth are we talking about here? That’s where Ayyub’s confirmation bias comes in.

First-person narrative in the Gujarat Files

I bought the Gujarat Files expecting an exposé that would finally to put rest the lingering doubts many have about what happened in the spring of 2002. I was shocked to see that the book was written in first person. Why would an experienced, award-winning journalist, used to writing in third person, write the most important book of her career in first person?

Because this book is no dry accounting of the facts and figures associated with those events. Instead, this book is a deeply personal account of Ayyub’s search for truth. She even expresses her regret at not being able to master the art of detaching herself from a story. And, attached she is. To her story, to her sources, and to her hypothesis.

I would argue the real plot of this book isn’t the story of the encounters, but Ayyub’s attempt to get that story. She gives us a glimpse into what it’s like to go undercover. She discusses how she built her cover identity, managed communications with her family and editors, and overcame the fear of discovery. But, most importantly, she talks about how she ingratiated herself to her sources and how the line between the reporter and her sources blurred as she went deeper and deeper undercover.

Ayyub’s sources took her into their confidence and treated her not as the filmmaker she was pretending to be, but as a daughter. Maya Kodnani’s words are touching:

“You eat, [sic] it will feel like my son is eating. You too are like my daughter, Maithili [Ayyub’s cover name].”

There’s a certain irony in publishing the transcripts that betray the people who trusted Ayyub with their stories. Ayyub’s entire undercover identity is built on lies, yet we are expected to believe her. At times, I am not even sure I can believe her sources, let alone her. I find her acceptance of Rajan Priyadarshi’s story problematic. She’s uncritical of his obvious boasting. He reminds me too much of that guy we’ve all met who claims to have spent a wild night drinking with Salman Khan, but in truth just happened to sit at the table next to him. As part of the narrative, Ayyub often wrestles with this issue. Sometimes, her assessments are quite harsh. She says of Sanjiv Bhatt:

“I remember telling Shoma [Ayyub’s editor] that somehow I could not bring myself to trust Sanjiv Bhatt’s account for many technical reasons.”

Which begs the question: if that’s the case, then why include his account? The simple answer is because it confirms her theory.

Usually, when I write book reviews, I do so in third person. By now, you’ve noticed that I’ve been using first person. I want you to ask yourself how my use of first person has affected your perception of my review so far? Are you inclined to agree with my assessment of Ayyub’s book or inclined to consider my review colored by my beliefs?

First person makes everything more personal, doesn’t? I am telling you what I think, what I believe. I am constructing a narrative that’s deeply personal. To me and to you. When people read first-person narratives, they often believe those narratives are true. But, for me, the opposite has always been the case. My training as a historian taught me that first person isn’t objective; it’s subjective. That’s one reason why it works well in fiction. First person allows a writer to draw us into the character’s mind and heart far more easily than third person does. We place ourselves in that “I” subject, don’t we? When a character says, “I sat on the bench and wept,” we weep. We connect to the story in a different way than if we read, “Diana sat on the bench and wept.”

What do I think of the book?

I think Ayyub tells a thrilling story of undercover work. I think this book has the potential to be a great book, but it’s not there yet. The editing isn’t strong enough; the narrative is too loose. I think that the first-person narrative hurts Ayyub’s argument and claims to objective reporting. I think that if you want an introduction to the 2002 riots, you have chosen the wrong book. I think this book is meant for those who don’t need that introduction. If, for example, you don’t know who Amit Shah is, you might want to skip this book.

Yes, I think you should read the book. I don’t care if you think Narendra Modi is the greatest thing since the invention of the teabag; I don’t care if you think he got away with murder. I don’t care if you are a BPJ, INC, or AAP party member. I don’t care if you are Hindu, Muslim, Christian, Sikh, Parsi, Buddhist, or Pastafarian. I urge you to read the book and assess Ayyub’s sources yourselves. I urge you to engage in intelligent dialogue about the questions she raises. I urge you to read critically.

Most importantly, I urge you to think, and this book has made me think. Think about my own biases. Think about the story-telling process. Think about the problem of memory in our search for the truth about an event that is now more than a decade old. And, that, after all, is what a good book should do: make you think. And, Gujarat Files is a good book.

But, Ayyub isn’t the only one with bias. Her readers are biased, too. Although the nature of that bias surprises me.

Reader reception and Amazon reviews in the age of trolls

How can I put this nicely? I hate Amazon reviews. I think they are garbage, but not because they are poorly written. The customer reviews were originally developed to give Amazon’s customers an honest assessment of a product’s features, functions, and failures. They were meant as an antidote to the product’s marketing collateral, which only extols the product’s awesomeness and not its craptasticness.

Book reviews were meant to help potential readers decide if they would enjoy the book before they bought it. Between the blurb, the sample chapters, and the reviews, the readers would be able to make a well-informed buying decision. But, then came the fake reviews. And, now, we’ve entered the era of troll reviews.

Nowhere is the impact of trolling better illustrated than with Gujarat Files. If Ayyub has her bias, it’s nothing compared to her readers. Or should I say “alleged readers”?

I retrieved the following data on 21 June 2016, approximately one month after the book’s release. I chose to examine reviews on Amazon.com and Amazon.in sites to compensate for the high number of trolls on the India site. I wanted to examine how using only verified purchase reviews would affect the overall Amazon rankings.

My hypotheses were as follows:

• The ranking would increase by up to one star.
• Most of the 1-star reviews would not be verified.
• Most of the verified reviews would fall into the middle star (2, 3, 4) categories.
• The percentage of verified reviews would be consistent across both sites.

Amazon.com reviews

Number of stars	Total reviews	Verified reviews	Percentage verified to total reviews	Verified reviews point totals
5	60	28	46	140
4	3	2	66	8
3	2	1	50	3
2	4	0	0	0
1	63	2	3	2
Total	132	33	25	153
Overall Amazon star ranking			2.9
Star ranking based on verified reviews only			4.6

 

Amazon.in reviews

Number of stars	Total reviews	Verified reviews	Percentage verified to total reviews	Verified reviews point totals
5	530	193	36	965
4	34	17	50	68
3	5	2	40	6
2	10	1	10	2
1	1020	6	<1	6
Total	1599	219	14	1047
Overall Amazon.in star ranking			2.4
Star ranking based on verified reviews only			4.78

 

My findings

First, I need to explain that the Amazon star ranking isn’t a simple weighted average. Amazon uses a machine-learning algorithm that gives higher weight to newer and verified reviews. That means a new 1-star review might have more weight than an older 4-star review, especially if that 1-star review is verified. As I’m not sure how the algorithm actually works (only Amazon knows that), and I am not a machine, I had to use the straight weighted average. An Amazon verified review is one that Amazon can verify as a legitimate purchase. If you’re in India and using Flipkart to buy your books, but putting your reviews on Amazon, then your review won’t be verified.

Second, I expected the ranking to increase, but the increase was far more dramatic than expected. On Amazon.com, the ranking increased by 1.7 stars. On Amazon.in, the ranking increased by more than 2 stars (2.38 to be exact). That means most people who buy books review the ones we like, not the ones we hate. Of the 33 verified reviews on Amazon.com, 28 are 5-star reviews, or 85%. Of the 219 verified reviews on Amazon.in, 193 are 5-star reviews, or 88%. My theory that most verified reviews would fall into the middle was wrong for several reasons. First, that people are more likely to review books they like. Second, that the polarizing nature of this topic tends to lend itself to readers either loving or hating the book. Third, that Amazon’s machine-learning process might be giving the newer 1-star reviews more weight. Fourth, that people who decided to buy the book probably were already inclined to like it. That’s their confirmation bias.

Third, as I expected, most 1-star reviews were not verified. What I didn’t expect was for less than 1% of the Amazon.in reviews to be verified. The actual percentage was .5, so I rounded up to <1% in the table. Many 1-star reviews were abusive, especially on the India site. I would urge Amazon.in to take a hard look at those reviews. In particular, this review on the India site in which the reviewer actually says he hasn’t read the book. Seriously. I know Amazon can’t verify everything, but clearly this violates the spirit of its review policy.

These troll reviews are doing serious damage to Amazon’s credibility. If it wants to differentiate itself from Flipkart, demonstrating a commitment to real, honest reviews is critical. I don’t trust Amazon reviews and don’t post my reviews there, even though I buy almost all my books on Kindle.

Fourth, I expected the percentage of verified reviews to be roughly the same. I was surprised that the US site had 25% verified while the India site had 14%. I thought the verified reviews would be closer to half. That means people are reviewing books on Amazon, but they aren’t buying them on Amazon. Whether that’s because they are buying the books from competitors such as Flipkart or in the case of the trolls, they aren’t buying the book at all, I can’t tell.

Fifth, to determine if my hypothesis holds up against scrutiny, I am considering examining the data on about 20 books on the US and India sites. I feel that the polarizing nature of this topic might be skewing my data. I leave it to you to decide if my arguments reflect my own confirmation bias or not.