Confirmation bias, or different priors?

Consider two people.

Person A is a person of color who grew up in an upper middle class household in a wealthy and crime-free neighborhood. This person has gone through their life encountering mostly others similar to themselves – well-educated liberals that believe strongly in socially progressive ideals. They would deny ever having personally experienced or witnessed racism, despite their skin color. In addition, when their friends discuss the problem of racism in America, they feels a baseline level of skepticism about the actual extent of the problem, and suspect that the whole thing has been overblown by sensationalism in the media. Certainly there was racism in the past, they reason, but the problem seems largely solved in the present day. This suspicion of the mainstream narrative seems confirmed by the emphasis placed on shootings of young black men like Michael Brown, which were upon closer reflection not clear cases of racial discrimination at all.

Person B grew up in a lower middle class family, and developed friendships across a wide range of socioeconomic backgrounds. They began witnessing racist behavior towards black friends at a young age. As they got older, this racism became more pernicious, and several close friends described their frustration at experiences of racial profiling. Many ended up struggling with the law, and some ended up in jail. Studying history, they could see that racism is not a new phenomenon, and descends from a long and brutal history of segregation, Jim Crow, and discriminatory housing practices. To Person B, it is extremely obvious that racism is a deeply pervasive force in society today, and that it results in many injustices. These injustices are those that sparked the Black Lives Matter movement, which they are an enthusiastic supporter of. They are aware that BLM has made some mistakes in the past, but they see a torrent of evidence in favor of the primary message of the movement: that policing is racially biased.

Now both A and B are presented with the infamous Roland Fryer study finding that when you carefully control for confounding factors, black people are no more likely to be shot by police officers than whites, and are in fact slightly less likely to be shot than whites.

Person A is not super surprised by these results, and feels vindicated in his skepticism of the mainstream narrative. To them, this is clear-cut evidence supporting their preconception that a large part of the Black Lives Matter movement rests on an exaggeration of the seriousness of racial issues in the country.

On the other hand, Person B right away dismisses the results of this study. They know from their whole life experience that the results must be flawed, and their primary take-away is the fallibility of statistics in analyzing complex social issues.

They examine the study closely, trying to find the flaw. They come up with a few hypotheses: (1) The data is subject to one or more selection biases, having been provided by police departments that have an interest in seeming colorblind, and having come from the post-Ferguson period in which police officers became temporarily more careful to not shoot blacks (dubbed the Ferguson effect). (2) The study looked at encounters between officers and blacks/whites, but didn’t take into account the effects of differential encounter rates. If police are more likely to pull over or arrest black people than white people for minor violations, then this would artificially lower the rate of officer shootings.

A and B now discuss the findings. Witnessing B’s response to the data, A perceives it as highly irrational. It appears to be a textbook example of confirmation bias. B immediately dismissed the data that contradicted their beliefs, and rationalized it by conjuring complicated explanations for why the results of the study were wrong. Sure, (thinks Person A) these explanations were clever and suggested the need for more nuance in the statistical analysis, but clearly it was more likely that the straightforward interpretation of the data was correct than these complicated alternatives.

B finds A’s quick acceptance of the results troubling and suggestive of a lack of nuance. To B, it appears that A already had their mind made up, and eagerly jumped onto this study without being sufficiently cautious.

Now the two are presented with a series of in-depth case studies of shootings of young black men that show clear evidence of racial profiling.

These stories fit perfectly within B’s worldview, and they find themselves deeply moved by the injustices that these young men experienced. Their dedication to the cause of fighting police brutality is reinvigorated.

But now A is the skeptical one. After all, the plural of anecdote is not data, and the existence of some racist cops by no means indicts all of society as racist. And what about all the similar stories with white victims that don’t get reported? They also recall the pernicious effects of cognitive biases that could make a young black man fed narratives of police racism more likely to see racism where there is none.

To B, all of this gives them the impression that A is doing cartwheels to avoid acknowledging the simple fact that racism exists.

In the first case, was Person B falling prey to confirmation bias? Was A? Were they both?

How about in the second case… Is A thinking irrationally, as B believes?

✯✯✯

I think that in both cases, the right answer is most likely no. In each case we had two people that were rationally responding to the evidence that they received, just starting from very different presuppositions.

Said differently, A and B had vastly different priors upon encountering the same data, and this difference is sufficient to explain their differing reactions. Given these priors it makes sense and is perfectly rational for B to quickly dismiss the Fryer report and search for alternative explanations, and for A to regard stories of racial profiling as overblown. It makes sense for the same reason that it makes sense for a scientist encountering a psychic that seems eerily accurate to right away write it off as some complicated psychological illusion… strong priors are not easily budged by evidence, and there are almost always alternative explanations that are more likely.

This is all perfectly Bayesian, by the way. If two interpretations of a data set equally well predict or make sense of the data (i.e. P(data | interpretation 1) = P(data | interpretation 2)), then their posterior odds ratio P(interpretation 1 | data) / P(interpretation 2 | data) should be no different from their prior ratio P(interpretation 1) / P(interpretation 2). In other words, strong priors dictate strong posteriors when the evidence is weakly discriminatory between the hypotheses.

When the evidence does rule out some interpretations, probability mass is preferentially shifted towards the interpretations that started with stronger priors. For instance, suppose you have three theories with credences (P(T1), P(T2), P(T3)) = (5%, 5%, 10%), and some evidence E is received that rules out T1, but is equally likely under T2 and T3. Then your posterior probabilities will be (P(T1 | E), P(T2 | E), P(T3 | E)) = (0%, 5.6%, 94.4%).

T2 gains only .6% credence, while T3 gains 4.4%. In other words, while the posterior odds stay the same, the actual probability mass has shifted relatively more towards theories favored in the prior.

The moral of this story is to be careful when accusing others, internally or externally, of confirmation bias. What looks like mulish unwillingness to take seriously alternative hypotheses can actually be good rational behavior and the effect of different priors. Being able to tell the difference between confirmation bias and strong priors is a hard task – one that most people probably won’t undertake, opting instead to assume the worst of their ideological opponents.

Another moral is that privilege is an important epistemic concept. Privilege means, in part, having lived a life sufficiently insulated from injustice that it makes sense to wonder if it is there at all. Privilege is a set of priors tilted in favor of colorblindness and absolute moral progress. “Recognizing privilege” corresponds to doing anthropic reasoning to correct for selection biases in the things you have personally experienced, and adjusting your priors accordingly.

Objective Bayesianism and choices of concepts

Bayesians believe in treating belief probabilistically, and updating credences via Bayes’ rule. They face the problem of how to set priors – while probability theory gives a clear prescription for how to update beliefs, it doesn’t tell you what credences you should start with before getting any evidence.

Bayesians are thus split into two camps: objective Bayesians and subjective Bayesians. Subjective Bayesians think that there are no objectively correct priors. A corollary to this is that there are no correct answers to what somebody should believe, given their evidence.

Objective Bayesians disagree. Different variants specify different procedures for determining priors. For instance, the principle of indifference (POI) prescribes that the proper priors are those that are indifferent between all possibilities. If you have N possibilities, then according to the POI, you should distribute your priors credences evenly (1/N each). If you are considering a continuum of hypotheses (say, about the mass of an object), then the principle of indifference says that your probability density function should be uniform over all possible masses.

Now, here’s a problem for objective Bayesians.

You are going to be handed a cube, and all that you know about it is that it is smaller than 1 cm3. What should your prior distribution over possible cubes you might be handed look like?

Naively applying the POI, you might evenly distribute your credences across all volumes from 0 cm3 to 1 cm3 (so that there is a 50% chance that the cube has a volume less than .50 cm3 and a 50% chance its volume is between greater than .50 cm3).

But instead of choosing to be indifferent over possible volumes, we could equally well have chosen to be indifferent over possible side areas, or side lengths. The key point is that these are all different distributions. If we spread our credences evenly across possible side lengths from 0 cm to 1 cm, then we would have a distribution with a 50% chance that the cube has a volume less than .125 cm3 and a 50% chance that the volume is greater than this.

Cube puzzle

In other words, our choice of concepts (edge length vs side area vs volume) ends up determining the shape of our prior. Insofar as there is no objectively correct choice of concepts to be using, there is no objectively correct prior distribution.

I’ve known about this thought experiment for a while, but only recently internalized how serious of a problem it presents. It essentially says that your choice of priors is hostage to your choice of concepts, which is a pretty unsavory idea. In some cases, which concept to choose is very non-obvious (e.g. length vs area vs volume). In others, there are strong intuitions about some concepts being better than others.

The most famous example of this is contained in Nelson Goodman’s “new riddle of induction.” He proposes a new concept grue, which is defined as the set of objects that are either observed before 2100 and green, or observed after 2100 and blue. So if you spot an emerald before 2100, it is grue. So is a blue ball that you spot after 2100. But if you see an emerald after 2100, it will not be grue.

To characterize objects like this emerald that is observed after 2100, Goodman also creates another concept bleen, which is the inverse of grue. The set of bleen objects is composed of blue objects observed before 2100 and green objects observed after 2100.

Now, if we run ordinary induction using the concepts grue and bleen, we end up making bizarre predictions. For instance, say we observe many emeralds before 2100, and always found them to be green. By induction, we should infer that the next emerald we observe after 2100 is very likely going to be green as well. But if we thought in terms of the concepts grue and bleen, then we would say that all our observations of emeralds so far have provided inductive support for the claim “All emeralds are grue.” The implication of this is that the emeralds we observe after time 2100 will very likely also be grue (and thus blue).

In other words, by simply choosing a different set of fundamental concepts to work with, we end up getting an entirely different prediction about the future.

Here’s one response that you’ve probably already thought of: “But grue and bleen are such weird artificial choices of concepts! Surely we can prefer green/blue over bleen/grue on the basis of the additional complexity required in specifying the transition time 2100?”

The problem with this is that we could equally well define green and blue in terms of grue and bleen:

Green = grue before 2100 or bleen after 2100
Blue = bleen before 2100 or grue after 2100

If for whatever reason somebody had grue and bleen as their primitive concepts, they would see green and blue as the concepts that required the additional complexity of the time specification.

“Okay, sure, but this is only if we pretend that color is something that doesn’t emerge from lower physical levels. If we tried specifying the set of grue objects in terms of properties of atoms, we’d have a lot harder time than if we tried specifying the set of green or blue objects in terms of properties of atoms.”

This is right, and I think it’s a good response to this particular problem. But it doesn’t work as a response to a more generic form of the dilemma. In particular, you can construct a grue/bleen-style set of concepts for whatever you think is the fundamental level of reality. If you think electrons and neutrinos are undecomposable into smaller components, then you can imagine “electrinos” and “neuctrons.” And now we have the same issue as before… thinking in terms of electrinos would lead us to conclude that all electrons will suddenly transform into neutrinos in 2100.

The type of response I want to give is that concepts like “electron” and “neutrino” are preferable to concepts like “electrinos” and “neuctrons” because they mirror the structure of reality. Nature herself computes electrons, not electrinos.

But the problem is that we’re saying that in order to determine which concepts we should use, we need to first understand the broad structure of reality. After which we can run some formal inductive schema to, y’know, figure out the broad structure of reality.

Said differently, we can’t really appeal to “the structure of reality” to determine our choices of concepts, since our choices of concepts end up determining the results of our inductive algorithms, which are what we’re relying on to tell us the structure of reality in the first place!

This seems like a big problem to me, and I don’t know how to solve it.

A shameful history

Perhaps there has been, at some point in history, some great power whose elevation was exempt from the violent exploitation of other human bodies. If there has been, I have yet to discover it. But this banality of violence can never excuse America, because America makes no claim to the banal. America believes itself exceptional, the greatest and noblest nation ever to exist, a lone champion standing between the white city of democracy and the terrorists, despots, barbarians, and other enemies of civilization. One cannot, at once, claim to be superhuman and then plead moral error. I propose to take our countrymen’s claims of American exceptionalism seriously, which is to say I propose subjecting our country to an exceptional moral standard. This is difficult because there exists, all around us, an apparatus urging us to accept American innocence at face value and not to inquire too much. And it is so easy to look away, to live with the fruits of our history and to ignore the great evil done in all of our names. But you and I have never truly had that luxury.

Ta-Nehisi Coates, Between The World And Me

I’ve had a bit of a revolution in my thinking recently. This revolution has consisted of becoming properly acquainted with the horror that is American history.

I am not the type of writer that can inspire emotion with the type of eloquence in the above quote. But it feels atrocious to me that the true nature of American history, which should be appreciated by every American, is not taught in schools, or is taught in a softened and white-washed form. I would like to share a few facts about the history of my country that I wish I had known earlier.

Legal Racism

Naturalization and whiteness

  • From the very beginning of our country’s history, Congress explicitly stated that only white immigrants could become citizens.
  • This stayed in place for over 100 years. In 1922 (Ozawa v. US), a Japanese-American man who had lived in the United States for 20 years was ruled ineligible for naturalization by the US Supreme Court.
  • From the decision: “In all of the naturalization acts from 1790 to 1906 the privilege of naturalization was confined to white persons.” This was used to conclude that Japanese could not be naturalized.
  • In both this case and a subsequent case (US v. Bhagat Singh Thind, 1923), the US Supreme Court specifically ruled that Japanese and Indian people do not count as white, and thus are racially ineligible for naturalized citizenship.
  • Bhagat Singh argued for his status as a white person by pointing to his Brahmin status. He argued that his people were originally the conquerers of the indigenous population of India, thus giving them an equal claim to whiteness.
  • Arguments from Bhagat Singh’s lawyers: Singh had a revulsion to marrying Indian women from the “lower races,” and had expressed a “disdain for inferiors” that characterized him as white.
    • “The high-caste Hindu regards the aboriginal Indian Mongoloid in the same manner as the American regards the Negro, speaking from a matrimonial standpoint.”
  • The court decided against the whiteness of high-caste Indians, because “while the invaders seem to have met with more success in the effort to preserve their racial purity, intermarriages did occur producing an intermingling of the two and destroying to a greater or less degree the purity of the “Aryan” blood.”
  • An irony: the Statue of Liberty was dedicated in 1886 with the famous quote…

    “Give me your tired, your poor, Your huddled masses yearning to breathe free, The wretched refuse of your teeming shore. Send these, the homeless, tempest-tossed, to me: I lift my lamp beside the golden door.”

    … At this time, only whites had naturalization rights, and the Chinese Exclusion Act had restricted all Chinese immigration to the country four years before.

Oriental exclusion

  • The Chinese Exclusion Act was passed in 1882, banning Chinese immigration for ten years.
  • In 1892 it was extended for another ten years, with the additions that that all Chinese residents must carry permits, that they could not serve as witnesses in court, and that they weren’t allowed bail.
  • In 1902, it was again renewed, this time with no ending date.
  • In 1913, California prohibited Chinese and Japanese immigrants from owning property. Other states followed suit.
  • In 1922, the Cable Act ruled that if an American woman married an Asian, she would lose her citizenship.
  • In 1924, the Oriental Exclusion Act prohibited most immigration from Asia, including foreign-born wives and children of American citizens of Chinese ancestry.
  • In 1929, the National Origins Formula completely barred Asian immigration.
  • In 1952, all federal anti-Asian exclusion laws are finally nullified by the Walter-McCarran Act, allowing for the naturalization of all Asians.

Interracial marriage

  • Laws banning interracial marriage (i.e. marriage between whites and non-whites) were enforced in many states up until 1967… a mere fifty years ago.
  • At least three constitutional amendments banning interracial marriage were introduced in Congress.
  • Quote from a speech given before Congress by a Georgian governor in 1912:
    • No brutality, no infamy, no degradation in all the years of southern slavery, possessed such villainous character and such atrocious qualities as the provision of the laws of Illinois, Massachusetts, and other states which allow the marriage of the negro, Jack Johnson, to a woman of Caucasian strain. [applause]. Gentleman, I offer this resolution … that the States of the Union may have an opportunity to ratify it. …

      Intermarriage between whites and blacks is repulsive and averse to every sentiment of pure American spirit. It is abhorrent and repugnant to the very principles of Saxon government. It is subversive of social peace. It is destructive of moral supremacy, and ultimately this slavery of white women to black beasts will bring this nation a conflict as fatal as ever reddened the soil of Virginia or crimsoned the mountain paths of Pennsylvania.

      … Let us uproot and exterminate now this debasing, ultra-demoralizing, un-American and inhuman leprosy.

  • Anti-miscegenation laws were deemed constitutional by the US Supreme Court in 1883. The argument was that both races were treated equally, “because whites and blacks were punished in equal measure for breaking the law against interracial marriage and interracial sex.”
  • Idaho banned interracial marriage and sex between black and white people in 1921, even though the state’s population was 99.8% non-black.
  • In 1967, the Supreme Court finally ruled these laws (that were enforced in 17 Southern states – all the former slave states) unconstitutional.
  • It took South Carolina until 1998 and Alabama until 2000 to actually remove the ban on interracial marriage from their Constitutions.
    • In the respective referendums, 38% of voters in South Carolina and 41% of voters in Alabama voted to keep the ban in place.

Racism now

  • A 2018 study found that Republican-appointed judges sentence blacks to 3 more months than similar non-blacks. This is 65% of the baseline racial sentence gap.
    • “These differences cannot be explained by other judge characteristics and grow substantially larger when judges are granted more discretion.”
  • A 2011 poll of Mississippi Republicans found that 46% think interracial marriage should be illegal, and 14 percent replied “not sure.”
  • A 2007 Gallup poll found that 17% of Americans explicitly disapprove of interracial marriage.
  • Black job applicants without criminal records have the same chance of getting hired as white applicants with criminal records.
  • If current trends continue, one of every three black American males born today can expect to go to prison in his lifetime.

 

There is so much more, and I haven’t even touched the brutality of the branch of US law dedicated to stripping Native Americans of their homes and their lives. Taking into account this disturbing legacy of white supremacy, one wonders how a slogan like “Make America Great Again” could be anything but a dog-whistle for the type of bigotry and hatred that has always been our nation’s past. It also casts the notion of American pride in an ominous light. The only attitude I can summon up thinking about the realities of our country’s past is shame. Is “proud to be an American” a sentiment borne of ignorance, or something more sinister?

 

You must struggle to truly remember this past in all its nuance, error, and humanity. You must resist the common urge toward the comforting narrative of divine law, toward fairy tales that imply some irrepressible justice. The enslaved were not bricks in your road, and their lives were not chapters in your redemptive history. They were people turned to fuel for the American machine…

Our triumphs can never compensate for this. Perhaps our triumphs are not even the point. Perhaps struggle is all we have because the god of history is an atheist, and nothing about this world is meant to be. So you must wake up every morning knowing that no promise is unbreakable, least of all the promise of waking up at all. This is not despair. These are the preferences of the universe itself: verbs over nouns, actions over states, struggle over hope.

Ta-Nehisi Coates, Between The World And Me

“Beating the drum for justice”

I’m going to quote a few pages from Just Mercy by Bryan Stevenson here, because I think they’re really important. It’s pretty crazy how no matter how many times I read them, a little lump in my throat never fails to develop.

I want to let the quote sit by itself without added commentary, so let me just say that I think this is a book that every person should read.

After making the three-hour drive back from Gadsden earlier in the day and heading straight to the office, it was once again approaching midnight as I left the office for home. I got in my car, and to my delight the radio came on as soon as I turned the ignition. In just over three years of law practice I had become one of those people for whom such small events could make a big difference in my joy quotient. On this late night, not only was my radio working but the station was also hosting a retrospective on the music of Sly and the Family Stone. I’d grown up listening to Sly and found myself rolling joyfully through the streets of Atlanta to tunes like “Dance to the Music,” “Everybody Is a Star,” and “Family Affair.”

Our Midtown Atlanta apartment was on a dense residential street. Some nights I had to park halfway down the block or even around the corner to find a space. But tonight I was lucky: I parked my rattling Civic just steps from our new front door just as Sly was starting “Hot Fun in the Summertime.” It was late, and I needed to get to bed, but the moment was too good to let pass, so I remained in the car listening to the music. Each time a tune ended I told myself to go inside, but then another irresistible song would begin, and I would find myself unable to leave. I was singing along to “Stand!” the soaring Sly anthem with the great gospel-themed ending, when I saw a flashing police light approaching. I was parked a few doors up from our apartment, so I assumed that the officers would drive by in pursuit of some urgent mission. When they came to a stop twenty feet in front of me, I wondered what was going on.

Our section of the street only ran one way. My parked car was facing in the proper direction; the police car had come down the street in the wrong direction. I noticed for the first time that it wasn’t an ordinary police cruiser but one of the special Atlanta SWAT cars. The officers had a spotlight attached to their vehicle, and they directed it at me sitting in my car. Only then did it occur to me that they might be there for me, but I couldn’t imagine why. I had been parked on the street for about fifteen minutes listening to Sly. Only one of my car speakers worked and not very well. I knew the music couldn’t be heard outside the car.

The officers sat there with their light pointed at me for a minute or so. I turned off the radio before “Stand!” was over. I had case files on my car seat about Lourida Ruffin and the young man who had been shot in Gadsden. Eventually two police officers got out of their vehicle. I noticed immediately that they weren’t wearing the standard Atlanta police uniform. Instead they were ominously dressed in military style, black boots with black pants and vests.

I decided to get out of my car and go home. Even though they were intensely staring at me in my car, I was still hoping that they were in the area for something unrelated to me. Or if they were concerned that something was wrong with me, I figured I would let them know that everything was okay. It certainly never occurred to me that getting out of my car was wrong or dangerous.

As soon as I opened my car door and got out, the police officer who had started walking toward my vehicle drew his weapon and pointed it at me. I must have looked completely bewildered.

My first instinct was to run. I quickly decided that wouldn’t be smart. Then I thought for an instant that maybe these weren’t real police officers.

“Move and I’ll blow your head off!” The officer shouted the words, but I couldn’t make any sense of what he meant. I tried to stay calm; it was the first time in my life anyone had ever pointed a gun at me.

“Put your hands up!” The officer was a white man about my height. In the darkness I could only make out his black uniform and his pointed weapon.

I put my hands up and noticed that he seemed nervous. I don’t remember deciding to speak, I just remember the words coming out: “It’s all right. It’s okay.”

I’m sure I sounded afraid because I was terrified.

I kept saying the words over and over again. “It’s okay, it’s okay.” Finally I said, “I live here, this is my apartment.”

I looked at the officer who was pointing the gun at my head less than fifteen feet away. I thought I saw his hands shaking.

I kept saying as calmly as I could: “It’s okay, it’s okay.”

The second officer, who had not drawn his weapon, inched cautiously toward me. He stepped on the sidewalk, circled behind my parked car, and came up behind me while the other officer continued to point the gun at me. He grabbed me by the arms and pushed me up against the back of my car. The other officer then lowered his weapon.

“What are you doing out here?” said the second officer, who seemed older than the one who had drawn his weapon. He sounded angry.

“I live here. I moved into that house down the street just a few months ago. My roommate is inside. You can go ask him.” I hated how afraid I sounded and the way my voice was shaking.

“What are you doing out in the street?”

“I was just listening to the radio.” He placed my hands on the car and bent me over the back of the vehicle. The SWAT car’s bright spotlight was still focused on me. I noticed people up the block turning on their lights and peering out of their front doors. The house next to ours came to life, and a middle-aged white man and woman walked outside and stared at me as I was leaned over the vehicle.

The officer holding me asked me for my driver’s license but wouldn’t let me move my arms to retrieve it. I told him that it was in my back pocket, and he fished my wallet out from my pants. The other officer was now leaning inside my car and going through my papers. I knew that he had no probable cause to enter my vehicle and that he was conducting an illegal search. I was about to say something when I saw him open the glove compartment. Opening objects in a parked vehicle was so incredibly illegal that I realized he wasn’t paying any attention to the rules, so saying something about it would be pointless.

There was nothing interesting in my car. There were no drugs, no alcohol, not even tobacco. I kept a giant-size bag of peanut M&Ms and Bazooka bubble gum in the glove compartment to help stave off hunger when I didn’t have time for a meal. There were just a few M&Ms left in the bag, which the officer inspected carefully. He put his nose into the bag before tossing it back. I wouldn’t be eating those M&Ms.

I had not lived at our new address long enough to get a new driver’s license, so the address on my license didn’t match the new location. There was no legal requirement to update the driver’s license, but it prompted the officer to hold me there for another ten minutes while he went back to his car to run a search on me. My neighbors grew bolder as the encounter dragged on. Even though it was late, people were coming out of their homes to watch. I could hear them talking about all the burglaries in the neighborhood. There was a particularly vocal older white woman who loudly demanded that I be questioned about items she was missing.

“Ask him about my radio and my vacuum cleaner!” Another lady asked about her cat who had been absent for three days. I kept waiting for my apartment light to come on and for Charlie to walk outside and help me out. He had been dating a woman who also worked at Legal Aid and had been spending a lot of time at her house. It occurred to me that he might not be home.

Finally, the officer returned and spoke to his partner: “They don’t have anything on him.” He sounded disappointed.

I found my nerve and took my hands off the car. “This is so messed up. I live here. You shouldn’t have done this. Why did you do this?”

The older officer frowned at me. “Someone called about a suspected burglar. There have been a lot of burglaries in this neighborhood.” Then he grinned. “We’re going to let you go. You should be happy,” he said.

With that, they walked away, got in their SWAT car, and drove off. The neighbors looked me over one last time before retreating back into their homes. I couldn’t decide whether I should race to my door so that they could see that I lived in the neighborhood or wait until they were all gone so that no one would know where the “suspected criminal” lived. I decided to wait.
I gathered up my papers, which the cop had scattered all over the car and onto the sidewalk. I unhappily threw my M&Ms into a trash can on the street and then walked into my apartment. To my great relief, Charlie was there. I woke him to tell the story.

“They never even apologized,” I kept saying. Charlie shared my outrage but soon fell back asleep. I couldn’t sleep at all.

The next morning I told Steve about the incident. He was furious and urged me to file a complaint with the Atlanta Police Department. Some folks in the office said I should explain in my complaint that I was a civil rights attorney working on police misconduct cases. It seemed to me that no one should need those kinds of credentials to complain about misconduct by police officers.

I started writing my complaint determined not to reveal that I was an attorney. When I replayed the whole incident in my mind, what bothered me most was the moment when the officer drew his weapon and I thought about running. I was a twenty-eight-year-old lawyer who had worked on police misconduct cases. I had the judgment to speak calmly to the officer when he threatened to shoot me. When I thought about what I would have done when I was sixteen years old or nineteen or even twenty-four, I was scared to realize that I might have run. The more I thought about it, the more concerned I became about all the young black boys and men in that neighborhood. Did they know not to run? Did they know to stay calm and say, “It’s okay”?

I detailed all of my concerns. I found Bureau of Justice statistics reporting that black men were eight times more likely to be killed by the police than whites. By the end of the twentieth century the rate of police shootings would improve so that men of color were “only” four times more likely to be killed by law enforcement, but the problem would get worse as some states passed “Stand Your Ground” laws empowering armed citizens to use lethal force as well.

I kept writing my memo to the Atlanta Police Department and before I knew it I had typed close to nine pages outlining all the things I thought had gone wrong. For two pages I detailed the completely illegal search of the vehicle and the absence of probable cause. I even cited about a half-dozen cases. I read over the complaint and realized that I had done everything but say, “I’m a lawyer.”

I filed my complaint with the police department and tried to forget about the incident, but I couldn’t. I kept thinking about what had happened. I began to feel embarrassed that I hadn’t asserted more control during the encounter. I hadn’t told the officers I was a lawyer or informed them that what they were doing was illegal. Should I have said more to them? Despite the work I’d done assisting people on death row, I questioned how prepared I was to do really difficult things. I even started having second thoughts about going to Alabama to start a law office. I couldn’t stop thinking about how at risk young kids are when they get stopped by the police.

My complaint made it through the review process at the Atlanta Police Department. Every few weeks I’d get a letter explaining that the police officers had done nothing wrong and that police work is very difficult. I appealed these dismissals unsuccessfully up the chain of command. Finally, I requested a meeting with the chief of police and the police officers who had stopped me. This request was denied, but the deputy chief met with me. I had asked for an apology and suggested training to prevent similar incidents. The deputy chief nodded politely as I explained what had happened. When I finished, he apologized to me, but I suspected that he just wanted me to leave. He promised that the officers would be required to do some “extra homework on community relations.” I didn’t feel vindicated.

My caseload was getting crazy. The lawyers defending the Gadsden City Jail finally acknowledged that Mr. Ruffin’s rights had been violated and that he had been illegally denied his asthma medicine. We won a decent settlement for Mr. Ruffin’s family, so they would at least receive some financial help. I turned the other police misconduct cases over to other lawyers because my death penalty docket was so full.

I had no time to make war with the Atlanta Police when I had clients facing execution. Still, I couldn’t stop thinking about how dangerous and unfair the situation was and how I’d done nothing wrong. And what if I had had drugs in my car? I would have been arrested and then would have needed to convince my attorney to believe me when I explained that the police had entered the car illegally. Would I get an attorney who would take such a claim seriously? Would a judge believe that I’d done nothing wrong? Would they believe someone who was just like me but happened not to be a lawyer? Someone like me who was unemployed or had a prior criminal record?
I decided to talk to youth groups, churches, and community organizations about the challenges posed by the presumption of guilt assigned to the poor and people of color. I spoke at local meetings and tried to sensitize people to the need to insist on accountability from law enforcement. I argued that police could improve public safety without abusing people. Even when I was in Alabama, I made time for talks at community events whenever anyone asked.

I was in a poor rural county in Alabama after another trip to pull records in a death penalty case when I was invited to speak at a small African American church. Only about two dozen people showed up. One of the community leaders introduced me, and I went to the front of the church and began my talk about the death penalty, increasing incarceration rates, abuse of power within prisons, discriminatory law enforcement, and the need for reform. At one point, I decided to talk about my encounter with the police in Atlanta, and I realized that I was getting a bit emotional. My voice got shaky, and I had to rein myself in to finish my remarks.

During the talk, I noticed an older black man in a wheelchair who had come in just before the program started. He was in his seventies and was wearing an old brown suit. His gray hair was cut short with unruly tufts here and there. He looked at me intensely throughout my presentation but showed no emotion or reaction during most of the talk. His focused stare was unnerving. A young boy who was about twelve had wheeled him into the church, probably his grandson or a relative. I noticed that the man occasionally directed the boy to fetch things for him. He would wordlessly nod his head, and the boy seemed to know that the man wanted a fan or a hymnal.

After I finished speaking, the group sang a hymn to end the session. The older man didn’t sing but simply closed his eyes and sat back in his chair. After the program, people came up to me; most folks were very kind and expressed appreciation for my having taken the time to come and talk to them. Several young black boys walked up to shake my hand. I was pleased that people seemed to value the information I shared. The man in the wheelchair was waiting in the back of the church. He was still staring at me. When everyone else had left, he nodded to the young boy, who quickly wheeled him up to me.

The man’s expression never changed as he approached me. He stopped in front of me, leaned forward in his wheelchair, and said forcefully, “Do you know what you’re doing?” He looked very serious, and he wasn’t smiling.

His question threw me. I couldn’t tell what he was really asking or whether he was being hostile. I didn’t know what to say. He then wagged his finger at me, and asked again. “Do you know what you’re doing?”

I tried to smile to diffuse the situation but I was completely baffled. “I think so.…”

He cut me off and said loudly, “I’ll tell you what you’re doing. You’re beating the drum for justice!” He had an impassioned look on his face. He said it again emphatically, “You’ve got to beat the drum for justice.”

He leaned back in his chair, and I stopped smiling. Something about what he said had sobered me. I answered him softly, “Yes, sir.”

He leaned forward again and said hoarsely, “You’ve got to keep beating the drum for justice.” He gestured and after a long while said again, “Beat the drum for justice.”

He leaned back, and in an instant he seemed tired and out of breath. He looked at me sympathetically and waved me closer. I did so, and he pulled me by the arm and leaned forward. He spoke very quietly, almost a whisper, but with a fierceness that was unforgettable.

“You see this scar on the top of my head?” He tilted his head to show me. “I got that scar in Greene County, Alabama, trying to register to vote in 1964. You see this scar on the side of my head?” He turned his head to the left and I saw a four-inch scar just above his right ear. “I got that scar in Mississippi demanding civil rights.”

His voice grew stronger. He tightened his grip on my arm and lowered his head some more. “You see that mark?” There was a dark circle at the base of his skull. “I got that bruise in Birmingham after the Children’s Crusade.”

He leaned back and looked at me intensely. “People think these are my scars, cuts, and bruises.”

For the first time I noticed that his eyes were wet with tears. He placed his hands on his head. These aren’t my scars, cuts, and bruises. These are my medals of honor.”

He stared at me for a long moment, wiped his eyes, and nodded to the boy, who wheeled him away.

I stood there with a lump in my throat, staring after him.

After a moment, I realized that the time to open the Alabama office had come.

Bryan Stevenson, Just Mercy

Overemphasizing disagreement

I’ve noticed a tendency in myself and others during debate to only respond to the parts of what others say that I disagree with, taking the agreement for granted. This makes some sense; if you agree with 95% of an argument somebody is making, there is the most progress to be made by focusing on the 5% remaining difference. But I think this also causes the perception on both sides that there is a greater distance to be bridged than there is in reality. Constantly focusing on subtle points of disagreement can also be perceived as being unresponsive and indifferent towards a significant part of the arguments being made.

This is an optimistic take on phenomena like the backfire effect – a lot of the sense that you and your interlocutor are getting no closer during conversation might be the result of this form of miscommunication. I think a good policy for reducing misunderstanding is something like Rapaport’s rules – explicitly stating points of agreement before going into disagreement. This isn’t only good for reducing misunderstanding – I’ve noticed that stating points of agreement, especially things I’ve just been convinced of, has the effect of actually making it easier to change my mind.

Regularization as approximately Bayesian inference

In an earlier post, I showed how the procedure of minimizing sum of squares falls out of regular old frequentist inference. This time I’ll do something similar, but with regularization and Bayesian inference.

Regularization is essentially a technique in which you evaluate models in terms of not just their fit to the data, but also the values of the parameters involved. For instance, say you are modeling some data with a second-order polynomial.

M = { f(x) = a + bx + cx2 | a, b, c ∈ R }
D = { (x1, y1), …, (xN, yN) }

We can evaluate our model’s fit to the data with SOS:

SOS = ∑ (yn – f(xn))2

Minimizing SOS gives us the frequentist answer – the answer that best fits the data. But what if we suspect that the values of a, b, and c are probably small? In other words, what if we have an informative prior about the parameter values? Then we can explicitly add on a penalty term that increases the SOS, such as…

SOS with L1 regularization = k1 |a| + k2 |b| + k3 |c| + ∑ (yn – f(xn))2

The constants k1, k2, and k3 determine how much we will penalize each parameter a, b, and c. This is not the only form of regularization we could use, we could also use the L2 norm:

SOS with L2 regularization = k1 a2 + k2 b2 + k3 c2 + ∑ (yn – f(xn))2

You might, having heard of this procedure, already suspect it of having a Bayesian bent. The notion of penalizing large parameter values on the basis of a prior suspicion that the values should be small sounds a lot like what the Bayesian would call “low priors on high parameter values.”

We’ll now make the connection explicit.

Frequentist inference tries to select the theory that makes the data most likely. Bayesian inference tries to select the theory that is made most likely by the data. I.e. frequentists choose f to maximize P(D | f), and Bayesians choose f to maximize P(f | D).

Assessing P(f | D) requires us to have a prior over our set of functions f, which we’ll call π(f).

P(f | D) = P(D | f) π(f) / P(D)

We take a logarithm to make everything easier:

log P(f | D) = log P(D | f) + log π(f) – log P(D)

We already evaluated P(D | f) in the last post, so we’ll just plug it in right away.

log P(f | D) = – SOS/2σ2 – N/2 log(2πσ2)) + log π(f) – log P(D)

Since we are maximizing with respect to f, two of these terms will fall away.

log P(f | D) = – SOS/2σ2 + log π(f) + constant

Now we just have to decide on the form of π(f). Since the functional form of f is determined by the values of the parameters {a, b, c}, π(f) = π(a, b, c). One plausible choice is a Gaussian centered around the values of each parameter:

π(f) = exp( -a2 / 2σa2 ) exp( -b2 / 2σb2 ) exp( -c2 / 2σc2 ) / √(8π3σa2σb2σc2)
log π(f) = -a2/2σa2 – b2/2σb2 – c2/2σc2 – ½ log(8π3σa2σb2σc2)

Now, throwing out terms that don’t depend on the values of the parameters, we find:

log P(f | D) = – SOS/2σ2 -a2/2σa2 – b2/2σb2 – c2/2σc2 + constant

This is exactly L2 regularization, where each kn = σ2n2. In other words, L2 regularization is Bayesian inference with Gaussian priors over the parameters!

What priors does L1 regularization correspond to?

log π(f) = -k1 |a| – k2 |b| – k3 |c|
π(a, b, c) = e-k1|a| e-k2|b| e-k3|a|

I.e. the L1 regularization prior is an exponential distribution.

This can be easily extended to any regularization technique. This is a way to get some insight into what your favorite regularization methods mean. They are ultimately to be cashed out in the form of your prior knowledge of the parameters!

Against the status quo

A good analysis of our society involves, I think, the ability to look at everything from the perspective of an alien – to remove ourselves from the contingent fact of our place in history and negate the biases and blind spots that might result from it.

If you were to be dropped out of a time machine into pretty much any time in American history, you would see enormous injustices happening everywhere, as well as status quo that is either blind or apathetic towards these injustices.

Social progress is so sudden and final that it’s easy to lose track of this truth. Women have only had the right to vote for the last 100 years, and the first woman to be admitted to Harvard Law School was in 1950. I’m very confident that it will never again be the case that women will not be legally allowed to vote in the United States, or banned from institutions of higher learning. We’ve moved as a society in a direction that we are never coming back from.

Racial discrimination in voting was only technically prohibited after 1965. Just fifty to sixty years ago in the United States, we have elected officials who rose to power by proudly preaching Jim Crow and segregation, affiliating with the Ku Klux Klan, and boasting about chasing black people out of their restaurants with ax handles. Lynching was a common practice in the United States for hundreds of years; Emmett Till was brutally tortured to death for wolf-whistling a white woman in 1955. Professional photographs of lynchings were published as postcards, to be sold as popular souvenirs. Etc etc.

The point of all of this is just to say that it looks like we should have a really strong presumption in favor of there being current massive flaws in the status quo. The argument being: at basically any point in history, the status quo has been blind towards and accept of stunning levels of injustice. While the trend of history has been towards progress and a widening moral circle, we should still suspect that there are aspects of society today that are massively unjust unless we think that we are in an incredibly unique historical moment. We should also suspect that we are vulnerable to the same biases that people have always historically been blind to in failing to see basic moral failings of the society in which they exist.

This raises the question: What are the massive injustices going on today? Where are our blind spots? 100 years down the line, what will people read about us that will make them wonder how we sat by and didn’t do anything?

This is a really hard question, and I only have a few answers that I’m pretty confident about. One of these is animal rights issues. We are inflicting enormous degrees of needless cruelty on animals, and I get the sense that the average American’s attitude towards this is essentially total apathy (although there has been a definite change in this attitude in recent years). Another possibility is mass incarceration. The US has the highest incarceration rate in the world, and we lock crazy amounts of people up for victimless crimes. Time in prison is brutally unpleasant, and for those that act out, solitary confinement is the punishment.

What other things do we accept in modern society that our descendants will look back on and hang their heads in shame about?