Blog Ipsa Loquitur

Like just about everyone who took an Intro to Psychology course in college, I learned about Philip Zimbardo’s “Stanford Prison Experiment” as a parable about the dangers of groupthink and the ease with which authority—real or imagined—led to the abuse of power. I didn’t know the experiment was fatally flawed until I read The Lifespan of a Lie by Ben Blum:

Zimbardo, a young Stanford psychology professor, built a mock jail in the basement of Jordan Hall and stocked it with nine “prisoners,” and nine “guards,” all male, college-age respondents to a newspaper ad who were assigned their roles at random and paid a generous daily wage to participate. The senior prison “staff” consisted of Zimbardo himself and a handful of his students. The study was supposed to last for two weeks, but after Zimbardo’s girlfriend stopped by six days in and witnessed the conditions in the “Stanford County Jail,” she convinced him to shut it down. Since then, the tale of guards run amok and terrified prisoners breaking down one by one has become world-famous, a cultural touchstone that’s been the subject of books, documentaries, and feature films — even an episode of “Veronica Mars.” ​ The way I learned it, the guard’s behavior was a natural byproduct of the prisoner/guard relationship. However:

Once the simulation got underway, [The “warden,” undergrad student David] Jaffe explicitly corrected guards who weren’t acting tough enough, fostering exactly the pathological behavior that Zimbardo would later claim had arisen organically.

“The guards have to know that every guard is going to be what we call a tough guard,” Jaffe told one such guard. “[H]opefully what will come out of this study is some very serious recommendations for reform… so that we can get on the media and into the press with it, and say ‘Now look at what this is really about.’ … [T]ry and react as you picture the pigs reacting.”

Though most guards gave lackluster performances, some even going out of their way to do small favors for the prisoners, one in particular rose to the challenge: Dave Eshelman, whom experimenters nicknamed “John Wayne” for his Southern accent and inventive cruelty. But Eshelman, who had studied acting throughout high school and college, has always admitted that his accent was just as fake as Korpi’s breakdown. His overarching goal, as he told me in an interview, was simply to help the experiment succeed.

I’m not a sociologist, and I don’t know that a double-blind experiment is the right approach here. (Though it’s fun to imagine a “prison” split in half by bars, and the people on each side are unsure whether they’re prisoners or guards.) But over and over in Blum’s examination of the Stanford Prison Experiment, the researchers seem to have pretty blatantly interfered with the process.

And this isn’t just modern science second-guessing the research methods of yesteryear:

Despite the Stanford prison experiment’s canonical status in intro psych classes around the country today, methodological criticism of it was swift and widespread in the years after it was conducted. Deviating from scientific protocol, Zimbardo and his students had published their first article about the experiment not in an academic journal of psychology but in The New York Times Magazine, sidestepping the usual peer review.

Famed psychologist Erich Fromm, unaware that guards had been explicitly instructed to be “tough,” nonetheless opined that in light of the obvious pressures to abuse, what was most surprising about the experiment was how few guards did. “The authors believe it proves that the situation alone can within a few days transform normal people into abject, submissive individuals or into ruthless sadists,” Fromm wrote. “It seems to me that the experiment proves, if anything, rather the contrary.”

For those of you keeping score at home, the silver lining here appears to be “the situation didn’t turn guards into monsters; some of the guards were monsters this whole time,” which is silver-ish at best. I’ll take it.

Remember the San Bernardino shooters? The husband and wife team that walked into his office holiday party and opened fire with a pair of assault rifles, and were then shot to death by police after a brief car chase? Detectives recovered the husband’s work phone, an iPhone with full-disk encryption enabled.

At the time, the FBI had a whole song and dance routine about how dangerous encryption was, and how companies like Apple needed to build the FBI a system to get keys to unlock any device at any time. But the FBI mishandled the phone in that investigation and made their own jobs a whole lot harder.

As I wrote about it back then:

Look, if local police in a tiny town in the middle of nowhere had screwed up this badly, we’d wonder why the FBI wasn’t entrusted with this. But the FBI made their jobs a lot harder; this can’t be their first time at the iPhone Evidence Rodeo, so how could they have locked themselves out of the phone? […] Sure is funny how the FBI was hilariously inept in their investigation, and now the only way they can get into the phone is by having Apple build a tool to circumvent its own security measures, right?

After all, if you were the FBI, and you wanted a test case, this whole “real live radicalized Muslim terrorist committing an act of war on American soil with pipe bombs and assault rifles before dying in a glorious shootout with the police” thing would be, like, the perfect test case. It sure would be a shame if the traditional investigation went awry, and the only way to get access to the phone was filing a completely novel test case.

That’s a pretty cynical take, I’ll admit. I more or less tiptoed up to the precipice of accusing certain members of the FBI of sabotaging their own investigation; it didn’t seem out of the question that the FBI was using that phone as ammunition in a broader war on encryption. Fortunately, cooler heads prevailed and the only backdoors in my phone are planted by the NSA. Ha ha(?).

Well, earlier this year, the FBI’s Inspector General issued a report on the San Bernardino iPhone investigation. That report is not much kinder to the FBI than my blog post from 2016. Susan Landau, law professor at Tufts University and one of the preeminent experts on this topic, breaks it down for folks like you and I:

For the FBI, the IG report brings some good news: No one deliberately withheld knowledge to prevent opening the locked iPhone. But that’s about the only positive revelation. The IG report chronicles foot dragging during the efforts to open the locked device and, in a critical instance, an aversion to finding a technological resolution of the issue outside of the court case. Above all, the IG report casts doubt on the argument that locked phones are “warrant-proof” devices preventing law enforcement from doing its job.

The FBI’s failure to open the iPhone was a result of bureaucracy and slowdown. Two units of the FBI’s Operational Technology Division (OTD) were key to eventually unlocking the iPhone: the Cryptologic and Electronic Analysis Unit (CEAU), which examines data on digital devices, working largely on criminal cases, and the Remote Operations Unit (ROU), which uses network exploitation techniques and appears to work largely in classified cases. […]

The CEAU and ROU weren’t communicating about this matter, but once they did, things moved pretty quickly:

The ROU chief reached out to his vendors, and on March 16, 2016, discovered that one of them was already 90 percent of the way toward a solution. At the FBI’s request, the vendor reallocated resources, moving work on opening the iPhone “to the ‘front burner.’” A month later, a vendor demonstrated a solution to the FBI, and the court conflict between Apple and the FBI was over.

Opening the locked iPhone should have been a good within the FBI. But that was not the view held by the CEAU chief; he apparently asked the ROU chief, “Why did you do that for?” The CEAU chief told the Inspector General “after the outside vendor came forward, the CEAU chief became frustrated that the case against Apple could no longer go forward.”

That’s a striking story. We have the FBI director testifying—and U.S. attorneys submitting a motion operating of of the same premise—that only Apple could unlock this terrorist’s phone. But it seems that what was really going on, at least on the part of some FBI investigators, was an unwillingness to really try.

Boy, that sure sounds like what privacy experts were publicly worried about back in 2016, doesn’t it?

I really miss the days when there wasn’t a whole month’s worth of news crammed into a week. Last week, the first details of Michael Cohen’s essential(?) consulting business have been leaked into the public. I don’t think there’s anything intrinsically illegal about selling information you have into the mindset of the most powerful person in the world; Cohen’s allowed to shop his thoughts on President Trump to whomever he likes. If your old boss became the President, you could charge AT&T a million dollars to answer their questions about your boss’s favorite model of phone, and what you know about his thoughts on 5G. But it seems some of those shoppers “hired” Cohen awfully soon after Trump tweeted vague threats to upend their industries, which looks a little shady.

More interesting than Cohen’s naked profiteering on the margins is what he probably knows about the heart of the Trump Organization. There were two stories published, by Buzzfeed and USA Today, in January that I think will ultimately result in more indictments than Cohen’s consulting business. The first is by Thomas Frank in Buzzfeed, titled Secret Money: How Trump Made Millions Selling Condos To Unknown Buyers:

In 2008, as the Great Recession cooled real-estate markets, Trump could not make a payment on a bank loan that he had guaranteed personally for $40 million. Trump Entertainment Resorts, which owned the Trump Taj Mahal casino in Atlantic City, faced a $53 million payment to bondholders. Trump forestalled the bank payment by suing the lender, Deutsche Bank. The casino filed for bankruptcy in 2009.

At the same time, Trump became financially entwined with Russians. In March 2008, a Russian billionaire paid Trump $95 million for a Palm Beach, Florida, estate that Trump had bought four years earlier for $41 million. Donald Trump Jr. told a Moscow real-estate conference in June 2008 that his father’s company, the Trump Organization, was planning to build condos and hotels in Russia. And he told a New York conference in September 2008, “We see a lot of money pouring in from Russia.”

Got that? Trump flipped a property for $54m in profit the same year he owed a $53m payment to bondholders, which is also the same year as the Don Jr. quote above. One does wonder.

But it’s not just the fact that there was “money pouring in.” It’s that shell companies hide the identifies of the buyers. Anyone can set up a company like this in an afternoon, and there are no laws requiring transparency into the real owners of the shell company. And there are tens of millions of dollars of transactions with shell companies during the years when Trump Jr. liked to brag about Russian money. Frank continues:

Trump Jr. was executive vice president of development and acquisitions at the Trump Organization, which opened two major condo towers in early 2008 after a four-year lull. By the time Trump Jr. made his now-famous comment in September 2008, cash-paying shell companies had bought $43 million worth of condos at the Trump International Hotel and Tower Chicago and at the Trump International Hotel Las Vegas.

At a Trump-licensed condo building in Miami-Dade, cash-paying shell companies had bought $32 million worth of condos.

During this time, the future president and his children also were heavily promoting the Trump SoHo, a lower Manhattan high-rise that has been mired in controversy. In his September 2008 remarks, Donald Jr. cited the project: “In terms of high-end product influx into the US, Russians make up a pretty disproportionate cross-section of a lot of our assets; say in Dubai, and certainly with our project in SoHo and anywhere in New York.”

And Nick Penzenstadler in USA Today reported earlier this year that most folks buying real estate from the President’s company are using shell companies to hide their identity:

The trend toward Trump’s real estate buyers buyers obscuring their identities began around the time he won the Republican nomination, midway through 2016, according to USA TODAY’s analysis of every domestic real estate sale by one of his companies.

In the two years before the nomination, 4% of Trump buyers utilized the tactic. In the year after, the rate skyrocketed to about 70%. USA TODAY’s tracking of sales shows the trend held firm through Trump’s first year in office.

Profits from sales of those properties flow through a trust run by Trump’s sons. The president is the sole beneficiary of the trust and he can withdraw cash at any time.

I understand some of the uproar about the not-hush-money portions of Cohen’s “consulting” business, but if I were looking to launder money or just bribe the President, I wouldn’t funnel the money through Cohen to do it. It seems much easier to set up a shell company and simply buy a $2 million condo for $10 million.

Maciej Cegłowski is one of the best writers about the internet you can read. In April 2017, he gave a talk titled Build a Better Monster: Morality, Machine Learning, and Mass Surveillance that you should watch or read in its entirety. At base, the talk is about Surveillance Capitalism, which is the economic basis of the Internet. As Cegłowski puts it, “every interaction with a computing device leaves a data trail, and whole industries exist to consume this data.”

Here’s his bit about the advertising industry:

Ads are served indirectly, based on real-time auctions conducted when the page is served by a maze of intermediaries. This highly automated market is a magnet for fraud, so much of the complexity of modern ad technology consists of additional (and invasive) tracking.

Curiously, despite years of improvements in the technology, and the amount of user data available to the ad networks, online advertising isn’t targeted all that well. You can convince yourself of this by turning off your ad blocker for a week. In a recent example, Chase stopped serving ads to 95% of its websites and saw no measurable difference in ‘engagement’ metrics.

Many advertisers are simply not equipped to use the full panoply of surveillance options. More importantly, adversaries have become very good at gaming real-time ad marketplaces, which introduces noise into the system. An uncharitable but accurate description of online advertising in 2017 is “robots serving ads to robots”. A considerable fraction (only Google and Facebook have the numbers) of the money sloshing around goes to scammers.

So robots bid against one another for the right to show ads on pages, and other robots visit pages with ads to drive up the value of the pages with ads on them in the first place. Of all of humanity’s creations, this quasi-ecosystem has to be one of the most baffling.

As an aside, even the biggest and ostensibly best surveillance companies still haven’t gotten the hang of this stuff. Facebook recently showed me that three of my friends had recently visited New York City, and encouraged me to visit New York City as well. Somehow, Facebook’s system failed to account for the fact that all three of those friends—not to mention myself—live in New York City.

That’s not to say that this demonstrates Facebook is somehow lousy at surveillance. This is just a funny outlier in the midst of surveillance so scary-good that it’s hard to say with certainty that Facebook isn’t listening to the conversations you have in front of your phone. Heck, IBM’s Watson answered a Jeopardy question in the “U.S. Cities” category with “Toronto” en route to crushing its human competitors. The more capable these systems get, the funnier the outliers.

But Also

The outliers serve a second purpose, according to Cegłowski. This is one of his best arguments:

The relative ineffectiveness of targeted advertising creates pressure to collect more data. Ad networks are not just evaluated by their current ad revenue, but by expectations about what new ad formats will make possible in the future, in a dynamic I’ve called “investor storytime”. The more poorly current ads perform, the more room there is to tell convincing stories about future advertising technology, which of course will require new forms of surveillance.

This trick of constantly selling the next version of the ad economy works because new ad formats really do have better engagement. Advertising is like a disease: it takes people time to develop immunity and resistance. Even the first banner ad had a 70% click through rate.

So long as advertising is the economic engine of the internet, the march toward ever more invasive surveillance technologies and ever creepier ads is inexorable. ​Toward that end, Cegłowski shares some meditations on what might make the ads of the future creepy in a way that’s hard to really wrap your head around. Advertising will be powered by artificial intelligences, but AIs are inherently alien, mostly because we don’t understand enough about brains to be able to reinvent them.

In the past, we assumed that when machines reached near-human performance in tasks like image recognition, it would be thanks to fundamental breakthroughs into the nature of cognition. We would be able to lift the lid on the human mind and see all the little gears turning.

What’s happened instead is odd. We found a way to get terrific results by combining fairly simple math with enormous data sets. But this discovery did not advance our understanding. The mathematical techniques used in machine learning don’t have a complex, intelligible internal structure we can reason about. Like our brains, they are a wild, interconnected tangle.

The result is that the algorithms that decide what we see (ads and contents) are smarter than us in some ways, and dangerously unfit to decide how to filter the word for us in other ways. The future’s going to be weird!

One of my favorite series on the Lawyers, Guns & Money blog is Abigail Nussbaum’s Political History of the Future, in which she analyzes the politics of science fiction settings. This bit from her essay on Iain M. Banks’s The Culture series is a wonderful introduction to something like Star Trek’s Federation, but simultaneously weirder and more realistic:

The Culture wants for nothing, and yet it is defined by a profound need for meaning. The Culture is the most radically, anarchically free society imaginable, and yet it is governed by AIs (known as “Minds”) who make decisions at a speed and complexity that human citizens could never hope to match. The Culture is constitutionally peaceful, and yet it constructs ships and weapons platforms capable of dealing out death and destruction on a galactic scale.

What’s more, the Culture’s covert operations wing, Special Circumstances, routinely interferes in the affairs of other societies, sometimes nudging them gently towards more equal, more benevolent forms of government, and sometimes orchestrating coups and civil wars in the hopes that these will lead to better results down the line. It can be hard to tell whether we’re meant to approve of the Culture or be horrified by it. Beyond that, it can be hard to tell whether the Culture is a utopian vision of the future, or a dystopian parody of the present.

​Nussbaum’s personal blog has a decade-long series of reviews of the individual books in The Culture series, if you’re already familiar with the setting.

Either way, don’t skip the link to the list of names which The Culture’s Minds give their spaceships. Whether or not superintelligent AIs are going to destroy humanity, I hope they have the decency to be as irreverent as the Minds are.

Rachel Louise Snyder in the New Yorker last month, writing about The Trial of Noor Salman and Its Shocking Disregard for Survivors of Domestic Violence left me pretty rattled. If you’re unaware, Salman is the widow of the guy who murdered forty-nine people at the Pulse Nightclub in Miami in 2016. When Salman was initially arrested and questioned on suspicion of being an accomplice, her husband—who domestically abused her—was alive. But over the course of her twelve-hour interrogation, the F.B.I. let her know he’d been shot dead by the police.

Salman entered the F.B.I. office believing herself the wife of an abuser, and learned that she was a widow. Suddenly, she no longer lived under the authoritarian rule of a man who watched grisly beheading videos on his phone while at work. Salman’s defense attorneys used very little of her history of abuse in their arguments, because the larger point for them was to convince jurors that she did not know of his plans before the attack unfolded. But from my viewpoint her victimhood was both entirely pertinent and shockingly disregarded by both the F.B.I. investigators and, later, by the federal prosecutors who chose to put her on trial.

Law-enforcement officials are not always familiar with the control that abusers have over their victims. They frequently encounter the following scenario: responders are called to a scene of domestic violence in a home. When they arrive, often to their dismay and annoyance, the victim begins to scream at them to go away, to tell them they aren’t wanted, even to holler obscenities at them. This happens even when a victim’s physical injuries—black eyes, bloody wounds—are obvious. Police often interpret this behavior as evidence that the victim is mentally or emotionally unstable. But this behavior is a message not to law enforcement, but to the abuser. It says, “I know you will be here when they are gone. I am loyal even in the face of your violence.” It says, “Please don’t kill me when they are gone.”

This left a crater in my heart. Ignorance is bliss, y’all.