1 month, 9 days ago

Trial by Ordeal

Link: http://feedproxy.google.com/~r/Soapbox/~3/w4cs_xJAosQ/trial-by-ordeal.html

Some people think that ethical principles only apply to implemented systems, and that experimental projects (trials, proofs of concept, and so on) don’t need the same level of transparency and accountability.

Last year, Google employees (as well as US senators from both parties) expressed concern about Google’s Dragonfly project, which appeared to collude with the Chinese government in censorship and suppression of human rights. A secondary concern was that Dragonfly was conducted in secrecy, without involving Google’s privacy team.  

Google’s official position (led by CEO Sundar Pinchai) was that Dragonfly was “just an experiment”. Jack Poulson, who left Google last year over this issue and has now started a nonprofit organization called Tech Inquiry, has also seen this pattern in other technology projects.

“I spoke to coworkers and they said ‘don’t worry, by the time the thing launches, we’ll have had a thorough privacy review’. When you do R and D, there’s this idea that you can cut corners and have the privacy team fix it later.” (via Alex Hern)

A few years ago, Microsoft Research ran an experiment on “emotional eating”, which involved four female employees wearing smart bras. “Showing an almost shocking lack of sensitivity for gender stereotyping”, wrote Sebastian Anthony. While I assume that the four subjects willingly volunteered to participate in this experiment, and I hope the privacy of their emotional data was properly protected, it does seem to reflect the same pattern – that you can get away with things in the R and D stage that would be highly problematic in a live product.

Poulson’s position is that the engineers working on these projects bear some responsibility for the outcomes, and that they need to see that the ethical principles are respected. He therefore demands transparency to avoid workers being misled. He also notes that if the ethical considerations are deferred to a late stage of a project, with the bulk of the development costs already incurred and many stakeholders now personally invested in the success of the project, the pressure to proceed quickly to launch may be too strong to resist.


Sebastian Anthony, Microsoft’s new smart bra stops you from emotionally overeating (Extreme Tech, 9 December 2013)

Erin Carroll et al, Food and Mood: Just-in-Time Support for Emotional Eating (Humaine Association Conference on Affective Computing and Intelligent Interaction, 2013)

Ryan Gallagher, Google’s Secret China Project “Effectively Ended” After Internal Confrontation (The Intercept, 17 December 2018)

Alex Hern, Google whistleblower launches project to keep tech ethical (Guardian, 13 July 2019)

Casey Michel, Google’s secret ‘Dragonfly’ project is a major threat to human rights (Think Progress, 11 Dec 2018)

Iain Thomson, Microsoft researchers build ‘smart bra’ to stop women’s stress eating(The Register, 6 Dec 2013)