Two years ago, I was struck by a talk at AIUK 2021 from Mahlet (Milly) Zimeta of the Open Data Institute. Zimeta presented on Decolonising the data and drew on classical literature to gain insights into designing better data infrastructure. The theme was set by The Fragility of Goodness by Martha Nussbaum, which uses Greek tragedy to examine the impact of luck on a person’s moral status. Within Greek tragedy, as much as in real life, circumstance can determine whether a person is ‘good’ or ‘bad’. These circumstances can be reversed, turning a hero into a villain, the blessed into the wretched.
At another AI conference, several years before, a delegate (I sadly cannot remember whom) illustrated this point specifically to me in what had started as an idle conversation about centralised citizen identity systems. “That was what the Dutch thought, and then the Nazi’s occupied”, she said and then went on to explain how benign Dutch record-keeping prior to Nazi occupation had enabled rapid persecution of Dutch Jews. The Netherlands had a long history of rigorous civil registration for military conscription and administrative records were “exceptionally comprehensive” (quotation by James C. Scott in Seeing Like a State, p78). The lesson is stark: even work done for advantageous purposes can lead to catastrophic outcomes.
These powerful insights led to a growing unease for me with a project at work that was attempting to map property boundaries in another part of the world. In my own country, I know that a property boundary is associated with physical concepts like fence and hedge, and legal concepts like planning permission and trespass. It is also associated with political concepts of agency and privacy, and cultural concepts of tolerance and sharing. However, what do I know of the physical, legal, political or cultural connotations of ‘property boundary’ in another country? This illustrates the Fragility of Good across borders. At very least, the project should have been asking what are the consequences of providing (accurate or inaccurate) information about property boundaries, to an actor in a society with different physical, legal, political and cultural conceptions. Thankfully, that work has now ceased.
I am becoming increasingly aware of how data and tools for public ‘good’ can have ‘bad’ outcomes. Evidently, good is fragile over both time and space but perhaps the most consequential fragility arises from a paucity of insight into the lives and luck of others. The stories told in both Peeters and Widlak (2018) “The digital cage” and Eubanks (2019) “Automating Inequality” attentively illustrate the discriminatory impact that systems designed for ‘good’ - from streamlining car registration and welfare payments to identifying the risk of child abuse - can devastate the lives of those they should be serving. These systems trap individuals and families in a “digital cage” or “digital poorhouse”, unable to rectify errors or inbuilt discrimination because the system designers did not imagine that such failings were possible or mattered.
There are two approaches to dealing with fragile borders. The first is to fortify them. This is the purpose of legislation and regulation that defines what is ‘good’. However, these are too often limited by the scope of the small group of people creating them. This means that even the most socially-minded laws and regulations rarely keep up with developing technology or society. Consider the damage that continuing lack of data privacy is causing to us all. Too often people suffer indignity and abuse before statutary instruments are created or updated.
Another approach is to embrace fragility and seek to imagine its every outcome. With specific reference to poverty, Eubanks proposes a gut-check:
Does the tool increase the self-determination and agency of the poor?
Would the tool be tolerated if it was targeted at non-poor people?
This is a simple step, but still somewhat externalises the experience of ‘the poor’*. Zimeta** takes us further by identifying that it is not just goodness that is fragile but also power. Public services tend to be designed by those with some power and applied to those with little. Public services, and data infrastructure, should therefore be designed as if power could be reversed at any time. This can be grasped as the Maximin Principle - the outcome of John Rawls’s thought experiment about the world that we’d create if we did not yet know our place in it. The principle is that society should maximise the welbeing of those at the most minimal level. Yet, none of us are ignorant of our place in the world and we cannot assume that stewards of data and creators of tools have the imagination to fully apply gut-checks, maximin or grasp the Fragility of Power.
Recently, I was introduced to the work of Jeni Tennison who founded Connected by Data. The vision of Connected by Data is:
We think organisations and communities should make decisions about data together
and their Theory of Change describes building capability by:
Working alongside communities to help them define how data governance should work for them
This participatory approach to data stewardship does not rely on the limited imagination of a few. Instead, if done well, it uses real-world experience and insight to decide the limits of ‘good’ and ‘power’ and to understand how they can be reversed. In my property boundary example above, a much better approach would have been for the work to be undertaken by a group that represented the diversity of that society. As fragile habitats are strengthened by increasing their biodiversity, fragile goods are strengthened by increasing the diversity of perspectives. Approaches to public participation is a rapidly developing area of research but there is already a wealth of resources on how to do it well.
Towards the end of her talk, Zimeta comments that the knowledge of fragility means “living with humility”. Public participation is a practical expression of that humility and I hope it will become commonplace in the coming few years.
*I don’t intend to be negative about Eubanks’ book - not only is it a much deeper study than I convey in this post, it is also an extremely accessible read and thus itself obeys the maximin principle
**If you have not seen Zimeta’s talk, please do - it covers so much more than I have touched on in this post.
Comments powered by Disqus.