Toju Duke: ‘Artificial intelligence amplifies systemic injustices that we’re supposed to have dropped by now’

There are at least six documented cases of illegal detention in the United States, all of them black people, due to automatic facial recognition systems that made a mistake. Also in the US, an investigation was opened against the Apple Card for offering a lower spending limit to women than to men with the same level of income. In the United Kingdom, the algorithm that determined A-Level grades during the Covid-19 pandemic in the absence of exams unjustly disadvantaged students from poorer schools.

“This is where we’re headed if we don’t fix it,” says Toju Duke after citing those cases of algorithmic discrimination, the unintended consequences of artificial intelligence. Born in Nigeria 41 years ago, this Briton worked at Google for 10 years as the Responsible AI program manager, in Dublin; now she devotes her time to dissemination and has founded an NGO, Diverse AI, to encourage diversity in that discipline. “The explosion of generative AI will increase the magnitude of the impact of this technology on society,” she adds. The expert spoke to EL PAÍS last Tuesday after her presentation in Madrid at EnlightED, a global conference on education, technology and innovation.

Question. In your experience, are ethics and AI really compatible?

Answer. They have to be. It’s about fairness, transparency, privacy and security, human rights protection, data ethics and other ethical considerations. There is a way to do it; we cannot have the excuse of saying it’s not possible. We can actually run fairness tests. We can make sure that there’s some form of transparency. The EU AI Act says to upload a dataset to a public directory. In terms of coding, we cannot code ethics into the program, but we can run lots of tests and we can fine tune the models, adjust the models based on the results of the tests, to make sure that they produce better outputs and results. Otherwise, things can happen, like the young man in Belgium who took his own life due to this terrible chatbot called Eliza.

Q. It all depends on the people who develop the algorithms.

A. Definitely. But I don’t want to just put the responsibility on them. We can put more pressure on policymakers; say “you need to protect this.” But beyond regulation, we need standardized bodies that actually help with these responsible AI frameworks and guide organizations like the EU to know how to do this properly.

Q. How can responsible AI be developed?

A. It is impossible if there is no diversity in the teams. And I think the next issue is lack of awareness. A lot of research scientists just want to have this cutting edge technology, they want to be able to say, “we did this first.” Developing a responsible AI slows down the launch process because now you have to start running some more tests. What if I’m from Malaysia? Maybe the image generators are not representing my culture, I can’t see anyone from Malaysia. Or this wedding picture does not really represent Malaysian culture.

Q. Do you think algorithmic discrimination is going to be a major social problem in the coming years?

A. It’s a problem right now. We just don’t identify it as such. And it’s going to amplify, especially with generative AI. There was a study earlier this year in which, if you did an image search for male CEOs versus female CEOs on a typical image search, I think it showed about 30% higher representation of males. When we use like image generators like Midjourney and Dall-E, it rises to about 70%. We’re showing our next generation that CEOs are predominantly male. These are systemic injustices that have existed in the world over the years that we’re supposed to have dropped by now. It’s very subtle, it’s very hidden. But if we don’t fix it today, the same problem that has existed in the world will just be amplified by the technologies that we’re using.

Q. The police use of automatic facial recognition systems has caused many problems in the United States. Do you think that will be corrected?

A. This technology is a source of flagrant cases of discrimination, always against African Americans. An eight-month pregnant woman was arrested and held for a few hours for a crime she did not commit, because she’s black. It’s still happening. That’s why I get a bit frustrated. We know these problems exist; why are we not paying attention to fixing them? We can come up with a solution. A couple of years ago, about five states stopped using facial recognition. But then two years later, they decided to pick it up again. That’s the typical US recurring cycle.

Q. There is no diversity, or multidisciplinary teams, in the companies that develop this technology. Why?

A. Executives see AI as a source of productivity improvement; politicians, a means to grow the economy. But no one thinks seriously about the challenges it poses to society. People are not aware of the problems of discrimination and injustice that AI brings. The debate has focused on whether AI poses an existential threat to humanity, when that is not really the problem we face today. We have more pressing matters to deal with.

Q. You worked on LaMDA, a large experimental language model similar to ChatGPT, developed by Google. How did you try to make it responsible?

A. One of the things that I worked on was a process called responsible AI maturity model, which is making sure that we have identified what different tasks that model is supposed to have. One of the tests that we ran was called the gender benchmark, which was basically just meant to see if the responses are actually going to be accurate, say, identifying a nurse only as a woman, or as a man as well. We also wanted to make sure we had a diverse team, fair representation of people from different sexual orientation, gender, ethnicity. We had to slow down because we had to add all these additional layers. Eventually it was launched three months later, but everybody felt much happier.

Q. Don’t you think that, no matter how many layers and variables you try to take into account, fair representation will never be achieved? The world is terribly complex, and the models are reductionist by necessity.

A. You can’t represent everyone, right? But at least you can decide to represent people from the global south, for instance. They’re always forgotten in these sorts of things. Once you run a test, you have to think about the impact that it has on certain people in society, especially the vulnerable communities, because by law, they should be protected. Pregnant women, children, disabled people, people from different religious backgrounds. However, it goes against the privacy values, because privacy anonymizes the data, so there are tensions between privacy and fairness.

Q. What do you think of ChatGPT? Would it pass your test?

A. It’s getting better, compared to when it just came out a year ago. It was rubbish. It was able to come up with sentences and predict the next word, but it was still a bit unintuitive. The main problem is the misinformation, right, and the hallucinations. I don’t know when it’s going to be solved, because we don’t know why it’s doing it in the first place. Maybe we’ll have to think about how we retrain these things, the techniques that we’re using. But it’s definitely a good step in the right direction. I still feel that we might be headed towards AGI [artificial general intelligence, that which can match or surpass that of humans]. In the future we’re definitely going to keep on making breakthroughs.

Q. Models like ChatGPT have been trained with internet content, which is made by people. But if generative AI continues to grow, future models may be trained with content written by machines, which are not completely reliable. How can we solve that?

A. That’s why we need to slow down. I think that’s where regulation needs to come into play. We need to be a bit more specific in our regulation to say: this is exactly what we don’t want you to do. And if you’re doing this, then make sure there’s a human in the loop process, make sure there’s a human involved in every phase of this lifecycle. It would actually be good to generate data made from AI generated data and build models based on it. Now, we need to be able to fact check it. We need to be able to check the data sources. We need to be able to review these things before we launch it.

Q. Do you trust AI?

A. I don’t trust AI as a technology, but I do trust its potential. I really believe that it can help us solve many of the world’s big problems. If I use ChatGPT, I have to go to Google to fact check it. In two years we might not have that problem. Maybe, maybe not, who knows?

Sign up for our weekly newsletter to get more English-language news coverage from EL PAÍS USA Edition