Državljan D

Državljan D je podcast za aktivne državljane. Za tiste, ki bi radi sodelovali v družbenih procesih in potrebujejo par smernic koristne in produktivne družbene aktivacije.

Play

Open in

20 episodes
Avg 42 min
Slovenian

Episodes 20

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Nov 15, 2025 · 37 min
112 Giacomo Zandonini and shady Europol
In this episode of the Državljan D podcast, we dive into the challenges and hidden dimensions of modern security policies with one of Europe’s leading investigative journalists, Giacomo Zandonini. For years, Zandonini has uncovered how extensive surveillance systems, data infrastructures, and technological ecosystems are being built in the shadows of EU institutions—shaping the future of security and its impact on fundamental rights. His latest investigations highlight Europol’s rapidly expanding powers, its growing ties with the security-tech industry, and rising concerns about transparency and oversight. Transcript of the episode: Expand the transcript 00:00:10 Domen Savič / Citizen D Welcome, everybody. It’s the 14th of November, 2025, and you’re listening to this episode of the Citizen D Podcast, airing on the 15th of November, same year. Today, we’re joined by Giacomo Zandonini, an investigative journalist whose work has exposed the unusually close and often opaque relationships between Europol and major tech companies. Welcome, Giacomo, and thank you for dropping by. To start us off, can you recap your investigation? There’s a link in the notes, but for people who don’t read, can you say a few words about what was the focus of your investigation into Europoling connections with tech companies and what are some of the main findings? 00:00:59 Giacomo Zandonini Yes, thank you very much, and thanks for having me here, Domen. The investigations I worked on with the UK-based group Statewatch stemmed from a recent initiative by Europol called the Research and Industry Days. It’s an annual meeting held at Europol’s headquarters, where industry representatives—mainly from various companies and a few research institutes—are invited to present their latest technological tools to Europol. This event is organized through an invitation process that takes place around this time each year. For example, the call for the 2026 edition has just been opened. Europol publishes a sort of wish list outlining the types of tools and sectors they’re interested in. Since the Research and Industry Days were launched in 2024, the focus has largely been on advanced technologies—AI-powered and machine learning-based systems. These range from tools for monitoring social media and the broader internet, to network mapping and analysis, as well as the use of drones, automated devices, and systems for processing, analyzing, and extracting data from phones or other devices. Many of these tools can be considered highly intrusive in terms of privacy, depending on how they’re used. They certainly have that potential. I chose to focus on this annual event because it offers a clear window into the relationship between Europol and the tech industry. It seemed like a good starting point to understand how these interactions work. From there, I expanded the scope of the investigation to provide background and context on how this relationship is regulated and how it has evolved—particularly over the past decade, during which Europol’s ties with industry have been notably close. That’s the main contribution I aimed to make. 00:05:02 Domen Savič / Citizen D To riff off your comment about how opaque these relationships often are—at the same time, Europol is essentially putting together a wish list that reflects the major selling points the tech industry is currently pushing out to the world. You mentioned AI and other tools… So, who would you say is actually steering this relationship? Who’s in charge? Is it the corporations, offering these tools to public decision-makers or are public decision-makers genuinely choosing which tools to adopt based on real needs and requirements coming from the field? 00:06:00 Giacomo Zandonini That’s a very good question, and I don’t think I have an exhaustive answer—because it’s really a two-sided story. Europol is an expanding agency within the European Union. Its annual budget isn’t as large as, for example, Frontex. For the past few years, it’s been around €240 million annually. The largest portion of that budget goes toward salaries for roughly 1,000 staff members, who are employed under various types of contracts. Beyond personnel, a significant share is spent on equipment and infrastructure—particularly IT systems. This includes the tools, software, and hardware Europol uses to transform itself into the EU’s Criminal Hub and Criminal Intelligence Analysis Center. So, the push to adopt industry products and maintain relationships with tech companies is central to Europol’s initiatives and programs—especially since 2015, when terrorist attacks struck the EU and France. We’re approaching the 10-year anniversary of the Bataclan attack, which marked a turning point for policing in Europe. That event triggered a major push for agencies like Europol to accumulate data and find ways to analyze it—primarily through software programs that, in recent years, are increasingly based on AI and machine learning. In this context, Europol began developing relationships with the tech industry. At the same time, companies became more proactive in pitching their products to Europol. One notable example is Palantir, a company revealed through investigations by two colleagues to have worked closely with Europol. Palantir’s software—particularly Gotham—is widely known and has been used across various security domains, from the war in Ukraine to Palestine, and within the U.S. security apparatus. It’s also present throughout Europe. Europol used Gotham to analyze and process its accumulated data. However, the relationship didn’t go smoothly, and eventually Europol discontinued its use of Palantir. That said, it’s still unclear exactly where things stand today. After 2022, Europol received a new mandate through regulatory reform. This mandate includes a focus on research and innovation, allowing Europol to develop systems internally—especially those based on artificial intelligence. They’ve since opened an Innovation Lab, which organizes events like the Research and Industry Days. Additionally, Europol houses the European Cybercrime Centre, which has been central to developing industry relationships over the past decade. So, while Europol is building its own in-house expertise, the industry remains essential. Europol actively seeks out AI-driven products, tests them, and sometimes licenses or contracts companies to use these tools. 00:11:24 Domen Savič / Citizen D From a privacy activist’s perspective, there’s often a tension between privacy and security. Whenever tools like biometric surveillance, IMSI catchers, or AI-based surveillance are proposed, there’s always a counterargument: “But what about our privacy?” That framing assumes that privacy and security are opposites—that these tools actually deliver on their promises, and that we must choose one over the other. But how do you view the idea that this sense of security might be misleading? Do you think there’s a lack of evidence showing these tools are truly effective in delivering security? Could it be more about the appearance of safety—what some might call “security theater”—rather than actual improvements in public safety? 00:12:50 Giacomo Zandonini Absolutely, that’s a key part of Europol’s relationship with the tech industry—especially when it comes to reusing tools developed by private companies. It’s important to remember that these companies are driven by profit, not by a mission to provide public security—at least not in our capitalist societies. Profit motives also mean aggressive marketing, and in this case, the primary clients aren’t citizens, but law enforcement agencies. These agencies often have access to significant funding, including from the European Union and the European Commission, to acquire and test new technologies and participate in research and innovation projects. There’s a broader movement pushing for law enforcement to adopt the latest tech tools to analyze data and identify potential criminal activity. But the risks are substantial. These systems often have vulnerabilities that can compromise the security of law enforcement agencies themselves. This has been well documented by scientists, researchers, and technologists. They also come with biases—facial recognition is a clear example—and high error rates. More concerning is that they can be used in ways that bypass democratic oversight. This is especially relevant when looking at the companies that participated in the last two editions of Europol’s Research and Industry Days. There are major players and smaller firms—some offering innovative tools, others partnering with larger corporations. One of the things I focused on was examining the track records of these companies and how their products have been used in other contexts. Of course, that doesn’t mean Europol will use them the same way, especially given existing EU regulations. But it’s still telling. Take Cellebrite, for example—an Israeli company publicly traded in the U.S. It’s one of the most powerful providers of technology for extracting data from phones, even without passwords. That’s their most well-known product, but they also offer platforms for analyzing and aggregating data, all aimed at law enforcement. Cellebrite’s tools have been used in authoritarian regimes and conflict zones to suppress basic freedoms. It’s one of the most controversial companies in this space. Even in Europe, there have been troubling cases. In Serbia, their tools were used to target journalists and activists. In Italy, they were used to extract data from activists’ phones—though that case is less well known. These companies are now central to modern policing. The problem is that we have virtually no public information about Europol’s relationships with these companies. In contrast, U.S. government procurement data—covering even secretive agencies like the FBI and ICE—provides at least some transparency. You can find contract amounts, product types, and renewal dates, even if the full content of the contracts isn’t disclosed. Europol, however, operates in a regulatory gray area. Its procurement and financial rules allow it to withhold contract details from the public under the justification of national or European security. Experts in transparency and EU law have told me this is highly problematic. These contracts should be subject to scrutiny. While it’s understandable that a law enforcement agency like Europol needs some level of confidentiality, its relationships with private companies raise serious concerns. These companies may access sensitive data and often have interests that don’t align with those of the European Union—or the security and rights of its citizens. 00:20:49 Domen Savič / Citizen D To follow up with a question—you’ve been focusing on Europol and its lack of transparency for quite some time. I’d like to hear your thoughts on why transparency seems to be such a burden for organizations like this. As a reporter, whenever you investigate intelligence or police agencies, you often run into issues with their lack of transparency—whether it’s about procedures, tools, or the effectiveness of those tools. Their usual justification is that revealing too much could help the “bad guys,” so to speak. But do you think that’s the whole story? Or do you see deeper issues behind this culture of secrecy? 00:21:55 Giacomo Zandonini Yes, I think this issue is deeply rooted in police culture. Of course, within law enforcement, there are many different personalities and perspectives. Some professionals and officials are more inclined to support transparency, and there are national norms across Europe that vary. But broadly speaking, there’s a prevailing idea that police institutions can be exempt from transparency. Formally, Europol isn’t a law enforcement agency, even though we often describe it that way. It’s an EU agency tasked with supporting national law enforcement bodies. Over the past decades, it has worked to legitimize its role by developing strong data analysis capabilities, and now it’s moving toward using advanced machine learning systems. National police forces send data to Europol, which analyzes it and provides feedback—essentially offering data insights as a service. Of course, it’s more complex than that, with joint projects, operations, and investigations also in play. Still, Europol seems to have absorbed the broader mindset that police work should be exempt from standard transparency obligations. This becomes especially problematic when private companies are involved. Europol often refuses to release information or grant access to documents held by EU institutions. The reasons given usually fall into two categories: commercial interests and national security. The commercial argument is understandable—a company shares internal information and doesn’t want competitors to know how its systems work. That’s fair. But the national security argument is used much more broadly. Europol often claims that revealing details would help criminals, who are supposedly watching closely. Sometimes, this logic implies—perhaps unfairly—that journalists investigating these issues are somehow aiding criminal organizations. The idea that transparency about relationships with private actors would benefit criminals is problematic, especially when applied so broadly. My team of journalists recently published a story on Europol’s AI program. We submitted a large number of access-to-documents requests, and most were denied. We now have five formal complaints pending with the European Ombudsman, all related to this lack of transparency. This matters because past investigations and questions from well-informed members of the European Parliament have revealed conflicts of interest—cases of “revolving doors,” where individuals move from Europol to private companies while continuing to work on the same issues. These are public officials with significant responsibilities, and such transitions raise serious concerns. So yes, some level of transparency is essential. I’m not saying Europol should disclose every operational detail or how a specific product is used in every situation. But we should at least know which companies they engage with, under what terms, and through what procurement processes. In fact, national police forces across Europe often provide more information about their contracts with companies than Europol does. That’s not a statistical claim—I haven’t reviewed every procurement system—but in countries I’ve looked at, like Italy, I’ve found clear traces of contracts with companies like Cellebrite, including renewal dates, contract amounts, and the types of software licensed. So yes, there’s definitely room for improvement 00:28:34 Domen Savič / Citizen D We’re slowly wrapping up, and I just have two more questions for you. The first one relates to an investigation—or rather, a collaborative investigation—you were involved in back in 2022. I’m referring to a Guardian article titled “A Data Black Hole: Europol Ordered to Delete Vast Store of Personal Data.” It describes how Europol was holding a massive trove of data, much of it gathered from hacked encrypted phone services and even from asylum seekers who had never been involved in any crime. Looking at that story and comparing it to your more recent work on Europol, would you say the trend is moving in the right direction when it comes to privacy awareness within Europol and similar organizations? Or is this just history repeating itself—more privacy-invasive tactics and the same kind of opaque, non-transparent operations? 00:29:52 Giacomo Zandonini I’d definitely say it’s the second option—it’s history repeating itself. Some of the issues raised by the European Data Protection Supervisor in early 2022, which we also covered in that Guardian article, were addressed to some extent. Europol had accumulated large amounts of data not necessarily linked to criminal activity—which means it could have included data on any of us. They did take steps to resolve part of the issue, at least from what I and other colleagues understand. For example, they began categorizing data more carefully. Europol is not allowed to retain personal data unless the individual is a suspect, a witness, or a potential victim of a crime. Any other data must be deleted. There’s also the matter of data retention—how long Europol can store personal data and the need to regularly review that. The problem is that Europol’s power to process, use, and store data is expanding. But we’re not seeing a corresponding expansion in oversight capacity. Despite what’s outlined in EU regulations, the European Data Protection Supervisor likely doesn’t have the resources or authority to fully monitor Europol’s activities. The internal offices within Europol tasked with oversight also raise questions about whether they have sufficient capacity, independence, and resources to do the job properly. So yes, we’re still facing the same challenges. And this lack of transparency could become even more problematic. Next year, the European Commission is expected to propose a reform aimed at turning Europol into what they call an “effective law enforcement agency.” While we don’t yet know the full details of the reform, one thing is clear: the proposed budget is expected to double—which is a significant signal of the direction things are heading 00:32:46 Domen Savič / Citizen D To wrap things up—and maybe take the conversation into a darker direction—how do you see this whole issue? Europol is collecting massive amounts of data and using AI tools to analyze it and make decisions based on that analysis. How do you view this entire field in relation to the concept of digital sovereignty? It’s a term that’s been thrown around quite a bit in recent years, and I’m curious how you interpret it—especially considering that most of the companies involved aren’t European. We’re talking about U.S. companies, Israeli companies—essentially foreign actors handling EU data. 00:33:47 Giacomo Zandonini Yes, that’s definitely an issue—and it’s becoming more visible. The European Union has launched strategies aimed at building digital sovereignty. One example is a recent proposal from the European Commission to create a dedicated European cloud for law enforcement. But so far, what we’re seeing from Europol suggests that powerful companies from outside the EU still play a significant role in its infrastructure. It may seem symbolic, but the fact that Microsoft employees now have a sort of permanent presence within Europol’s headquarters is quite telling. It raises serious questions about tech sovereignty. This collaboration with Microsoft could be a topic for further investigation, if colleagues decide to explore it more deeply. As I mentioned earlier, companies like Cellebrite—an Israeli firm listed on the U.S. stock exchange—and Palantir, which has grown even more powerful in recent years, are key players in this space. So yes, there’s a real issue here. To be fair, Europol does appear to be trying to move toward greater digital sovereignty. Some member states, like Germany and possibly France, are pushing in that direction. These are countries with strong ties to national and private security industries, and they’re advocating for more European control over tech infrastructure. For example, when it comes to Europol’s IT systems, they are increasingly contracting European companies to build and maintain them. Still, the EU remains heavily dependent on U.S. and Israeli tech markets. These companies are dominant—not just in terms of software, but also infrastructure. It’s definitely something we need to keep a close eye on. 00:36:54 Domen Savič / Citizen D Thank you very much, Giacomo, for joining us and sharing your insights on this important topic and best of luck with your future endeavors. And to you, dear listener—thank you for tuning in, we release one episode each month, so stay connected and catch the next episode of the Citizen D Podcast in December. Thank you again, Giacomo, and all the best. Citizen D advice: Demand transparency from agencies and support oversight efforts. Engage in public debate and back privacy advocacy groups. Push for stronger data protection and effective oversight. Promote digital sovereignty and question reliance on foreign tech. More information: Behind closed doors: Europol’s opaque relations with tech companies – article Europol Sought Unlimited Data Access in Online Child Sexual Abuse Regulation – article Europol and the Fight for Europe’s Big Data Ark – article Europol’s deepening aversion to transparency – article About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!
Oct 15, 2025 · 49 min
111 Nika Mahnič in sistem avtomatiziranega odločanja
V današnji epizodi podcasta Državljan D se z Niko Mahnič pogovarjamo o avtomatiziranem odločanju – fenomenu, ki vedno bolj oblikuje naša življenja, pogosto neopazno in v ozadju. Nika Mahnič raziskuje digitalne tehnologije in etike umetne inteligence, posebno pozornost pa namenja vplivu algoritmov na družbo, zasebnost in pravice posameznika. Skozi svoje delo opozarja na pasti avtomatiziranih sistemov odločanja in raziskuje, kako lahko posamezniki in institucije ohranijo nadzor ter transparentnost v svetu, kjer odločitve pogosto sprejemajo stroji. Državljan D svetuje: Zahtevajmo zakonske varovalke in pripis odgovornosti za avtomatizirane sisteme odločanja Bodimo pozorni na politične razprave o uvajanju takih sistemov Dodatne informacije: Trapped by Automation: Poverty and discrimination in Serbia’s welfare state – članek Suspicion Machines: Unprecedented experiment on welfare surveillance algorithm reveals discrimination – članek Disablism, racism and the spectre of eugenics in digital welfare – članek Automated Hardship: How the Tech-Driven Overhaul of the UK’s Social Security System Worsens Poverty – članek O podcastu: Podcast Državljan D je podcast za produktivno preživljanje časa, v katerem igramo vlogo državljana. Državljan D v pogovorih s strokovnjaki določenega področja informira in aktivira. Da se. Gremo naprej!
Sep 15, 2025 · 40 min
110 Aljoša Ajanović Andelić and the spyware society
Aljosa Ajanovic Andelic is a Policy Advisor at European Digital Rights (EDRi), where he covers issues related to illegal state surveillance like spyware, journalists’ digital rights, and the effects of techno-solutionist and securitisation policies on minoritised communities and people on the move. We sat down with him to discuss the current situation in the field of spyware, the role of nation states and private entities and the role of European union. Transcript of the episode: Expand the transcript 00:00:10 Domen Savič / Citizen D Welcome everybody. It’s the 15th… No, it’s the 4th of September 2025 and you’re listening to this episode of Citizen D podcast on the 15th of September same year. W ‘re back after a prolonged, let’s call it a vacation and with us today is Aljoša Ajanović Andelić, policy advisor at European digital rights EDRi, where he covers issues related to illegal state surveillance like spyware, journalist digital rights and the effects of techno solutionism and securitization policies on minoritized communities and people on the move. Welcome, Aljoša, it’s good to have you here. 00:00:55 Aljoša Ajanović Andelić / EDRi Thank you. Thank you for having me. 00:00:56 Domen Savič / Citizen D Let’s start with the opening salvo. We’re going to be talking about the spyware issue and I was wondering, why is spyware getting so much attention in 2025? Like, if you followed this this topic like a regular person, not like a digital activist, you’d hear probably something about Pegasus a few years back and then everything went sort of silent. But now, 2025, spyware is literally everywhere. Why is that? 00:01:32 Aljoša Ajanović Andelić / EDRi Well, the answer to that question, I would say that that is because of scandals. So that would be the same answer if you had asked me five years ago because nothing has changed. So from time to time, we come across these big scandals of illegal surveillance to different types of activities to political opposition in different parts of Europe, and that is when this topic always comes back to the front pages. In 2025, the biggest scandal has been the Paragon scandal, which was a revelation earlier this year that said that more than 100 people were spied across Europe with this spyware called Graphite that’s developed by the Israeli company Paragon. And this is something that that has made the front pages, because in Italy, some of those victims whose identity hasn’t been fully disclosed, but three or four of them have come, have come to the front and it happens to be journalists and humanitarian aid activists in Italy who have been spied illegally by… we don’t know who. So what happened? After these revelations came and these people spoke as one of the affected victims is what we have seen in all other scandals. So we’ve seen that in the past in Catalonia, but also in Poland and Hungary, which is that first the state denies all type or any type of responsibility on that and then starts considering a bit. Well, yeah, we spied on these ones, but not on the other ones and so forth. This is where we stand at this point in Italy in in which, you know, they have recognized that those humanitarian workers from the NGO Mediterranean have been actually spied by the state because they say that their activities qualify as crimes because they have been allegedly promoting human smuggling, but in the case of the journalists, the Italian government is saying that they are not involved, even though all the evidence points to that direction. What is happening in 2025, again, is that many scandals are unfolding. There’s also another case in Serbia that was revealed in which the Serbian Government has developed its own spyware. So it’s not just private companies, it’s also state developed spyware, and it was used in this case to also go or investigate the phones of journalists and also politicians. What makes this issue so problematic, or why should everyone, even if you are not a digital rights activist, why would you care about this? It because the fact what these programs do, which is to be able to access anything in your phone. This is something that’s very appealing to any common citizen, because we all now rely massively on our phones, they are kind of where all our information is stored and we all can imagine what kind of privacy violation would that be: that the state accesses you. 00:04:48 Domen Savič / Citizen D You’ve mentioned the state several times. Are they still the main users or the main actors in this field or are there other actors? Maybe somebody from the private sector, from the criminal underground that is also a part of this spyware ecosystem? 00:05:11 Aljoša Ajanović Andelić / EDRi Well, mainly with the biggest issues on particularly spyware have come all of them from state authorities, private companies that are the biggest sellers of this. So, we can think about NSO group, but also Paragon and others, they all claim to just be selling to state authorities, so it doesn’t need to be the state itself, sometimes is police force. Sometimes is the judiciary power, but it’s mainly addressed to state authorities and the main scandals that have come, they all come from targeting from States and we do believe that’s the most worrying practice and the one that we as human rights defenders to care the most, because the state is the one that should be protecting your privacy and your digital rights and not attacking them. We’ve also had some cases in in countries like China or Russia in which it is not the state itself that conducts these attacks, but mercenary hackers or groups of hackers who are committed to the state in some way because the state is sometimes paying for their services or is servicing them with the programs, but they are not directly part of the state. So we’ve seen that groups of mercenaries using these type to conduct smear campaigns or to intimidate certain groups. One good example is the regular minority in China, which has been attacked massively by these types of mercenary groups, but yeah, I would say that the biggest problem we’re facing is when states use that because when private, you know, criminals could be using that, that’s something that of course we should worry about, but what we cannot tolerate is that the state is using this without our knowledge and without any transparency. And of course you know, bringing to this union of companies with very worrying practices such as the set and the subgroup with state action, which is, for example, the police or law enforcement investigation, which is something of a public service so to say, and then it’s tied up with these companies that have virtually no oversight, so we don’t know what’s happening. 00:07:36 Domen Savič / Citizen D You’ve mentioned transparency, right, and seeing how nation states are still the biggest or the most important actor in the field. How do you see this conflict of interest on the issue or on the on the level of transparency? So on one side you want your state or your elected leaders to be transparent about the activities, at the same time I’m guessing revealing too much information about the usage of spyware by nation states could harm their defense strategies and police tactics. Is there any middle ground on this issue so that at the same time the state still has this competing advantage against legal usage of spyware in order to prevent crime or terrorism and at the same time to calm the people down so they’re not using it indiscriminately, for, you know, tracking journalism journalists, activists and other parties? 00:08:51 Aljoša Ajanović Andelić / EDRi Well, that’s a very good question. So first of all, I would like to start by saying that when it comes to this type of spyware—so we’re thinking Pegasus, Graphite, and all these intrusive spyware that can gain control of your device, see all its historical logs, but also see where you are, activate your phone, activate your camera and so on—for this type of capabilities, we as a digital rights organization struggle to see how this can be compliant with human rights at all. To break down one human right, such as the right to privacy, you need to make sure that what you’re doing is necessary and proportionate. And this type of capability is completely unable to be proportionate because once they access your phone, they can access everything. So not just the conversations, let’s say, that can be necessary to get to know the activity of someone contacting criminal activity—so to say, it’s not accessing just that. It’s accessing the whole of the phone without any limitations. And what’s more worrying, without any log for that. So we don’t know. It leaves no traces on the phone most of the time. It tracks the proof that it has been inside that phone, and also we don’t have any logs of which information has been extracted, where it went, and so on and so forth. So starting from this premise, with the current state of play, we would say that there is no lawful application of these types of tools. I would also say another thing, which is: we understand the national security angle. Of course, you cannot reveal everything if you want to make sure that you are keeping your population safe. But at the same time, what these scandals have shown is that national security has not been an excuse for all of them. We’ve had scandals, as I said, in Poland and Spain, which have been targeting, for example, political opposition or journalists. And the problem of using this type of programs is that there isn’t a log of what has been done. That means that maybe, for example, five years after the Catalangate scandal in Catalonia unfolded, we have no information about what data was extracted from those phones. Where is that data stored? Is it stored in the Spanish police servers? Does NSO Group, an Israeli company, have a copy of all that data from Spanish citizens? And what can they do with the data? With this lack of transparency at all levels—there’s no transparency on whether a state has acquired this, so procurement opacity, total procurement opacity—but there is also a total lack of transparency on its use. Therefore, we would say that to start, for the victims that have already come forward, they need to have this transparency in order to have their rights upheld and to have remedies for the violations they have suffered. But again, there is a lot more to ask from states. So first of all, we need to know which are all the contracts that countries have signed with private vendors, but also which tools they have developed in-house, let’s say. And also we need to make sure that the victims who have come forward have this right of knowing what has happened with their data. Because in these cases of journalists or political opponents, I would say that no one can disagree if we say that these are unlawful uses of spyware. And even though someone might not hold this absolute stand as we do—which is that spyware should be totally banned—even those people could agree with me that these cases were not used for national security. So national security cannot be an excuse to not reveal these types of practices: what data was taken, from where, where it is, and which company was used to do that. And then there’s the security angle — like, the state keeping an edge in spyware use by not revealing too much. Because, you know, if you tell people how it works, or which version or type you’re using, you might end up warning the criminals too. So, is that something that resonates? Like, is that actually in place? Or do you think — as you already said — since we don’t have, what’s the word… any real stats on how successfully spyware is being used, we can’t really know if this is even working? Yeah, that’s definitely one of the key things — if we don’t have information about what’s happened in the last 10 years in Europe, we can’t really draw any solid conclusions. But we do agree — and you can check out the paper we wrote in 2022 on state hacking — in that paper, we outlined 11 steps that states should follow to lawfully try and hack into the phone of someone suspected of committing a very serious crime. But none of that is actually followed when it comes to spyware. And that’s partly because of the capabilities of the tech, but also because of the entire structure that enables spyware use — starting with this commercial market. It’s a super opaque market that thrives on opacity. These companies need secrecy to operate. What they do is look for vulnerabilities — in programs or operating systems we all use. Think iOS from Apple, or WhatsApp, or Meta apps. They actively try to find flaws in those systems so they can use them to infiltrate phones. And all of this is driven by demand from states, who turn to private vendors to access citizens’ phones. That creates a whole other market — the market of vulnerabilities — and this market moves millions behind the scenes every year. From the last 25 major vulnerabilities Apple found in its own system, 21 were used by spyware companies to infect phones. That shows how this whole thing is a major driver of the vulnerability market — and that puts everyone at risk. Because once these vulnerabilities end up in a company’s hands, we’re all potentially exposed. So, by allowing this private market to thrive, states are not only putting citizens in danger — they’re also putting themselves in danger. These same vulnerabilities can be used against them. They can target state officials, or even be used to extract secrets from European companies — which is a huge issue in this context where everyone’s talking about economic sovereignty. This can backfire in so many ways. Now, of course, states have always had secret services. They’ve always had tools to investigate and stay ahead of criminals — that’s part of what police are supposed to do, and that’s fine. But when it comes to digital rights and privacy, it shouldn’t be a carte blanche to access any phone, any time. That’s neither necessary nor proportionate. What we’re saying is: there are ways to hack phones that don’t violate human rights — if certain requirements are met. And that’s the direction things should go. Instead, what’s happening now is the opposite — the market is shaping how states use these tools, instead of states saying, “Let’s find a human rights-compliant way to do this,” and then turning to the private sector for help building it. But that’s not what happened. The industry created these monsters — tools that let you access anything — and states are just buying them. And we think that’s deeply problematic. Because even in democratic — or so-called democratic — countries, we’ve seen these tools used to avoid scrutiny, or to go after political opponents. Not just to fight crime. So they’re dangerous tools by nature. Which, for us, makes them incompatible with fundamental rights. And what needs to change isn’t just how states use them — the entire market and the whole concept of spyware needs to be rethought. 00:19:03 Domen Savič / Citizen D And you’ve mentioned private companies, right? So, mobile phone or device developers — that’s basically the origin of this whole horror story, right? These zero-days, these issues and bugs in software and hardware that enable the development of spyware tools or spyware software. So how do you see the role of these companies in all of this? You’ve got Apple, Google, Samsung, and other mobile developers who, on one side, are very eager to patch and fix flaws when they’re notified. But do you think they’re doing enough — proactively — to sort of stave off the dangers of, yeah, the whole spyware economy, so to say? 00:20:04 Aljoša Ajanović Andelić / EDRi Well, in this case, I’d say — as you might know — in our work as digital rights activists, we’re most of the time actively fighting against big tech. We have totally different interests, and we believe this whole structure of monopolization and oligopoly in the sector is really dangerous for all citizens. But when it comes to spyware, it’s true — we’ve seen some good practices from these companies. One example is the recent Paragon scandal. It came to light because WhatsApp — so, Meta — actually informed people that they’d been targeted. They found the vulnerability and let those users know. Apple, too, is now publishing a report every three months with all the bugs they’ve found and patched in their system. So yeah, it’s in their own interest to keep their apps as secure as possible. And that just makes sense — if people lose trust in their services or in the strength of their end-to-end encryption, they’ll switch to something else. That’s pure capitalist logic. But in this case, it actually helps us, it helps having these companies actively involved in finding and patching vulnerabilities, and being vocal about it. We’ve also seen, for instance, when the UK tried to push for mandated backdoors in some systems, Apple pushed back — hard. And in the end, the UK government backed off. Because not just experts, but even the companies themselves, were saying: forced backdoors are basically just vulnerabilities — they make devices less secure, and open the door to anyone who can find a way in. So in this case, yeah, I’d say there are some good practices. Of course, more could be done. But it’s in their interest to protect the integrity of their encryption — and that’s why, in some cases, the only information we have comes from these companies. Because detecting spyware infections is really difficult. If you’re a user who’s been infected, you’re not going to know just by looking at your phone. There are very few ways to detect it. There’s no proactive investigation by Member States, or by the judiciary. So we end up relying on organizations like Citizen Lab — who, in good faith, and because they care about this issue, are actively looking for infected users. They’ve even developed tools to help detect infections, but we also rely on the companies themselves to tell us who’s been targeted and whose devices have been compromised. That’s just the reality. And there is some legislation, like the Cyber Resilience Act — adopted by the EU a couple of years ago — that requires products on the market to actively search for and patch security flaws. So there’s at least some kind of framework pushing companies to be proactive. But still, we’d urge them to double down on those efforts. Because right now, when states aren’t willing to be an active ally in this fight, the companies are pretty much all we have. So that’s something we definitely recognize — and we do welcome all efforts from these companies to help detect who’s been compromised. 00:24:03 Domen Savič / Citizen D Because that’s the issue in the end, right? It’s so niche, so specialized, so problematic — and at the same time, it’s commercialized. It’s literally available to almost anybody with enough funds. So it really begs the question: how do we move forward from here? Is everything that’s happening now — the legislative proposals, the patches — is this a genuine attempt, or maybe a valiant attempt, to actually prevent this from happening? Or is it all just a bit of, you know… whatever color you think spyware is — greenwashing, basically? Like, everyone knows this will keep happening, but all you really need to do is signal a little that you’re “against it,” while at the same time not doing nearly enough to actually, you know… stop it. 00:25:10 Aljoša Ajanović Andelić / EDRi That’s a very good question. And it’s funny because, in the case of spyware, there isn’t even an attempt to greenwash it. Like, no one’s even trying to say, “Yeah, we’ll address this.” What we’ve seen at the EU level over the last five years is just total inactivity. The only thing that’s happened is the Parliament set up an inquiry committee — and to be fair, there were some MEPs who were really involved, really pushing to denounce this and calling on the Commission to take immediate action. Because this is a major human rights scandal. That committee concluded its work two years ago. Its final recommendations were adopted by the European Parliament. And two years later? Nothing’s happened. They asked for things like setting up a European version of Citizen Lab, and proactive legislation to control the use of these tools. The Commission? Completely inactive. They promised they’d issue a communication — just a non-binding legal text — to analyze how spyware interacts with GDPR and data protection laws. And they haven’t even done that. At the national level, with this whole “securitization” agenda spreading across Europe, the argument we keep hearing is: “We need this to fight crime, so it’s fine.” And what’s dangerous about that is, first, it ignores the fact that this shadowy market puts all of us at risk. But second, and maybe more importantly, it shows a huge disregard for human rights. Even the known victims — people who’ve already been targeted and have been facing years of judicial inaction — even they haven’t had their rights restored or protected. And forget about future victims — there’s been no move to prevent it from happening again. We haven’t seen a single state that’s been caught up in one of these scandals come out with a strong effort to investigate, to expose what really happened, or to hold anyone accountable. Not one. And we haven’t seen any meaningful commitment from the Commission or from Member States to actually make spyware use compatible with human rights. That just hasn’t happened. And it’s worrying. Because right now, Europe is basically becoming a hub for these companies. There’s no legislation stopping them from setting up shop, selling these tools internally, or exporting them. The legal tools we do have — like the dual-use regulation — they’re failing. They don’t cover these kinds of advanced spyware tools. So what we have is a total lack of legislation. And weirdly, the Commission doesn’t seem bothered by it. Which is strange to me. Because I compare this kind of spyware to someone breaking into your house. These days, all your most personal information — your whole private life — is on your phone, not in your living room. And yet, the Commission might come out and say, “Yes, we’re very concerned,” or “We’re taking this seriously,” but then they do nothing. I keep thinking: what if there were a pandemic of people getting their homes raided by police all across Europe — would they act the same? If this was something physical, something you could see, would the response still be this passive? And honestly, spyware is comparable to a weapon. We’re basically letting companies set up shop in Europe to produce and sell weapons — without any oversight. And if you’ve spoken to any of the victims, or looked at the impact this has had, it’s clear how serious it is. Yes, there are the human rights violations — privacy, freedom of expression, freedom of assembly, and so on. But there’s also real psychological harm. We’re seeing victims across Europe suffer from PTSD, anxiety, all kinds of mental health issues. This level of state intrusion into someone’s private life can not only stop them from engaging in political action — it can destroy their sense of safety altogether. And no one has paid the price for that. So yeah — I’d say that’s a huge problem. 00:30:28 Domen Savič / Citizen D And one final — or pre-final — question for our conversation is about the media framing of this issue, right? Journalists are one of the most important — or at least one of the most common — targets for spyware developers or users. Do you think the way the media frames the spyware issue helps… or maybe doesn’t help… the public understand how serious it really is? Because, the way I see it, you only really hear about it when something happens — in a specific country, or to a specific group, like journalists. But there’s no real broader connection being made — no ongoing conversation that ties this to state representatives, Ministries of Interior, or other institutions, right? So, do you think the media framing of this issue actually helps keep it alive and unaddressed — especially from a human rights point of view? 00:31:55 Aljoša Ajanović Andelić / EDRi I’d say it probably depends on the country. The role of journalists in this issue has definitely been relevant — in some cases, they’ve been the ones who actually uncovered the scandals. So sometimes it’s not even companies like WhatsApp or Meta who inform the public first — it’s journalists doing the digging and bringing these violations to light. So in that sense, I’d say they play a really important role. And then, as victims, journalists have also been very active in denouncing cases — especially when other journalists have been targeted. So they’re involved on both sides. At the same time, though, we see a classic divide — the same one we see in politics — where media coverage varies depending on the political alignment of the outlet. So, depending on where they stand ideologically, their opposition to spyware is either stronger or weaker. In countries I’m more familiar with — like Spain or Poland, for example — journalists have definitely helped keep the issue alive. Politically, spyware doesn’t get much sustained attention — there’s always noise when a scandal breaks, you get the headlines, some debate… but once that first wave passes, it’s often journalists who keep following the story and bringing it back to the surface. So in some cases, yeah, we’ve seen good practices. In other cases, the seriousness of it — the scale of the harm — has been downplayed by certain kinds of coverage. And that might be the most damaging effect: when journalism minimizes the threat or doesn’t make the public fully aware that this is a serious human rights issue. So maybe what’s missing is more personal coverage — stories about the victims, the real consequences in their lives. That kind of reporting could help people understand just how deep the impact is, and hopefully get more attention on the issue. But again, it really depends — on the country, and also the type of media outlet. Because, as usual, the big outlets owned by conservative groups tend to be more state-friendly. Their coverage often leans toward empowering law enforcement and justifying these kinds of surveillance practices. On the other hand, you’ve got independent investigative outlets — the ones funded by readers or private donors — and they’ve been really strong, really vocal against surveillance in general and spyware vendors in particular. So yeah, I’d say there’s a bit of both. 00:34:58 Domen Savič / Citizen D I’m not even going to dive into the topic of personal responsibility or what individuals can do to protect themselves from spyware. But let’s wrap this up a little differently. What do you think is going to happen in this whole constellation — the EU Commission and the EU Parliament, in relation to spyware, but also everything that’s going on globally? You’ve got the Israeli bombing of Palestine, the Russian attacks in Ukraine… you’ve got these actors who are some of the biggest developers of spyware, but at the same time, they’re doing things that clearly don’t align with the European Union’s values. So, what do you think will happen? What do you see in the next 2, 3, maybe 4 years in the spyware landscape, and with the EU’s approach to it? 00:36:10 Aljoša Ajanović Andelić / EDRi Well, this last mandate started off really poorly. In the past year, since it began, we’ve seen the EU and the new Commission really falling into the trap of surrendering to US pressure and big tech interests in some areas. I want to believe that the EU will realize the only way to stay competitive globally is by holding on to what made it unique — its commitment to human rights. For example, there are some regulations in the digital rights space now, like the DMADSA Act, which, when we look at where the world is heading, could be seen as really innovative. These are wins for the human rights or digital rights community because they’re putting some limits on these big tech giants, while the US is going in the opposite direction. So, I think if the EU recognizes that its commitment to human rights is actually its strength, not its weakness (as it’s been framed in this first year of the mandate), then the natural next step would be to start taking these issues seriously. First, they need to release the communication we’ve been waiting for — it’s been blocked for over two years now, and we don’t even know why. This communication should analyze how spyware use interacts with EU law. Then, I’d say the EU could go further. It could tackle the biggest threat we face now: the commercial spyware market, which is thriving and fueling a market for vulnerabilities that puts us all in danger. If I were in the Commission’s cabinet, I’d say the first priority should be stopping the EU from becoming a hub for these companies. It’s self-destructive because it puts all of Europe, even its national security, at risk. My second priority would be ensuring that victims get remedies. We know of victims in at least seven countries in Europe, and no one has found any solution for them. The EU has the mandate and the competence to do something about this at any time, so it’s unacceptable that they haven’t. I won’t say they’ll do it, but they should. If the EU wants to maintain its relevance globally, the only way is to stick to what made it strong — protecting human rights, collective rights, and being innovative in defending those rights, not deregulating them. If they keep handing over tech regulation to the oligarchs, they’ll be playing into the US’s hands. The US will always be the biggest player in that field. Strategically, it’s a bad decision. So, yeah, that’s how I see the EU’s path forward. 00:39:45 Domen Savič / Citizen D Excellent. Thank you, Alyosha, for dropping by. And thanks to you, listener, for tuning in to this episode of Podcast Citizen D. We’ll be back next month. Best of luck with your work in this field and everything you’re doing in the digital rights space. Citizen D advice: Personal responsibility should not be the only approach, we need political solutions Spyware industry is not just a political tool, technology is not neutral More information: Open letter by NGOs to the EU asking for a ban – website Would you click?? – website Serbia: Authorities using spyware and Cellebrite forensic extraction tools to hack journalists and activists – article Paragon scandal by Citizen Lab – article Documentary: The Dissident, on the role of spyware in the murder of Saudi journalist Jamal Kassoghi – documentary Documentary: Surveilled, a documentary on teh effects of spyware on daily life – documentary Short video – Pegasus: the spyware technology that threatens democracy. – Youtube video YT Documentary – How Israeli technology became one of the world’s most feared spyware | Al Jazeera World – documentary About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!
Apr 15, 2025 · 49 min
109 Žana Erznožnik in preverjanje informacij
Foto: Matej Povše Z Žano Erznožnik, novinarko centra za preiskovalno novinarstvo Oštro in urednica projekta Razkrinkavanje.si, kjer že več let sistematično preverjajo dejstva in spremljajo dezinformacije v slovenskem javnem prostoru, smo se pogovarjali o problemu preverjanja dejstev v svetu dezinformacij, vlogi novinarstva v družbi in vplivu družabnega spleta. O politični komponenti dezinformacij, domači in tuji dezinformacijski sceni in kreativnosti, pa tudi o temu, koga Meta ni dovolila preverjati glede lažnivosti izjav in kaj je to pomenilo za dejanski učinek programa. Za konec pa še o tem, kaj je danes na področju dezinformacij največji problem in česa se lahko na tem področju nadejamo v prihodnosti. Državljan D svetuje: Preverjanje dejstev je samo del enačbe boja proti dezinformacijam Družabna omrežja so politična, ne tehnološka kategorija Dezinformacije so danes del političnega diskurza Dodatne informacije: Some truth about Meta fact-checking program – članek Učinek skupnostnih zabeležk na omrežju X – povzetek analize Odzivi na ukinitev Metinega programa preverjanja dejstev – članek Scena dezinformacij v Sloveniji – analiza O podcastu: Podcast Državljan D je podcast za produktivno preživljanje časa, v katerem igramo vlogo državljana. Državljan D v pogovorih s strokovnjaki določenega področja informira in aktivira. Da se. Gremo naprej!
Mar 15, 2025 · 43 min
108 Erik Tuchtfeld and the ANTIFA tech
We are talking to Erik Tuchtfeld, the co-chair of D64 – Center for Digital Progress and the co-author of the “Call for a digital firewall against fascism“, trying to find out the solution to the fascisization of technology and tech services. We touch upon the issue of big intermediaries, the different policies regarding tech regulation in the US and EU, the role of different interested parties, from the tech industry, the regulator and finally the end users. What can we do to prevent the political misuses of big platforms, why tech sector needs to pick a side and why time is running out. Transcript of the episode: Expand the transcript 00:00:02 Domen Savič / Citizen D Welcome everybody, it’s the 13th of March 2025, but you are listening to this episode of Citizen D podcast on the 15th of March same year. We are again, you know, catching the deadline… This time we are joined by Erik Tuchtfeld, co-chair of the D64 Center for Digital Progress, here to talk about the recent Chaos Computer Club post that calls for digital infrastructure resilient against fascism. Hello Eric and thanks for dropping by. 00:00:36 Erik Tuchtfeld/ D64 Thanks a lot for the invitation, it’s a pleasure to be here. 00:00:39 Domen Savič / Citizen D Let’s start with the obvious – what does fascism crave in relations with digital infrastructure? 00:00:47 Erik Tuchtfeld/ D64 So, what they’re craving for is on the one hand opposing any kind of resistance, any kind of opposition, so what they’re looking for is a totalitarian and absolutist form of government, basically, and at the same time, to reach this aim, they need to surveil and monitor society. Both is necessary for them to achieve their aims and with the fascist movement movements we currently looking at a particular in the United States, but also in in in Europe, and in particular also in Germany, where the right-wing party, the AFD, is also rising. We can see that right, populist, authoritarian movements are on the rise and that there is increasing connection between private surveillance, capitalist big tech companies and actual states, as actual state governments. 00:01:45 Domen Savič / Citizen D Hmm, so how did we get here, how did we change enter this field, where we were often told that tech companies or technology in general is apolitical, it doesn’t have anything to do with parties or political movements, it’s there for all it serves the greater good of humanity to this really stark and harsh reality where we can see parties and autocratic government cooperating with big tech companies in order to, as you say, surveil and monitor the general population? 00:02:31 Erik Tuchtfeld/ D64 Yes. First of all, I don’t really think that technology was ever neutral, so that is maybe some kind of myth, a myth which is very popular also in the tech scene, the civil society engaged in questions of digitalization…. But technology is never neutral, technology can be used for certain purposes and the way technology is, normally foster certain goals and impedes others. And that’s fine, I think, because there are certain things we do not want to have in the Internet, and this can also be something where like bipartisan issues and many people agree on, this could be something, for example, spam like people across the political spectrum agree that there should be less spam, or that the distribution of material of child sexual abuse should be impeded. Where the discussion begins then, is how to do this and for what other purposes and purposes, which are more political, this technology can be used. You asked… how did we get here? I think what the EU lacked in the past decades was a will to deal with the stark accumulation of power, the concentration of power in the hands of very few. Talking about big tech is not something we do for five years or so, but since 10 or 20 years more or less, there’s talk about the concentration of power, about surveillance capitalism, but the EU only reacted by reforming them, by trying to tell them that they need to deploy their power in a certain way, but it didn’t take the power as such, the concentration as such. And I think that led to the situation we’re in right now, we have a concentration of power which has very little precedent in the history of basically humankind… that so few people can control the communication spaces of so many, of effectively billions of people. Currently, I think we see that these people who are in power, who have so much power in their hands, fear somehow threatened, and they react to this threat by trying to exercise total control and trying to diminish any kind of protest and opposition they are observing on their on the platform. 00:05:20 Domen Savič / Citizen D Hmm, first follow up question. Why was it so hard for the EU to react or act effectively in this field? 00:05:28 Erik Tuchtfeld/ D64 I mean, I think in the end it was a question of political will and the EU, I think, could have acted and we see that in, in certain areas they have also acted… I mean, they are sanctions against, for example Microsoft, also Facebook/Meta, which go into the billions. But there was never this very strong assumption that the concentration of power as such might be a problem, it was always about the concrete way these powerful oligarchs or these powerful companies exercise their power. Do you try to regulate the way they’re exercising their power first with the data protection regulation and also, with some antitrust competition measures they’ve taken against the platforms, but the general assumption was always “OK, the system as such is not the problem, but the way it currently works… if you reform the system to a certain extent, this will fix our problems.” But there was not the recognition that the system as such is flawed and that additional public sphere, which is oriented toward profit, but is currently the case with private companies and not towards the common good… that this is such a problem and that it needs alternatives, that the concentration of power needs to be attacked, for example, with antitrust measures, that something which maybe could have happened more actively in the past, that also mergers… I mean, we currently see these products which are used by millions of people, are not developed by the platforms or by the companies which own them at the moment, but were simply bought and these kinds of acquisitions, these kinds of mergers could have been controlled more sharply in the past. The European Union could have also invested more in building alternatives, alternatives to the big tech companies, which are all based in the United States, which are all creatures of Silicon Valley and Silicon Valley’s ideology, apart from Tiktok, something which is also coming from China, but still mainly these companies are based in the US, and in particular, not in Europe, they’re not inventions inheriting European values. 00:07:52 Domen Savič / Citizen D This is an often-repeated question or a comparison between the US and the EU in terms of innovating in the digital economy or the Information Society, right? We usually say or there’s the general perception that the US is the cradle of innovation while the EU is the cradle of regulation, right? Would you say that is factual, that is true, or is there some other… are there some other powers in play that the enabled the US to be the cradle of innovation? If it is in fact, and on the on the other hand, that it tasks the EU with regulation? 00:08:48 Erik Tuchtfeld/ D64 I mean it is a very common, it is repeated by hundreds of scholars and politicians, it’s very popular and I guess there is some truth in it, but still, I don’t think that it is that easy. First of all, I’m not really sure if the United States are so good at innovation, there’s a lot of innovation also going on in the European Union and we have a very active open source and civil society scene here in in Europe as well, maybe even more active in certain parts. But what American companies are very good at is scaling, getting big really, really quick, and then also maybe not caring so much about the problems they’re causing for others in this process of scaling and this process of getting bigger and bigger. I mean this is permanently coined by the term “move fast and break things” and maybe exactly that is something which is not possible in the European Union because there is more regulation and there is more supervision over rules than in the United States. But that’s not, that’s not a bug, it’s a feature, right? We don’t want certain practices, we don’t want companies to break things and I mean break things somehow sounds harmless, but when we talk about Facebook, for example, there is this concrete evidence that the way Facebook moderated content led to a genocide. So that’s not simply breaking things, but it’s harming and being responsible for the death of many, many people. And EU regulation, maybe that is the hypothesis I would like to phrase, impedes this kind of innovation, this kind of scaling, but it’s a scaling which does not really benefit humankind and, I am being a bit dramatic, I don’t think that something is missing when we don’t have this kind of scaling, but the question is how do we protect ourselves then against the scale factors of other companies of us American companies? Because with the scaling there comes a lot of interesting side effects which makes it more difficult then to switch to other alternatives. There’s for example the network effect that you want to go there, where many other people are because there’s the interesting content, they are locked in lock in effects so that it’s really difficult to leave certain platforms to follow the content from other platforms because they’re normally not based on general standards, they’re not interoperable… So, all of that, all of those are then certain measures, measures which were consciously taken by the platforms to hinder competition and to make it more difficult for alternatives which are not oriented toward profit to enter the market. 00:11:57 Domen Savič / Citizen D I want to stay at the genesis phase of this current situation that we find ourselves in… So, so would you say that the political discourse or political agreement with these, as you said, mostly U.S. companies that were growing like a virus on a global scale sort of suited the decision makers in that these companies were at one point providing a service that maybe, not exactly, solved a lot of issues, but was very fitting also, for the decision-makers to sort of reach out to the public, to sort of communicate with them, to sort of have this benefit of this, pulse measurements of the public opinion through social media and other digital inventions? And would you say that was the reason the regulators, left them alone for so long and did not address these issues that that are or that were popping up for the better part of the last ten to fifteen years? 00:13:21 Erik Tuchtfeld/ D64 I think that might be a relevant aspect… I mean social platforms and also US, American-owned social platforms are not bad as such, I would argue they’re also connecting people and are the basis of platforms where many social movements, liberal social movements, movements which are critical of power, are forming and have formed. So, there are relevant communication space for many, many people, and they’re also relevant communication space for politicians because they can reach citizens, they interact with them, they can discuss matters with them and they, as you said, they get a grip on the pulse of society. There are a lot of flaws, but it’s always important to me to emphasize this aspect that not everything is bad and what we need to look for is alternatives and not simply shutting the system down as such, because some argue at the moment for some kind of service shutdowns, there are particular countries which have even introduced Internet shutdowns as such, and I think there are concrete dangers for fundamental freedoms and fundamental rights, in particular freedom to expression, freedom to impart information, but also many other fundamental rights. I think you’re right, that it is interesting that is and was interesting for politicians as a tool to communicate and as a tool to interact with society, and at the same time also from a regulatory perspective, it is, to a certain extent, it’s convenient to have a single point of contact, a single point of access, when you want to enforce certain rules. So, in a decentralized Internet where there are many different institutions, many different companies, organizations all providing their own public spaces with their own rules, maybe even when the regulator finds that something is needs to be stopped a certain behavior, or that a certain measure needs to be enacted, enforcement becomes very, very complicated because you have so many different actors you need to address, and it’s just it’s just difficult and now when you don’t want to have, for example, a certain application to be provided to smartphones, there are only two companies you need to go to and when they don’t comply, you can issue fines and sanctions. So, it makes regulation and the enforcement of rules in certain ways substantially easier when there is a concentration of power, and so there is only a very limited amount of counterparts you have to talk with when you want to enforce your laws online. 00:16:19 Domen Savič / Citizen D Because looking back at the role of the Internet or the social networks, these big, mostly US-based companies in different say social movements or demonstration or protests… There’s always this “yes and no” position in terms of how useful these tools are in generating or keeping the social movements alive? So, for example, looking way back, you know, the Arab Spring was one of the first maybe or one of the most important social movements or protests or waves of protests in the Middle East, where everybody in the beginning were saying, you know, social networks, digital platforms were, you know, harbingers of democratization in in those societies, right, Egypt and neighboring countries. Then after a while, these claims were scaled back in terms of, “Oh, no, no, no, this wasn’t as big as it was reported before, the networks played some role, but people would still find a way if social media wasn’t present in the society”, and then you had, let’s say in the 21st century, you had the wave of Russian propaganda and all of the sudden every aspect of social networks, of digital platforms that was before used as an argument for the platforms was now used as a critique of platform – they’re brainwashing people, they’re spreading disinformation and so on. So, in terms of regulating this field, why do you think the regulators were so slow to catch up to the fact that, “OK, if we can spread the right message and are effective at doing that, somebody else could spread “the wrong message” and also be effective at that. So, this, as I said before, this mythical neutral technology versus technology as a tool to effectively spread a particular political message that is fitting or isn’t fitting with the ruling structures? 00:19:24 Erik Tuchtfeld/ D64 I mean it’s a very good and very difficult question because it’s not only about what would be the best regulation, but it’s also about what are the motives of past legislators to not regulate so much. Before the European Union became very active, in particular at the beginning of the 2020s, but also before with the GDPR, which was kind of the starting point of digital regulation of the new wave of digital regulation in Europe, my impression is, which is also still valid when it comes to current regulation like the Digital Service Act, that there is a certain hesitancy because we don’t really know what we want or legislators don’t know what they want, because there’s not a clear image of how good digital public space, where people meet and convene and can exercise their protest and can communicate political issues and stuff like that… There’s not a clear answer how does it look like? There might be certain shared values on how it should not look like, what should not be there, like constant surveillance, which is currently the case. Maybe the EU should have put more emphasis on what should not be there and then try to prohibit this completely. For example, as I said, the constant surveillance on digital platforms, the personal advertisement and stuff like that. They’re doing that now but slowly and obviously, also with a lot of opposition from industry. So, I think what the past has shown is that social media can be a force for the good, that there are good things happening on social media, you’ve talked about the Arab Spring, but we can also talk about Black Lives Matter movement, we can talk about Fridays for future and the climate movements. As such, I think many in particular movements which are somehow established with younger people, youth movements, they have certain roots also in social media and I’m not saying that without social media we had no protest or we had no movements. Obviously, there were social movements before and there will also be social movements when this kind of social media does not exist anymore, but we at least facilitated by the way social media works. Also, apart from these big movements, social media companies or social media platforms are spaces where marginalized people meet, where people who are maybe living in the countryside and who are not aware of any other person who shares their certain characteristics and who feel really lonely, they can meet friends online and they build a community and feel to be part of a community and be empowered through these tools. I don’t know about you, but also many in the civil society, many people who are organized in civil society and who are very critical of big tech are still using their platforms because certain aspects connecting people works and that’s I think something what also regulators have acknowledged in the past and they have seen there is something good and the platforms are doing. At the same time, they’ve also seen the problems. But there was a lack of a vision of how it should look like when it shouldn’t look like as it is and I think this lack of vision is something which is still a problem. I don’t have the impression that the European Union and also the politicians at the top of the European Union, have a clear vision of a digital public space which is committed to the public good. They just look at this at the current commercial public space and say “That’s not the best we need to improve this, there needs to be a bit less hate speech, there needs to be a little less disinformation, but we don’t really know how it should look like, which is really a different model and which is an alternative”. 00:23:52 Domen Savič / Citizen D If I can play a devil’s advocate a little bit to what you just said… so, if you think about the role of social media as a force for good, as platforms organizing, bringing people together and at the same time you wonder from a position of a digital activist… How much of this usage was actually the necessary evil? If you talk about or if you look at the role of the general mainstream mass media, right… in terms of telling us what’s going on in the world, contextualizing the happenings around the world, focusing on different perspectives and let’s say social groups or minorities… you can clearly see that in the last few decades, they were basically asleep at the wheel, because of many problems from advertising to the whole profit model of mass media and similar stuff. And then you had on the other side, you had this upcoming new actor in the field that offered as networks usually do, free, nice, happy, “just be yourself” type of agenda, but at the same time, you know, looking at it from today’s perspective, we were basically sheep brought to the slaughter because, you know the whole system was based in in surveillance capitalism, right? It was based on our personal data… And this was something that was always built with an intention of bringing together huge swaths of personal data and using it for purposes of advertising, of state security and stuff like that. So, thinking about alternatives and thinking about the role of our, let’s say, functioning society… what are your thoughts on that? Where do we go from here? It’s 2025, you have uprisings of far-right movements literally all across the globe, you have these digital platforms playing favors with them. What’s next? 00:26:26 Erik Tuchtfeld/ D64 Yeah, I think… the law as it stands at the moment won’t save us, so there are many currently asking for very strict enforcement of the Digital Services Act against the US platforms which are now so aligned with the right-wing government in the United States, and I don’t think that that will be enough. It’s necessary, but it’s not enough. What we need instead is, I think, a more active approach towards these platforms where the concentration of power as such is a problem. So, I would say, we really need to break up big tech, there shouldn’t be platforms which are that powerful. At the same time, this is not very realistic, because I don’t think the EU wants to escalate the conflict with the United States further at the moment. What might be a possibility is a more active interoperability obligations for the platforms so that the effects so which lock users in at the moment, so they have to stay there because they don’t get the content anywhere else…that these effects are diminished. A reform like that that could be implemented, for example in Digital services Act or the Digital Market Act, might be more realistic than a complete break up and I think what we need is alternatives and we have, I mean some alternatives like fediverse which conceptually I would say are really, really convincing. I mean when it comes to ditch the public spaces committed for the public good, the way the fediverse works and also its most prominent service, Mastodon. I find it really convincing; there’s a lot of possibilities to experiment with certain governance structures to legitimize the rules which are governing these spaces by the users which are affected by the rules, which is, I think, a super nice and kind of a novel idea for internet communities, at least when it comes to social media communities. So, I think there’s a lot of potential, but at the same time we can clearly observe that it doesn’t really fly, so there’s so much going on and still it’s not like all of our neighbors, at least not my neighbors here are using it… if at all they use any alternative, they use bluesky, which is of course… I mean it’s not-for-profit company, if I’m informed correctly and theoretically, it could also be decentralized, but it’s very difficult so it’s better, but it’s not really a super good alternative. So, they preferably using this and why is that the case? I actually think mainly because of algorithmic creation, because, I mean, Mastodon prominently does not use any kind of algorithm apart from chronological sorting and I think that people want to have some creation over the content of social media platforms, they simply do not want to be online all the time to see what has been posted right now. And then I see it in my feed and when I come online at the end of the day, I don’t see any more what was posted. In the beginning, I would say, one of the weaknesses which needs to be improved in the very concrete design of Mastodon and to enable such developments, I mean that’s not a very concrete proposal, maybe that’s also not the fix for everything, but there are also other flaws in the software as it is at the moment, which hinders some people from using it, and I think the software could be more accessible. What we need for that I think is also funding, so there needs to be a European idea of how social media platforms should look like and then there needs to be public funding when we say it should not be profit oriented. When we say a society say, it should not be profit oriented but committed to the public good. And I think with this funding, there comes more development, the tool becomes more attractive and then it attracts more users, more interesting content, which will then hopefully be a vicious circle in the best sense and to make the alternatives more prominent than the current established platforms. What I don’t think the European Union or any other big stakeholder in this field should try, is simply to build up the next Facebook, the next Instagram, the next Twitter/X and try to copy and create some kind of European giant, just a European centralized platform which at a certain point might have the same issues when it comes to the concentration of power as we currently face towards the United States owned companies, but it needs to be an alternative, where decentralisation of power and autonomy in the sense of self-government of the users is somehow embedded at its core. 00:31:42 Domen Savič / Citizen D So again, playing devil’s advocate, you focused on the platforms themselves, on the way they were built, on the way they used algorithmic content moderation to grow, gain influence… but what about the broad situation in a society, right? Do you think part of the reason for their growth was also a sort of social optimism that was present during their period of expansion of growth, there was talk about technology saving the world, there was talk of digital literacy, there was digitalization happening all over the place… and now in 2025, we are facing, I’m not going to say completely, but very different reality in terms of what the tech is actually doing, how is it solving or addressing some issues and do you think that plays a factor that even if a new platform or a new way of communication literally drops from the sky tomorrow, the same thing that happened in the early 2000s will happen again? 00:33:23 Erik Tuchtfeld/ D64 Yeah, I mean that’s a fair point… I think that infrastructure which is organized in itself against… which has some mechanisms which try to make it more difficult to abuse power, which try to make it more difficult to concentrate power, that would be a feature which was not in mind 20 years ago, because you’re completely right the way the big tech platforms grew, it was very much founded on this idea of techno optimism that technology would somehow save the world. This is also something we can currently observe when in conversations about artificial intelligence that this will somehow just like magic save the world and solve all our problems. And I think what we have to acknowledge is that those are social technical systems, they’re embedded in society and society has its flaws and problems and there will also be reflected in the technology at the same time, we are aware of the last 20 to 25 years. So, I hope that when we create something new, when we design something new, we are a bit smarter than we were 20 years ago and that we would try to at least to make it more difficult for the abuses we are currently experiencing to happen. Again, so that gives me some optimism, that we’ve learned something from the past and that when we create something new, we our new designs are informed by what has happened in the past. But at the same time, I don’t think that the new social media and alternative social media for the public good will not in itself solely fix the rise of authoritarian movements and I also think that it’s not only social media which is important or which is responsible for the rise of these right-wing authoritarian movements. We can currently also observe sometimes in the discussion that one has the feeling if only big tech wasn’t there then all problems would be solved solved and Donald Trump would definitely not be president and the AfD in Germany would not have 20% or maybe would not even be in Parliament… I don’t think it’s that simple, I think it’s a very complex interdependence and I think that social media platforms have this certain share in the current rise also of authoritarian movements, but I think it’s also about bigger problems with society and we somehow… we all do this as activists because we are, I mean at least my understanding, we are political activists, fighting for more progressive future and our current field of application is digitalization. But we’re part of broader movements who are fighting for different and a more open society and that is something we… I mean we don’t need to fix just social media, but we need to work for a transformation of society which is more just, more open and also takes into the account the limitation of growth and climate justice and many different aspects. I think that that is important to keep in mind that there are certain flaws in social media we can and should fix, but that this won’t be enough to fix society as a whole. 00:37:07 Domen Savič / Citizen D And since we’re slowly wrapping up, I just want to return to the original reason for this episode, the Chaos Computer Club post… So who did you have in mind when or while you were collaborating, writing this call for a tech resilient against fascism cravings. Was it focused on the decision makers, was it aimed at the tech sector, the end-users, the activists? Who do you wish would answer it? 00:37:54 Erik Tuchtfeld/ D64 So, the letter or the post which was published in the English version by the Case Computer Club in the German version also on our website at D64, it’s a joint effort of more than 20 civil society organizations in Germany and the very current trigger to write such a request letter are the coalition negotiations in Germany. A new government is forming in Germany and we try to influence the way they’re doing tech policy… It’s directed towards decision makers at a political level and it has three main sections. First is against surveillance, one is emphasising that protection, security for all people, in particular marginalized people, is necessary and the third one is on fostering democracy and democratic participation in the digital space. The very current trigger are these negotiations and they’re addressed towards policymakers in Germany. But there is a reason why we published it in English as well, because we think that that the concrete proposals, we make are not only something for German policymakers, but for policymakers and also civil society organizations and even end-users around the world. We think that surveillance as such is a constant problem, and I think it’s something which needs to be fixed on the policy level and which should not be individualized. That being said, there is some kind of resistance, which is also possible as an individual or as a civil society organization, the use of alternative tools which are not based in the US, the use of alternative platforms which are not based in the US but which are maybe self-governed, we’ve already talked about Mastodon. Understanding the themes of this letter; protection and security for all, in particular marginalized people, democracy and democratic self-governance in the digital space and the commitment against surveillance is something which cannot only be fulfilled by policymakers, but something all of us can try to improve and can try to support in our daily lives, also as tech users. 00:40:28 Domen Savič / Citizen D This is this is lovely but I just have one devil’s advocate question for the conclusion. Do you think all of this makes business sense? So do you think this can be a convincing argument for the industry that was built around for the better part of the last two decades on the exact opposite of what you just described? 00:41:01 Erik Tuchtfeld/ D64 I mean, I don’t think it’s really a business case we are describing in this letter but at the same time, the industry also needs to decide on what side of history they want to be, and currently with the alignment of big tech companies in the United States, with a fascist and authoritarian government, I think that those Members of industry which are not yet aligned with the right wing movement, they have to know that there is no neutrality in this situation. So when they want to protect themselves from a government which seems to know no limits anymore, they need to change their products in a way that they are not a danger to users anymore, they simply must not collect any data which are not completely and absolutely necessary for their services, and I think being aware of this responsibility towards society, also as a company and also as industry is something which might in at least some cases, might respond to that, I have some hope. I don’t think that that Meta will change its business model, I don’t think they Elon will read our letter and then will realize he should really step down and somebody else should take over. But I think that also in industry, that industry is not just one block which is completely aligned and which is homogeneous, but that they are diverse voices and with this call, we are trying to address those voices which are maybe currently in quiet opposition towards what is happening in the US at the moment and that they need to become more active and that they have a responsibility towards their users to protect them by making sure, for example, not to connect collect any data which can be abused by state authorities. 00:43:08 Domen Savič / Citizen D This was great, thank you so much Erik, for dropping by, for, for sharing your thoughts and wisdoms, thank you, dear listener, for listening to yet another Citizen D podcast episode, where we publish an episode every month, thank you all and see you soon. Citizen D advice: We need political solutions for techno-related problems The tech sector must choose which side of history it wants to be remembered on Civil rights must be a part of the tech development political conversation More information: ‘The New Fascism Is Here – And Big Tech Is Running It’ -article “Headed for Technofascism”: The Rightwing Roots of Silicon Valley’ – article How the Roots of the “PayPal Mafia” Extend to Apartheid South Africa – article Aufstieg rechter Unternehmer: Galionsfigur des Tech-Faschismus – article Call for a digital firewall against fascism – website About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!
Feb 15, 2025 · 56 min
107 Joe McNamee: Disinformation is not freedom of speech
We sat down with Joe McNamee, senior policy expert at the EU DisinfoLab, who was previously leading the European Digital Rights network. The topic of the discussion? Everything from digital rights to fighting disinformation, the current EU political climate and more. How does transparency come in to play when addressing the digital rights and countering disinformation, why the current EU situation does not bode well for guardrails in both fields and what can we do about it? Transcript of the episode: Expand the transcript 00:00:02 Domen Savič / Državljan D Welcome, it’s the 11th of February 2025, but you’re listening to this episode of Citizen D Podcast on the 15th of February same year. With us today is Joe McNamee, a senior policy expert at the EU Disinfo lab who’s been working on topics related to Internet regulation for over 20 years. From 2009 to 2018, he led European digital rights, so-called Adri, the Association of Digital civil Rights organizations in Europe, working on major topics such as adoption of the general Data Protection Regulation and the Copyright Directive. So welcome Joe, welcome to the show. 00:00:43 Joe McNamee Thank you. 00:00:45 Domen Savič / Državljan D Let’s start at the beginning… there’s tons of things happening in the disinfo area of EU policy and practice as well, but I want to start with your previous or pre-previous employer; European Digital Rights Association. How did this field of digital rights change in the last, let’s say, 10 years, the so-called digital decade. What were some of the challenges that were, let’s say, present at the beginning and what were some of the solutions we saw as the as the as the decade wrapped up? 00:01:30 Joe McNamee Well, I’d say the problems are easier to define than the solutions because the problems are inspired by things that seem to last forever, and solutions change as technology changes. So, I think over the last 10 years, as you mentioned, the general Data Protection Regulation getting across the finish line was already a success, and it being a reasonably solid instrument was also a success. We had the adoption of the Digital Services Act and the Digital Markets Act on a European level which served to strengthen the role of the of individuals in society in the digital society, we had things like the latest revision of the Audiovisual Media Services Directive, which is more important than people think in regulation. What we see in here in the audial space, particularly with video streaming services, both based on profiling like YouTube and paid for like Netflix… So, there’s the one thing I think that is a fundamental change is that I remember in 2014, two 1015, there was a lot of objections to the notion that there were digital rights. Whereas now I think the question is answered that the rights valued in the pre-Internet age have a different character and a different value in the in the digital age, in 2025 and we have to think about what it means to transfer those rights and to protect those rights in the digital environment compared with the more analogue life that we had prior to 2015. 00:03:50 Domen Savič / Državljan D This was actually my follow up… so how do you see or how did you see the change in, let’s say, civic organizations or the general public, the change of perception of this field. So, I remember like back in was it 2011, 2012, one of my first, let’s say campaigns was against the ACTA Treaty and back then it was really hard to sort of broaden this this issue of treaties of surveillance of, you know, monitoring, monitoring the different digital past. It was very hard to translate this into a language that the general public would understand, right… as we progressed it seemed in a way that the public is getting more in attuned to these issues, to these values. There’s a bigger connection between, as you said, human rights and let’s say digital rights. But again, looking at the past 10 years, how did you see this area changing or adapting to the to the new reality? 00:05:04 Joe McNamee I think it’s a question of people and society being better able to map what they consider to be unquestionable values of society like democracy, like privacy, like freedom of expression, mapping that onto the digital society. And I think ACTA… it was perhaps one of the first examples of a certain part of society going “We can’t just leave this to arbitrariness…”, we have to have better control of our freedom of communication online and since then, we’ve had different challenges, different opportunities, like the update of the Data Protection Directive which became the GDPR, where there’s a lot more understanding among policy makers among politicians than there would have been two years before, or even or three years before. I remember when the GDPR was proposed first by the Commission, there was… It was met with sort of a degree of bafflement, and the degree of misunderstanding and deliberate misunderstanding to a certain extent. So, I think it’s society becoming more in tune with old rights in a new digital form, I think that’s really the big change over the past 10 years. 00:06:46 Domen Savič / Državljan D OK. And what were some of the drivers of this attunement? 00:06:54 Joe McNamee I think, it came to something like data protection, I think it became more obvious just how much data was being generated about each individual and what could be extrapolated by computers … if we imagine the foundational work for the GDPR was done in the early 1990s and the number of computers that existed and their computing power was fairly minimal and we were talking about protecting personal data that was largely on sheets of paper. And I think in the course of the… between 2010 and 2020, it became more embedded in society that it’s a whole different level of magnitude; personal data being protected now with huge amounts of data being collected and data that can be compared with each other. I wrote a paper for EDRi, back in the day, where I talked about data that had no parents. So, data that was generated by comparing two other pieces of data. If you’re this, and if you’re that, then chances are you will be… If you’re A and you’re B, chances are you’ll be C and the awareness of that change in what those that hold the data and those that are allowed to process the data can do with it was finally understood in that period. And I think even in 2014, I think it was when the initial conversation started, society wasn’t there yet, hadn’t quite understood and I think now you know that your data can be used in this way and I think that is a fundamental shift that allowed good things to happen, even if implementation of the GDPR is sometimes disappointing. 00:09:33 Domen Savič / Državljan D And how do you see the difference between, let’s say for quite some time the, let’s say the good guys were the big intermediaries, right? Facebook and other US-based companies and the bad actors were these shady companies that nobody really knew about. As the time moved forward, you had this… I felt like a change between the perception of these big intermediaries that weren’t all good and at the same time the issue of foreign states came into play, right? You had the US, let’s say China, Russia in in the last couple of years… so do you think this change from commercial to geopolitical sense of digital rights helped the progression of this field? 00:10:49 Joe McNamee I’m not sure there was ever a possibility to make a complete distinction between state and private because we saw… most simply were in the Data Retention Directive that showed, once data is stored and once data is processed it will be kidnapped for and repurposed for state reasons… So, I think one leads to the other rather than one being different from the other. Of course, China went off in in one, the US, with its big tech base, went off in another direction and is going off in a new direction now, but I wouldn’t see them necessarily as a distinct phenomenon. 00:11:53 Domen Savič / Državljan D I mean, sure, they’re connected. But at the same time, I get the feeling that, when we were discussing, let’s say, the GDPR, the main actors that the data needed to be protected against were the big intermediaries. Right now, you have… after or maybe even during or just before COVID, you had this shift to a more geopolitical sense, where companies were maybe replaced by country names or regime names in a way, right. So, you were talking or we were all talking about, you know, EU digital sovereignty in regards to both China and the US, you had this whole push towards more EU-centric development of private, but also public sector in this regard. And on the other side you, you also had these quarrels between the West and the east, right, so you had the US countering the push of let’s say, the Chinese, then you have Russian companies in in this regard. So, I was trying to lead into the next topic on the agenda, which was: are our digital rights, something that pertains mostly to the consumer or are our digital rights, something that pertains more to, let’s say, a citizen? 00:13:50 Joe McNamee I think… if something is a value, then it’s a value regardless of time to a point. So, the rights that we have are rights that we see as values that underpin our society, even if the challenges to those values, whether it’s privacy or freedom of expression or whatever, the right to life… The things that need to be protected so it doesn’t… it’s easier to explain that in one year by explaining the threat from big tech and another year, it’s easier to explain because from a threat from government perspective, but the actual underlying value that our society holds dear is timeless to a point. For example, there was the profiling case of the Dutch Social Security Office, I can’t remember the what it’s called anymore, that did some illegal profiling of individuals in order to assess whether or not their applications were appropriate and it became easier to explain to Dutch people the danger of state profiling, because there was an example that was fresh in their minds as regards what had to be avoided and what the underlying values of society as reflected in human rights law are required to be protected, whereas if you if you go to a Portuguese person or a Spanish person trying to explain the same thing you need, you can’t explain in the same way because the society’s experience is different. So, it… I don’t think it’s better to focus on the value that the right represents rather than it necessarily being something that needs to be protected, either from private business or from the from the state… it changes and it will continue to change. 00:16:38 Domen Savič / Državljan D But do you think the approach to protecting these values is the same or… are there some similarities between arguing for privacy against Google and big intermediaries or against a democratic or somewhat democratic state? 00:17:04 Joe McNamee I think… nothing is so good that it can’t be done badly and try as we try as we might, we always find ways of doing good things in bad ways. So, if you say a state democratic state by definition doesn’t need the accountability of data rules, then that is that democracy will be eroded by the lack of accountability. So, it’s a very dangerous and counterproductive thing to say, we can trust these people, but not those people, because that’s, that’s where the erosion of morals and values happens. No government structure, no private structure is good enough without accountability to withstand the temptation to take shortcuts. 00:18:18 Domen Savič / Državljan D Looking at the EU level, but also how does this regulation and these protective frameworks transfer between the Brussels level, the EU level to the to the Member States?Do you see this as an effective system of translation in implementation or is this something that is actually even more important than the regulation itself but is then sort of diluted or spread so far apart that it’s very hard to follow it to the actual enactment of all these EU regulation in practice? 00:19:12 Joe McNamee And that’s a question that I would be very rich if I could answer it effectively. The GDPR relies on national data protection authorities and they vary quite a lot in in quality… and this is traditionally the way the EU did business and traditionally works quite well, if the issue is within the borders of a country. So, if you take telecom regulation, for example… some countries implemented well, some countries implemented badly. The countries that implemented well, have fast and innovative companies with low prices, the countries that that don’t have higher prices, less competition, less quality telephone and Internet connections. That’s not good, but ultimately, it’s a problem that is or it’s either a problem or solution that is chosen by that country for that country and so at a given moment, before the GDPR was thought of it made a degree of sense. So, when the GDPR came along, they followed the same approaches, the 1995 data protection directive and they said, “OK, we’ll leave it to the Member States to implement”, for companies that are based in those countries. But what happened was that that made the traditionally weak Irish data protection authority responsible for Google, Facebook and every other big tech that wanted to establish a base in in Europe and an underfunded unsupported authority became responsible for vastly more than it could ever have been able to manage and that has led to significant problems for the protection of personal data for individuals in the EU and beyond. And you saw that in the in the discussions around the Digital Services Act, for example where one of the things that was repeated often over and over again in the in the political discussions was we can’t have what happened to the GDPR happen to the Digital Services Act and there was a push from the center and the right of the European Parliament to say we can’t have a new agency, ideologically, we just we can’t defend new agency. So, the compromise that was reached was that the European Commission would be the regulator for big tech, taking that away from the risk that it might fall to Ireland again allegedly and the national authorities would be responsible for everything else on the national level. So, there’s been an evolution of the thinking of the of the EU from a time of pre-networked age where everything could be left to national choices that would have national consequences to a network page where we have networked data, which and networked freedom of expression and so on, that requires a higher authority… which for good or evil, is the Commission in this case. 00:23:34 Domen Savič / Državljan D And you know, looking at or comparing the field that you’re working in now, so the disinformation or the tackling of the spread of this information and the digital rights, so your previous job… What are some of the similarities and what are some of the differences? Because in a way I thought about this really weird connection where one could argue that on one side the digital rights were more about freedom of expression and the uninhibited transfer of information across the Internet, while the disinformation field is slightly different in a way that not everything that moves freely is actually good. 00:24:30 Joe McNamee Well, that goes back to my frequently repeated mantra that everything can be done badly if you try hard enough and defending digital rights and the basis that nothing has any limits is as wrong as saying that everything needs to be restricted by default. The truth lies somewhere in the middle. If you leave it to big tech to control what you see and hear on the basis of reverse censorship of pushing the information that they see as profitable up and pushing everybody else’s freedom of expression down… that’s not defending freedom of expression either. So, a balance needs to be to be achieved where voices are heard and without undue interference by the companies. And in the same way, there are fundamentally anti-democratic forces that are organized in in spreading disinformation which removes the freedom of expression and democratic rights of individuals and groups that need to be addressed without necessarily preventing anybody from any human being from speaking. I thought that there was when I before I started this job, that there might be a friction between my strong belief and freedom of expression that I defended in EDRi and my strong belief in democratic rights, which I defend in my current job, but actually there isn’t because you have to both can be defended at the same time and it’s the. If you look at the pre-censorship of algorithms that are promoting toxic content or foreign information networks, those are both contrary to democratic rights and country to the freedom of expression, rights of individuals and groups. So, I I’m happy to say with complete certainty that the two go hand in hand, and they’re not contradictory to each other. 00:27:49 Domen Savič / Državljan D Because I was wondering about that… let’s say these autocratic forces are usually using the freedom of speech protections to sort of further the spread of disinformation, while you have the counter-argument of companies that are, you know in charge of algorithmic content management are sort of, you know, left alone or they’re usually, you know, defended by “Oh, no, no, these are private, you don’t need or you don’t have meddle in them”, right? So how do we address the issue of at least on paper or in practice or on private companies that are now becoming more and more essential and more and more inescapable, if I can use this term in a modern communication sphere. 00:28:59 Joe McNamee I think we can solve multiple problems with one effort and the effort is to be found in the shortest article in the Digital Services Act, and I think possibly the shortest article in the history of European legislation, which is Article 38, which says that there must be non-profiling based options available in very large online platforms that are providing services. If you give people the power to build their own feed of information that they that they want to see and people that they want to hear from and get recommendations from those people about other people, they want to hear from rather they want to hear from, you have an increase in media literacy because these because people are able to understand what a menu is and what choosing your based on your tastes is. And that means that different people would have different feeds and there wouldn’t be the single point of failure at which we have at the moment, which is the monolithic algorithm feed which is designed not for the interests of the user, but for the commercial interests of the platform. And I think if we could, if we could break that monopoly of information feed in the platforms, we would have platforms that would be serving society in a more transparent way, in a more safe way, because it’s comparatively easy to work out the basics of how an algorithm works and exploit it, it’s quite another thing working out how to overcome an algorithm that is based on the tastes of many million or a billion subscribers? That’s a personal opinion rather than one of either former employers or my current employer. 00:31:55 Domen Savič / Državljan D But is that, you know, feasible in… Because then you’re usually left with the question of these platforms, they are running these algorithmic content decision-making solutions on the other hand, you have, the end users and state entities. But how do you make the whole thing work? So, we’ve met back in 2006? and the one of the first issues even back then was these self-regulatory models do not work when push comes to shove, so we need stronger role of the state impacting this field, right? And as we moved on, we saw even in the digital decade that the States aren’t really pulling their weight, but you know, compared to always reverting back to self-regulatory models and codes of conduct… there was a lot of the latter and not a lot of the former, right. So is there… I know it’s $1,000,000 question but is there an effective way to change the practice in this in this regard… to make it more functional and effective, and actually doing what it says it does on paper. 00:33:36 Joe McNamee Well, I would first of all ask whether it’s possible to make it less functional than it is at the moment because the monolithic algorithms are causing harm and any improvement would likely be a change for the better. I think one important thing to understand to accept is that self-regulation doesn’t always work, but self-regulation does sometimes work, so the Council of Europe adopted guidelines on best practice for content moderation, for example, and that sets out some very basic principles for what rules need to be followed in order to have a self-regulatory approach that achieves public policy objectives for the good of society. And we need to move beyond embarrassing levels of naivety of governments when they when they approach this topic. If it’s to the economic disadvantage of a company, to self-regulate in the way that governments want, it will not do it well, it will not do it at all possibly and having a press release and having a photo opportunity saying something different is not going to work. So, in environments where self-regulation is not going to work, then regulation has to work and the DSA Digital Services Act from the EU tries to strike a balance between flexible, rapid measures that can be taken in a self-regulatory environment together with several articles on transparency in order to ensure the appropriate levels of evidence and data are produced to show when regulatory intervention is needed and necessary and proportionate and rules that apply in all cases. I think that the architecture, if not necessarily the outcome, is an admirable one and one that potentially could have been done better, I think. I think it’s a model that isn’t a failure by default. 00:36:37 Domen Savič / Državljan D And how do you see the issue of, let’s say talking or staying on the topic of EU regulation, right? How do you see the issue of these changing decision-making bodies on the EU level? We’ve just got a new European Commission; we got a new European Parliament. They’re now deciding what they will be doing for the next 5 years… How do you see the next five years in the area of digital rights? 00:37:23 Joe McNamee That’s a very big question… the only thing that we see clearly at the moment is the noise around enforcement of the of the Digital Services Act, the noise around the creation of the Subcommittee on the Democracy Shield, the majorities in the Parliament are quite difficult to build because of the election results. Plus, if anything is possible on a scale similar to the GDPR of the DSA seems quite limited, the Commission is under tremendous political pressure to not do anything that would be perceived as undermining financial security or the economic well-being of the Union. So, every direction that you look in, there’s a constraint on the Commission as a whole, the strange division of powers that the Commission President has imposed on the Commissioners is a restraint on all of them to act freely in the best interests of the people of the European Union, in my opinion. And even if there was something strong on any subject that was going to come out of the European Commission, there are forces in the European Parliament now that are partly there because of external forces that don’t want Europe to have effective regulation on anything and therefore building majority to pass legislation that the European Commission will have difficulty proposing is limited as well. There are significant barriers to there being any major, meaningful policies adopted in the next five years, in my personal opinion. 00:40:01 Domen Savič / Državljan D So, will there be more bad policies or as you said before, it’s going to be a stalemate where nothing will move at all? 00:40:15 Joe McNamee I think there will be an effort to deregulate, to move backwards in the name of bureaucratic simplification. And how far that goes, I think that sounds nice, it sounds sellable, “we’re simplifying bureaucracy where we’re getting rid of red tape,” but the red tape was designed for the achievement of a public policy objective that was shared by a majority in the Parliament and by everybody in the Commission. So, it’s a seductive term with… the least bad consequence is the removal of measures that were designed for the good of the people in Europe and businesses in Europe, in and with the majority in the three institutions. So, I see more political potential to move backwards, being seduced by simplification and getting rid of bad red tape than I see moving forward with good legislation, good policy development and legislation. 00:41:56 Domen Savič / Državljan D And if we in our last part of the show, if we move on to the disinformation and compare all of the work that’s been done in the last digital decade on the issues related to digital rights and now try to talk about the current situation in the field of disinformation… Do you think the next five years of the new European Commission will provide some solutions, some regulatory frameworks that will address the issue of disinformation or is this, with the current powers at play sort of a lost cause or an issue that you know nobody’s going to touch? 00:42:55 Joe McNamee Ideologically, I don’t like the notion of condemning somebody for a failure that they haven’t failed at yet. I think in principle, the Digital Services Act, Digital Markets Act and GDPR give us all of the tools that we need to within the European context, within the EU context to address to, to make major take major strides in the fight against organized efforts to undermine our ability to choose by undermining our ability to know what our choices are. It is still a little bit too early to tell if the implementation phase will fail, I hope not. But I think we have a duty to not condemn those who are trying their best to impose, to implement legislation that was adopted for the good of our society, because even because the best way of ensuring legislation will fail is by talking it to death before it has it has had a chance to succeed. 00:44:26 Domen Savič / Državljan D And staying in the disinfo area, what would be some of the key issues or the most pertaining problems that the EU is facing in the current situation? 00:44:48 Joe McNamee This is again more of a personal opinion, but I think disinformation, the economics of it, of how disinformation is funded and shared is central, if we start with the algorithmic choices made on social media that promote certain content and demote other content, is a problem that is inherent to any system that is reliant on an algorithm which is not in the interest of the user and not is not in the interest of quality, but in the interests of driving traffic for the purpose of generating data. And that’s why I come back, always to the recommender systems… So, I think that’s the Achilles heel, on which a lot is based. Then you have a quasi-duopoly in the online advertising space on which online media and newspapers in their digital or hybrid format rely on, because that siphons off of revenue from the newspapers to the online advertising industry, that needs to be addressed. The fact that you generate data for your newspaper by generating clicks, which means that your headlines and your content have to be based on or leaning towards clickbait and lower quality rather than the type of high quality that a subscriber would rely on, is a problem in that it drives quality down and it’s a problem in that it drives revenue away. And on top of that, you have the untransparent nature of the online advertising market and there was a report only this week about child abuse material being rewarded by advertising from major brands and from government departments in some cases. So, these are all interconnected problems and all generated by the attention economy and I think this is the place to look if we want to solve many problems by addressing that one source of problems. 00:48:15 Domen Savič / Državljan D And who do you see as the leading actors in this field? We have already talked about the civic sector, the decision makers, the companies in the digital rights field… is the situation similar in countering disinformation, or is there somebody who’s leading the pack a bit more compared to the field of digital rights? 00:48:46 Joe McNamee So, from the digital rights side, there are so different NGOs in the space, have different expertise and different priorities… however, there is a lot of energy in the data protection and freedom of expression fields and there’s a lot of energy in the disinformation field, and I think we all have a shared interest in getting to the root of this, this whole range of problems by requiring transparency, by requiring alternative recommender systems, by requiring an end to the fraud in in the advertising sector, by helping wean newspapers off low quality and back, as some are beginning to do back towards quality driven, subscriber based models. So, I think there’s a big overlap, I wouldn’t like to pick out anybody as being the champion in in this area, there are a lot of champions, thankfully. 00:50:04 Domen Savič / Državljan D No, but so my question wasn’t yeah, to sort of put you on the spot… I was just trying to hear your thoughts on… are there differences between the ecosystems, right? Digital rights, at least in the beginning, but also in in the last few years, had a very strong support from the digital industry sector, that was present in this field to promote their own goals, but to sort of also expand and to sort of normalize the discussion right in the area of disinformation, you kind of don’t have anybody who’s actively or who’s openly for disinformation. You have various, you know, interest groups that wouldn’t mind if disinformation was flowing freely, right? So, in addressing this issue, is the majority of the burden again on the NGOs, the fact checkers, on journalistic organizations? Or is there somebody else from the field of academia, from the field of state institutions that are helping address this issue? 00:51:31 Joe McNamee I think I’m not going to be able to provide a simple answer to that because I think anybody that’s working on rigorous implementation of data protection law is working towards an environment where disinformation is less hard to drive through the attention economy. If they’re in their NGO’s working on implementation of the DSA and for example measures against systemic risks and measures to ensure adequate transparency and transparency of algorithms, they’re not working directly on disinformation, but their success will be a success for disinformation. There are a lot of people working on analyzing strategies behind disinformation campaigns and these are not… nobody has got the free speech rights to build a billion bots to drown out everybody’s voice that is not a free speech, right? And there are a lot of people working on analysing the networks and the structures and the technologies behind disinformation that are also working more explicitly in the disinformation field. I have a particular personal dislike for the word ecosystem as a metaphor, but iIt really is harped in this case, that we’re all in the different battles we’re fighting in the different areas of digital rights, we are all playing our part to make the environment better for everybody. 00:53:43 Domen Savič / Državljan D OK. And one final question, I know it’s just February, but the annual EU Disinfo lab conference is coming to Ljubljana this year in October, you’re still, filtering through the program proposals, but let me ask you this… why did you choose Ljubljana for the next iteration? Was it because we at some point had a place in the disinformation universe? 00:54:23 Joe McNamee Well, sadly, every country has got their place in in the disinformation universe at the moment. It wasn’t me that made the choice, so I can’t be precise, but I think from my experience that area of Europe has got an unusually sharp and vibrant civil society as well as experience in a sort of political crossroads. Ljubljana has got a lot going from a location, it’s a beautiful, small city, it’s got ancient neighboring countries have a strong civil society and everything we heard from local government agencies was also very positive and we’ve had very productive discussions on topics related to us with government officials and those implementing the DSA. So, I can’t think of any argument that I heard against Ljubljana. 00:55:55 Domen Savič / Državljan D OK, thank you very much, we’ll wrap it up at this point. Thank you, Joe, for dropping by, for sharing these thoughts, best of luck going forward and if we don’t see each other before, we’ll definitely see each other in in October. 00:56:14 Joe McNamee Much looking forward to, it will only be my second trip to Ljubljana. Citizen D advice: Demand political action in the field of funding disinformation Push back against reducing legal protections of digital rights Join us in October 2025 at the Disinfo annual conference in Ljubljana! More information: Big tech funding disinformation – article Countering disinformation – booklet Digital Rights Worsened in Central, Southeast Europe in 2024 – report About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!
Jan 15, 2025 · 42 min
106 Maximilian Gahntz: AI is not (just) tech!
We sat down with Maximilian Gahntz, Mozilla Foundation’s AI Policy Lead, working on questions around the regulation and governance of AI around the world. Previously, he has also led work around data governance and platform accountability. Before he joined Mozilla, he was a fellow of the Mercator Fellowship on International Affairs, working on the EU’s AI Act at the European Commission. We talked about the currents in the AI policy development, the issues with media representations and political interpretation of the AI issues and more. Transcript of the episode: Expand the transcript 00:00:11 Domen Savič / Citizen D It’s the 9th of January 2025, but you’re listening to this episode of Citizen D Podcast on the 15th of January same year. With us today is Maximilian Gantz from the Mozilla Foundation, he’s the AI policy lead, working on questions around the regulation and governance of AI around the world. Previously he has also led work around data governance and platform accountability, and before that he was a Fellow of the Mercator Fellowship on International Affairs, working on the E US AI Act at the European Commission. Happy New year, welcome and so good of you to drop by. 00:00:48 Maximilian Gahntz / Mozilla Foundation Happy new year, thanks so much for having me. 00:00:51 Domen Savič / Citizen D Well, to jump right into the gist of it, we’re obviously going to talk about artificial intelligence, the AI act, the regulatory frameworks and maybe for an opening question is: How will the EU AI Act help or prevent the big tech capture of the AI field that we are facing today? 00:01:19 Maximilian Gahntz / Mozilla Foundation It’s a good question and I think it’s important to also take a step back and think about what the AI act actually is and what it isn’t, because in in the end it’s a Product Safety law, not competition law. So, like the primary intention of the legislators here was never to make sure that there is no concentration of power or to, you know, curb the power of big tech. The intention was to essentially make AI products in the usaver and prevent harm first and foremost, so I think that’s just like important context here because obviously there are some implications. There are some tiered rules that might apply to bigger companies, but not to smaller companies, there are some exceptions, for example, when it comes to open source. But ultimately, it’s a Product Safety law, so I think if we want to talk about concentration of power and market power here, we need to look at this in a broader context, because there are other levers we can pull and that regulators in the EU, in the UK, in the US are trying to pull as well, and in general, when it comes to the success or not success of the AI Act, I think it’s also important to say that it’s too early to know really because there are so many vague provisions in the AI act is so much implementation work to be done, standardization, secondary legislation, there is a big code of practice process currently underway for developers of large language models. So, in the end, what the rules that the AI act put forward will look like, we’ll only know in a few months, a few years, and then how these rules are enforced is actually the long game here, so it’s too soon to judge how well the act is working out at this… we’ll actually have to wait a few years and then also just be… As activist and civil society organizations very vigilant about how these rules are implemented, how oversight authorities are enforcing the law and then what companies are doing in the market. 00:03:39 Domen Savič / Citizen D So currently like the broad perspective of, let’s say, the AI landscape consists, if you look at it from, from, from a perspective or through the eyes of let’s say a regular digital user, it looks like you have a couple of big companies that are battling over for market dominance and then you have some localized or individual tools that are being used by smaller actors. Would you say that assessment of the AI landscape is correct or are there other players who we as activists should pay attention to when we’re looking at this situation? 00:04:31 Maximilian Gahntz / Mozilla Foundation Yeah, I think naturally a lot of the attention right now is on the big AI models and their providers. If you look at what the people in the press are writing about and people are talking about is ChatGPT and Gemini, and what those tools can and cannot do and how they’re implemented into different services, but I think it’s important to keep in mind that, sorry, this is the tip of the iceberg of the AI value chain, which is pretty long and complex, but if we’re talking about the distribution of power and concentration here, it’s important to go down or go up the value chain and think about whose strike dominance in these different markets that are really important for AI. Because in the end, obviously like what, end users, consumers, people are going to be using or going to be affected by, it’s the applications and how these tools are put into practice and deployed, whether it’s in like a consumer setting or even by governments, so naturally like a lot of the attention is there. But if you think about what you need to build these applications, there’s a lot that goes in there, so you have the big model provider is Google open AI anthropic and they train large language models. If we’re talking about generative AI here, using a ton of data and a ton of computing power and raw processing power and to do that, they need a lot of data. So, who has the benefit here? The companies that already do collect a lot of data from other services, for example or that have the capacity to enter into licensing deals with publishers or other rights holders or who have the capacity to build a web crawler who can crawl large parts of the web and download their data and feed it back into the model. The other part is computing power and there, if you look at who the big cloud providers are, who provide that computing power, it’s roughly the same companies or many of the same companies that you also see strike being very active in the AI industry. It’s AWS from Amazon, it’s Google… So they actually have a lot of market power in the cloud market as well. There is the chip providers where I think when it comes to a specialized AI chips, NVIDIA controls large parts of the market, I think it’s like somewhere in the 80 to 90% market share range when it comes to AI chips. So, the further up you go, you still see market concentration and you can’t really look at any one of those levels in… you do need to look at each one of those in combination, because only that way you will actually see who’s like if. If you want to, you know, put it in a poignant way, like who holds power over AI? 00:07:48 Domen Savič / Citizen D You previously worked on data governance and platform regulation… How is let’s say the area of AI and maybe more specifically generative AI, how does this differ from the situation that we’ve been having with digital platforms, with social media, with trying to regulate them for the past few years? Listening to you naming all of these actors, it would seem to me that it’s just a little bit of history repeating, right? 00:08:28 Maximilian Gahntz / Mozilla Foundation I think in parts, yes and parts no. Obviously the problem space is still somewhat different, because, for example, if you’re talking about safety and bias in AI, it’s a bit different with many AI products compared to online platforms, social media platforms who have their own problems of bias that’s like deeply enmeshed in content moderation and recommendation algorithms. And obviously this is a very salient topic right now, again because two days ago, just announced that they’re going to change their content moderation practices, so I think some of it is similar, but obviously if we’re talking about content moderation, there’s been in the past 10 years a lot of debate around the limits and you know, liberties of free expression and the value of preserving free expression that’s been politicized in different ways. It’s probably something that will come up in the AI debate as well at some point, and we’ve already had some discussion around like, do chat bots display political biases, for example, but that’s not the core of the issue. So I think there’s definitely lessons to be learned and we should look at the different experiences we’ve made in different digital policy and other policy fields in the past ten, 20-30 years to learn more like new debates on AI, but it’s not going to be, you know the same thing all over again, because there’s just different equities, different changing political circumstances, obviously. What the European Commission and the new US government and the new UK Government are going to be talking about in the coming years is going to be somewhat different from our political focus areas have been determined in the years. So, there’s a… I’d say there’s a shift in the grandeur political context as well here. 00:10:38 Domen Savič / Citizen D Yeah. And is that going to, let’s say, positively or… I know I’m a bit naive… positively or negatively influence the way we’re addressing all of these… you’ve already mentioned some of them, you know, dangers or pitfalls of regulating this field? 00:11:00 Maximilian Gahntz / Mozilla Foundation I would say yes. Umm, if you listen to policymakers in Brussels right now, for example, there is a lot of emphasis on European competitiveness. If you’re talking to people in the UK, there is a lot of focus on enhancing productivity, if you’re speaking to people in the US, there is a lot of focus on national security and AI, and especially in relation to China. So, I do think that changed significantly… and if you’re an advocate, you kind of need to adapt to that, because fundamental rights and consumer protection to the people who are emphasizing that often aren’t the primary concerns. The challenge for us here is to figure out, how we can still insert those concerns and make sure that like despite a broader competitiveness agenda, which is, you know, OK to have, that despite that fundamental rights aren’t kicked to the curb, but rather it’s like integrated as a fundamental part of that. And then the same is true in other countries and other contexts as well. But I do think it will like drastically change how we’re having these policy debates in the coming years because I think… in the past European Commission mandate, for example, curbing platform power and discretionary… that a lot of platform companies, for example, were able to make that was very important to decision makers. I think it still is as we’re currently seeing playing out debates around, you know, Elon Musk and Mark Zuckerberg and how they’re inserting their voice in the EU but overall, the debates are going to be different, and there’s going to be a lot more talk about competitiveness, national security productivity, etcetera. 00:13:14 Domen Savič / Citizen D OK. And how would you, let’s say rate the impact of AI on these topics; on national security, on the competitiveness, on the labor force. Because on one side you have these public conversations where the AI group is saying, “OK, AI is here, it will solve all of our problems, increase productivity…”, on the other hand, you have skeptics who are saying, “Oh no, the current state of AI market isn’t sending very strong signals that these tools can actually do the jobs, they’re being highlighted for” and then you have the pessimists try who are saying, “OK, they’re going to take our jobs and we’ll be left with nothing”. Based on these three camps, where would you see yourself in? 00:14:24 Maximilian Gahntz / Mozilla Foundation It’s good question, probably it oscillates a little bit as well. Historically, I’ve probably been like a bit closer to the skeptics’ corner, but I think it’s also important to disentangle like a couple of different things here, right? There is a lot of hype around AI, is it like a bubble? Maybe yes, but I think we’re also seeing that at some things it might be… if you’re talking to people in the creative industries, you’ll hear that you know, like AI isn’t going to replace everything, but it might replace somethings. If you’re talking to someone who’s writing… I just listened to a podcast with an essay of a composer who writes jingles for advertising sounds like, you know, generic background music. That’s something that AI can like do reasonably well by now and I think there are more and more areas like that where we like starting to figure out what it might be good at. There are many areas where it’s not good at and it’s like it’s made out as this magic bullet and eventually, we’ll find that maybe it’s not that magic bullet. So, I think we’ll also over time, just have to figure out what the nuances here are and also stop using AI as a blank canvas that we can like project anything we want onto. If you look at like party manifestos in Germany, we’re voting in a couple of weeks and parties just released like draft manifestos, everyone’s mentioning AI in very vague terms, it’s supposed to solve a lot of challenges. Will it solve those challenges? Probably not, at least not the way that people think it will. So, I think overall, as the debate is maturing and the hype is maybe dying down, we’ll be able to have a slightly more grounded conversation about where AI is going to really change things. And obviously also I mean with that… the benefits and the risks of AI, because I think, it’s not necessarily a net positive in every respect of AI is really good at a job if we’re talking about applications, used for surveillance or used at the European borders in the migration control context. There are things were like AI might actually be quite effective, and we still might not necessarily want it deployed at scale in those contexts, which I think is also important. 00:17:14 Domen Savič / Citizen D How do we direct the public debate or you’ve mentioned political, political campaigning before the elections, right… So how do we direct the public debate around pros and cons of implementing AI in in in particular fields as opposed to just going all out and saying “OK, just put an AI on it’ll work”? Who should contribute to debate? Or how do you see different actors meshing together on this topic, like the public framing of the power or lacks of artificial intelligence solutions? 00:18:04 Maximilian Gahntz / Mozilla Foundation I mean, big caveat here is that I’m not a campaigner. I’m a policy wonk, and I’m constantly struggling with, like the messaging aspect of doing this work. How do you actually change like a dominant frame and a policy debate? And I would say often it’s not up to… It’s just something that happens and that suddenly drastically changes how debates are being had in Brussels or DC or London or Berlin or wherever. Case in point is like working on AI policy and working for example on the AI Act was a very different job before the release of ChatGPT and after the release because that that really kicked off the frenzy, right? That really changed the discourse and changed the emphasis of a lot of policy debates, although I think it’s important to educate decision makers about how this technology works, because there is a lot of talk of AI as the sort of like mystic thing, and it’s computing. It’s non-deterministic, which is why the unpredictable stuff happens, but in the end it’s hardware plus software plus data. So I think demystifying AI is important and also, you know being mindful of the hype merchants are, if you want to call it that, who the players are, who have a vested commercial interest and playing up this technology and talking about future scenarios such as “AI is going to upend everything,” because they might stand to benefit from it, and because it’s good marketing. And then also, make sure that there is space for people to think and talk and debate about it’s like where we want AI and where we don’t want it and then how… because obviously this is going to be, I do think it’s going to be relatively pervasive in different areas of life and sometimes it might just replace human decision making. And then it’s a really important question in what domains? What decision? What’s at stake there for people? Especially given that there is often like a tendency to experiment with those technologies on marginalized, vulnerable, disenfranchised communities and to make sure that even when the zeitgeist currently dictates that the EU needs to watch out for being competitive and having a strong position vis-à-vis other geopolitical actors that doesn’t that that doesn’t void the need to talk about risks and fundamental rights because ultimately, that’s the baseline for a flourishing society, right? That people retain their rights and liberties and that technology does not interfere with that because it’s not a deterministic development that we’re having here. And ultimately, like it’s, it’s up to us to help shape how this technology is being developed, how it’s commercialized, how it’s deployed, that’s not something for like a few techies to decide, something for a broader debate within society in the long run. 00:21:51 Domen Savič / Citizen D You’ve mentioned policy development and we’ve been talking about the EU, but what are your thoughts if you, let’s say, compare the US, the EU and China in this regard in terms of regulatory frameworks or policy developments that are focusing on the artificial intelligence. What are some in your opinion main differences and what are some similarities in that regard? Is there a common ground that literally the world agrees on? 00:22:35 Maximilian Gahntz / Mozilla Foundation I’ll have to say I’m not an expert on… there are people who know much, much more about tech policy and tech in general in China, so take what I’m saying with a grain of salt. I think over the past maybe 5-6 years like one very common narrative we’ve heard is that of the three different pathways and tech and tech policy, right? It’s the US, that’s completely laissez faire deregulatory. We’ll just let our companies do whatever they want and it’s China, which is putting its foot down and being very prescriptive and then Europe is the middle ground where we’ll have innovation, but also preserve European values. There’s probably still some merit to that, but I also think it’s a bit of oversimplifying things because if you look at the US at this point, at least for the past couple years, it hasn’t been completely lassie affair. Obviously, there’s always been a lot of gridlock in Congress, so Congress doesn’t really pass a lot of new laws also on AI, even though they introduced a lot of bills on AI in the past year or two, but there are many different states which are trying to regulate AI so if we are talking about EU, that’d be like a lot of Members state regulation in the absence of European wide or in the US Federal Regulation. If you’re talking about California and I think Colorado, there is a lot of debate and some adopted laws on AI already, there was a big executive order by the Biden administration in late 2023 that put a lot of rules on, for example, government use of AI… So, I would say the idea that like nothing is happening in the US is slightly outdated. Obviously, less happened in the US than in EU, for example and then we’re going to see what’s happening with the new administration, the Biden Executive order on AI is probably going to be repealed, but it’s kind of hard to foresee right now what the Trump administration will do on AI because they’re also competing camps. Obviously there’s a lot of venture capitalists who supported his campaign, there is also Elon Musk, who’s like a big AI safety proponent and talks a lot about existential risks from AI, so it’s going to be interesting to see that play out, but overall, we can probably expect a bigger shift of focus away from like AI ethics and civil rights to national security and American competitiveness. And then in China, when people in EU have always been saying like, you know “We’re the first to regulate AI”, that’s also only semi-true because China has put forward rules and regulations for platform recommender systems for example, but also for large language models a while ago and I can’t really, I can’t really speak to the exact nature of those rules and how effective they’ve been, but China’s legislators and policy makers are very active on a AIaround the world, not just in EU. And that’s just talking about the US, the EU and China, right, where that hasn’t even mentioned like a lot of what’s going on at the international multilateral level because the G7 has adopted its code of conduct on AI, the OECD, the UN is increasingly inserting itself into policy debates and they’re building out that capacity. So much more will happen at the international or transnational level as well in the coming years. 00:26:50 Domen Savič / Citizen D And speaking of this happening on many levels, is there like a way to sort of dissect the actual chamber of power, where decisions are being implemented and are put together? So again, listening to public debates, listening to political speeches, listening to discussions from the industry, it seems that they all, these different sectors, all have their perception of where the real decision is being made, right? The industry is saying, “Oh no, you know, the regulators will just come up and mop up everything that we’ve been doing for years”, the regulators are saying “No, no, no, these decisions on regulatory frameworks are being written by us and we are the smart ones” … so how would you see this flow from innovation towards regulation that is actually working out for the general population? 00:27:59 Maximilian Gahntz / Mozilla Foundation I think, at least for us in Europe, it’s also just democracy at work. And with all its flaws and benefits and intransparencies, because obviously if we learn about our political systems and high school or at university, studied political science back in the day, you get a very formalized process that you can draw upon a sheet of paper, and that’s how it still works. But obviously, like the reality is much messier and there’s a lot of lobbying which I would also use as a value neutral term because there is a lot of people lobbying for important and good things as well. There is a lot of backroom, there is a lot of public back and forth depending on how politicized an issue is. And I don’t think that necessarily varies with AI from other policy… Obviously, that’s the one I’ve been following the most, but I think the old blame game about how regulators are over regulating, U.S. companies are profit driven and are thus trying to preserve their interest and then wherever civil society and other actors fit in this, I think that’s largely the same. Obviously if we’re looking at the AI Act for example or like other tech policy files in EU and elsewhere, I do think it tends to be an uphill battle for civil society and everyone who works in civil society and has tried to engage on tech policy files will know this because there are other actors with much more resources to put into this and big public policy and government relations teams and agencies who just got a leg up. So, the challenge of coming together as civil society and despite maybe having fewer resources, making your voice heard. That’s just one that won’t change, we just have to come up with strategies and scheme together. 00:30:18 Domen Savič / Citizen D Speaking of this exact issue, working as an activist or as an advocate, do you think it’s better to focus your intentions or your energy on the local level or the national parliaments and governments or is this a Brussels game exclusively? 00:30:43 Maximilian Gahntz / Mozilla Foundation Definitely not a Brussels game exclusively, I think it really depends on the issue and on political context. Obviously, if you live somewhere where your local politician or the rapporteur for a big legislative proposal in the European Parliament that gives you different leverage than like someone from a very different country in EU being responsible for that file. I think there’s always value in engaging locally, but I think, when it comes to tech policy, I will say a lot happens in Brussels and Brussels has its own logic and structures and I say that as someone who’s doing this job from Berlin. There are people who are much more in the center and I’m more of a tourist who every once in a while, goes to Brussels. I think it really varies and it really depends, but it just like requires you to be become familiar a little bit with how the process works and like who the people are that you want to talk to affect change. 00:32:03 Domen Savič / Citizen D And looking forward and slowly wrapping up our conversation… What are some of the things you are paying attention to as the year progresses, right? We’re doing this in in the beginning of January, so what are some of the policy developments or interesting topics that you are focusing on in the field of artificial intelligence? 00:32:31 Maximilian Gahntz / Mozilla Foundation I mean, I think there’s the legislative side of things because we’re still waiting to see what the new European Commission will do. For AI, there’s debate about an AI liability directive, which basically sets rules for a civil liability for AI developers and we’ll see where that is going or whether it’s going anywhere. The UK is working on an AI bill and I think they will release on Monday an “AI opportunities plan”, where is currently unclear what exactly that means, but it’s more about the investments side and less about the regulation side. We’ll have a lot of debate about copyright rules, for example and how those might need to change or not and what the other plumbing is in our like copyright and rights enforcement system to make sure that there’s sort of balancing of interests between tech companies and creatives and rights holders and journalists, for example. So, I think that’s the nitty gritty policy side. Obviously, there is an overarching development that we’ll all be watching, which is like how is the new US administration going to work? How are they engaging with for example EU in general, but also on issues like tech because this has the potential to be very contentious and we’re already seeing strike some posturing right now around, for example, the EU imposing rules on US tech companies and the Trump administration may not like that, and there’s a lot of US tech executives who are also like starting to play into that narrative a bit more. So, we’ll actually have to see, you know, whether UK policy is going to suddenly be part of trade policy debates because if the you start sanctioning American companies, that might or may not trigger a response from the US… So, I think like at the geopolitical level, things are going to get much more complicated and probably a little messy and I think it’s important to get in that a better understanding of how that might pan out and then what different actors, both at the government level, also at the civil society level can do here. 00:35:07 Domen Savič / Citizen D And before we wrap, up just one more question… so AI went through several periods in the history going back to the 70s in in the 20th century. Looking at the current situation, the players, the regulatory frameworks, the situation in the in the industry, can you sort of describe how the next AI winter looks like. 00:35:42 Maximilian Gahntz / Mozilla Foundation I you I think you’d have to ask someone else. I think that whoever gives you an answer to that, it’s a creative exercise. There’s… it’s hard to predict how hype might die down or not, like what technical developments are looking like… It’s good to remind yourself of what, like the market forces are here and like, think about some of the trends in AI, because I think like, what’s safe to assume is that this debate and this landscape is going to continue to change. I think there’s a lot of people who talk a lot about acceleration, there’s a lot, some people who talk about a bubble bursting, but I think if you look a bit closer and like you know what might happen within those things. There are so many different scenarios of how this might pan out, AI might just become a commodity that gets like at some point much more efficient than it is right now, much less energy intensive. We won’t be talking about big models as much anymore because we’ll all have a small model on our phone and then it’s just, you know, regular computing. It’s a new technique and it’ll be good for something, then bad for other things. I think that’s one scenario, obviously, like we might also continue on “bigger is better” trajectory that we’ve been on for like the past couple years where companies are just continuing to throw a lot of money and capital into this to build ever bigger and bigger and more powerful models. And then we really have to also talk about resource constraints both like economic resources but also natural resources because we know that developing AI models and actually using them as super energy intensive, has real environmental impacts and it’s no coincidence that a lot of big American technology companies are currently talking about or already doing it, bringing nuclear power plants back on the grid, building their own power plants. So, I think… everything might get bigger, everything might get smaller, or somewhere in between. And then what? I think well, to be honest, we’ll have to just like be wait, have to wait and watch and be vigilant because I think as someone who’s like not at the on the technological front line, but rather like an observer of the technical ecosystem, to then start thinking about like what societal and political consequences of that are you just got to make sure that you understand what’s going on in order to be able to act and then ideally shape where things are going. 00:39:05 Domen Savič / Citizen D And that goes for, I would assume that, the self-aware AI. So, I would love to hear a few more thoughts on this matter, because this is constantly popping up every time the industry it seems needs a PR boost, Altman and others are starting to talk about self-awareness. So, is this a feasible or is this a realistic path or is this just something that the PR department needs to put out just to ramp up the groupies? 00:39:46 Maximilian Gahntz / Mozilla Foundation I mean I have at this personally, a lot of skepticism, but ultimately, I don’t know, right? I think on average, humanity is probably not super great at exactly predicting the future, so I’d have my doubts that, you know, Altman, or some tech executive or, you know, some civil society advocate is actually going to predict what’s going to happen in 2040. I do think there’s a lot of hype, I do think there is a lot narratives that are being pushed because someone has an interest for pushing them. I think what’s also clear is that, like the technical community, like very much doesn’t agree on what’s happening there or maybe some voices that get a bit more airtime because they’re like, you know, powerful CEOs or because they want a Turing award and are famous computer scientists. And I think these are all positions that have merit, we also need to look at them as a prediction of what might happen rather than someone actually knowing what’s going to happen. I think there’s been a bit too much of that in the press, I think we all just got to learn how to deal with uncertainty, to be honest, because… It’s just an AI, it’s a bit crazy right now and everyone’s imagination is running wild. In the end, we… just got to see which person, just like by pure coincidence, happens to be right. But that only applies if we’re talking about the very long term… if we’re talking about what’s going to happen in two years, that might be different. But I’m not going to make a prediction about 2050 ever. 00:41:35 Domen Savič / Citizen D …unless it the year is 2048, right? 00:41:39 Maximilian Gahntz / Mozilla Foundation Yes, we can meet again in 2048. 00:41:44 Domen Savič / Citizen D I’ll pick you up on that. Thank you, Max, for dropping by, this was the first 2025 episode of Citizen D, we publish an episode every month and we focus on different range of topics from digital policies, regulation and human rights advocacy. See you next month! Citizen D advice: Pay attention and listen to arguments, not wishful thinking Political arguments stem from media representations AI needs a wide discussion on different social impacts More information: Mozilla’s research: Unlocking AI for everyone, not just Big Tech – article Online life is Real life – podcast series Redirecting Europe’s AI industrial policy – analysis Accelerating Progress Toward Trustworthy AI – whitepaper Generative AI and Labor: Power, Hype, and Value at Work – primer Critical Dependencies: power consolidation of digital infrastructures – report About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!
Dec 15, 2024 · 1 hr 9 min
105 Katja Koren Ošljak: Vsaka tehnologija je medij
Za konec leta postavljamo piko na i teme mladih in informacijske družbe s Katjo Koren Ošljak, ustanoviteljico zavoda Vsak in asistentko na katedri za medijske in komunikacijske študije na Fakulteti za družbene vede, ki se že skoraj deset let na različne načine ukvarja s fenomenom mladih in informacijske družbe. V pogovoru se dotakneva fenomena neukih digitalnih domorodcev, razlik med digitalno in medijsko pismenostjo ter vpliva tehnologij na spreminjanje medijskih navad mladih kot posledica pandemije, vseprisotnih družabnih omrežij in drugih digitalnih fenomenov. Dotakneva se še generativne umetne inteligence, razlik med vzgajanjem digitalnih potrošnikov in digitalnih državljanov in razvojem politik informacijske družbe. Državljan D svetuje: Podatkovju informacijske družbe manjka kontekst demokratične družbe Tehnologija ni samo tehnologija, temveč tudi medij Vir financiranja raziskav in kampanj je ključen za razumevanje problematizacijske fokuse Dodatne informacije: Raziskave “Mediji in mladi” – povezava The Gentrification of the Internet: How to Reclaim Our Digital Freedom – povezava in recenzija Andrew Tate factor – novica O podcastu: Podcast Državljan D je podcast za produktivno preživljanje časa, v katerem igramo vlogo državljana. Državljan D v pogovorih s strokovnjaki določenega področja informira in aktivira. Da se. Gremo naprej!
Nov 15, 2024 · 56 min
104 Mackenzie Funk and the birth of surveillance capitalism
We sat down with an award-winning investigative journalist Mackenzie Funk, whose book, “The Hank Show: How a House-Painting, Drug-Running DEA Informant Built the Machine That Rules Our Lives” tells a story about the origin of surveillance capitalism we know and hate today. We talk about the book and the man behind the story, but we also touch upon his legacy, the surveillance capitalism that stems from data economy and big data intermediaries and the way we have to address this on a local but also on a systemic level. Also included in the conversation are the critiques of predictive policing, the issue of almost unchecked business of data analytics and the way forward. Transcript of the episode: Expand the transcript 00:00:06 Domen Savič / Citizen D Welcome everybody. It’s the 23rd of October 2023, but you’re listening to this episode of Citizen D podcast on the 15th of November same year. With us today is Mackenzie Funk, an award-winning investigative journalist whose second book, and it’s a mouthful, “The Hank Show how a house painting drug running DEA informant built the machine that rules our lives”, tells a story about, well, I’m just going to say it, it basically describes the birth of surveillance capitalism. Hello, Mack. Welcome to the show. Would you say the description is accurate? 00:00:44 Mackenzie Funk Yes, you’re right. That is a mouthful. I never had an easy way to explain what the book was about except that it was… Yes, it was the person who we’ve all never heard of, who started so much of this world we now live in. 00:01:01 Domen Savič / Citizen D And before we start with the book, so the first question I’d like to ask is before this… You were so you’re an award-winning investigative journalist and you focused mostly, or you focused mostly on environmental stories, global warming and stuff like that. What made you pick the the Hank Asher story? 00:01:21 Mackenzie Funk I think, as crazy as it sounds, that surveillance and climate change are very similar in some ways, especially to an audience that doesn’t know very much about either one. They can seem very dense and very boring at first and so you need to find a way to tell a story that is not just about the facts of it. So, in the case of climate change, it was just about here’s the science. And so, for many years before, I was working on my climate change. You would have these arguments in the United States between the people who believed in the science and the people who did not, and it was not a very good way to win the argument, just to be right, just to just to have the scientists have the right facts. And so, I was trying to find a way with the climate change book to tell a story that would maybe show the stakes show why people should care. When it came to surveillance, that was my same technique as I focused on a person because I figured even if you’re not interested in privacy or surveillance, you might be interested in this person. But the second thing is that they’re both these big systems and the reason they that some people find them hard to… they find them boring or hard to understand is because they’re it’s so complex. But there are these big systems that seem to fall the heaviest on the people who are poorest among us and the most vulnerable. It’s becoming more obvious with both climate change and privacy that the poorest people in the world, the poorest people in each of our countries, those are the people who are bearing the brunt of this and that there can even be winners in these new economies from climate change and from surveillance capitalism. And those winners are not the poorest, they’re the usual winners in our societies, and I find both of these both climate change and our lack of privacy to be accelerating some of the worst inequities in society. 00:03:25 Domen Savič / Citizen D Before we jump into that, I saved this part the last part of the show. But first let’s talk about the book. So how did you find this guy? Hank Asher, the guy who… I’ll let you tell the story or a brief recap, but this was… Reading your book this was the first time I heard about it and I am working in the field of, you know, digital privacy and digital activism for the last fifteen to twenty years 00:03:56 Mackenzie Funk Yeah, I would not say that I was by any means a privacy expert when I started this, but I certainly paid attention from reporting abroad, especially in in places like China and Russia, I was very careful about… I tried to understand where my information is going and who will see it and even after the Snowden revelations in the United States, I became a little careful about even what my own government was looking at. And I had never heard of Hank Asher. His name first came up when my… it’s a complex story, but a magazine approached me. An editor I knew, and he said I have a story for you. And the story was about this group of people who were trying to stop child predators as they called them, and they were using this software built by a person named Hank Asher, who they described on their website as the father of data fusion. So, I was looking at this group, and I saw that note on their website. And I said, who, what? And that is that is what it began my journey, and I then googled the name and then I saw that he had been a cocaine smuggler, that he made a fortune multiple times and then lost it in Florida in the 80s, I saw that he had this crazy story that he was involved in the security build up after 9/11 in the United States. That his technology undergirded some of the biggest surveillance companies in the world and certainly in this country, and that it was just this character nobody had ever heard of. And I found that amazing. But what really did it for me was when I stumbled upon his obituary page because he had died by the time I heard his name and the things that people wrote about this man. Just on his online obituary website that his company set up for him and then another one that the Funeral Home put up are just not the kinds of things you see written about most dead people you know. You read obituaries, you read the comments people make, and they say nice things about people. But the ones that they said about Asher was like, he changed my life. He paid for my kids’ college. He fixed my broken teeth, or he yelled at me. He swore at me. He was the craziest person I ever met. He changed… He changed this country forever. He changed my life forever, and he just seemed to have this outsized impact not only on privacy, but also on the people around him. 00:06:39 Domen Savič / Citizen D The book is full of these anecdotes or happenings in in his life when you, I mean, the title sort of tells a pretty good story. So, you have a guy who’s been drug running for, for the DEA, or who was an informant for the DEA, but at the same time, he was saving lives helping people, you know chase down, child molesters and stuff. But what was in your view the thing that surprised you the most? Like when you were doing, when you were doing the research for the book? 00:07:16 Mackenzie Funk Of course, everything about I got perhaps a little too obsessed with understanding what exactly he was doing in the 1980s when in the United States, the center of all drug smuggling was Florida because of its proximity to Latin America, to Colombia, to Jamaica, to the Caribbean, and I became obsessed with that. But that was interesting, but not surprising. The biggest surprise was how these open records laws in Florida were a good thing in this country. We wanted transparency and the States, especially after some scandals, Watergate and others. They decided to make sure that the public could know what the government was doing in their name and so many states opened up their records, opened up their books so that citizens could see what the government was doing and people like Hank Asher, in states like Florida, which had very open public records, exploited this. They were able, using this state law, they were able to go in and get all of the driver’s license records and all of the vehicle registration records, marriage records, birth records, divorce records, all the housing record. Everything, everything you can imagine that the government, local government would use to sort of have a citizen move through their life. This was a public record and because the governments wanted this to be they, they wanted citizens to know what they were doing. They opened it up to a new species of person, which is these data, data aggregators. And the technology, by the time Asher came along in the late 80s and 1990s, you were suddenly able to scoop all this up and make sense of it in a way that I don’t think you could have when these laws were written. So that was the big surprise that this, this very progressive policy was became something very different and would… say if we compare the Hank Asher’s period where he was buying up or gathering all of these data points and putting them in a in a database in a searchable database, would you say there are similarities if we draw a comparison with Mark Zuckerberg or Elon Musk or any of the other big intermediaries, digital intermediaries’’ bosses… 00:09:54 Mackenzie Funk Yeah, I have similarities and an important difference in the big trick with Hank Asher’s era. We’re talking in the 1990s and early 2000s is that they are going out to all these different databases and trying to make sense of these little tidbits of information they get from each one, they’re trying to connect all of them to a single person. And to do that, they ended up assigning a tracking number to each American citizen and each American resident, and also many people across the world, so that each of us has our own our own bar code or something like this. And then they had very tricky rules that became algorithms or and now are different through machine learning, they’ve perfected these but different rules that assign a new data point that comes in to an individual with some degree of faith that this is going to the right person so I am not the only Mackenzie Funk in the United States. It turns out there are many of us and they tend to know when I’ve moved or when I’ve bought something. And they do so because they’re able to see, of course, my address, but also the people I’ve lived with and my age, my gender, these kinds of details. So, they can with some faith put this new data point with me. Someone with a name like John Smith, it’s two very common names in this country, it’s much more difficult, and yet they would have ways to make sense of this. Contrast that with what Zuckerberg built, which is you have to use your real name on Facebook, and the moment you you do, you become your own aggregator. Zuckerberg did not need a fancy algorithm or a fancy set of rules to attach each new data point to an individual because we did that for him, we did it by logging in under our real names. We did it by putting Facebook on our computers and on our phones, and so it almost turns the problem on its head from the perspective of the of the aggregator and I think that’s the biggest difference with what’s come out today and Asher’s day is that back then the problem was information was coming in from all sides and we need to find a way to really attach it to a person. In Zuckerberg’s era, it was how do we make people build their own databases and then give us access to it and I think right now in a maybe post Facebook era, we’re a little bit in a more of a hybrid space between the two. The same problem Asher solved is now being solved again from the from the data broker perspective, they’re trying to do it again. 00:12:51 Domen Savič / Citizen D And just to continue your train of thought… So, what are some of the lessons that we can learn from his story right regarding data economy regarding these data points regarding data aggregation regarding I don’t know use of data in, in policing and surveillance? 00:13:19 Mackenzie Funk Yeah, I. Well, one thing is that Asher, I think had high minded goals with what he was doing and why he was doing it. Yes, he wanted to get rich, but like many people in Silicon Valley, say 15 years ago, he kind of believed he was the good guy and that he was aggregating all this information to do good things, and that included, especially in his case, using this information to go after child predators, to go after child molesters and to help the police do that. Then after the attacks of 9/11, he used it to go after supposed terrorists, he thought it would help the police solve crimes. He thought it would eventually, you know, help companies determine lower rates for people who had good driving records, for instance, or otherwise look good. And it was hard for Hank Asher to see the downside of what he built. But I think he was an early stand in for people like Zuckerberg, who may have claimed high minded principles. At the beginning, you know bringing the world more connected or whatever the slogan was in in the early days of Facebook and yet all this stuff has such a dark side that if you build this this weapon, eventually you’re not going to be the one in control of where it’s aimed or it will become take on its own. And with Asher, you could see the dark side of this pretty quickly, his technology was used not by him, but by the people who took control of his first company to change the results of one of the most important elections in this country by kicking black Americans off the voter rolls in Florida, they did this in the name of keeping fraud out of elections. But the end result was that, it appears that George Bush beat out Al Gore in the 2000 election in Florida and therefore won the entire country. You can draw a line from what Asher built to that moment, not something that he would have wanted. He actually was in favor of the Democrats in favor of Al Gore, but that’s how his technology was used, I don’t think that he would like how it was used much later in Florida, again disenfranchising and taking away the vote of black Americans. I don’t know that he would have liked how it misidentified various Muslims in this country as somehow being terrorists. But it did. I don’t think he would like how people were wrongly accused because a police officer who was badly trained in this information would pick the name of somebody from his database and say “Oh, this has got to be our suspect”, but that’s happened. So, I think that’s the parable you can draw from what he built is that this stuff has such power and eventually its creator won’t be in control of where this power goes. 00:16:44 Domen Savič / Citizen D Do you think… what you just said… that this lends itself to a theory that, you know, technology is neutral and that it matters who’s running the show if I can borrow a phrase from the title of the book, or would you say that technology has some inherent biases and inherent problems built in, and the driver behind the steering wheel isn’t the only, let’s say good or bad switch? 00:17:20 Mackenzie Funk That’s a good question. I think the reality is that it is both things, I think the driver matters quite a bit, but also the technology can have something built in that makes it inherently worse or better, and I don’t think that it has to be one or the other. And that again is the lesson I would see here, if you think about what Hank Asher built and why he built it, in some sense, it’s a suspicion machine. Keep in mind that this is a former cocaine smuggler who suddenly is wanting to work with the police, he knows that he has something hidden in his own background, he thinks like a criminal. He knows how the how the Drug Enforcement Agency, the DEA, has gone after other drug smugglers, and he’s seen how they put information together and draw connections and so what he kind of built a machine that would, that would show hidden connections between people or between people and what they were trying to hide. For instance, say their assets. And a lot of the machine was built based on his own psychology and because of that, it’s something that surfaces, things that most Americans would think would be hidden, like if you’ve lived with the same person three times in a row and they know that because his machines have your entire address history and they can see “Oh, these people were roommates.” What if in a time that this country was a much more anti-gay, what if that was your husband? If you’re a man and that’s your husband, but you were keeping your relationship a secret… Well, that’s the kind of hidden information that his machines would surface. And they basically remember every little thing that you might want to hide, or think would be hidden. And that’s based on in part who the maker was and how he started to think about what would be useful to investigators and what criminals or other people might be hiding… So yeah, there there’s some inherent danger to the technology, but also how it gets used is very much depends on who’s in control of it. 00:19:44 Domen Savič / Citizen D And if I can just backtrack a little bit, you’ve mentioned that after Watergate, the government wanted to be transparent and open. So nowadays you have digital activists, you have human rights activists arguing for more transparency in the government works in police works in in these supposed closed system systems. But on the other side, transparency on its own… I sometimes get the feeling it it’s not doing much good, I would say, in a way that OK, sure, you open up all the databases and people are drowning in these data points. And every once in a while, you have this Hank Asher type of type of person who, you know, makes sense and a little bit of money on the side from all of these. So would you say, how would you argue or what would be the correct way to sort of argue for transparency, but at the same time not argue for, OK, you just open up the floodgates and let it all out. And, you know, we’ll figure it out as we go… 00:21:08 Mackenzie Funk I am no expert in the data privacy laws in the United States versus Europe, but I do know that you have a much better understanding that data collected should be used for the purposes it was collected and not for other things, and one of the big things that went wrong in the United States and I think continues to go wrong and round around the world is that this idea of transparency also means that companies can purchase or otherwise acquire data it was never meant to be used for, say, advertising. When it’s something that a local government has collected, but because there are no controls on how it is used. Two or three purchasers down the line, then it becomes transformed into something very different. So I think the answer to much of the transparency and privacy problem, which is real, is some sort of control on… is this used for what the person who gave up their information to their local government, for instance, intended if I give my information to the Department of Motor vehicles in my state here, I don’t expect that they are going to then sell that information on to LexisNexis, the data broker. I don’t expect that LexisNexis will then send that information onto immigration authorities and that the immigration authorities then could come and come to my home if I am not a say, I’m not a citizen and I don’t have the right papers to be here. Those are not things I expect when I go to get a driver’s license. And I think some sort of… something like that is built into the laws, can go a long way to helping… I tend to agree that transparency isn’t enough, I tend to think that that governments are collecting too much information and they are keeping it far longer than they need to because then. If there is transparency, if there are transparency laws and it happens to be in the database still, well then, they might have to give it to whoever asks for it. If they don’t collect it in the first place, or if they don’t retain it for a long time, that’s often safer. But I think the tension is always going to be there, but I think a lot of it is about the very the gut check. Is this what I thought was going to happen to my information? 00:23:47 Domen Savič / Citizen D And do you see things changing like now, like in the past? Like maybe after the pandemic or even during the pandemic, it seems that you know the data industry, all of these big digital intermediaries, they finally lost its, I should say user-friendly halo, right? And people are sort of pushing… in the United States and also in Europe and also in other places across the globe, they’re pushing for a for a different approach to these to these issues of privacy of, you know, companies knowing almost everything or everything about the person. So do you see that happening as we move away from from, let’s say, the COVID pandemic and go forward? 00:24:36 Mackenzie Funk Yes, I do see it happening and I think again, I know the situation in the United States better than I know in Europe, but in the United States, the big change was after the election, in the first election of Donald Trump. After the 2016 election and Facebook’s role in that election, the Cambridge Analytica scandal, those things seem to very much change the public perception of these of social media in particular. And then you see more controls on our phones, both Android and Apple have done a better job as to not letting other companies track us all over the web all the time. Not that they’re perfect, but we have a lot more control than we did eight years ago. And I think that is that is a good thing. That said, I think we’re coming back, especially in in the United States from 20 years of almost no control on privacy and, and so even if there’s some incremental gains on the digital space… A lot of this information is already out there and a lot of the information that companies like Hank Asher built and those are big parts of LexisNexis and Thomson Reuters and TransUnion, those databases are still there and so they’ve got20 going on 25 years of information about each of us and they never got rid of it. They didn’t purge it, they kept it. And so even if governments are collecting less, even if our phones are giving a little bit less information about us, the length of that history that some of these data brokers have on us and that still includes Facebook, that really matters because they can get a sense of where your life is going and where you’ve been and those are… That’s pretty important information when they’re trying to help other companies make decisions about your life. I looked at healthcare in the United States, doctors began to look at not just your clinical conditions, but at the conditions of your life now in general, where do you live? Do you have access to a vehicle to get to your doctor’s appointments? Or a bus or a train? Do you drive a certain kind of car? Because certain kinds of cars can be, you know, can you can judge someone based on their car. You can kind of guess what their general health is. Where do you get your information? Because if you’re getting all your news from Facebook versus of newspaper, you might be lower information and you might not follow the directions of your healthcare provider as much. That kind of information or even things like where did you live when you were growing up nd was that area worse in terms of air pollution… that kind of information, going toward healthcare decisions now is very… You’re not going to easily get away from that just because your phone isn’t telling Apple or Google exactly where you are right now. 00:27:59 Domen Savič / Citizen D And just to follow up… So do you think that all of these data points about the person or the information packets you just described, do you think these are actually used by people or by companies by industries collecting it or do you feel that this data economy is basically I’m not going to say a mirage, but basically something that looks nice, looks fancy, looks I don’t know, important, but at the same time you just get these, you know, data dumps that that nobody uses and then, you know, when somebody hacks a big, you know pharmaceutical or health related database, then it becomes a problem, right? 00:28:47 Mackenzie Funk Yeah, I think they are using it. In fact, I know they are using it, but most of the Hank Asher’s products, most of the world’s biggest banks and most of the biggest companies use those products. Most of the law enforcement agencies in this country use these products and I’ve seen… I mean, imagine it’s not just the little bits of data, right? It’s tied to a person’s identity, all the information you might want to know about them. For figuring out how much to charge them for insurance or like I said, healthcare decisions, or if you’re a police officer trying to track someone down, where do they live? Where have they lived? Who are their friends and who are their relatives? If you’re trying to find them, where would they go? That kind of information is in there if you are an immigration officer trying to track people down as, as could happen in in this country and has happened in this country. It’s the same kinds of questions, and also when did they arrive? When does their name first appear in this database? Does this show that their recent arrival in this country? And I found evidence that it’s being used all the time. I don’t know how banks work in Europe, but when I log in from a new location, say I have a new computer or a new phone and I want to access my bank information sometimes, it will send me a bunch of questions: which of these street names have you lived on or which of these streets have you lived on in your past? Who of these people have you lived with? Which of these people have you lived with? Little details about my biography that only I would know. You would see things that stretch back 15-20 years into my history and that’s a Hank Asher product built right in and used for something that I think is pretty benign. It’s making sure that you are really you, but it’s a show of the reach that these products have everywhere in the background of our lives in this country. And nobody really has any idea… 00:31:10 Domen Savič / Citizen D So, the case that you just described with the banks it perfectly I think describes the issue between privacy and security, right? So, you have these data products that are a boon for security, but at the same time they are literally almost or not even almost literally killing our privacy. Right? So, do you see this conflict? You know, resolving in in the in the future, in, in any shape or form, or do you think we’re just going to go back and forth between, you know, this is too much security. We want more privacy or this is too much privacy, we want more security? 00:31:54 Mackenzie Funk Yeah, I don’t see it resolving and nor do I see the sister of that one, which is the privacy and convenience question. And we see of course that that one all the more with the new AI products that say Microsoft or Apple is coming out with how much of your information do you want to give to these companies, especially if they’re uploading it to the cloud to make your life more convenient to complete your emails or calculate the distance between when you have to leave so you can get to your next meeting or to reschedule a meeting. These APIs, these AI products can already do this if you let them have access to your calendar or all of your emails and I find that scary and very interesting because I think a lot of people will choose convenience over privacy here and that the privacy controls will always come late. And as for security and privacy, it goes back to your earlier question about are they actually using these products? Do they actually work? I think we’re finding that as much as, at least in this country, as much as they were used for, say, counterterrorism, or for predictive policing or all these things where the idea was, if you get enough information about everyone in in a country or in a city, and if you really build these algorithms to help us predict who’s who among the general population. You can stop crime before it occurs or stop a terrorist attack before it occurs. So far, that hasn’t worked out as well as it was supposed to so far… yes, police officers are using Hank Asher’s products to say, figure out someone’s address history or who their relatives are, but they’re not using it in sort of an AI powered way… let’s let’s just give everybody a score, and I mean they are using it, but when they’ve used it, it hasn’t worked very well. So, a lot of them have dropped it. So, a lot of the predictive policing projects have been dropped in major U.S. cities because it didn’t work as well as they thought it would. It just turned out to be racism at scale. It just turned out to wrongly tar huge swaths of a city with the idea that they might have a higher risk of being a criminal, which didn’t always match up with reality and was certainly unfair to those people who were caught up in the algorithm. So, I do think that there’s been… Yeah, I sometimes think that is showing that, especially with law enforcement and counterterrorism, that it’s not as useful as they thought. 00:34:49 Domen Savič / Citizen D I was asking this because, you know, we in so this podcast is a part of the NGO with the same name and we focus on privacy, security and amongst other things, surveillance capitalism. And we’ve been doing this investigation in the CCTV cams that are set up by municipalities across the across the country and we realized that it’s basically… it’s a promise of security. They never deliver because the whole pipeline that follows detection, we have somewhere around 500 CCTV cameras across Ljubljana, and at the same time, they’re not doing much in terms of crime prevention, but they’re doing a lot in terms of, you know, privacy invasion, right. And if you see, if you look at across the world, you see that, you know, some technologies work in particular cases, but they sure don’t work in in other cases, right. So the debate around security and privacy becomes very targeted, I should say. Or you know, you have to look at individual cases, individual technology and so forth. So, would you say the data, the data policing or the policing with the use of data, is that something similar or would you say the success rate is much higher or higher than that? 00:36:18 Mackenzie Funk Hmm… we have this similar debate with traffic cameras and surveillance cameras here. I would say that the use of data in policing, it has been useful and hat Asher brought and what his companies bring to local police departments has accelerated the kinds of investigations they would do. For instance, they I was talking to these detectives who worked in Florida in the 1990s or in the 1980s, about how they would go about their casework and they would go and track down each of these individual pieces of information from different police departments across the state, and they would not necessarily have access to say the stuff you could get from credit cards, which always have a good address history, or they would or they could get it, but they would have to get a court order and the court order would take weeks. They could get someone’s phone records to know who they’ve been calling. But that again would take weeks or months and so to just to get enough information to try to track down a suspect in a crime, it would take them a very long time. And all of them found these products that Hank Asher built to be transformed and to help them do the work they were doing much faster and I as much as I was skeptical of where the technology has gone. I wanted to be fair and recognized that that it really did help some of these police and do their jobs and I and I think in a way that it didn’t necessarily hurt the privacy so much, no more than they already could. It was just happening faster, but things changed. When Asher’s company and others began to make this more algorithmic and to start assigning scores to someone’s criminality or their likelihood of being a terrorist, and I think where the a lot of the failure has been, is that when taking humans out of the loop or in saying that we need all of this so that we can solve these crimes, I don’t think you need all of this. And you certainly don’t need the scoring technology to help you surface people you think might be more likely to be criminals than others. It’s very different to try to predict crime than it is to try to track down suspects in a crime that already occurred. And I think that once they started to try to predict the future using this data, that’s where many of the biggest harms have happened and then when it goes away from these narrow uses. I can see that someone investigating a murder, for instance, might it might be more fair to give them this information than someone investigating a minor crime. One thing we have in the United States, not just the surveillance cameras around the around cities, but are the license plate readers on all of the police cars and on many city cars all over the country. And many of those were installed by a private company and that private company has taken all these scanned licenses and they’re able to give predictions is about where someone will be. For instance, if you always see someone’s car parked in front of a certain address at a certain time of day, you can predict that they’ll be there pretty soon, and some of that, that just seems like too much. Like who actually needs that? You know, who needs that is real estate companies or banks trying to use data for things very different than policing and these same companies were selling to the police and they were also selling to business. And I think the selling to business part doesn’t need to happen and the retention of all these scans over the course of many years, does not need to happen. Other people using it were immigration authorities trying to track down people. In this country, who might who might not have come legally, and my opinion on that is that in the United States, there are so many people who came to the country without the right papers and there are only so many they’re going to want to arrest. And it seems like if you are going to try to prioritize people, it’s the worst idea to go after people who have been here for so long that they might have children who are citizens in this country and who have real lives and who are living like any citizen, if you have to prioritize. And yet these systems, they will surface the people who are living the most, who are not trying to hide right? The ones who engage in banking, the ones who properly registered their car and got a driver’s license and got insurance, bought a home. All that information, that all goes into these systems. And so, it turns out that the people who are not trying to hide, who are living the non-criminal, very average lives are the ones who are easiest to find in these systems. And so, if they’re used for these purposes like that for immigration enforcement, for instance, that just seems like a total perversion of what what they’re for. 00:42:08 Domen Savič / Citizen D I have two more questions before we wrap up. So, the first one is basically a continuation of this debate. So why do you think these technologies, these data silos that are basically telling us where a person is going to be at a particular moment, why do you think the usage exploded in literally every corner of our society, so if you like, take Hank Asher, for example, right at in the beginning, he was basically helping out the police to catch pretty basic criminals, right? And then, the usage extended to anti-terrorism and then it extended to anti child molester. So why do you think the people are so excited about copy-pasting the same basic solution to, you know, places or fields that that may not have very much or much in common with. You know, the previous or the original field that this technology was used in? 00:43:20 Mackenzie Funk That’s a good question. I don’t know the full answer, but I can say that, before this was used for policing, the original product that Hank Asher had and that many of these things grew out of was not policing, but insurance. Private insurance companies trying to get a sense of who am I selling this insurance policy to and how can I reduce my risk? And the idea of reducing risk is basically look at the past to try to understand the future. Try to do a calculation about what someone has done before in order to predict what they’ll do in the future, and then if you can do that and whoever does that best they’ll make the most money, if you’re an insurance company to those clients who are riskier and they’ll charge much less money to clients who are not really going to have any problems. So basically, they’re getting money for nothing, and that that’s the whole idea. It seems like with Hank Asher, they took this idea of risk management from insurance and applied it across all of society. This idea that you can just look at someone’s past to predict the future, or more broadly look at people like this person and what they’ve done in the past to predict the future. That it’s a very seductive idea, because imagine for one if you can predict the future, you can make money off it or you can protect against whatever coming danger is. So, we all love to know what’s coming down around the corner. Human literature is full, is full of stories about time travel, or the person who could see the future or the seer and whoever knows what’s coming is almost the most powerful person in the world. And that’s what these systems promise to police and everyone else and counterterrorism investigators is a chance to predict the future. And maybe they get it right, for the most part, but the problem is, this isn’t advertising. In a lot of these systems are not used just to send somebody an advertisement for a tennis shoe, or a handbag they might want. These are used to make life and death decisions about that really affect people’s lives, who gets insurance, who gets a job and who gets arrested and getting it right most of the time, I don’t think should be good enough. I think it’s very different if you send the wrong ad to the to somebody, no big deal, you’ve wasted a little bit of money. But if you arrest somebody because you think they’re going to do something wrong, that’s a very big deal. And so, we can’t have the logic of insurance or the logic of advertising which is similar to that of insurance, which is your risk pool just… If you get it 70%, that’s great, if you get 50%, that’s great. We can’t have that logic be the same logic for the rest of our lives. 00:46:36 Domen Savič / Citizen D And since you mentioned healthcare… OK, two more questions. Since you mentioned healthcare, why do you think this data surveillance or data economy isn’t regulated as well as health field, right? So, you’ve just mentioned 70% isn’t good enough… imagine if you would have those options going to a doctor, right. And the doctor would go well, you know, I can heal you, but I can also kill you, so let’s see, let’s see what happens next. Why do you think the like, the regulatory frameworks, and this is the second part of this question, still heavily rely on the end user. So you’ve mentioned Apple and Android building all of these protections in in in our devices in our, you know, online services and stuff, why do you think there wasn’t a bigger, let’s say, systemic push for, I’m not going to say corporate protections of our privacy, but let’s say state protections of our privacy. So why are we at the end of the day David versus the Goliath? 00:47:54 Mackenzie Funk Good questions. The first one… healthcare. It’s very easy to understand that these are life and death decisions. It’s very easy to understand that what happens with the doctor or at the hospital is going to change everything for a person. I think it was harder for people to understand and imagine that what happened with privacy would have such impact, and this goes back to your earlier question about why did you go from or how do you go from climate change to privacy. Early on and climate change, everyone was talking about polar bears, right? They couldn’t imagine they knew they cared, but they couldn’t imagine that really that climate change would really affect their lives. They couldn’t imagine the wildfire smoke or the glaciers disappearing or anything really changing for them. So sure, they cared about it, but not in a way that was visceral and close to them, not like a healthcare decision. And I think with privacy, a lot of people are still stuck in the imagination that it’s these advertisements that are that it’s about advertising that it’s about these ads following you around as you surf the Internet and that, yes, it’s a little bit creepy, but what does it really have to do with my life? How does it really change my life and that the public has been slow to wake up to the reality that it has everything to do with your life and more and more the information about you is out there being used to make decisions about your life and what opportunities you’ll be given. And once that understanding is there and it’s becoming more there than maybe data will be regulated a little bit more like healthcare because we’ll see that it’s the same level of decision. As to the second part of the question – why is it always the end user? Why is it not systemic? I think it’s related a little bit, I think it especially in the United States, people think well, it’s your choice to use Facebook or it’s your choice to give up this information online and if you really care, you can do something about it and it’s almost the companies that still dominate the privacy landscape, which are American companies are built with this American individualist ethos. That we are a country where we pretend that the individual has lots of power and that we are supposedly self-reliant and that the government shouldn’t step in and regulate, and that if you really care, you’ll take action on your own. And of course, that’s nonsense. That’s nonsense with climate change and it’s nonsense with privacy. The idea that people would have any way to meaningfully counteract the most powerful forces in our lives without banding together without asking government to do what government is supposed to do, which is step in when it’s a collective problem, it’s nonsense, but it’s still permeating. I think the debate and that’s the biggest issue is that that that we’re stuck with this individualist sort of American ethos. And another point on that, the idea that there is any meaningful choice anymore that you that you can somehow avoid the digital economy that you don’t have to give up your information if you don’t want to, that you don’t have to use these tools that their tech companies built, that’s not… I mean for, for young people, for anybody who wants to participate in society as it is now, you have to be using these tools and therefore you have to be giving up information for the most part. And that’s not really a meaningful choice, when a when a gun is at your head. 00:51:54 Domen Savič / Citizen D We’re trying, we’re not always successful in wrapping up on a positive note, so please if you have like a like a… I’m not going to say an uplifting message for the end and I’m not even going to ask you about how’s the current US election going and what do you think will happen after November in this field… but is there a place to or is there a thing that you notice regarding data economy regarding you know these protections, that is, let’s say improving in the last couple of years? So, is there a light at the end of the tunnel? 00:52:34 Mackenzie Funk Yes, I do think as we’ve talked about that the public is much more skeptical of Silicon Valley and of these technologies in the United States in particular, then the public was five years or eight years ago. And I think it we are slowly waking up to the fact that this has life and death consequences for people that privacy is not just an ad that follows you around the Internet and that knowledge, even if it’s too late, even if it’s a little slow, will eventually translate to change, I think. I think it must, we’re not stuck just doing whatever these companies have us do entirely because we are, they’re still voting in all of ours countries and there are still elections, and those elections, if voters care about these issues and voters care more and more about these issues, I think there can be change and little things. I saw from Hank Asher’s technology for instance, the utility companies, we’re talking Internet companies and electricity companies, water companies for many years they were all getting together and they created a database of all these different customers and what their address was, what their names were, and they did it in the name of making sure they were credit worthy so you couldn’t switch from one electricity provider to another. And if you were someone who didn’t pay your bills that that they would know about that. But they started taking this information and selling it to the data brokers which you would never imagine that you move into a house and you decide to get electricity and because of that some corporation knows where you are or that they sell that to police. And that was one of those things that once it was uncovered in the United States, a few years ago, there was outrage and they stopped selling it to the data brokers that went away. It’s little things like that, there is a change. There are in this country and I know also in the EU there are lots of small fights against overreach by the surveillance companies, not all of them are victorious, but there are more and more, and I think that is a sign for some hope. 00:55:10 Domen Savič / Citizen D Excellent, that’s a perfect way to end this debate about the really horrifying situation everywhere. The book is called The Hank Show, thank you Mackenzie for dropping by, for, for sharing your thoughts, it’s really nice to see these ideas of privacy and the ineffectiveness of the security capitalism echoing around the world. Thank you so much, dear listener, this has been the Citizen D podcast, we publish an episode every month, so feel free to subscribe and we’ll talk to each other next month. Thanks again, Mackenzie. Citizen D advice: Demand new regulation of data economy Pay attention to the local surveillance capitalism implementations Focus on re-contextualising the debate between privacy and security More information: LAPD ends another data-driven crime program touted to target violent offenders – article Editorial: The problem with LAPD’s predictive policing – article Chicago PD automated policing program: Heatlisted – article New Orleans ends its Palantir predictive policing program – article Health Insurers Are Vacuuming Up Details About You — And It Could Raise Your Rates – article Automatic License Plate Readers: Legal Status and Policy Recommendations for Law Enforcement Use – article How ICE Picks Its Targets in the Surveillance Age – article Hank show excerpt – article About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!
Oct 15, 2024 · 26 min
103 Meredith Whittaker: Technology acts political and hides behind objectivity
With us today is Meredith Whittaker, president of the Signal Foundation who serves on its board of directors. She was formerly the Minderoo Research Professor at New York University (NYU), and the co-founder and faculty director of the AI Now Institute. She also served as a senior advisor on AI to Chair Lina Khan at the Federal Trade Commission. Whittaker was employed at Google for 13 years, where she founded Google’s Open Research group and co-founded the M-Lab. In 2018, she was a core organizer of the Google Walkouts and resigned from the company in July 2019. She now runs Signal, the leading global privacy-orientated NGO. Transcript of the episode: Expand the transcript 00:00:06 Domen Savič / Citizen D Welcome everybody, it’s the 10th of September 2024, but we are releasing this episode of Citizen D podcast on the 15th of October 2024. With us today is Meredith Whittaker, president of the Signal Foundation who serves on its board of directors. She was formerly the Minderoo Research Professor at New York University (NYU), and the co-founder and faculty director of the AI Now Institute. She also served as a senior advisor on AI to Chair Lina Khan at the Federal Trade Commission. Whittaker was employed at Google for 13 years, where she founded Google’s Open Research group and co-founded the M-Lab. In 2018, she was a core organizer of the Google Walkouts and resigned from the company in July 2019. Welcome to the show, Meredith! 00:00:55 Meredith Whittaker / Signal Thank you so much for having me, it’s great to be here. 00:00:58 Domen Savič / Citizen D Let’s start at the beginning. I’m curious to hear your thoughts on the consequences or the whole Google walkout situation. Back then, it seems to me that it was your personal convictions that started everything, and I want to hear your thoughts on the balance between personal involvement, personal responsibility and the push for systemic change in a particular area. Does one happen without the other? 00:01:32 Meredith Whittaker / Signal That’s a big question and I’m not sure I have an easy one-size-fits-all answer. Ultimately, we’re all individuals and when we act, we need to act on our own volition. We need to recognize what we believe and take it seriously and be accountable to our analysis. But I wouldn’t actually say that the walkout was the very beginning for me, the walkout was a culmination of a lot of work, a lot of thinking, a lot of conversations that I’d had over more than a decade. And the walkout also wasn’t just me. It was thousands and thousands of people. It was apparently the biggest labor action that has happened in tech, with 20,000 people leaving work in protest, you know, against the unethical business conduct at Google and against a culture that persistently valued some people more than others and developed products that often caused serious risk for those who were devalued, so to speak, due to that culture and those design decisions. I think the walkout was one way in which throughout my career, in many, many ways I have endeavored to be accountable to my analysis, I have endeavored to do what I can to change things when I saw them going in in a bad direction, but I had worked for many years and many different ways, from the inside trying to influence trying to shape policy and many of these things I still do… So again, I think the walkout wasn’t the beginning. It was one manifestation of a theory of change that looked to collective action from below to remedy some of the dangers and harms of the concentrated tech business model. 00:03:52 Domen Savič / Citizen D I was wondering because it’s like an example of this personal engagement that is usually pitted against “please don’t regulate us”, and “the industry is capable of regulating itself”. So, would you think that with these types of personal engagements and campaigns from people who are working from within the industry… do you think that’s enough to resolve these issues revolving around privacy, security, surveillance and other pertaining topics 00:04:42 Meredith Whittaker / Signal I don’t think there is one trick we’re going to find that solves these issues at once. For all the labor organizing and the ability of the structural leverage that employees/workers have relative to the institutions and people who employ them is a well-known lever before the 1980s, before the hegemonic ascent of neoliberal ideology, it was very well accepted, on the right and in the center. Conservatives, liberals, leftists… recognized simply structural check on toxic capitalism and labor power involved, the workers having some say in what they work on and how. I don’t know that this is individual so much as going back to some of the basics and recognizing that we have an industry that is making some decisions and putting revenue and growth above the common good in ways that could be really, really dangerous given the power and information possessed by this industry. 00:06:11 Domen Savič / Citizen D And speaking of power and controlling power, you’re now running Signal which is, for privacy activists an a journalists and many others across the globe the app or the service to use if somebody wants to protect privacy and have a decent level of security and I would like to know how do you generate trust in the app, in the system, in the symbol that is Signal currently representing in terms of privacy, security, and other protections of human rights especially in the area or in the time where you distrust in institutions, in each other is ultimately growing? 00:07:15 Meredith Whittaker / Signal Well, I think… look, trust is earned. It’s slow to earn. It is fast to erode, and it reflects a pattern of behavior and trustworthiness overtime. So in fact, I don’t, we don’t ask people to trust us, we endeavor to be as trustworthy as possible, to be as open as possible to develop our code in the open, open source to make our encryption protocols open, to allow people to vet and scrutinize the math and the implementation, and to ensure that in fact, insofar as possible, we are never asking someone simply to blindly trust us because they like what I say or they like the way Signal looks or operates. We are asking people to validate our claims and we are making insofar as possible everything available for them to do that and I think that is why Signal is so trusted, because in fact we, we are going above and beyond to be trustworthy in a way that most actors in the ecosystem can’t or are unwilling to for a number of reasons. 00:08:36 Domen Savič / Citizen D An easy follow up question: how hard is it? I mean how hard is it to go above and beyond and do all the things you just said Signal is doing to encourage or to be trustworthy? 00:08:55 Meredith Whittaker / Signal Well, it is difficult and it’s difficult for a couple of reasons. First, the tech ecosystem, the tech industry as it exists now, as it’s been built since the 1990s and before is structured around making money off data collection, it’s structured around monetizing surveillance. That’s the business model. So, you collect data, you use that data to target advertisements, or you use that data to train an AI model… but the assumption is built into all of the tools we use, just the narratives. We think with the libraries that we can access, that we would want to collect as much data as possible. And so, it is difficult to do the opposite. We actually end up having to rewrite parts of the stack, so to speak, in order to enable privacy, in order to reject data collection as a norm. So that is difficult because we are swimming upstream against a massive current in a trillion-dollar industry, where privacy has not been something that was prioritized and trust around privacy is certainly not been part of the business model. Now it’s also difficult or related to that it’s difficult because there isn’t a business model for privacy at this point in the tech industry, and this is one of the huge harms that we are, we are grappling with. The profit motive is oppositional to privacy, data collection is oppositional to privacy. So it’s difficult from that perspective in that we have to really think about our structure and protect ourselves from the imperatives of profit and growth, not necessarily because they’re bad in and of themselves, but because following those imperatives, would at this point lead us down a path towards surveillance toward data collection. So, this is why Signal is structured as a non-profit. This is why we really go out of our way to take the incentives for surveillance off the table when it comes to Signal again. So, we’re structured for success in the long term, so we stay laser focused on our mission. 00:11:29 Domen Savič / Citizen D And speaking of economy or surveillance capitalism… that’s one part of the equation, right? The industry is focusing on gathering information, repacking it, reselling it or selling it to the highest bidder. On the other hand, you have politicians, who are worried about privacy eroding security. So, is it hard for you, for Signal to argue for privacy when faced with a fake dilemma of choice between privacy and security? 00:12:17 Meredith Whittaker / Signal I think that debate, sadly, has gone on since, you know, we can date that debate to the mid-1970s and before when public key cryptography was introduced as a way to secure a network. And we’ve seen that debate resurface under multiple pretexts from the good guys need to undermine privacy, so we can stop terrorism to stopping child abuse to whatever else it is. But the motive is always the same. And the motive is that there are some among governments and law enforcement who feel that the fundamental human right to private communication should not be available to people online, that there should be no communications network that is not tappable, that law enforcement or governments aren’t able to surveil. And I think that is… It’s just simply incredibly dangerous, and it flies in the face of the long-standing expert consensus that knows there is no way to create a backdoor, create a way in that only “the good guys” can access, that anytime you create a flaw in these infrastructures, anyone with the tools and expertise to exploit that flaw will, and so you are corroding the very same cyber security measures, the very same private communications networks that your government also relies on, that your law enforcement also relies on and you are making those vulnerable to hackers, to hostile nations and to whoever else might want to infiltrate those. So, it is a very pernicious line of argument, but I don’t think it’s always in good faith and I don’t think that we’re ever going to win this battle simply by being correct simply by force of argument. We’ve been correct for multiple, multiple decades. The facts have not changed, but the will to create some magical formula that lets the government spy on everything does not seem to die. 00:14:35 Domen Savič / Citizen D And currently it seems like the debate, or the situation is getting worse on both sides of the Atlantic, right? You have several legislative proposals from the EU, from the US government, from Australia and other countries to sort of corrode the privacy, to enable backdooring for the good guys and so on. Why do you think the current situation is going from bad to worse? 00:15:09 Meredith Whittaker / Signal Well, I don’t know. I don’t have insight into the rationale of each individual government, but if we look at the arc of recent history, we see that the situation was pretty bad in the 2000s. Large surveillance companies, the big tech companies we talk about now, had open relationships with governments, were handing over huge amounts of data. You know, that still happens, but in 2013, the Snowden revelations really shone light on this imbrication between the large surveillance companies and the business model that we touched on and the US and other governments. And I think that that provoked a kind of counter reaction. You saw a number of the platform companies, you had iOS and Android adding encryption to their operating systems, you had a turn to privacy from the industry that wanted to in effect, save their reputation if we’re going to be cynical about it, one to distance themselves from government spying by adding privacy features, and immediately after that in 2015, you see a showdown between the FBI and Apple in the US over the encryption on the iPhone. And you begin to see an escalating campaign, as it were, to undermine the privacy guarantees that have been put in place post Snowden, most profound among these is Signal and the Signal protocol. The Signal protocol pushed the state-of-the-art forward significantly by making private communication truly private communication possible on mobile apps and after it was released in 2013, WhatsApp integrated the Signal protocol and now protects the contents of your messages, not every piece of data, but the contents of your messages, using our technology and there has been, since then and as private communications become more and more mainstream as people begin to recognize in the wake of data breach after data breach after data breach that it really is important to prioritize privacy, there is a mounting anxiety that I see on the part of some in law enforcement and some in government. Often people who are perhaps a bit parochial or confused that want to undermine and walk back these changes. Now there are many kind of dynamics that I think have helped or hindered this, but I see this as one more salvo in an ongoing battle and no sign that we are losing the war… and in fact in the last couple of years we, those of us in the privacy world who are pushing for these fundamental human rights, have had a number of wins, have pushed back on a number of pieces of very bad legislation in the face of often incredibly emotional and compelling narratives that are difficult to fight against, particularly when someone is bringing a heartfelt story to the table and then you’re on the other side debating about the nuances of cryptographic mathematics or something, right? It’s difficult battle, but it’s, you know, nonetheless, it’s incredibly worthwhile. 00:18:50 Domen Savič / Citizen D And one of the issues, of that battle is also the commodification of privacy. So now, as people are more aware of privacy first or privacy savvy technology analogies, privacy becomes a business offer, right? You trade off your privacy for usability or you trade privacy for data, for product. Do you think that poses any significant danger moving forward? So, in terms of companies saying “OK, you can have privacy, but you either have to pay for it or you do not get to use all of the services and features.”? 00:19:37 Meredith Whittaker / Signal I think this pattern, it requires a bit of nuance or a bit of understanding of the particulars of every case, but, what I can say in the context of communication is that requiring people to pay for privacy simply doesn’t work because, of course, communication requires more than one person constitutively it requires,at least it’s me and other people who are going to be involved in whatever message or call or what have you and if only one party pays, the other party is not benefiting from privacy either, and the example of e-mail is a good one. There are great services for private e-mail like Proton, but Proton is also interoperable with other major e-mail services like Gmail or Outlook which means that if I e-mail somebody on Gmail from my Proton account, Google has all of that information shared on their servers, right? Similarly, if I were to pay for privacy, I’m paying the subscription fee so that you don’t collect my data, say, but I e-mail somebody or I connect with somebody, communicate with them in in some way and they don’t pay for that, I would also not be kept private, so you know communication networks are really something that shows us just how interdependent we are when it comes to privacy. And some of these standards, that a “pay to play” model breaks down pretty quickly when it comes to communication. 00:21:25 Domen Savič / Citizen D Speaking of networks and big players in this field, we already touched upon Google and its monopoly in the communication digital communication sphere, it seems to me that on the one side you have an issue with privacy, personal data, all of these dangers that we’ve talked about, but on the other hand you can see Google, Apple, Amazon and other big intermediaries almost becoming essential services to the netizens and to basically everybody around the globe. So how do you see or what would be in in your words the best way to dismantle this or to start doing things differently on a universal scale? 00:22:19 Meredith Whittaker / Signal Yeah, well, there isn’t one answer to that question. In my view, in part because these services have infiltrated and become core infrastructure for so many heterogeneous functions of daily social and economic and even governmental life. Imagine trying not to use Google or Amazon at your job? Imagine having no social media presence and trying to apply for a loan or get government benefits, right? When we begin to engage in those thought experiments, we begin to recognize just how deeply these services have infiltrated so many domains across the globe, and so I think to answer those questions, we need to get a little bit more granular. We need to ask to efficiently fulfill the role of governments in these core services is, what type of technical infrastructure that is not controlled by monopoly actors? Would we need to be able to govern technology in a way that is more democratic? What type of governance structures do we need to put in place that allow participation at the level of design and features and how a technology behaves, how it respects or fails to respect fundamental rights. And those are questions that are ultimately very exciting and I do think we are in a time when there’s no longer any debate over whether this business model is good or bad. You even have institutions like Combinator, which has done, you know, arguably as much as anyone to cement and promote the toxic Silicon Valley business model, is now coming out and saying like, “Hey, we’re actually not very into big tech. We’re looking at little tech now. We want to promote the, you know, the small players!” and whether that’s good faith or not, I think that really shows us that there is a sea change in terms of sentiment and that there is an opportunity to think through. How would we dismantle and disarm the centralized power that is held by these platform companies? You know, what would independent cloud infrastructure look like? What would independent communication networks look like, what would interoperable protocols that enable more flexibility and independence at the application layer look like and how do we find the capital to fund these things and maintain them forever over time? And how do we put in place governance structures that don’t behave like the boardrooms of big tech with their focus on profit, revenue and growth over everything else, but have more civic minded duties and processes that are working to aerate the tech ecosystem and make it more amenable to building technologies that actually serve beneficial futures. 00:25:46 Domen Savič / Citizen D Who should be the driver of this debate, of this big rethink? It sometimes seems that this debate sort of formulates in one area, let’s say one part of the industrial ecosystem, one company, sometimes one person thinks about a concept and then others just parrot it. And when you listen to political debates, when you listen to media discussions, it seems that a lot of people aren’t thinking about this, they’re just copy pasting arguments and counter-arguments for a specific case. 00:26:33 Meredith Whittaker / Signal There isn’t one actor who should be driving this because what we’re talking about is a massive scope. Scope that is affecting people in heterogeneous domains across the globe, where we are in a situation where what Facebook means in Myanmar will be very different than what Facebook means in Stockholm, for example, you know the use of TikTok in Nairobi may look very, very different from the use of TikTok in Connecticut. So, I think in a sense that when I say we need to get down to a more local level, that’s not fetishizing the small and the local. That’s really recognizing the function of these platforms, the role they play vis-à-vis government services, vis-à-vis commerce and communications really does vary across contexts and that the people in those contexts are almost certainly best positioned to answer some of these questions. We don’t want to repeat the mistakes of one size fits all billion-user platforms, but simply do that kind of interventions right, because there is no one-size-fits-all. 00:28:09 Domen Savič / Citizen D And do you see a danger of activism fatigue or even personal fatigue in this regard, because these problems or these systemic pushes are just too massive and are taking a toll on our basically health, right? So, do you see that as an issue in terms of that the NGO or the civic sphere are not replicating fast enough to counter the industry and the political pressures or pushes in this area to be an effective counterweight to privacy versus security and everything else, we’ve talked about? 00:29:11 Meredith Whittaker / Signal Yeah. Well, I certainly do see that. And I think we do need to look long and hard at the political economy of the NGO sphere and this whole society sphere because you know what I’ve seen? I’ve been in this work for almost 2 decades now and I’ve really seen it sadly is often very susceptible to trends. In the US, which I know is different than many other places, but it’s also generally funded through philanthropy, so you don’t have long term sustainable funding in most cases you are at the whim of whatever foundation or your donor might think it is important at that moment, and of course that is also susceptible to trends and to whim and to hype, which makes it very difficult to pursue a long term strategy, particularly when you are rowing upstream against vested interests that may frankly, may have a lot more access to some of the leaders in philanthropy than some of the activists who are on the ground doing real work but not being seen and appreciated. So again, I think the political economy of NGO work and civil society needs a lot more scrutiny. And I think we need to be a bit bolder in frankly demanding the kind of support and capital that we need to do this work. And there’s a lot of really good ideas out there, really good architectures, incredibly brilliant thinking around how we could build tech differently, how we could build more respectful tech, but an idea is not the solution. An idea is a possible template and what is not generally understood or let’s say respected is just how much work and how much money it costs to build reliable tech. It’s never just built once. This isn’t two guys in a garage, who come up with something genius and the world changes. No, it’s two guys in a garage, a really good idea and then billions of dollars of capital and hundreds of thousands of hours of labor who make that idea real, to maintain that idea in a volatile and dynamic environment and to do that forever or until that idea dies, or that tech doesn’t exist. I think we also need to reframe our understanding of tech and recognize that we can’t have Sam Altman be the only one who’s talking big money, right? We also need to recognize that we’re serious about this change. We need to be at the table, and we need to be demanding a cut of that. 00:32:19 Domen Savič / Citizen D Part of that issue is also this illogical mantra about tech being neutral and non-political. That it sort of just flies out there in the world and smart guys are gals are just reaching for it. So, do you think that’s one of the stereotypes that prevents us from moving forward or think differently about the whole development cycle and uses and misuses of information technology nowadays? 00:32:57 Meredith Whittaker / Signal Oh, absolutely. And I think that mantra or that narrative that often couches highly political decisions in the veil of objectivity is not new to computational tech. If we are going a little far afield, there is an enlightenment era or a paradigm that saw certain forms of knowledge and certain forms of certain people as neutral and objective and able to discern and represents the facts in a way that was simply stenography of the world. And this is a paradigm that has been very, very well studied by a number of scholars looking at the way this guise of neutrality was used to mask often very political and very brutal regimes and determinations. We can look at things like race, science, that couched structural inequality as neutral biological destiny, they were just observing differences and then determining what those differences meant in ways that were pernicious and harmful for the world. I think we need to question narratives of neutrality to begin with, and then particularly in tech, I think that this has been a conflation of computational technology with scientific progress, which has been promoted by the tech industry. The reason we’re all suddenly using Google or we’re all hosting on Amazon is not because those companies were successful during the primitive accumulation stage of tech but is simply because what they discovered is that significant scientific advance and they are introducing that to the world and as such, they bear no responsibility for that as such, that what they are doing is neutral and inevitable, it cannot be changed and as such, if you were to question it or if you were to say desire to regulate it in a way that wasn’t beneficial for those companies, you are anti-progress or anti-science, you’re putting your finger on the scales of human advancement, and I think that narrative has done as much as anything to really chill our ability to grapple with and meaningfully regulate these technologies over the past number of decades. 00:35:49 Domen Savič / Citizen D It is funny because when you talk about using Signal or be a Signal supporter, it seems that especially in in the political arena, there’s a bit of a hypocrisy going on. So, you have, at least in Slovenia, but I bet all over the world, you have many politicians, MPs, ministries and so on using Signal for their own communication efforts, but at the same time they don’t see a problem in the Signal being the solution to whatever problem they’re trying to solve with it – usually the privacy of communication. So how do you see the current political arguments going on both sides of the Atlantic regarding the privacy on one side and everything else on the other? 00:36:50 Meredith Whittaker / Signal Well, I, I mean, I don’t think the hypocritical, but nonetheless, the desire to have it for me, but not for thee is very new to politics. Right? And that’s basically what I see happening here, “Yes, my privacy is important, but you must understand as a responsible steward of our society, I need to invade your privacy.” You know, these are people who imagine themselves as always, in a position of power, and thus don’t generally question in the type of levers of power that we are creating and the way those could be misused, if somebody with more pernicious intentions were occupying their seat, so, I think this is an age-old pathology and it’s why we need to hold anyone who has a position of power to incredibly stringent standards and recognize that it’s really not personal, but that if you are going to take that kind of responsibility, you need to be held accountable and the people who are worthy of that responsibility should be embracing that. 00:38:05 Domen Savič / Citizen D And if they don’t? 00:38:10 Meredith Whittaker / Signal Well, if they don’t… read history. There are many, many, many things that can be done if they don’t, but I don’t have a specific example. I just think that you know, they’re clearly not suitable for the responsibility of the office, they have endeavored to occupy. 00:38:34 Domen Savič / Citizen D We’re slowly wrapping up, so for the last part, I just have a few, let’s say Signal specific questions. So first off, could you describe your design process when deciding on a new feature being implemented or not being implemented? You hear a lot about Signal being offered on Graphene OS as a native app or you should not be using telephone numbers as an ID. There’s a lot of different ideas but how do these ideas get streamlined into the design process? 00:39:18 Meredith Whittaker / Signal Well, we are very serious with reading and thinking about the feedback and the ideas that come from the massive community of people who use Signal. And we’re also serious about ruthlessly prioritizing so that we can focus our, our small but mighty energies on the things that really matter. So, there are many, many things, I would say 99% of the things that we would like to do are not things that we choose to do, because we do really value focus and we balance, we think long and hard about new features, we think about whether we can build those features in a way that that meets our very strict privacy bar, we think about whether those features are useful to people. Are the features that are common in communication apps in one region or another? When people pick up Signal to use, it is the absence of that feature something that is going to make it feel like it’s incomplete or like it’s broken or that people can’t use it? For example, Signal introduced Stories a couple of years ago, similar to Instagram, but our Stories are actually private, right? And while they’re not a hugely common feature in the US, they’re massive in South Asia and in Brazil and we were hearing from people that pick up Signal and it feels broken, because it doesn’t have this feature that has been a core way of communicating among the people who use Signal there. So, it’s a lot of conversations, a lot of collaboration with our Chief Product Officer Clancy Childs, who is very brilliant and very experienced and has been working on messaging now for over a decade, who really has a lot of instincts there and we try to do some market research. We do don’t collect user data, we don’t collect telemetry and analytics the way almost every other communication service does, so we often don’t have or we almost never have, the kind of signals that our competition does, but we do have other ways of fetching information and doing user research in the field that gives us a sense of how are people using Signal, what might they enjoy and we go from there. 00:42:08 Domen Savič / Citizen D What are some typical technology limitations you have to jump through or overcome when you’re deciding to improve Signal or how to sustain the current quality level of the service? 00:42:47 Meredith Whittaker / Signal As a rule, doing things with strong encryption takes more resources, computationally it is slower. It often means that you can’t do the obvious thing, you have to do a work around so for example, we’ve talked about in the past is something like gif search. People want to be able to send reaction gifs on Signal. This is again one of those features that people feel is really missing if it’s not there. The company Giphy was collecting all that data and then was acquired by Meta, so we weren’t going to simply shove in a Giphy library and call it a day, we had to do a lot of work that involved some pretty deep re-architecting of parts of our system in order to make gif search available to people in a way that met our privacy bar. Now the end result for a normal user is gif search, right? It looks exactly the same as if we just shoved it in there and said “Hey, we’re giving all the data to Meta”, but in fact we spent orders of magnitude more time, creativity and rigor doing that than the competition. 00:44:14 Domen Savič / Citizen D You were repeatedly quoted that Signal will not be available in the markets that will pass privacy-invasive legislation or back doors for the good guys. What do you think it will happen by the end of the year or in 2025 in this area? Are you seeing some movement that politicians are sort of backing off of these terrible ideas or do you see do you see the opposite? 00:45:04 Meredith Whittaker / Signal Well, I am not a wizard or a psychic, so I don’t have a clear prediction. I do think more and more that politicians are hearing the consensus from the expert community, the arguments that we and others have put forward. Now, I don’t think those arguments alone are going to stop the push from some who don’t really care about privacy, they just want the ability to surveil. But what I can guarantee is that whatever happens, we will continue to fight and our position is not going to change. We’re simply not going to comply with any regulatory or legal provision that would force us to implement a backdoor or otherwise undermine the privacy guarantees that the people who rely on us rely on often in life-or-death situations. 00:46:02 Domen Savič / Citizen D And just one final question. So, I’ve labeled this part controversies in order to sort of encompass everything that has been floating around the Signal app and service around the web. So, from rumors that the Signal is supposed to be a CIA front, that it’s not secured, that is being mismanaged or something like that. So, is there a way for the company to sort of present itself as a trustworthy? We already said a few words about that in the beginning, but I’m curious to hear your thoughts on users choosing between Signal and Telegram or other services, while in fact not having the necessary technological experience to sort of really go into the the nitty gritty of individual services and sort of decide for themselves? It often seems like the choice for privacy, it almost seems like a fashion statement or a fashion choice, right? You either wear blue or red. 00:47:20 Meredith Whittaker / Signal Yeah, but it’s sadly much, much more serious than that, and we do see misinformation campaigns very frequently that we believe are attributable to in some case nation states who would really prefer that people be using a very insecure communications app like Telegram over Signal which is not secure. How do we put the disinformation to rest and how do we push back against it? Well, we really do go above and beyond. We are open source. You can validate our claims if you have the skills, you can go into our repos, there are people who watch every commit we do and analyze it on Reddit, looking for new features, discussing what we’re doing, submitting bug reports, so we are one of the most scrutinized apps and one of the most scrutinized and audited cryptographic protocols in the world. How does that translate to a popular message? Well, we are out there talking about Signal security as much as we can. We spend a lot of time when these campaigns of disinformation that are saying that Signal is a CIA asset or whatever the nonsense is. We spend a lot of time pushing back, even though we know this is fully fabricated and kind of, I would say almost lazy. We also recognize the stakes are really high, that there are people who don’t have the expertise to validate these claims themselves and can get really worried by those claims. So, we also ask that the overzealous community of people who may be amplifying those claims or making them in service of getting attention or going viral. please find something else to do, because digital security is life and death for a number of people who use Signal in authoritarian context, and these kinds of rumors can have real harmful impacts on people, even if they are completely baseless. 00:49:54 Domen Savič / Citizen D Well, I think that’s all for us today. Thank you so much Meredith, for taking the time and thank you for all the answers and all the expertise. Best of luck going forward, fingers crossed we at some point reach the situation where Signal won’t be necessary anymore because everybody will understand the importance of privacy, surveillance capitalism will die a horrible death without or with very little victims and life will be great again. 00:50:32 Meredith Whittaker / Signal Well, from your lips to God’s ears. Thank you, Domen, thank you for having me on the show. It’s been great. Citizen D advice: Use Signal Engage in a political debate around privacy Rethink. More information: Time100 AI: Meredith Whittaker – article Signal president Meredith Whittaker criticizes EU attempts to tackle child abuse material – article ‘You cannot do mass surveillance privately, full stop’: Signal boss hits out at government encryption-busting moves – article About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!
Sep 15, 2024 · 36 min
102 mag. Andrej Tomšič in “magičnost” javnega videonadzora
V jesen zakorakajmo z videonadzorom javnih površin in trendih na temu področju. Z namestnikom Informacijskega pooblaščenca, mag. Andrejem Tomšičem v pogovoru predebatiramo zakonske varovalke zasebnosti, lažno dilemo med izbiro varnosti ali zasebnosti ter premišljujemo o politični komponenti videonadzora ter zakaj na področju videonadzora še vedno pristajamo na nedokazane prednosti videonadzora na področju varnosti, hkrati pa se avtomatično brez pomislekov odpovedujemo lastni zasebnosti. V pogovoru še o domačih inšpekcijskih praksah, pravilih transparentenega objavljanja lokacij nadzornih videokamer in mednarodnih trendih na področju videonadzora ter vedno bolj invazivne tehnologije in vloge civilne družbe ter posameznika, ki ima veliko moč pri oblikovanju družbenega diskurza o tej temi. Državljan D svetuje: Zahtevajte poročila o učinkovitosti in funkcionalnosti javnega videonadzora Podpišite našo peticijo MOL za transparenten videonadzor Opozarjajte na lažno dilemo med zasebnostjo in občutkom varnosti Dodatne informacije: Only 5,000 of 7,500 CCTV cameras installed under Safe City project working in Bengaluru – članek Cost-Effectiveness of CCTV Surveillance Systems: Evidence from a Polish City – študija Javna razprava o občinskem videonadzoru – posnetek razprave O podcastu: Podcast Državljan D je podcast za produktivno preživljanje časa, v katerem igramo vlogo državljana. Državljan D v pogovorih s strokovnjaki določenega področja informira in aktivira. Da se. Gremo naprej! Financirano s strani Evropske unije. Izražena stališča in mnenja so zgolj stališča in mnenja avtorja(-ev) in ni nujno, da odražajo stališča in mnenja Evropske unije ali Evropske izvajalske agencije za izobraževanje in kulturo (EACEA). Zanje ne moreta biti odgovorna niti Evropska unija niti EACEA.
Jul 15, 2024 · 25 min
101 Sacha Altay and the misinformation circus
Sacha Altay is a post-doctoral fellow working on misinformation, trust, and social media in the Digital Democracy Lab at the University of Zurich. We sat down with him to discuss the perception of disinformation, the failed attempts of self- and co-regulatory frameworks that try to limit the its spread and the way we should be addressing this problem. Transcript of the episode: Expand the transcript 00:00:06 Domen Savič / Citizen D Welcome everybody. It’s the 25th of June 2024, but you’re listening to this podcast episode of Domen Savič / Citizen D podcast on the 15th of July same year. With us today is Sacha Altay, postdoctoral Fellow at the University of Oxford within the Reuters Institute working on misinformation, trust and social media. So of course we’re going to talk about football. That’s a little opening joke. So welcome, Sasha. Thank you for being with us. 00:00:38 Sacha Altay Thank you. Thank you for having me. I’m not with the Reuters Institute anymore, now I am at the University of Zurich. I was, you know, in Oxford last year. 00:00:45 Domen Savič / Citizen D OK, excellent, things change so fast. And speaking of you working in the field of psychology and disinformation and trust in social media and so forth. It seems that our world nowadays seems to run on disinformation in various areas. You have disinformation in politics and economy and environment and public health. There are numerous attempts in the EU, in the US, all around the world to sort of level out the playing field for the media consumer, you have regulatory attempts, self-regulatory protocols, increased efforts in education. My opening question to you would be how did we get here? Was it always like this or did something change in the recent past so that disinformation became so prevalent and so influential in so many areas of our lives? 00:01:49 Sacha Altay So, I’m going to answer this question first by talking about how people talk about it rather than whether there is more disinformation or misinformation today than before. Just how do people talk about it and whether people talk about it more today than before? And I think it’s pretty clear when you look at the scientific literature or the number of news articles published or Google searches, that people are more interested in myths and disinformation or conspiracy theories now than before. And by now, I mean broadly, after the 2016 U.S. presidential election and the Brexit in the UK. After these two major events, interest in misinformation, disinformation and conspiracy theory really spiked both in the news headlines, in intellectuals’ circles, as well as in scientific research. And I think clearly the COVID-19 pandemic, where the director of the World Health Organization, said that, you know, there was an infodemic, like a lot of misinformation about COVID-19 etc. So, I think interest also piped during that time and more recently very recently with the release, or at least the democratization of the ChatGPT and worries about the power of generative AI. There have been new fears again about the impact that generative AI may have on elections. For instance, during the 2024 elections that are being held almost everywhere around the world. So, I think, yeah, clearly people are more worried about it. Recently at the World Economic Forum, for instance, missing misinformation was considered the number one risk in the next two years for democracies in front of climate change in front of war in front of any other risk. So clearly people, scientists and leaders are very worried about it, and I think it’s unprecedent. But of course, we don’t have very good data. About 100 years ago, we were also extremely worried about this stuff. I doubt it, but if it’s possible, we don’t have very good data on it. But let’s say at least that yeah, now we are very worried about it, more than in the past as we can document it. 00:04:13 Domen Savič / Citizen D Would you say that these threats or the perception of threats of this and misinformation are credible? So, is it really that big of a problem then, like the media and the politicians and everybody I guess is saying? 00:04:28 Sacha Altay Yeah, because I’ve talked about perceptions. Now let’s look at the evidence. Let’s say that before 2016 there was some work on it, but the work was quite limited compared to today, and I think that since 2016 there has been a lot of great empirical work to look at. The prevalence and impact of miss or disinformation, and most of this work, at least in Western democracies, like the US or Western, Europe has shown that mis- and disinformation is very, very small. It’s consumed by a very small number of people who have pre-existing attitudes that basically predispose them to consume and accept the messages in the mist and disinformation. So, in the US, for instance, we know that it’s mostly, I don’t know, for Trump supporters that are consuming mostly for-Trump misinformation, and you can say the same for the other. It’s the same the other way around, so that’s what we know because we know that the average news consumer doesn’t consume or even stumble upon much misinformation. So that’s that has been well established. And regarding the impact, it’s a bit trickier, but all the attempts we’ve done suggest that the effect is small and smaller than most other things like even just following the news like of course, when you follow the news, you get more informed about what happens in the world. But you also may develop some biased perceptions of the world because of course, the news doesn’t cover everything perfectly and they are not completely neutral etcetera. So let’s say that the impact of mis- and dis-information is very small compared to just media bias effects and of course a lot of people just don’t consume the news and just not very interested in the news or politics and so are just broadly, uninformed about many of these things, and this has a much stronger impact than misinformation could ever be. 00:06:24 Domen Savič / Citizen D So, it sounds like we don’t have a problem there. 00:06:30 Sacha Altay I mean the way I see it is that we have problems. For instance, I don’t know, we have people who deny that climate change is happening and that it’s human cause. And I think it’s a problem that people disagree with, that it’s a scientific fact that has been established people disagree with. It’s a problem. And these people will also say that they believe in misinformation, etc. And so, I think often people jump to the conclusion that “people who vote for populist leaders also believe in fake news”. It must be because of false news that they vote for populist leaders or like the same for Brexit, for Trump, etc. All these people, they say they believe in fosters. And so, I think we tend to attribute this bad stuff to false information, and I think false information is most often a symptom of other problems. For instance, we know that at the country level in countries with more corruption like I don’t know, countries in the Middle East compared to northern Europe, countries in the Middle East are more corrupted than countries in northern Europe like Denmark and belief in conspiracy theories is much higher in like the Middle East than in Denmark, for instance. And that’s because it makes sense to believe that elites are corrupt or competing against people in corrupt countries. So, there is some rationality to it. And we also know that people who distrust institutions for various reasons, some good, some of the, the less good, are more likely to believe in misinformation, conspiracy theories… literally because they are looking for information that goes against the establishments against institutions, and sometimes people are warranted to do so. But let’s say that in Western democracies, where elites are often right, it leads to bad, bad outcomes. 00:08:15 Domen Savič / Citizen D So, you would say that that this whole, let’s say one of the main reasons or important reasons for the prevalence of let’s say belief in disinformation is actually decrease in trust towards let’s say public institutions, governments, mass media outlets and so forth. 00:08:37 Sacha Altay I think it depends. I think, I don’t know, for instance, in times of war, I don’t know if you look at Russian propaganda, some of the best predictors of believing in Russian propaganda is identifying strongly as Russia. And I don’t know, believing in the Great Russia narrative, for instance, that you want a great Russia. So, you’re going to buy the Russian propaganda. But if you’re Ukrainian and have a Ukrainian identity, you’re not going to believe any of the of the Russian propaganda. So, in that case, it’s mostly about identity and I think most of the time identity plays a very important role. People believe stuff that aligns with identity, people and various identities. You can have political identity, national identities, many kinds of identities. But yeah, as you mentioned, I think at least for conspiracy theories, they are often constructed really in opposition to events that are covered in mainstream news. And they very rarely come up with their own stuff. Most often they just look at what’s happening in the news, and they say that’s false. It’s actually something else that’s happening. But what they mostly do is they wait for mainstream news to do something, and then they say, oh, it’s actually the opposite. And so, I think in that case, for conspiracy theories trust in institutions is, yeah, a very strong picture. But if you look at other stuff, like for instance naturopathy or alternative medicines, then it’s mostly distrust of health institutions, not political institutions for instance. But yeah, trust is key to understanding belief in misinformation. 00:10:11 Domen Savič / Citizen D Do you have any thoughts on the general distrust phenomenon? If you look at the way or if you look at all the areas that disinformation is rampant in, it’s basically you know, we don’t trust literally anything, right. We don’t trust the as we said, the governments, we don’t trust the media, we don’t trust the researchers in the field of yeah, environment and other issues. So why is this mistrust so prevalent? 00:10:46 Sacha Altay I don’t know. I think it’s complex. It depends on the context. I think one potential explanation is that it’s hard for us to evaluate how trustworthy many institutions are, even evaluating how competent some people are. For instance, when I tell people that they do behavioral sciences, a lot of people don’t even know that it exists. Basically, you can study human behavior in a scientific way because for them science is mostly about, I don’t know, biology, geology, physics. And so, I think now the division of cognitive labor, like how experts, people can be, is extremely high in today’s society. So, I think we have trouble understanding, you know, how experts. Some people are on vaccines, on GMOs, on nuclear energy or stuff like that, and that it really goes beyond our own experience and that basically with our own eyes. Or our own brain, our own experience, we cannot come up with conclusions to write about this and we need to trust other people. But then we need signs that we can trust them, and often we don’t see these people, they don’t talk to us. They are far away. They may be anonymous. They may have weird names. And so, we I think it’s hard to trust people that far away and so that may be one reason but honestly, its they are there are many different reasons. For instance, during the COVID-19 pandemic, the authorities in many countries did not communicate well about guidelines about scientific evidence, and so it forced the distrust. They also had some measures that were very restrictive and then not restrictive, and all that a lot of people didn’t like that, and it affects trust, so there are there are many ways to lose trust and one thing is that it’s easier to lose trust than to gain it. So that’s why also institutions etcetera need to be careful because they can very easily lose people’s trust. 00:12:45 Domen Savič / Citizen D So, what does that tell us about, let’s say the countering of disinformation pandemic or infodemic. So, are these attempts of, let’s say, regulation, co-regulation, fact checkers, everything that’s been going on since, let’s say, yeah, since the election of Trump… are these countermeasures sort of focusing on the right problem? Are they addressing the issue or are they mostly greenwashing? Greenwashing in the simplest of terms, saying “Ohh, you know, people are dumb because they believe disinformation, and now we just have to give them knowledge or science and everything will be OK.” 00:13:33 Sacha Altay I mean, I think these people mean well most of the time and in my opinion, I think many of these measures like fact checking labels etc… They mostly target the symptoms, and they can help a little bit. I mean I’d rather have sack checks than have no fact checks. I think it’s a good thing that they are fact checkers, it’s just that we need to not lose track of the bigger picture and remember that fact checks are effective at correcting misperceptions, but they are they are not effective at changing people’s minds about who to vote, how they feel about some politicians and so that that that’s a problem like you can fact check as many as Trump’s statement as you’d like, people are still gonna vote for Trump if they have some pre-existing values, attitudes and reasons to vote for Trump, and we need to understand why some people are attracted to populist leaders to, I don’t know, people who prone anti vaccine stuff. We need to understand the reasons why that’s the case and so that that way I’m just worried sometimes when leaders think that, yeah, if we address the disinformation problem, the deeper problem are going to go away, like populism is going to go away with fact checking. I mean, no other thing is really defending that. But I think we need to keep track of that there are some deeper factors that affect this, and we need to target them. Some of them may be, yeah, lack of trust or polarization, just how people feel about political opponents, etc. And then of course, there are some even deeper factors that I mentioned. Corruption, inequalities, poverties. There is stuff like that that do affect belief in misinformation. And of course, I’ve been advocating to target this because if we reduce inequality or if we reduce poverty, it will have all the benefits, then reducing misinformation. 00:15:27 Domen Savič / Citizen D But that’s hard, right? 00:15:30 Sacha Altay Yes, of course. But I think it’s something we need to be clear about is that there is no easy solution. There is no technological fix that will solve these problems. There is no magic solution or whatever, like it’s a tough problem because social political problems are often very complex and populism, anti-science attitudes et cetera are not going to go away with fact checking or media literacy. 00:15:54 Domen Savič / Citizen D But so. So, we’ve just had the EU election, right. And did you feel… I’m coming from Slovenia, you’re now based in Switzerland. Did you feel the debates, the discussions before the election sort of highlighted these issues that are that are that are underlying the disinformation pandemic? Did you have the feel of political representatives really knowing what’s going on in regard to as, as we said before, crisis, lack of trust in in several different areas for them to sort of change the way we’re addressing this issue as we move yeah, into a new European Commission mandate? 00:16:42 Sacha Altay I’m not sure. I’m not sure, to be honest, I’ve not followed it very closely, but I’ve seen a lot of technological solutions being proposed. I mean, I’m not thinking about the elections right now, but stuff like adding labels on social media to say that it will hurt your mental health or holding tech companies accountable and it’s good to hold them accountable. But I think a lot of solutions are a bit too focused on technological fixes and not enough on the functioning of democracies, institutions and deeper factors, but often as politicians, it’s easier to blame Mark Zuckerberg or big tech and say that you’re going to go against big tech than to say that the problem is politicians lying or pushing lies or instrumentalizing some facts or political. 00:17:35 Sacha Altay I think it’s easier for them to blame big tech than themselves and the system, so no, I’m not sure there has been really a wake up call. 00:17:50 Domen Savič / Citizen D And what are your thoughts on on exactly this issue of this techno deterministic solutionism, that tech is is the problem, but at the same time, it’s also the the grand solution to to every problem that that we have what what is fueling this idea or this way of of proposing solutions in in in this area? 00:18:17 Sacha Altay I mean, I’m not totally sure I’m more and more interested in in this area and the more I read about it, the more I realize that we have always done that as humans to blame new technologies for so many problems, like we’ve blame writing for losing our memory. Like if we if we write, we don’t need our memory anymore with blame books for disconnecting us from reality, we blame every new technology for many reasons that now we would consider very silly. But at the time we took it, at least some people took it seriously and pushed it. But no, I’m fascinated by why we do so. In France, for instance, at the last year’s election, the extreme right party did well and a lot of commentators they’re saying that it’s because of TikTok. It’s these stupid kids basically on TikTok being influenced by a young leader and of course it’s a very simplistic explanation that contradicts most of what we know about the effects of mass communication and how people use new technologies and how people decide for who to vote. But for some reason these are very, very attractive and we tend to always blame the same people. It’s like young people who have this weird new information technology, tick tock or whatever and that may explain why they are different from us and do stuff that we don’t understand. So, I’m not sure why there is this focus on technology… there are many theories. Some say that yeah, it’s intentional. It’s like politicians, for instance, pushing these narratives so that we don’t blame them, but I think also these explanations tend to be quite intuitive, like a lot of people don’t really know what TikTok is. They think it’s just young people who dance. And if suddenly you have politics, then people may be influenced by it. Because in psychology there’s this well-established finding that we tend to overestimate how much people are influenced by bad media effects like advertisement, propaganda, etcetera? We tend to say no, I’m not so much influenced by it, but other people, and especially by political opponents, are extremely influenced by it. So, I think, yeah, we have this tendency to think that. So, these explanations are very, very intuitive and most people of course are not super interested in the truth. So, for them the job is done. You know, why did Trump get elected? Well, because they watch Fox News and they are a bit stupid, and they’re not very well educated and then they hope this is the end of the explanation. And so, the problem is Fox News and of course Fox News maybe a bit problematic, but removing Fox News will not remove populism. I think it’s also a bit of laziness to some extent. 00:20:55 Domen Savič / Citizen D Hmm. So, addressing this issue moving forward, then we can also talk about not just the co-regulatory practices or the regulatory practices, self-regulation, whatever, but also about media literacy, about training, about changing the way people interact with media outlets or with the social media and other informational faucets so. So how would how would you what needs to be done or what would you do to sort of change, change the tie the tide of this, yeah, of this stalemate where you always have, you know, fact checkers playing whack a mole with, with the disinformation spreaders. 00:21:44 Sacha Altay I’m not sure. It’s also why a lot of people don’t like my work is that I also don’t have solutions to to offer because the problem is very complex and I’m just advocating for realization that the problem is complex, then the solutions that a lot of people are offering that are very, very small set and are not going to save the world. But at least when it comes to media literacy, I think a lot of the media literacy programs, they assume that people are gullible and too trusting, whereas we know that a lot of people are either just completely uninterested in the news and avoid it altogether, or do not trust the news and start from basically cynicism instead of gullibility. And so, I think it would be perhaps more interesting to try to foster interest and trust in reliable information, not necessarily the news, but also some news influences, Wikipedia or high-quality sources of information in general, rather than to alert people about misinformation like they did during the COVID-19 instead of saying there is misinformation everywhere, be careful, say look there is reliable information and you can find it here and you can trust us for these reasons and we are being transparent and we are being accountable etc. So yeah, I’ve been advocating for that for this shift in focus. But again, I don’t think it’s going to necessarily have huge effects. It’s just a small, small change intervention that already has small effects. 00:23:21 Domen Savič / Citizen D So, is there a cause and effect of these underlying reasons that are that are then birthing the era of disinformation? Would you say, if you look back at, let’s say historical developments moving, past Trump, but by going further down the line? What caused this or what are some of the happenings that pushed us into the situation that we that we have today? 00:23:55 Sacha Altay I mean, I’m not sure, to be honest, I’m not exactly sure it’s often very complex and I’m not an expert in the rise of populism in the US or the rise of anti EU sentiment in in the UK. It’s some people are experts in this domain and I don’t think they have clear answers. So no, to be honest, I don’t know. It’s just complex. I just know that information and information practices often downstream of attitudes and values and political identities and stuff like that and so, they rarely play an important causal role. They are most often there to, like, rationalize people’s attitudes and behaviors. But no, I’m not sure why populist leaders are rising in the world, and why Trump got elected, and why the Brexit happened. It’s complex, I’m not sure. 00:24:53 Domen Savič / Citizen D Do you think it has something to do with role separation? So this is partially my theory, or this is something that I subscribe to, looking at this field from, let’s say the disinformation researcher, but also from someone who’s focusing on media literacy and is interacting with people on on this issue. Coming from Slovenia, let’s say from the Balkan region, Central Europe, whatever it is, it almost seems like that that on one side nobody wants to talk about political engagement or the role of a citizen, because this has some bad connotation moving backwards to Yugoslavia or the east versus West. But at the same time, everybody’s emphasizing this individual role of a consumer that you are in charge of everything. You vote with your wallet basically, and you’re the one who’s making things happen, right. And then at the same time, every time something, let’s say, bad happens, everybody’s escaping the responsibility or part of their part of responsibility of certain issues, be it surveillance, be it disinformation, be it anything or everything else that’s happening around us. 00:26:20 Sacha Altay I mean, I’m not sure, to be honest. Because you’re saying that there’s been a shift in seeing consumers, the information consumer…yeah, I’m not so sure. 0:26:40 Domen Savič / Citizen D It’s just, thinking about it, seeing how how people on one side they recognize that the problem, let’s say of disinformation or issues related to this information s very broad and very composed out of different outlets and at the same time they want, as you said, this techno deterministic solution. Just push of a button and and everything will go away, right? And then maybe if we if we move on to let’s say the relationship between politics, between party politics or governments and disinformation, would you say there is a link between those two? Or should we focus or should we talk about this information in connection with, let’s say, political agendas and other? Yeah, other yeah, with things related to politics. Or is this completely separated? Are these two issues completely apart? 00:27:50 Sacha Altay No, no, of course they are very much related in the case of disinformation. So like false or misleading information spread with the intent to cause harm. Very often it’s spread by government or like foreign governments, for instance, doing like information operations and so they are clearly political. Like, I don’t know, increasing polarization within the society, reducing support for some states, or increasing support for some other states. So there are some clear political actors behind disinformation campaigns, that that’s for sure. In the case of misinformation, not all misinformation is related to politics. But let’s say that scholars have focused more on political misinformation than other types of misinformation. For some reason, it’s not totally clear why some people think it’s more impactful, even though others have agreed. I have argued that health misinformation is also probably problematic, and that we should focus a bit more, but yeah, a lot of misinformation, at least misinformation that matters, and that people see and that has the potential to be impactful is spread by politicians because they do have a wide coverage, and sometimes they even do so in the media. When Trump was in power in the US and he clearly spread a lot of falsehood through mainstream media because he’s the president and the news have they have to cover him and sometimes he tells lies and that was a difficult situation, but of course, in general politicians instrumentalized facts or spread lies to gain supporters to political gains in general. So, and of course, not all politicians do that, but many people have agreed that when politicians do it, it’s more problematic than when a random user with 300 followers does it because of course the impact is not the same, and because politics can have an impact even if people don’t really believe it. For instance, if Republicans in the US say that they are against masks, then Republicans in the US can use a mask as an identity factor. They can say, oh, I’m not wearing masks so that I identify as a as a Republican and then it becomes this bookmarker, this party line and then people wear, not wear masks, just to identify like that. Even if they don’t really, even if they have nothing against masks. So that’s problematic. 00:30:30 Domen Savič / Citizen D And focusing on, you’ve mentioned it a few times, the role of mass media, right? So, on one hand you have the theory of media watch, public watchdogs of the fourth estate. On the other hand, you’ve just mentioned that you have them as megaphones that are amplifying, let’s say the dis- or misinformation conspiracy theories. How do you see the role of mass media in all of this? Are they friends or enemies of the people? 00:31:02 Sacha Altay I mean, first I’d like to say that the news media plays a very important role still, despite the rise in social media, despite the advent of like the Web 2.0 and everyone being able to be a content creator, etc. We know that when people consume news, they still are mostly turn to mainstream media so they still play a very important role and then the question is, are they friends or enemy? I think it depends on the context and the country. I’d say that in Western democracies where there is a strong public media ecosystem that’s free and has money to do good work in the UK with the BBC, for instance. Then I think they have a good media ecosystem and that the news is mostly friend, even though for instance in the UK there are also a lot of tabloids that probably don’t help people being informed that much if not create a bit biased perception. But overall on average, let’s say that in countries like the UK, the news is a friend in the sense that on average they help people being more informed about what’s happening in the world and perhaps make more informed decisions and be better citizens. But then in other countries, I don’t know, like of course, obviously like China for instance. Many news outlets are controlled more or less by the state and are not totally free to do what they want or like in Russia, or to some extent in in India. In these countries it’s not totally clear whether the news is a friend or an enemy. Because if you’re if your friend of the government and the government is a is a dictator dictatorship, then of course you’re not serving the people, but you’re just serving the the the regime. And in countries like the US, for instance, where they don’t have a strong public service media on average, let’s say it’s all right following the news in the US is better than totally not following it, but it’s also less good than than I think in other countries like like the UK or France. So it depends. And of course there are some good news outlets. There are some bad ones and and and of course they it’s hard for user outlet to give a completely perfect representation of reality and cover topics fairly and satisfy all audiences. And so I think they have, they have a lot of work to do and then they should do it better, but I think a lot of them are trying. Trust in the news has been declining at least a little bit in the in the last 10 years, but more the the largest drop has been in interest in the news, where people just are less and less interested everywhere in in the news and participation with the news. So just like talking about the news sharing news online etc. I have done some work on it and it’s declining in many countries, so people are just turning away from the news, but hopefully we can create a new form of news that attracts people and that interest people. And I think there are a lot of benefits or at least potential gains in like news influencer like on TikTok. In France, for instance, we have Hugo decrypt, the news influencers that do a very good job at summarizing what’s going on. And they also have deeper explainers and I think it better fits how new, how young people consume news and information in general online, so the news also needs to adapt to its audience and be better to gain trust. 00:34:33 Domen Savič / Citizen D Just to follow up… What do you think are some of the reasons the people stop engaging with with mass media or with, with, with news outlets? Is there something like you can or we can point a figure at and say, OK, it’s this, it’s this or it’s just yeah, a complex situation that you can’t really describe? 00:34:55 Sacha Altay There’s a lot of work on that and there are some clear reasons that come up all the time. One of them is that the news is too negative and it brings down people’s mood. It happened, for instance, in COVID-19. A lot of people consume more news, but at the same time, the news was mostly negative. And so, lots of people turned away from the news because it was too negative. And in general, that’s something people who start avoiding the news or who avoid the news mentioned. That’s one of the main things. A lot of people also feel just overwhelmed by the amount of news that there is and they think there is too much they cannot follow everything. That’s also the reason… I forgot the other reasons I think they are slightly less important. I think negativity in the news is one, but at the same time, we know that negative news is also more interesting. So you know, it’s also there are some positive news outlets, but the positive news outlets are not very successful. And I think it shows that people are actually not very interested in positive news. They they say they want it, but I mean a lot of people say they want stuff, but then when you give it to them, they they don’t like it. And of course, we are interested in the trains that are late and not the trains that are on time so it’s difficult. There’s no easy solution I think to remedy this problem. But I do think that maybe the news sometimes covers negative events too much and doesn’t give the broad picture enough, like even about stuff like poverty, hunger around the world. Most people think that, yeah, poverty is increasing, that hunger is increasing and that’s totally false, like extreme poverty has been reducing a lot in in the last 50 years. Same for hunger and many of the of the world’s problems are actually getting better. Many people don’t know that, and I think maybe the news just needs to also give the big picture a little bit more. But again, I don’t know, it’s complex. 00:37:00 Domen Savič / Citizen D And wrapping up. Are there some, let’s say low hanging fruits in this fight against disinformation about media perception, about reality perception that that you are, let’s say, paying attention to when you’re looking at it? Are there some things that should be happening or will happen that will make you say OK we did something right or this is something that that was worth pursuing or that is worth pursuing if we want to wake up one day in, to put it very stereotypically, in a brave new world – in a positive way? 00:37:49 Sacha Altay I’m not sure, but to be honest, some example I see are like really Hugo decrypt in France, this French influencer I think is doing a lot of good to interest people in the news that are usually not interested, and I see a lot of value in that, probably more than fact checking. I’d say that just the work of Hugo decrypt reduces misperceptions in French more than all fact checking in France just because a lot of people who would not have consumed news now consume news because of him and are aware of a lot of stuff and get some context on events, etcetera. And so, I think this is going in the right direction and it makes me optimistic. I also very much trust the European Union to do some work to hold big tech accountable. Of course, I don’t think big Tech is the cause of all our problem, but I do think that we need to hold them accountable. And they are so powerful that at the country level in Europe, we cannot do much against them. But united with Europe, I think we can increase, for instance, the sharing the data or there are a lot of stuff that we can push these big companies to do, that would help us better understand the ecosystem. Because one thing I haven’t said is that we don’t know much about the descriptively about the disinformation ecosystem in general in most countries, we know a lot about the US and some Western democracies, but because we don’t have good access to data, we don’t know much and a lot of people are jumping to conclusions proposing solutions, et cetera, but might take has mostly been guys we don’t know much yet and in most countries that matters and even countries like India, we don’t know much about the new ecosystem. We don’t know much about what’s going on with the WhatsApp group and in these countries and now we are proposing solutions based on, I think, a biased understanding of information, ecosystems and so, I believe the European Union could help us have a better overview of information ecosystems, but I think, yeah, that’s mostly what makes me hopeful. But when I’ve seen discourse and news coverage during the European elections, I’ve not seen much misinformation. I’ve not seen much AI generated misinformation. So, I think it’s pretty good news and it was the same in many other countries, even like India or many countries like there. There was of course some false information as usual but there was no massive information operation that worked and influenced people. So, it also makes me pretty hopeful that, at least for now, I think things are going fine. Of course, we need to make it better, but it’s not like here. 00:40:45 Domen Savič / Citizen D Excellent. We’ll end up on a on a positive note. Thank you Sasha for dropping by, for sharing your thoughts on the issue. We are off the next month, so see you all again in September. And yes, thank you again Sasha for for dropping by. 00:41:02 Sacha Altay Thank you very much for having me. Citizen D advice: Instead of information on disinformation, breed trust in the legacy media Focus on social issues that are the basis for effective disinformation Reform the media industry to adapt to new way of communicating the news More information: Misunderstanding the harms of online misinformation – article Misinformation on Misinformation: Conceptual and Methodological Challenges – article News Can Help! The Impact of News Media and Digital Platforms on Awareness of and Belief in Misinformation – article Not Born Yesterday – book About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!
Jun 15, 2024 · length unknown
100 Filip Dobranić in diskurz o umetni inteligenci
V stoti epizodi se s filozofom in programerjem Filipom Dobranićem pogovarjamo o pogovoru o umetni inteligenci. Kako to področju uokvirjajo množični mediji in kako politiki, zakaj je to lahko težava in kako bi jo morali najprej reševati na področju jezika, ki pogojuje področje politike in v nekaterih primerih celo področje tehnološkega razvoja. Od tega, da bo umetna inteligenca rešila svet, do tega, da ga ugonobila, o pomenu pojma umetna inteligenca in drugih podrobnostih tega področja, ki uokvirjajo razpravo na temu področju. Državljan D svetuje: Več pogovora o področju umetne inteligence in vključevanje večih družbenih skupin Več pogovora v času prostega teka odločevalskega procesa Več priložnosti za učenje in neškodljivo delanje napak Dodatne informacije: How to talk about AI (even if you don’t know much about AI) – article How people view AI, disinformation and elections – article AI Poses Risks to Both Authoritarian and Democratic Politics – article O podcastu: Podcast Državljan D je podcast za produktivno preživljanje časa, v katerem igramo vlogo državljana. Državljan D v pogovorih s strokovnjaki določenega področja informira in aktivira. Da se. Gremo naprej!
May 15, 2024 · 45 min
099 Nina Jankowicz and the political aspect of disinformation
We sat down with Nina Jankowicz, an American researcher and writer, currently working as the Vice President at the UK-based Centre for Information Resilience, to talk about fight against disinformation and online harassment, the role of different actors in this area and the differences between EU and USA in this field. Nina is the author of How to Lose the Information War, on Russian use of disinformation as geopolitical strategy, and How to Be a Woman Online, a handbook for fighting against online harassment of women. She briefly served as executive director of the newly created United States Department of Homeland Security (DHS)’s Disinformation Governance Board, resigning from the position amid the dissolution of the board by DHS in May 2022. Transcript of the episode: Expand the transcript 00:00:06 Domen Savič / Citizen D OK, welcome everybody. It’s the 12th of April 2024, but you’re listening to this episode of Citizen D Podcast on the 15th of May 2024. With us today is Nina Jankowicz, an American researcher and writer currently working as the vice president at the UK Based Center for Information Resilience. She’s the author of how to lose an information war on Russian use of this information as geopolitical strategy and how to be a woman online, a handbook for fighting against online harassment of women. She briefly served as an executive director for the newly created United States Department of Homeland Security disinformation Governance board resigning from the position amid the dissolution of the board in May 2020. Welcome, Nina. First of all, thank you for joining us. It’s really a pleasure. 00:00:56 Nina Jankowicz Thanks for having me on Citizen D, yeah, I’m excited to be here. 00:01:00 Domen Savič / Citizen D Excellent. First let’s start at the beginning, right. This information government board, your brief tenure, the critique which you wrote for Foreign Policy where you highlighted the issue of political say squabbles that are limiting the fight against this information. Uh, it seems that no matter where you go in the US, in the UK, it’s always the same, right? Every time somebody tries to do something in terms of regulating the disinformation, fake news, propaganda and other issues, the backlash, it’s always, you know, they’re going to take our freedoms, we have to defend democracy, free speech and everything else. Would you say that this is something that just needs to be taken into the account or is there a correct approach to these types of regulatory bodies where people actually feel that the body will have an actual benefit on the entire of society, not just the political parties or sides or whatever. 00:02:08 Nina Jankowicz Sure. So let me start with kind of a little description of what the board actually was and what people say it was. For the listeners that may not be familiar, so prior to joining DHS, I’ll say a little bit about myself too. I had been an expert on disinformation and analyst, focusing particularly on Russia and Eastern Europe, but increasingly especially during the pandemic in the lead up to the January 6th insurrection, looking at some of the domestic disinformation that we had seen so that’s my background. I had testified before Congress for both Republicans and Democrats. I had worked with members of Congress across the aisle as I was a fellow at the Wilson Center and I had done a lot of work kind of supporting policymakers all around the world and trying to put forward solutions to disinformation that upheld democracy right that made sure that our democratic freedoms were protected. I came to the job optimistic because I didn’t really expect to get a political appointment from the Biden administration, and so when they came and said, you know, will you serve your country? I said, of course, this is in my area of expertise. It seems like there’s actually room to do something here, so let’s get something done. And I’m bringing with me everything I’ve learned in my research, including a lot of policy analysis of the things that had been tried in Central and Eastern Europe that that may or may not have worked right, which is what I write about in my first book. The Disinformation Governance Board was meant to be an internal coordination body bringing together all of the different components as we call them of the Department of Homeland Security, which have very disparate missions. It’s a huge organization, and in some ways of Frankenstein, right? It includes everything from our emergency and disaster management agency, FEMA, to our Customs and Border protection to TSA, the guys that make you take your shoes off at the airport to our cyber security and Infrastructure Security Agency, SISA, which actually dealt with election security and kind of beefing up local and state election partners ahead of elections and potential incursions by the Russians or the Chinese, so a lot of different missions and the idea was to bring everybody together, make sure that we had a shared definition of disinformation, make sure that in the work that each agency or component was doing, that it was upholding civil rights, civil liberties and the right to privacy which Americans of course hold dear. That was the brief that I was given that I was going to bring these people together. I was going to assess what was going on in the agency and make recommendations for the board to adopt or not adopt. And we would go forward and make policy as a policy making body. How was to in the policy part of DHS, right, the policy shop, as we say. When the board was announced, I had been working at DHS for about 8 weeks at that point, at the very beginning of my tenure, I said to my bosses, I think it’s important that we announced this and that we announced it transparently. There had been incidents for similar agents announced that they there were I wouldn’t even call it backlash, there were rumors there were lies about them. I’m thinking in particular of the Czech Center against terrorism and hybrid threats when they were announced, there was a lot of excitement among disinformation researchers and, you know, people in the national security space. And then people on kind of who were skeptical of the government in in the Czech Republic really didn’t like the idea of this center against terrorism and hybrid threats, which in part was dealing with disinformation. And that was because they communicated poorly about it. They weren’t going to be doing any fact checking, they were very narrowly focused on the mission of the Ministry of Interior and, you know, threats related to terrorism. But they didn’t communicate that very, very well and I write about that in my book. So I said to my bosses we need to make sure we communicate well, that did not happen. You know, I wasn’t actually a very high-ranking person within DHS, I wasn’t confirmed by the Senate. There were, you know, a lot of people who outranked me there. And my job is to give them advice. And I gave lots of advice about different approaches to communication that I thought we should take. And they were all kind of rejected in favor of an approach that was announced to the board but didn’t give a lot of detail about it and the problem with that was that we left a vacuum for the adversaries, both political adversaries. But I will also say Russia wrote about this to fill in the blank and what they filled in the blank with was wise. They said that this information governance Board was going to govern what was true and false on the Internet, which was not true. They said that I was going to have the power to send men with guns to the homes of Americans with whom I disagreed – also not true. I was not a law enforcement official. I had no budget. I had no operational authority. That operational authority lied with all lay with all the components, but even they were not going to send men with guns to the homes of Americans with whom they disagree. That was just preposterous. And also, in that vacuum, because there was so little information about the board it was ironic, of course, having written about online abuse and studied online abuse that I would then be subjected to a very widespread hate and harassment campaign that looked it. I mean, my family was doxxed, our personal information was released on the Internet at the time I was pregnant, people were saying horrible things about my reproductive status, my baby, my husband, whatever. There other members of my family were targeted. There was wide skepticism about my personal life, I got death threats and all sorts of other nasty, unsavory, violent threats. And also, it’s not the same, like the violent stuff is worse, of course, but people just lied about me, they lied about all sorts things they said that the Hunter Biden laptop was a Russian disinformation operation. I actually never said anything like that. I urged people to be skeptical about the laptop given when it was released and who was shopping it around. Rudy Giuliani. He does not exactly have a very good track record of telling the truth. Right. And I said, listen, we don’t know what this is. It can’t be independently verified right now, treated as a Trump campaign product, and somehow that has been construed to say that I believe that it’s Russian disinformation that’s been repeated over and over. They said that I believed in the Russia gate conspiracy theories, if you read my book I kind of say we don’t really have the evidence that says that Russia, Russia was colluding with Trump, but we do have evidence that Russia tried to interfere in our election, right, that’s open-source evidence, we know about that. So, my opinions, my thoughts were totally misconstrued. I was painted as a young woke liberal woman who was coming away to take your coming into governments, take your rights. And so, to this day, people believe that I committed treason against the United States because I was attempting to violate the 1st amendment and take that away from my fellow citizens. And I’m sure my grandfather is rolling over in his grave. So, my grandfather was 10 years old when he was deported to a gulag by the Soviet Union, made his way through Central Asia and ended up in the UK in the early 50s and immigrated to the United States. Kind of looking for the American dream. So that is very much an experience that I carry with me and that is in my DNA, and for somebody to say that I wanted to take away Americans fundamental rights and freedoms is just so anathema to me. And of course, what happened with this in the end I was pregnant at the time I decided to resign because the department really didn’t have my back. They were not pushing back vociferously on these lies. They thought that strategic silence was the right thing to do, that like this would blow over. I kept being told that it would blow over the next weekend or the weekend after and as we waited and time grew and, you know, issued a fact sheet that wasn’t very good. They had, you know, people give various statements to the media. But I wasn’t allowed to speak for myself. And it was my record and my family that was being disparaged and endangered. So I decided to leave. They had asked me to stay on as a policy advisor and I just decided it wasn’t worth it, then they disbanded the board after I left later that summer. And you know, I was at home with my baby trying not to think about anything. This is a really frustrating and sad moment for me, and I think it’s one that’s important for your audience to understand as well as much as there has been political backlash against some of the counter disinformation in attempts in Europe, there’s never been anything like what we’re seeing in the States right now. And it’s not just about me, it has now expanded from the disinformation Governance Board and me in 2022 where kind of the Democrats presented the board as a fait accompli to the Republicans and said, OK, you win. We’re not going to fight you at all and that’s kind of an admission of guilt, right? It looks like one, even though nothing there was no wrongdoing and then. Having had that proof of concept, the Republicans went further and said look at all the rest of this censorship happening, and they basically redefined the term of censorship as any person studying the information environment. They’re a censor. Any cooperation or communication between an academic or somebody at a think tank and a social media platform or the government that’s also censorship, right? That’s the claim right now. And I think that that is a completely false definition of censorship. Censorship has a clear definition of what that looks in the American context. But B, it is it is doing the opposite effect of what it says it is. It’s not defending people’s fundamental rights and freedoms, it is having a chilling effect. I’m the researchers who are doing this work and just trying to stand up for democracy. So that’s where we are right now in the states. It’s not a not a pretty picture and to me, it represents the most fundamental threat to academic integrity and freedom of expression since the McCarthy era. 00:12:01 Domen Savič / Citizen D Why do you think looking back on your involvement with the DHS, the backlash, the response because it sounds like the arguments you wrote about in your op-eds and everything everywhere else. They sound like there is a little black book on how to discredit people, right? And anybody can rent it out in a library, read the paragraphs, and then use it in in different contexts in the US and EU all around the world, right? So why do you think this is such a transferable procedure? So why do you think the same things people are using to you know, media assassinate type person or gather up gather up like reasoning for personal attacks. Why do you think this works like literally anywhere in the world? 0:13:05 Nina Jankowicz A good question. I mean, I think ultimately, whether it’s online or in a newspaper, or just gossiping with coworkers around the water cooler, right, people are interested in personal stories, and the more salacious, the better. And so, you know, claiming that I’m this crazy woke, you know, feminazi, working in the DHS, that’s a good narrative, and it makes money for people. It makes money for people. I know because it’s been repeated over and over and over again. Whenever anybody has a chance to just mention me, that’s how they describe me. Right? So, it clearly is resonating with people. And I think also there’s been a normalization, at least in the United States and our politics of these sorts of personal attacks since Donald Trump’s presidency. We have seen that sort of personalization not just happen within him and his office, but also being normalized in the Republican Party to some extent. I wouldn’t say all Republicans do this, but people there are certainly people within the party who have enormous influence who act this way. And it’s interesting because they do it bombastically in public, but when you get them behind closed doors, they don’t act that way and so you know that it is – it is something that is done for show. It is something that is an act and it’s really depressing to me. And I have actually said directly to members of Congress who have attacked me in this extremely personalized way and put me and my family in danger that I thought they were engaging in a dereliction of duty as elected officials. They have an important role to play in setting the tone of discourse in our country and by acting the way that they do, they are telegraphing to their constituents that that is OK and generally some of the constituents, not all, but some will go farther. They will say worse things they will, you know, send violent threats. They might show up to people’s houses. They might engage in violence like, that’s not beyond the pale. We’ve seen that already in the United States over the past four years. And so I really think we need to call that out and make sure that people recognize how strange that is. I remember there was a conversation with a governor and I forget which governor; it was a state governor here in the States and he was asked about the gendered and misogynistic attacks on Nikki Haley, who was running for president for the Republican Party nomination at the time. He said that’s just politics and actually if you look at other countries, other developed democracies, it’s not as accepted in other places, it still happens. Yeah, there are still misogynistic comments, but they are not the norm. And I think we really need to get back to that. And of course, the Internet getting back to the global nature of all of this. Internet really supercharges that ability to be mean and personalized, and downright violent, right? There’s no consequences for any of this stuff. 00:16:25 Domen Savič / Citizen D So, so would you say, you know looking at potential, I’m not going to say remedies because that would be too optimistic, but the right way to sort of address these issues, right? So we’ve had like spanning from let’s say the first election of Donald Trump, the Golden Age of, this information writing and fact checking and active citizenship we’ve had nothing but failures up to up to today, right? Nothing that we’ve advised on in 2015, worked. In fact, if you compare, the first notes about, you know, the media is lying to you, you have to check everything, everything is disinformation and propaganda. I think it’s sort of… like to me, it seemed that the biggest blowback came during the pandemic crisis, right? I usually joke in a very sarcastic way about this and say that the best fact checkers or the best people that really took fact checking to heart were the antivaxx community during pandemic, because they said nothing is true, everything is a lie, we have to figure out everything on our own, right? So moving forward, are there some lessons, not to be learned, but are there some lessons we can take as a basis to sort of reformat the discussion the framework in which we are addressing the fight against this information, a fight against propaganda. 00:18:03 Nina Jankowicz Yeah, I’ve always been. I don’t want to say skeptical of fact checking. I think it has a time and a place and serves useful purposes, but I think we need to be very clear about its limits, so a fact check, a reactive fact check of a piece of disinformation or misinformation is not going to necessarily reach the people who already believe the mistruth, right. In fact, it often causes them to double down on that mistruth and so I am much more in favor of investing in information literacy and people react poorly, especially in the states. That’s beginning to be another kind of divining or lightning rod issue here because they think that means that people are gonna be told, OK, this media outlet is trustworthy and this one is biased, this one just publishes false news. And that’s actually not what media literacy is. It’s giving people the tools that they need to navigate today’s information environment and while I don’t want to put all of the emphasis on people, right, I think there needs to be other parts of the solution. I do think that if people understand how social media platforms work, why they’re being shown certain content, if I’m looking for a new pair of shoes and then suddenly I’m getting Instagram ads about different pairs of shoes that are, you know, that have the characteristics of the ones that I’ve been looking for, like there’s a reason it might not be the best shoe for me, it’s just that they’ve got a very good online advertising team and similarly, you know, knowing that the platforms are targeting you based on your interests and the things that you’ve engaged. Before I wrote a piece in 2020, about how it’s so easy to go from in our context, we had a big movement to reopen businesses in about, let’s say, April or May of 2020. And even if you just have that as a purely economic argument, if you knew like, if you didn’t believe in any COVID misinformation, you didn’t think vaccines, you know had 5G we didn’t have vaccines back then but you know all these crazy conspiracy theories? Yeah if you didn’t believe in all of that, but you just wanted to reopen your business, it was a couple of steps removed from joining a a reopening Facebook Group to going into like crazy conspiracy theories about heavy metals and like the vaccine was a conspiracy to get people to become sheep. And it was just a few steps removed and that’s the sort of thing that I think people need to be… they need to understand. So I’ve always advocated that governments, in addition to taking kind of the national security precautions that they all should take in addition to making sure that they’ve got like one belly button of policy coordination within their government so that that can all go out to different ministries or agencies that they that they need to be investing in information literacy as well. And it wouldn’t be prescriptive… it wouldn’t say Fox News bad, New York Times good, it would say you know today’s information environment is very polluted and we all need to kind of recognize that navigating it is difficult and here are some tools to do that. So that’s part of it. I think the other thing that we have seen from the countries that have actually been successful at this is that they have publicly prioritized that disinformation is a problem, like they’ve publicly declared that. So if you look at Sweden and Finland and the psychological defense that they do with their populations or Ukraine, obviously Ukraine is in war, so it’s an extraordinary situation. But, you know, people recognize that this information is being used to undermine them and undermine their efforts as they try to resist this enemy. And so that’s part of it. And then also, and this is something that governments struggle a little bit more with, again creating that policy center somewhere at the heart of government with full political buy in from the very top of an administration and something that’s going to stay from administration to administration that can’t quickly be undone by political upheavals, is incredibly important and no president since 2016, while Trump didn’t prioritize it and Biden didn’t figure out a way to do it right. I think DHS what I was supposed to do was a smaller version of this within the department itself, and it failed miserably. And now, as a result, a lot of the counter disinformation work that the US government is doing has been rolled back. So, it’s difficult and you have to telegraph it the right way, you have to be very transparent about what you’re doing and say we’re not doing any fact checking. You know, we’re going to be putting good information out there. We’re looking at what’s happening online, and we’re going to provide Americans with information about how to get to the polls, how to vote, where to find their voting place. You know, we see a lot of misinformation and disinformation about that sort of stuff that that might be all it means, but you need to be transparent about it and for some reason the US government couldn’t figure out how to do that. I will say one final thought, which is that when the Salisbury poisoning happened in the UK in 2018, that was, in my opinion, a very good example of how to be transparent, how to declassify information quickly? How to work across government from very different parts of government, from the police to the foreign and Commonwealth Office to the people who dealt with chemical weapons to get information out to the public quickly and I think we could all learn a lot from that. I know there are probably still people who believe the Salisbury operation was a conspiracy or a plant by MI5 or whatever, but I think the majority of the world and certainly the majority of Brits believe the truth, which is that Russia did this on purpose as retribution against Scripal. 00:23:47 Domen Savič / Citizen D So, my next question would be like the difference or looking at the different, let’s say sources of this information, right. So, you have these let’s say let’s call them like homegrown disinformation outlets that are operating within a certain interior politics or party politics, within a specific geographical area like a country or a region and then you have these, you already mentioned them, foreign ops, influencing or trying to lower the attention of the enemy, of the foreign countries. Would you say that dealing with these different types, so to say of disinformation could be the same or is there like a huge difference between addressing these two issues. Because we’ve been, like the pandemic was sort of like a highlight of everything mashed together, from local disinformation outlets that were basically just monetizing the pandemics informational black hole, then you had the Russian foreign operations that were trying to sort of, yeah, dissolve or inject sort of incoherence and stuff. 00:25:03 Nina Jankowicz Yeah, it’s difficult to draw the line right. I think everyone wants there to be a very clear line between what our foreign adversaries are doing and what’s going on at home, and this is something that’s been incredibly misunderstood by almost everyone from the press to policymakers since the very beginning of the kind of current era of interest in disinformation and I talk about in my book how the most successful disinformation is the disinformation that manipulates pre-existing grievances in society. So, if you have again to use a US example, if you’ve got people who believe Texas should secede from the rest of the United States, that’s a really easy grievance to identify and manipulate. If you are Russia and you might identify some local actors who believe that, like you know, local civil society groups or a local media outlet that that takes that kind of position for themselves as an editorial position and you can work with them. And that’s when you get information laundering. Right, so we see narratives or pieces of, quote UN quote, intelligence introduced to local actors and then they are able to spread it without kind of the Russians or the Chinese or the Iranians, showing their hands that aha, this is a foreign actor and it’s so it gives them a layer of insulation, plausible deniability. And it also makes it harder for at least in the American context because of our free speech laws, which I agree with, just to put that out there so no one can slice this and say that I didn’t say that it makes it much harder to respond, because if you’ve got somebody who fervently believes something, they have a right to say it right. So how do you respond to it then? And that’s where you get into kind of the idea of counter speech. You know, somebody putting out that information and hoping that the story that they tell is more compelling than the lie. So, it’s really difficult and I guess the other thing that I would say is that the nations that have been good at this and I think the UK is another good example again and also probably the Nords and the Balts. They have an actor agnostic approach, right? It doesn’t matter if it’s coming from Russia or coming from inside the house. It means that you know they’re going to respond to disinformation that affects public safety, public health or democracy in the same way, but and I think that’s the right approach, I don’t think that in any of these cases, well if it’s a, it’s if if it’s a fake Russian account, by all means take it off, take it off of Twitter. But we need to be sure of that or Facebook or TikTok or whatever. I don’t think that you know, I have in fact if you go back to my very first New York Times op-ed in 2017, September 2017, I say that playing whack a troll is not the right approach to countering disinformation, we need to do better strategic communications. We need to tell better stories. We need to, you know, prioritize this as a policy area and talk about it transparently, not playing whack control, not removing those accounts. And unfortunately, you know, I think as you and I have talked about, that’s where we’ve ended up that people still think that the response to disinformation is censorship, that the response to disinformation is removing stuff online. And that is just not true. 00:28:21 Domen Savič / Citizen D And how would you compare like the current, let’s say regulatory uptick in in the EU that is sort of, you know, I mean they’re on the way out. You know, elections are coming in two months, but they’re still trying to sort of push out a lot of regulatory frameworks that address, you know, financing of the media transparency of the operations, the role of digital intermediaries… They’ve had some success in terms of addressing or highlighting these issues, California and other states in the US started, you know, writing their own versions of, let’s say, the same regulatory frameworks. Do you think this is part of the solution or is it just like a PR stunt for the politicians to sort of show off and say yes, we’re doing something? 00:29:17 Nina Jankowicz DSA and related pieces of legislation are good. I don’t know if they’re gonna work. I think we have to wait and see to, you know, see if the implementation is as good as the idea. But I think we need to start somewhere and particularly with the transparency and oversight question that one, I think is the most critical. So, when people ask me what I would do to regulate social media, I tell them that we can’t even have a conversation until we’re all using the same set of facts and right now, that’s true, right? And researchers, I don’t have to tell your audience, I’m sure, but researchers access to data on social media platforms has been almost entirely cut off or monetized to the point where they can’t get access to it. So, what we’re dealing with now is basically hearsay. It’s like when Elon Musk decides he wants to release snippets of conversations without their full context to pre-selected journalists. That’s part of what we get or whistleblowers or stuff that’s been gotten through. You know, these congressional investigations that are going on here again released in a retaliatory or untransparent manner. And what I think the DSA can solve and other regulatory regimes, including the E Safety Commissioner in Australia, who has the power to essentially, ask questions of social media platforms based on safety related issues that allows us a shared set of facts. It means that we will know how the platforms are dealing with content moderation questions. We will know how they’re dealing with foreign interference questions. We will know in in issues where people feel aggrieved that their content was removed or demoted or they have their account shut down. We will know why that happened. And until this point we’ve not had anything that has allowed us to do that. We could make inferences even when we had access to more data, but we would. We would only have kind of what the data showed and the inferences that we could make and then what the platform said. The truth is probably somewhere in the middle, right? So, I’m excited about that part of it. I think there’s still a lot of work to be done thinking about the way that our information environment today has effects on public safety and the online safety bill in the UK makes some attempts to square that circle, but it also has a couple of concerning elements to it, including the requirement to break encryption, things like Signal, who have been really pushing back on that and I think rightly, but on the other hand, it also, you know, I think makes it legal deep fake porn and other types of harassment that are based on protected characteristic. So, it’s a difficult dance to do, but I think that that is going to be the longer term harm that we have to kind of iterate on and see as things are developing, what’s going on the transparency issue. And if we solve that and we make sure that there is a non-partisan intermediary body overseeing that data and distributing it to researchers and journalists, frankly, then then we’ll have a conversation. We’ll have the basis on which we can have that conversation and make it more productive. 00:32:32 Domen Savič / Citizen D And what would be the result of this conversation? You often hear about transparency that is the cornerstone against fight against disinformation, but then they never mention the next steps, in terms of, OK, we have transparency, we know what’s going on behind the curtain. What’s next? 00:32:59 Nina Jankowicz Yeah. I mean, I think then we have some sort of intermediary regulator who can respond to exigent threats on the platforms and either give guidance or set regulations for what they must do if a certain thing happens. So, I like to compare it to the airline industry. I like to actualize what goes on the Internet, because I think a lot of people still believe that there is a firm line between the online and the offline worlds and that just isn’t true anymore. So, Boeing obviously has been having a lot of problems over the past couple of years recently here in the States ad door blew off an airplane and immediately we had an investigation by the FAA into Boeing and what was going on there. I think we need regulation similarly for technology platforms that have such an immense influence on our lives and have so much of our personal data as well, right? We need to make sure that they’re safeguarding that appropriately, and that if they are not safeguarding it appropriately, if they are not, you know, investing enough in making sure that child sexual abuse material or terrorist content, or if we ever get there, you know, other online harms that deal with adults and everyday people. Deep fake pornography is one example, right? If they’re not doing their due diligence and expending a certain level of effort, then they get fined or then they have some sort of penalty. That’s what happens in every under every other industry, from finance to cars to airplanes to food. And to think that we are giving over so much again of our personal information to these companies for access to something that we use in our daily lives to connect with people, to do business, to, to stay in touch with families, and there is nobody, nobody who’s watching what they’re doing. I mean, it’s shocking. It’s shocking. So, I think that would be the logical next step. But again, we need that shared set of facts to start from a common ground there, because otherwise we’re just going to get hoodwinked by political actors and private companies who are trying to make a buck. 00:35:00 Domen Savič / Citizen D It seems to me like this is the perfect topic or this subject of regulating, let’s say intermediaries or online platforms. It’s like the perfect storm situation where everybody can or anybody can add to its own two cents. The political actors are trying to sort of lessen the encryption, as you’ve mentioned, and trying to sort of convince us that you know, terrorism will be gone if they just have access to everything. Then you have the private sector that is just “No, no, no. Just leave us alone. We will handle this perfectly on our own because we are the only ones who know what to do.” So, who do you think of all these actors should, let’s say, start the conversation about transparency and about next steps like you’ve mentioned political actors, you know, we’ve talked about the market forces… who should be the like the instigator of this conversation? 00:36:05 Nina Jankowicz I think it’s got to be researchers. I don’t. I don’t think either. Nobody, nobody is unbiased enough of the political actors and the and the private companies to really drive the conversation, and it’s the researchers who have worked with the data. It’s the researchers who have uncovered some of the harms online, and it’s the researchers who want to be able to continue to do some so without this exorbitant cost? And I think they they’re starting to do that. The problem is that with due respect to my beloved colleagues in academia, academia moves very slowly. It’s really difficult to get consensus on things because everybody has very strongly held opinions based in their own research and their own experiences and they don’t communicate very well. So, I think most academics would agree with that, so you know, I think there’s a lot of work to be done to make those points to the public as well. And I’m doing some work with some academics to start to do that in the future. But it is difficult. It’s a really difficult battle because everyone turns off their brains as soon as you start thinking about something technical, but I think again, that’s where the analogy to the airline or the food industry or whatever comes in very handy, because I think everyone hopefully agrees now that what happens online does not stay online and that means that it needs to be regulated. 00:37:27 Domen Savič / Citizen D Yeah. And speaking about regulation, just one more topic before we wrap up… your latest book talks about online harassment, it describes your experience, it talks about other women’s and other types of harassment online. What’s your take on the current, let’s say regulatory situation in that area? Is this similar to this information or fight against this information? Is it something completely different? I’m guessing harassment is as old as this information operations if not older so? How do we address this issue moving forward, especially now that we are, you know, walking, maybe even sleepwalking in in the era of generative AI, deep fakes, and everything else that just you know, turns the volume up to 11 in terms of harassing. 00:38:23 Nina Jankowicz I mean, I could have spent an entire hour on that question alone, but I’ll try to be quick. I mean, I think the state of online harassment and online harms online safety, especially in the United States, but really, most places around the world is primitive at best, the state of regulation and in most cases, I think really discounts the harm, particularly that women and minorities face when they encounter online harassment, right? The intent of this harassment, whether we’re talking about just, you know, trolling someone or deep fake pornography or violent threats, is to silence them. So, it is a speech issue. And yet, when people turn 18, especially when women turn 18, right, we have a lot of conversation in, in the online harm space about girls and I had a student at Oxford asked me once, why is it that when girls turn 18, we no longer care about them and I think that’s a really, really good and trenchant question. They’re not minors anymore. Sure, but are we asking just by raising our voices to receive rape threats, to be targeted in this way, to have to fear for our lives when we’re walking around, you know, outside? I mean, I have a cyber stalker who I had to get a protective order against, I had a bad experience with law enforcement when I did that. The detective who handled my case basically said, well, he lives in New York. Why do you think he’s gonna show up here in your home in Virginia? And I’m like, because he has my address and he has a habit of showing up at people’s homes and places of employment and events that they’re at. And I have a baby, and I don’t want him to, you know or see my baby or be in proximity to him. There are many law enforcement agencies that react similarly. So, I think I think we need much more help there. The fact that deep tech pornography has not been criminalized or even civil penalty instituted at the federal level in the United States is shocking to me. When the harm is so, so, so clear. And it has become so easy to create stuff like this and you asked if this is a similar question to disinformation or not. I mean, I think the response that I hear from skeptics is that this is just, you know, you should have to deal with it. You should just buck up and deal with it. Often, that’s white men who say that. They just don’t get the same sorts of harassment that their female counterparts do. There’s a lot of data on that. And so, what do I think needs to happen? I mean it’s tricky, right? Because it runs into speech questions. But I think in regulating the platforms we might be able to find a happy medium there. Obviously like these, these platforms have terms of service that they are meant to implement that already say that you’re not allowed to harass somebody based on protected characteristics like gender, age, religion, ethnicity. The problem is they don’t enforce them, so perhaps there’s an enforcement mechanism that says if you don’t enforce your own terms of service, you are not upholding your duty of care to your users and that’s a market problem, right? That’s kind of a consumer protection problem. There’s a lot of different ways to do it, but I think until people recognize the harm that online harms do cause that it’s not just sticks and stones, but there is a real physical threat that that comes from a lot of this stuff, then we’re going to be going around in circles. But I am hopeful. Danielle Citron, who’s a privacy scholar, a former MacArthur fellow, she has two great books. One of them is hate crimes in cyberspace and the other is her newer one is called the fight for privacy, and she mentions how in the 70s, of course, it was legal, perfectly legal and normal to get sexually harassed at work. And women came together and we banned it against that, and now it is very much not legal. Right. And you can you can have harassment claims against your bosses, your coworkers, for creating an unsafe work environment. We just need to bring America, bring the world up to speed. That that’s not OK online either because it is a speech issue, like I said, coming back around to it, it means that people, women, marginalized communities are silencing themselves because they are afraid of these horrible things that might happen to them online if they do speak out, so that’s where we are. I’m hopeful that we can change it for right now, situation is not very good. 00:42:51 Domen Savič / Citizen D And just one more question before we wrap up. Like there is a, there is a ton of political movement in the US and the European Union, EU, US election is coming up the EU election is coming up. Do you see this these topics well addressed or during the election cycle? I sometimes think it’s a generational issue, like the older people, let’s say above 50, they just don’t have, as you said, the same experience or it almost looks like they don’t live in in the same world. So, is this a generational issue? Do you see like working in this field working with younger or young people? Do you see a change of narrative and perceptions of these issues as people go, you know, are younger and are more involved in these types of situations? 00:43:59 Nina Jankowicz Yeah, I think so. I think younger people understand the harms that the Internet can cause more, more acutely than people, a couple of generations ahead of them. But I’ve also had some really enlightening and encouraging conversations with people who are many years my senior, so I wouldn’t entirely write them off, though I do think we’ll see more and more legislation coming from kind of younger MP’s, younger members of Congress as we start to address these issues. 00:44:33 Domen Savič / Citizen D Thanks so much Nina, for dropping by. This has been really informative and really great. Best of luck to you in your future endeavors. And yes, fingers crossed me move, you know upwards, not yeah, downwards. 00:44:48 Nina Jankowicz Yes, I agree. I’ll be watching the elections in the EU and I wish you all the same as well. Bye bye. Citizen D advice: Rethink the connection between disinformation and political parties Develop new approaches to financial ties of disinformation research Empower disinformation researchers with political support More information: New Group Joins the Political Fight Over Disinformation Online – article How disinformation fuels violence — and makes our politics worse – article ‘A surreal experience’: Former Biden ‘disinfo’ chief details harassment – article About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!
Apr 15, 2024 · 36 min
098 Mark Dempsey and digital activism fatigue
We sat down with Mark Dempsey, a Senior EU Advocacy Officer for global free speech organization ARTICLE 19. Prior to ARTICLE 19, Mark consulted for the European Commission on a project focused on data protection laws in non-EU countries. ARTICLE 19’S work in Brussels is driven by the goal of ensuring that the European information environment is free, fair, accessible, inclusive and decentralized. During our conversation we touched on the current regulatory frameworks within the EU, digital activism fatigue, the role of the end-user and the omni-present role of digital technologies and services that need to be put in check by regulation. Transcript of the episode: Expand the transcript 00:00:06 Domen Savič / Citizen D Welcome everybody. It’s the 5th of April 2024, but you’re listening to this Citizen D podcast episode on the 15th of April same year with us today is Mark Dempsy, Senior EU advocacy officer for Global Free Speech organization, Article 19. Prior to Article 19, Mark consulted for the European Commission on the project focused on data protection laws and in non-EU countries, and even before that, Mark, you’ve worked in the financial sector. Is that right? 00:00:39 Mark Dempsey / Article 19 Yeah, correct. Correct. Yeah. 00:00:41 Domen Savič / Citizen D Before we start our conversation, I just want to know what made you switch from the financial sector to digital rights, which one is worse? 00:00:53 Mark Dempsey / Article 19 Which one is worse is a funny way of putting it. So, I worked for a long time in development, finance, and then regulatory finance. And I was an EU policy advisor for the Financial Conduct Authority in London, where I would go to Brussels, and I’d sit in Council working groups discussing nascent proposed legislative proposals from the Commission and I think really what made me make the move is an awareness of the growing encroachment of big tech on our lives and the controlling of the narratives. Not just in the sort of public space perspective, but also commercially and the reliance of so many businesses like Amazon, Google, etcetera controlled the various data flows and the commercial relationship? So it was, it was the encroachment of big tech in our lives. But it was also the realization that there was there was an extent of regulatory capture in finance, which was never going to change. And I think coming from Ireland, seeing the financial crisis and the way taxpayers have basically footed the whole bill, no one in the US went to jail, there is still this aura of invincibility around finance and I just became, I think, quite disillusioned really and it was very hard to see the tangible effects of regulation to people on the street, and I felt that digital rights was more of an area where I could make more of a difference, so I took the time out to go to Hertie and to do a masters. And I was lucky to work closely with Joanna Bryson, who has this unique view as a technologist, where she looks at society and the human rights impact and the digital rights that come attached to it. So, I think though she was a bit of an inspiration. I’m sure she’d like it; she’d be heartened to hear that. And then, of course, there was such a huge avalanche, as you say, of regulation in the EU and it felt like a very good moment to be involved, even if I did come in a little bit late. When I came into the DMA/DSA in terms in terms of their negotiations were being finalized as I joined Article 19, but the GDPR, of course was a good experience with them. The Commission was a good learning experience and set me up nicely for the Article 19 role. And where of course I came straight into the European Media Freedom Act, which was in the middle of negotiations, which in itself is a is a fascinating piece of legislation which we can talk about later. 00:03:32 Domen Savič / Citizen D Sure. Because my next question would be exactly that, right. So, the EU Commission mandate is wrapping up, they’ve done a ton of work in terms of regulatory proposals or regulatory laws in terms of digital rights. You’ve mentioned in the DSA the DMA, the European Media Freedom Act previously the GDPR. Is this the right way or are we heading in the right direction with these proposals or with these laws that are that are, as you’ve mentioned, focusing on big tech on, on intermediaries, on everything that’s happening in, in the, in the digital world right now? 00:04:12 Mark Dempsey / Article 19 I mean, I don’t know whether I’m the right person to ask. That question is is in itself questionable. I think the EU can be commended for making the first move and I know that they sometimes get criticized as being mainly regulatory cage, so to speak, and that they seem to specialize more in regulation. But that’s because they do have this immense capacity of civil servants within the Commission. If we’re going in the right direction, it’s too early to tell. I mean what is disappointing is that the process of agreeing proposals ultimately at the end of the trilogue is still very non-transparent, and I think with the EMFA, with the DSA probably less so the DMA, there were agreements made, or at least there were. There were positions for justice by the Council at the end, which undermines the initial legislative proposals, so I think that process is simply, it shouldn’t be tenable anymore, but I think it probably will be because I don’t see any changes happening. So we are dependent very much on a strong Parliament, and you’re very dependent on having your champions in Parliament to push the civil rights aspects of any legislation. But if you don’t have those people, they go into the rapporteur, for example, I, the person who oversees the file, if they go into final negotiations and they’re not particularly, how do you say, favorable towards civil civil rights, then our position is severely weakened. So I think the proof will be in the pudding in terms of enforcement, I mean again, and I know for some they’ve heard this ad nauseam, but this all lies in the enforcement and particularly with these new legislative pieces that have come in. There are mechanisms which gives the EU a strong regulatory position, so they are going to be regulators for DSA/DMA, AI act and we just have to see how that plays out. I mean, the GDPR has been a lesson for the EU, I think they realized that. So, let’s see if they put the right resources work closely with civil society actors and that look, that will all happen in the next couple of years. And then it’s a case of understanding how all these pieces of legislation interact with each other. I mean there could be unintended consequences, they’ll have to work very closely. The different units, like the G Connect to GG comp. The new AI office. They’ll have to work very closely together and coordinate and in a non-siloed fashion and I think the EU is not known for our coordinating well amongst different units. Things tend to be quite silent, so iit’s really up in the air as to whether as to where all this goes. But I do go back to commending the EU for at least taking the step because, I mean the US, you don’t even have a federal privacy law, so I think certain states as we know like California, they’ve learned from the GDPR and they’ve created their own strong privacy laws, but I think the EU is really being the first mover here, but I think they tend to love saying that and praise themselves. But we’re beyond that now, now it’s about whether they can actually enforce. 00:07:45 Domen Savič / Citizen D OK, I’m going to, I’m going to flip the question and ask you a little bit differently, so, is there a a phenomena or is there a, let’s say a mistake in the field that should be corrected by these EU legislative regulatory frameworks? And if they don’t do it, then it it will sort of signal that the regulatory framework has I guess I could say failed in in in its attempt? 00:08:20 Mark Dempsey / Article 19 Well, something that Article 19 is looking at is the concentration of market power, so if the DMA doesn’t do what it’s supposed to do then I think we will have an issue. You could say that that particular piece of legislation will have failed. I mean the whole point of it is to create fair and contestable markets and that means ultimately as well opening up these platforms, and I think there are avenues within the legislation to do that. But the Commission hasn’t mandated that so far yet.For example, if they were to open up recommender systems, for example, and allow third party recommended systems into your social media feeds, I think that would be something that would certainly disrupt the business models and provide the choice which the DMA proposes to do. So, I think that there are ways for them to decide that something is not working and therefore we will have to make some fundamental changes to the business models to make sure that these markets are fairer, more open and more contestable. But again, this all goes down to a certain level of political will and the second level of courageous leadership. Because if you look at the, if you look at the parallels of finance, those steps weren’t taken, I mean that there wasn’t political courage to say. You have these large institutions which are deemed too big to fail, but post financial crisis, there has to be ways of breaking them up so they don’t have a dominant position in the market. And if you look at your Goldman Sachs and you look at the the large American banks, they’re still making remarkable profits and their share prices have held and gone up and the level of pay compensation that executives get is just absurd when you compare the ratios going back over the decades. 00:10:22 Domen Savič / Citizen D Would you say that’s something that compares or that’s something that is similar between the financial and the big tech industry or area in terms of big players positions that do not change no matter what because politicians that are afraid of them? I used to say that politicians are in love with intermediaries, with big tech giants and they don’t want to hurt their lovers in a way, so they dance around them and sort of not face enough pressure from the public. They know they usually do something small and say OK we’ve done our job and now the rest is on the end users. 00:11:11 Mark Dempsey / Article 19 Yeah, I think there’s actually absolutely an element of that. I mean, unfortunately it seems to be sort of human nature that there is this admiration for the ability to make money, so when an executive from a big tech firm walks in, there’s fawning over that person, and that is definitely a parallel that exists for the banking sector. I mean, we need legislators and regulators to think about how are these companies improving society and I think it’s quite clear they’ve been doing the opposite for some time on so many different levels, not least when it comes to undermine democracy. But still, when we you’re not getting that forceful language by regulators to change this, yeah, the regulation is the regulatory proposals to the DMA they say are a step, but really that’s all they are and I think the DSA is probably more limited in what it can do than the DMA. But there is a huge cultural shift that needs to happen, and it’s admittedly difficult because they’re entrenched monopolies at this point and it’s hard to change that. It means that to change that, you have to disrupt business models and that take that will take a certain amount of strong leadership and willingness to push against the grain of the usual argument of, you know, innovation is being stymied by regulation. I mean, none of this is proven and when you have… I mean there are so many incidents that have happened as a result of big tech and then of course you’ve got AI and AI adds a whole new component because they’ve already started reinforcing their dominant positions by forging partnerships, which are basically acquisitions and taking over all the all, all the staff of the company that they’re going to partnership with and I know then that there’s investigations into this. But yeah, does it bode well for the future? I think it’s hard to say that it does at the moment given how how quickly generative AI seems to have become something that’s again completely been taken over by big tech. 00:13:23 Domen Savič / Citizen D Speaking about the big tech and you’ve mentioned GDPR in in the beginning, so yeah, another interesting party, so to say in this fight is the end user right? So you’ve had the GDPR put complete control over our own data into our own hands and you know, that proved a bit of a challenge for people not being used to be in the driving seat. So on the other side, you have these big tech giants, you already mentioned these firms that are that are that’s basically a a monopoly, if we’re lucky… So how do you see the role of the individual user moving forward? Should we as activists sort of try to move away from this logic that we have to empower the people and everything else will follow and focus more democratic institutions and more on the systemic players? 00:14:29 Mark Dempsey / Article 19 Yeah, I think we absolutely want to focus more on the systemic players because, has GDPR ever worked? I mean has more people understood the value of their personal data… I mean, that’s really the question, right? And I’m not sure that they have, I mean that there’s the whole cookie fatigue, there’s the determined services, there’s the ability for big tech to hover up data in a transaction which no money is transferred. They’ve managed to to yet again control that sort of transaction and I think the question is always like how do you get people to value their personal data, what needs to be change to have a simpler choice? I mean cookie fatigue; we all know that a very small proportion of people actually go to each of the cookie options and then decide to reject them all. I mean, as digital activists were still very much a minority when it comes to valuing personal data, yeah, there needs to be systemic changes made beyond the idiosyncrasies of GDPR. I mean I could flip the question back to you, right? I know you are interviewing me for this podcast, but I’d be kind of curious to hear what you think on that as well. 00:15:59 Domen Savič / Citizen D I mean… So my first campaign was back in, what was it, 2012? I think we’ve organized the the first anti ACTA protest in in the country and and back then, the field was very, you know diluted, you had you had different lawyers and let’s say tech engineers working on this subject and they came together saying, you know, well, this ACTA thing is really troublesome and we have to pull our weights together. But then afterwards they sort of drifted off in into their own, you know, spheres and this is something that was to me, a signal that, OK, we need more of a focused pressure on these issues because putting together a coalition just to have it fall apart after the fact, it’s not very effective, right? You can’t do that every time. And then then it was like from 2015 to 2018, yeah, exactly to the to the GDPR, it was this, I’m going to call it the golden era for activism, where you had small individual fights popping up and you could, you know, hammer them down individually on your own by motivating the crowd every time something came along now. And moving forward, you said it with with the AI and everything, I don’t see NGOs or activists being the right way to fight, because there’s just too much of everything, it’s happening constantly. It’s on all the time and at the same time people, you know you, you don’t have the time, the energy, the funds to sort of brief the people in real time about these issues and encourage them to sort of go against the big tech or the bad players in. And this is something that, that, that was very memorable to me when we met in Brussels recently and you spoke about the digital fatigue right and I love the term, because I was, you know, I am in the process of questioning that that very same question, that this is just untenable moving forward. Like, are we just, you know, destroying our health and whatever you know, because there’s just too much of everything and at the same time, there’s a very asynchronous feeling to the field that you said before in the trialogue and everything. You know, it all goes someplace you don’t want it to go right and you have no influence over the… 00:19:04 Mark Dempsey / Article 19 Yeah… And I mean, you’re absolutely right. I mean, digital fatigue is… things have changed so quickly, the results in terms of flops, civil society is looking for are relatively minimal, so of course there’s a level of cynicism, skepticism that creeps into the conversation, that brings to the question like, how do you mobilize civil society to be as effective as possible? Like why are we choosing our battles wrong? Are we, are we choosing too many battles? I think we must fight on. I mean I think we have to… there has been success successes, they have and there’s definitely a growing awareness of what civil society can bring to the table, at least within this EU bubble, but it is a challenge to bring about a culture where fundamental rights and everything that encompasses become a serious part of the conversation. And if you look at the DSA, we have been pushing the Commission to create a sort of more formal structure or a forum for CSOs to have a regular dialogue with the Commission, and we’ve had one or two workshops, but nothing has been formalized yet. But. I think they realize that, one, they don’t really have all the capacity that they need and that the CSO community, along with the academic community and all the various researchers attached to it, bring a huge amount of expertise, and I think with the DSA, a lot of this hinges on the provision which allows for data access. If that process happens properly, which is obviously not at all guaranteed, because I mean there will be ways which Facebook and Google use this process as a means of further controlling access to data. Yes, they give access to data, but it will be a long, laborious process till they get to that point where you can access it. And I think this is why the Commission really has to come in quite strongly and make sure that this access happens relatively seamlessly and to a wide number of people, so the right research can be done. So I think there are elements of hope there which should make our job easier, but ultimately, as long as they’re considering their bottom line, I the platforms they will be doing their best to make sure that that isn’t the case, but there’s the twin goal of making sure that people beyond our community understand what their fundamental rights are when it comes to the platforms. Then there’s the bringing in the culture and the common and the digital services can either reflect fundamental rights, how they need to be respected and how they can do that as quasi regulatory gatekeepers. But I mean, I think the digital fatigue also comes comes purely from the fact that you have all this change happening extremely quickly in the private sector side along with this huge number of legislative proposals which are very dense and no one still knows how they work with each other. So that’s where you got to take a step back holistically and understand the consequences of each piece of legislation. But then of course, there’s all the case law that’s going to naturally follow an that case, though, will be very important. But again, that’s a slow process as well. 00:22:42 Domen Savič / Citizen D True, yeah…I mean, I wasn’t trying to sound like defeatist, I just realized very, very soon that that in this area, like in the digital rights, you always have to question yourself from time to time just to sort of open up to, to new ideas or to new venues or to new approaches. 00:23:04 Mark Dempsey / Article 19 I keep referring to the EU public and the EU is really a bubble, there is not nearly enough voices being brought into the conversations in the global South, for example, and that’s a whole different discussion. But as we both know, there is a huge amount of data exploitation happening when it comes to the building of data centers in Kenya, for example, and the lack of frameworks to properly control power, the large platforms are working in local economies such as that of Kenya or in other African countries, there’s a there’s basically no governance framework there whatsoever and that doesn’t get discussed enough. And then of course the sort of the far-reaching consequences of the EU’s digital proposals, the so-called buses, in fact, but there are negatives negative externalities, which the Commission is is presumably becoming more aware of, but as more players from the global South get brought into the conversation and I think with Article 19, the European work is a very, very small piece because ultimately it’s a global organization which is working and protecting journalists in Mexico, Brazil, where arguably its work is more impactful, because then there’s a much greater need to do something now because of the actions of government, which tend to be more imposing themselves and the people in negative ways, which largely isn’t the case, and bar a few countries in in the EU. 00:24:53 Domen Savič / Citizen D Do you think that’s a good aspect of, this is going to sound weird, of areas, of countries that do not have formal regulation in place? So do you think activists can do more in the jungle than in the urban centers where everything is written down and stamped and approved? And yeah, you have all these institutions that are supposedly there to help you, but in practice… 00:25:29 Mark Dempsey / Article 19 Yeah, it’s just an interesting question, what I suppose they can learn from the failures of… I mean do they have more opportunities to change things quicker? Possibly. Possibly. I mean, I’d have to think of an example. 00:25:56 Domen Savič / Citizen D Yeah, I mean just just looking at at everything that’s happened in in the last, 10 years, I remember a couple of instances where you had working in inside the EU or working to change the EU proposals but you’ve often realised that the length of the process, when you have the a very narrow window of impacting the legal frameworks or the regulatory process and then, you know, two years go by, three years go by and this law comes into power and then everybody’s like, wondering, yo, what’s this? And then you go well, remember, like, three years back when you said it was nothing and we said it was a big thing and you should act on it then. Well, this is it, right? So, I’m thinking the whole systemic legal, the length of the process sort of you know helps the bad guys to sort of have the upper hand while activists are battling from day-to-day and then realizing that OK, now we have 2 years of nothing before people realize what’s going on in the first place, right? 00:27:26 Mark Dempsey / Article 19 Yeah. I mean, are you speaking to the the sort of speed of how things change? And then by the time it becomes law, it’s more or less, I wouldn’t say outdated, but less relevant than it might be? 00:27:36 Domen Savič / Citizen D So there’s that, and there’s also the whole legal process or the regulatory framework process takes so much time that people lose attention in between or people stop following the issue and then realize after everything’s done that, whoops, this is something we should pay attention to two years ago. 00:28:00 Mark Dempsey / Article 19 I mean, I think it’s very hard to have a fast regulatory or a fast legal process, right? Because whatever one says about democratic deficits, I do think way the Parliament is structured and the various committees and the rapporteurs and the champions, I think it’s actually quite a good process. I mean I think that the level of detail that one is forced to go into to have a provision the way you want it to be, I think it’s actually not bad. I think it’s just the trialogue process that really can become very deflating and I think again it sort of comes down to strong personalities who care about what we care about being in the position to push back against the Member States. And then of course, as the voting modalities, I mean in Council, it’s generally anonymity and then there’s this horse trading that’s done, for example, when Poland was in, was leaning towards an autocracy, you had it supporting Hungary at different junctures. And then of course when that starts happening that it reminds the initial piece of legislation. So I think there are real changes that need to be made in voting modalities and how the trial logs operate. But I think as I said before, it’s it’s hard to envisage that happening. 00:29:30 Domen Savič / Citizen D Before we wrap up and you’ve mentioned Poland and Hungary and this perfectly segues into the into the last piece of of the conversation. So, so you’ve mentioned it before the the European Media Freedom Act. How’s that going? Yeah, in terms of, when will we or can we actually already see some movements in in this area regarding the effectiveness, the usefulness of this proposal or is this something that is again ruined, so to say, by the trilogue, by exemptions, by doing everything and nothing at the same time? 00:30:14 Mark Dempsey / Article 19 I think it’s the the eventual outcome certainly wasn’t what we as a civil society activists would have liked. I think for some, it’s a bit late. I think it will be interesting to see how the Commission reacts to what’s happening in Slovakia, because arguably it’s already acting against the spirit of the regulation, which is on the statute book, so it’s effectively… they need to be acting with the letter of the law, or at least the spirit of the regulation and they’re not. So I think the Commission has to come down quite hard and fast to stop Slovakia going down the route of Hungary. I think it’s a piece of legislation, it was always going to be very difficult because you’re bringing together a huge number of actors and you’re regulating media in ways that you didn’t do before from a European level. I think the juries out. I mean again… you have this new European Board of Media services, you’ve got a lot of responsibility on the national regulators, the Commission has to make sure that there’s a proper effective, well resourced secretariat in place to service the board and my understanding is that none of these are European level. There’s still a lot of institutional building to be done to make sure that all these are properly in place. I honestly it’s difficult for me to say how this is going to work out, but I think it… on a very on a basic level, it’s good that it’s there, that the law is now is now in place and there are there are elements of law, like the Merger Assessment test, to retain plurality and make sure the media freedom stays in place and that mergers don’t just happen without other economic avenues being sought to make sure that this an independent media outlet can exist independently. Uh, I think it’s great that they’re there, but it’s going to be interesting to see how such a complicated regulation can be properly enforced and again, the Commission has put itself in a position where it’s sort of in the eye of the storm. That if these regulations which largely… they are there to protect democracy, particularly the Media Freedom Act, don’t work as intended, I think they’ll really end up undermining themselves in a big way. So again, if we have, with the impending elections and you’ve got a right leaning Parliame, at least these legislative pieces are there, but have you still got a right leaning Parliament and not strong leadership in the Commission, these won’t be particularly effective. 00:33:08 Domen Savič / Citizen D Yeah, this was actually going to be my wrap up question to this debate and it’s all very interesting so we’re coming into this, pre election or pre pre election cycle for the European Parliament and I was wondering to hear your thoughts on how should we, let’s say, as NGOs, but also maybe as as journalists or as media, as public opinion shapers, how should we frame the the issue of of digital rights? What is the difference, talking about digital rights now compared to, let’s say, the last the last pre election pre election cycle that happened 2018, 2019? So what should we change to sort of bring more impact to these topics and not just brush them away as “these are kids on TikTok topics”? 00:34:07 Mark Dempsey / Article 19 So I’m going to like, I feel quite strongly that the way over the last five years, digital has become even more… it’s become a much an even stronger part of our daily lives. It’s everywhere. I mean it’s remarkable how it’s intertwined in everything we do, and arguably it already was in 2019, but even more so now. So for me, digital rights cannot be something that is apolitical, I think digital rights has to be has to become explicitly political and what do I mean by that? I mean that if we look at what is happening in Gaza, if we look at the +972 media outlet article the other day about the use of AI when it comes to targeting civilians in Gaza, much of that AI was designed by platforms like Google. There is a complete lack of accountability when it comes to the interaction of geopolitics and the platforms role in it. So I think that as digital rights activists, we have to start embracing the political implications of this and calling out the platforms for involving themselves in conflict, for example. Because in conflict you’re taking sides, so these are not neutral actors and I think as digital rights activists, this is something we need to become much more involved in. And yes, it’s more, it’s of course more sensitive and of course it has to be more nuanced, but I think that there’s a leadership role there to be taken by digital rights bodies and communities and at least having that conversation with geopolitical actors. I think that for me, that’s the major gap that exists and that was for me that was very much highlighted when it came to what’s happening in Gaza. 00:36:07 Domen Savič / Citizen D Yeah, that sounds perfect, thank you Mark for the pleasure, this was this was really interesting, lovely to to talk to you. Best of luck going forward and hope to see you again sometime in person. 00:36:23 Mark Dempsey / Article 19 Thank you very much and pleasure. I really enjoyed this. And yeah, there’s lots we could talk about even further, of course. But yeah, see you next in Brussels. Citizen D advice: Politicize technology and digital services Connect the individual active user with systemic safe-guards and authorities Rethink the role of digital activists More information: Joanna Bryson: The Past Decade and Future of AI’s Impact on Society – pdf Article19: Watching the Watchmen – pdf Taming Big Tech: a pro-competitive solution to protect freedom of expression – pdf When bodies become data: Biometric technologies and free expression – pdf About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!
Mar 15, 2024 · 34 min
097 Alja Isakovič in etični razvoj tehnologije
Alja Isakovič, razvijalka programske opreme in uporabniških procesov, se v zadnjem času posveča izzivom etičnega razvoja informacijske tehnologije in tehnološkemu okoljskemu odtisu ter poudarja aktivacijo posameznika in povezovanju v skupine za doseganje družbeno relevantnih ciljev. V pogovoru se dotakneva tem vključevanja žensk v razvoj informacijske družbe, problemu tehnoloških revolucij, ki jedo svoje otroke, načneva pa tudi probleme gradnikov umetne inteligence, o pomanjkanju volje in moči posameznika ter lokalni aktivaciji ter razlogih za optimizem na temu področju. Premišljujeva tudi o prihodnosti, evropskih volitvah ter politični komponenti tehnoloških tem, ki bi morale priti na tapeto novih odločevalcev. Državljan D svetuje: Potrebujemo več razprave o politični komponenti razvoja informacijske družbe Pomembno vlogo igrajo razvijalci in programerji, ki lahko spremenijo dejanske poslovne prakse Vedno bolj pomembno je povezovanje večih različnih panog Dodatne informacije: The Cult of AI: razmislek o kultu umetne inteligence in povezano Can Myth Teach Us Anything About the Race to Build Artificial General Intelligence? With Josh Schrei o skoraj religioznih motivih tistih, ki zagovarjajo razvoj umetne inteligence za vsako ceno These Women Tried to Warn Us About AI: o raziskovalkah, ki že leta opozarjajo o negativnih vplivih umetne inteligence in ceni, ki jo za to plačujejo v svoji karieri AI is dangerous, but not for the reasons you think: pregled trenutnih negativnih vplivov umetne inteligence, z okoljskim vidikom na čelu A polar bear cub is harmed every time you use ChatGPT: moj uvod v problematiko okoljskega vpliva umetne inteligence AI Countergovernance: primeri, kako se lahko s skupnimi močmi upremo Black Mirror vizijam prihodnosti Pathfinders Newmoonsletter: Tethix mesečnik, ki vsak mlaj posaja navdih za drugačne tehnološke prihodnosti O podcastu: Podcast Državljan D je podcast za produktivno preživljanje časa, v katerem igramo vlogo državljana. Državljan D v pogovorih s strokovnjaki določenega področja informira in aktivira. Da se. Gremo naprej!
Feb 15, 2024 · 25 min
096 Katarina Bervar Sternad in boj za človekove pravice
So človekove pravice zadnje orodje revnih proti finančno premožnejšimi elitami ali gre za vrednostni sistem, ki ga sprejemamo vsi udeleženci družbenega okolja? Z direktorico Pravnega centra za varstvo človekovih pravic in okolja ter eno od pobudnic Pravne mreže za varstvo demokracije, Katarino Bervar Sternad smo se pogovarjali o logiki človekovih pravic, vlogi nevladnih organizacij, problemom delovanja v prid človekovim pravicam in poti v svetlo prihodnost. Kakšna je situacija na področju človekovih pravic v Sloveniji in kakšne so razlike v primerjavi s tujino, kako so v Sloveniji pozicionirane nevladne organizacije in kaj je političen problem človekovih pravic? Državljan D svetuje: Človekove pravice morajo postati politični standard Nevladne organizacije so tu zato, da držijo ogledalo odločevalcem in ne, da jih podpirajo Preizpraševati je potrebno model financiranja nevladnih organizacij na področju zagovorništva Dodatne informacije: PRIPOROČILO KOMISIJE (EU) 2023/2836 z dne 12. decembra 2023 o spodbujanju udejstvovanja in učinkovite udeležbe državljanov in organizacij civilne družbe v procesih oblikovanja javnih politik – povezava Mnenje Evrospkega socialno-ekonomskega odbroa o podpori in financiranju civilne družbe na področju temeljnih pravic, pravne države in demokracije – povezava Koalicijska pogodba 15. Vlade RS – dokument Monitoring zavez vlade, Glas Ljudstva – povezava Strateški svet za preprečevanje sovražnega govora izdal 57 priporočil vladi – članek Evropski parlament nagradil slovensko Pravno mrežo za varstvo demokracije – članek O podcastu: Podcast Državljan D je podcast za produktivno preživljanje časa, v katerem igramo vlogo državljana. Državljan D v pogovorih s strokovnjaki določenega področja informira in aktivira. Da se. Gremo naprej!
Jan 15, 2024 · 53 min
095 Urška Henigman in spolnost med politiko, mediji in družbo
Za začetek leta se z Urško Henigman, novinarko uredništva izobraževalnega, otroškega in mladinskega ter dokumentarno-feljtonskega programa na prvem programu nacionalnega radija pogovarjamo o medijski reprezentaciji spolnih praks v Sloveniji, ujetosti spolnosti med politiko, mediji in družbo ter o uredniških premislekih pri oblikovanju s spolnostjo povezanih medijskih vsebin. Pogovor teče o zgodovini medijske reprezentacije spolnosti v Sloveniji, težavah z iz tujine plačanih aktivistov, ki spolnost spreminjajo v temo strankarske politike, o vsebinskih premislekih in potrebah občinstva na tem področju. Kaj vse je spolnost, kako nastajajo s spolnostjo povezane medijske vsebine, kako pomemben je senzibiliziran novinar in zakaj je takih vsebin še vedno premalo? Državljan D svetuje: Spolnost je politična tema, o kateri je potrebno govoriti tudi skozi medije Politično uokvirjanje spolnosti je delo interesnih skupin Človekove pravice niso predmet večine, ki bi rada omejevala manjšino Dodatne informacije: Evolucija užitka – podcast Slovar spolne vzgoje – serija Na pravi strani – podcast Glasovi svetov – oddaja Prvič – nova serija O podcastu: Podcast Državljan D je podcast za produktivno preživljanje časa, v katerem igramo vlogo državljana. Državljan D v pogovorih s strokovnjaki določenega področja informira in aktivira. Da se. Gremo naprej!
Dec 15, 2023 · 39 min
094 Claudio Agosti and the algorithm economy
Claudio Agosti is a fellow researcher at the University of Amsterdam. He has a hacker background and security professional. He researches, implements, and promotes technology in the public interest. In the last decade, he addressed issues such as communication protection for whistleblowers, analysis of the web-tracking phenomenon, and algorithm analysis. In 2019 he started to work with the ETUI Foresight Unit on research and training courses aimed at understanding the ‘technicalities’ behind Artificial Intelligence. Five years later, the ETUI is releasing a technical report which meticulously outlines the techniques used by researchers to observe the internal logic of the app used by riders in Italy and documents its actual behavior in terms of harvesting their personal data. We sat down with Claudio to discuss his investigation, the consequences of unchecked share economy and the way forward. Transcript of the episode: Expand the transcript 00:00:23 Domen Savič / Citizen D OK, so welcome everybody. It’s the 8th of November 2023, but you’re listening to this episode of Citizen D podcast on the 15th of December 2023. With us today is Claudio Agosti, algorithms explorer and digital rights evangelist at the AI forensics NGO and the topic of today’s discussion is worker rights in the digital economy. So first of all, hello. 00:00:52 Claudio Agosti Hello and thank you people of the future. 00:00:56 Domen Savič / Citizen D Let’s start with the quick recap of the report. It’s titled “Exercising workers rights in algorithmic management system”. What does that mean? What was the topic of the report? What did you investigate and what were some of the findings? 00:01:15 Claudio Agosti Thank you. The report, it talks about a story, an investigation that began in 2019. I founded a project Tracking exposed. That was a project meant to do algorithm analysis, and we were analyzing platforms of social media or of Amazon and other web platforms. But I get in touch with the Aida of the European Trade Union Institute because I met her at the privacy camp, the event organized every year in in Brussel to put together privacy folks and other people considered on this right. And she was fascinated about our approach to analyze algorithms because it’s a black box analysis. And it was in 2019 we met. I started to collaborate with that institution to teach a bit to trade unions how they should be skeptical of the apps that run on the, let’s say, on the riders phones, to organize their work, because those apps may violate certain kind of privacy rights or also some kind of labour rights. Initially it was just an insight, an intuition, but only in 2020 or let’s say around the time we start to try to make some investigation. And through that we approach it in two ways. The first was making a survey, a set of questions that were meant for riders, to understand if they felt that the techniques that the technology that organize their work was discriminating on them, that’s because if you want to use some, if you want to bring it to court a company, you need to have evidence that this company did something bad, and the evidence of the violations. In the other path instead, we were doing analysis that was only technical. So, try to do the best engineering of an app, but to do a reverse-engineering of a rider app you need to have a login and password of a rider because the app starts to execute and do all the potential privacy leakage or the surveillance of the worker only if you log in properly and we needed to find a rider willing to share and a login and password, and that was particularly difficult. We also tried to subscribe ourselves to be rider, but we did not get accepted. I don’t know if it was because of the place we were living. But after 18 months and that’s this huge amount of time… So one of the important cost of this investigation that could have been reduced and will be reduced in the future, but after 18 months, we found a person willing to share those logins and passwords. So we set up a mechanism and this methodology that initially was some static analysis by Exodus privacy, which is an online service. I suggest you consult because it shows by doing static analysis how many known trackers are present in every mobile app. Then we meet them, a proxy is a software that allow you to intercept the traffic that the app is performing to the outside, and then with the freedom is a system that allows to run the application on a sort of special environment, where the calls made to the system can be intercepted and recorded or modified and in that way we can observe when the app was actually accessing to the GPS or to other peripherical. So, with these three methods we start to observe how the app was behaving and we start to realize that. As first was revealing, the location of the worker, even outside of the working shift. The second is that inside of the communication, about the profiles, so we were intercepting traffic and inside of this communication between the app and the global infrastructure you were seeing that some requests were made by the app to understand who’s the rider that is using the app, and so you are seeing the profile and the information tied to this profile. And then there were other requests more focused on getting new orders. Inside of the profile of the rider, there was a present. A score. That’s it. I mean, it was not the number you could have expected in the sense that official League Global acknowledged the existence of an excellent score. And we realized that the excellent score was a different score. So therefore, there was a hit then scoring mechanism present or let’s say present in this communication, then how it was used by the app. But infrastructure, we don’t know. But that’s was an evidence that even if it’s not that surprising for a Labor Unionist, it’s very important to have this evidence because Labor unionists in the past hugged, they requested that the worker should not be subjected to, let’s say, the vote or it can be subject to the vote, but the voting that they get from the customer should not impact their ability to win. And that is part of the labour right. It’s not that your worker can stop because someone starts to vote you poorly, even if this seems to be a standard in the online market, because it’s normal for me when I buy something I don’t know on eBay to check the task worthless. The seller is not OK that uh, if uh, your life and your work depend on a system. Someone can game this system and start to downvote you and make you suffer a loss of business. And last but not least, we saw that there were also third parties not declared in the contracts or in the privacy policy that we’re getting all this information about the user profile and their location and everything that we’re doing in the app. So, you click it here and you move this panel where you were when this happened. How much you are moving in that moment, all those kinds of detailed information were given to third parties, that’s, is another problem. But I don’t want to keep reassuming it, there is the report that is a 60 page long and there is also a video of 50 minutes around with me and Joanna that talks about it and you can find it on the website https://www.reversing.works because tracking exposed the project that I mentioned before closed this year and it became two different projects, one is AI forensics. The one you mentioned that carry on the algorithm analysis toward the influential algorithm. So we look at TikTok and being chart and the language model. So some part of the algorithm analysis is carried on by forensics, reversing works focus more on the impact of algorithm and surveillance capitalism in worker rights. So this kind of effort is captured by this different group and the website reversing works contain also more reference about this report. 00:10:08 Domen Savič / Citizen D So in your investigation you’ve investigated one app and one company basically on one market, right? So how fair would be the assumption that other providers of these types of services and the same providers in different markets? Are they using the same, let’s say, tactic, social scoring, hidden grades and so forward. Do you think it’s this? Is this limited to 1 market or is it present everywhere? 00:10:43 Claudio Agosti I believe there are insights that make us assume this is a frequent condition because one sided analytics third parties are in between offering analytic service to them. So showing how the app is actually used, but they’re also tied to the marketing campaign, so they use the data collected this way to resell them in the advertisement or don’t know customers certification or brand analysis market… This is still part of surveillance capitalism in the sense that for me, if you install a third party that monitors my behavior and uses this information to make a product out of it, that is the exact definition of surveillance capitalism. And those companies, let’s say user the offering of analytics or other analysis services as a reason to be in the device. And that’s, I believe, a phenomenon that is present in many apps. The exodus privacy website that I was mentioning can allow you to inspect which trackers are present on the app and then this tracker in all the apps is present and that can give you an idea on how this phenomenon is very widespread. This phenomenon is wide spread in the market and the problem for me is when this also has an impact on an app that you have to use to do your work because you are not a user but you are a worker. So worker rights apply to you and not just to the service. So because those apps are born under the surveillance capitalist Internet is fair to assume that this behavior is quite spread. Then not all the apps have the same amount of trackers, all of them, or let’s say all the one I checked have a heavy presence of Google services, so there is some Google infrastructure that run together with the company infrastructure and the Google infrastructure also receives a large amount of personal data. Now we can talk about Google and the current EU US privacy agreement. But that is a different topic. Other kind of behavior like secret scoring or other kind of discrimination are all part of the business optimization that made the economy effective and now is the time for the worker and for the labor union. And also for the regulators to be updated on how this problem is affecting the population and perhaps make better regulation because we cannot expect someone that start to make a tool that to optimize and the delivery and use this tool on different continents of the world. Only if it’s force, we start to implement a different policy, a different regulation and perhaps something more protective for riders rights. In a specific country only if it’s forced, at the moment they are not yet forced. That’s why I assume what happen in a market also happen in all the market where this app is present. 00:14:27 Domen Savič / Citizen D Before we move on to the topic of workers’ rights. I just want to ask a follow-up question. So, this app is workers or writer specific, right? So, it’s not the same app as it is when you know consumers are using it, I’m guessing, right? 00:14:45 Claudio Agosti Correct, this is a different app and it requires login and password. 00:14:51 Domen Savič / Citizen D So my question is, would you say the consumer app is using similar, let’s say strategy and approaches to getting more information about their users than it says on the label, so to speak? 00:15:08 Claudio Agosti I believe that, uh, in the consumer app this is granted and that is also written in the fine print. Because in excuse privacy, you realize how the, UM, analytics, uh and a marketing campaign is the same that is present in the Courier app. So, it’s really a third party that is collecting data from couriers and from consumer in the consumer you have this term of service when you create your account. Let’s say we’re going to use your location because they must deliver food to you and therefore when you have the app open, it’s normal that they know your location and they’re going to study what you consume. So let’s say it’s not surprising that this is happening. This is, let’s say, more justified as a way to give you a service is that for the for the rider that your location is surveyed outside of the working shift, that is not justified. 00:16:11 Domen Savič / Citizen D Mm-hmm… and what was the reaction of the unions. You’ve mentioned you’ve cooperated with workers unions on this on these issues. What are they worried about this? Are they paying attention? Is this a topic that they are, let’s say, actively pursuing? 00:16:34 Claudio Agosti Partially, in the sense that I tried to work with the Union and explain how that was a potential problem, but they don’t have those kinds of technical skills. That’s why we started to make analysis independently to show that there was evidence and potential then because you have also to think how your finding can have an impact and we went to that protection authority. In 2021, we start to collaborate with a lawyer, but this lawyer took two years to present a complaint and this complaint initially was rejected by the Italian authority because the data were too old and we show that the version of the app was changing nearly every week. They initially rejected this application. This summer, after this rejection, we made a test and instead of making a legal filing that was articulating potential violation. We just submitted a technical analysis and that allowed us to make the analysis and report in one week. This has been accepted. So now there is an investigation open in the election authority in Italy. Why is important because in 2021 are related from our investigation, the authority already fined global for 2,000,000€ as they did not have our insights, but they had other kind of insights because they were running an investigation, they proceeded as official, which is the authority can investigate on what they want because they feel it’s important and because in 2019 the Italian Government, the Labour Ministry, was making some high-level agreements with the rider company, then that artists are to make some investigation to verify how much they were compliant and in 2021 that investigation was concluded and the fault global faulty of many different reasons. Interestingly, Glovo appealed and in 2022 and the judge said to Glovo, you don’t have to pay the fine because it’s too high, but the authority not only issued the fine, but also requested many remedies. The solution is that the app stopped to be abusive that the true solution and the global apparently ignored all this remediation because we were doing our technical analysis in this time and we were seeing that the problem was still present. Now I know after we submit our technical evidence this July, at the end of September, the Constitutional Court comment on the appeal and revert it and say the “No, no Glovo, you have to pay the fine.” So now after two years they use the global how to pay the fine, how to implement remediation 2 years ago and that there is an investigation in progress that showed that they did not implement this remediation and there were more problems than before. So we tried to use GDPR to enforce labor rights, and that’s even if for us price activist seems to be a linear behavior for labor union. It’s not because normally a labor union protects collective rights and they go to defend the rider in a different strategy. They want to have their contract be fair, and then the company can do the business they want as long as the work is stated fairly. What we want to argue is that if you want to talk about power dynamics in the modern work environment, you need to understand how the technology works because it’s in the technology that the power will be exerted. So it’s not just enough to protect the contractual obligation and the agreements. And so it’s a new thing for the labor law labor union thing that GDPR or privacy right can be used in in this collective frame and that’s why it has been a bit complex initially to engage with them. At the moment we don’t have any kind of active projects with unions yet. We are trying to show that they need to be trained to learn about this and show that this problem is present in every European country. So now is the replication and scalability of this experiment. That is our goal at harvesting works. 00:21:34 Domen Savič / Citizen D You’ve mentioned GDPR. Do you think the current European legal frameworks in this area, the Digital Service Act, the Digital Markets Act, all this that is coming up or has come up in, the past few years. Do you think these are effective legal tools to pursue these types of privacy abuses. 00:22:07 Claudio Agosti I will relay only in GDPR. The Digital Service act or also the currently debated platform worker directive, they may offer additional tools, but the digital service act is not meant for those that kindplication and services and also I don’t know yet I did not make a full assessment on what are the new legal tool that we have in our ability to speculate how we can use them for worker rights. At the moment GDPR is already giving to us a lot of tools and there are many parts that can be explored. One of the experiments that we made was also to use data subject access request because if you start to get data about you, you can start to understand what is preserved and based on what you get about. Based on what you get some service offered or not, and those form of transparency, they should be more enforced by the Platform Worker Directive, but this directive needed to be as first need to be completed in this cycle. And at the moment the trialogue is stuck between the contract or obligation so should be there either be a full-time employee with some contractual agreement that protects them or not. That is where the focus of the trialogue is. There are some articles that talk about algorithmic transparency and possibility of oversights, and those articles need to be translated in national law. I believe there will be some potential of lobbying in that phase because algorithmic transparency is something that nobody ever saw. If you look at what Instagram does, for example, is making a blog post, let’s say algorithm works in this way but but for me algorithm transparency means that every time I get something assigned, I know based on what and even if I get excluded by something, I need to know why. When I’m doing some activity I should be aware if this has an impact or not on my ability to work in the close future. So transparency means full accountability, full control and the constant feedback between what the app record about you and how this will impact your life and you see this spectrum is not defined in the regulation need to become part of the demands at work. Also, perhaps the labor union, should also promote those kinds of demands because they can be part of the collective bargain. At the moment it’s not, that is why the fight for algorithmic transparency, if not for more control, is still to be thought, and the current regulation help enough, and really, even without new regulation, we have to explore many paths that GDPR gives us. And uh, more than having privacy activists exploring those regulation, these matters we have a strategy that is with the labor union is with other representative of the gig economy workers is with the worker itself. 00:25:53 Domen Savič / Citizen D Hmm, I’m asking because the election year is coming up in the in the European Union or in the Europen parliament and I’m curious to hear your thoughts on the, let’s say, the media representation of these types of issue. Is this a topic that is being discussed in the media reporting? The reason I’m asking is at least in Slovenia, the political decision-making processes and the media reporting go hand in hand, right? The politicians are usually picking up topics they want to represent or they want to address in, let’s say the Parliament or in their work based on the amount of media reporting. So if an issue isn’t reported enough or isn’t reported regularly, then the politicians are saying, OK, I’m not going to get any brownie points for addressing this issue so they just don’t, and I’d like to hear your thoughts or your analysis of situation in Italy in this field. 00:27:12 Claudio Agosti Well, it’s quite weak in the sense that we had some coverage when we raised the report and section of the GGL, which is one of the important labor unions, talk about it and organized a strike in Milan where the riders were asking for algorithmic transparency. So that was the cyberpunk moment when the actual last layer of the pyramid demanded algorithmic transparency. But besides that, that event, this topic is not touched, not discussed, maybe because privacy or general data processing is abstract and it’s not that compelling. It’s not that seen as a as a problem and the media did not catch up. It’s different in Spain. In Spain there is El Diario, a media outlet that is dedicated to cover a lot of the rider issues. But that’s also because Glovo is there and there are other companies that developed the gig economy based in Madrid or somewhere else in Spain. And then there is the layer rider, so theydedicated the law that protects more labor rights in the economy. So let’s say Spain was a bit more advanced in the public debate, and that has also converted these demands into a better regulation. So what you say is, is correct. Perhaps if there is more coverage that will be also more progress in in labor rights, but at the moment in Italy we don’t have a government that care too much about this topic, yeah. Other countries I’m not that aware of special movements. 00:29:16 Domen Savič / Citizen D The reason I’m asking is exactly because we’re slowly wrapping up and I’m trying to wrap up these discussions in, in a sort of looking into the future, right? So what would be an effective way to to address these issues going forward? We need more cooperations between privacy, digital activists and trade unions. Do we need better journalistic training in this regard, to sort of, you know, hone the journalistic sensibilities around this issue so they’ll be able to write about them, to investigate them? Do we need better regulators so that, you know, activists and other players in this field from a non-government sector will basically stop worrying about the atomic bomb because it will be regulated well enough. Is it all of the above? 00:30:13 Claudio Agosti Yeah, a bit all of the above especially have to say also in the privacy activism sector, despite in theory, those problem have been looked they were not considered different from any other privacy. Instead, they can give to us more tools, because if surveillance, privacy evaluation, sorry, affects regulation, a law that protects riders, that’s sorry workers and this law is stronger than others, then is a legal tool that you can use for social privacy improvement. So this path has not been that explored and also when I try to advocate for it or pitching to some to some grantor, let’s say very few are keen on this idea, especially grantor that came from the US, never saw the labour rights as something important and most of their accountability effort is focused toward this information or other form of platform accountability Labour has not been a compelling keyword, even in our sector of activism now, I hope it will be how we can help that I believe improving the amount of evidence is through that we made an analysis by involving one rider in one app in one nation, but that can be easily replicated. We understood what were this low point like finding a rider and to give to us login and password and the legal filing instead of filing a legal analysis, we can just provide the technical assessment to the authority and then they have the legal expert that will make a case out of it. So now that we identify the two slow points, we can train other technologists to do this kind of reverse engineering trainer, union unionist or rather advocate to do at least superficial analysis and understand what’s potentially problematic and talk more to the workers. But I mean it’s clear that a worker that has already the problem of working for an app perhaps is not that interested in the technical analysis of that app? Is a kind of reporting and analysis that requires a lot of technical information or a lot of abstract thinking on how this affects society, how these affects your ability and your autonomy, and be a worker. Now at the end, the algorithmic influence is not a problem that is compelling as other physical problems. It is a quite abstract problem and we’re still there. That’s why, for example, there is a documentary that I forget the name now, but I will offer you the link. It’s a documentary produced by some colleague in Herman Center. They are part of Europe media initiative in Italy that investigates the gig economy, and they made a documentary that, for example, interview a a worker, and they asked her, so what do you feel about the algorithm? And she honestly answers, what is the algorithm? I mean at the end they should just get order in an app, and explaining what there is behind the scenes which impact, which power has that is not that granted for us. That’s why the privacy advocate I believe are the community that more easily can start to catch up and eventually give us some thought and effort in this domain. But thelarge-scalee communication has a lot of complexity. 00:34:33 Domen Savič / Citizen D That’s true. And just one more, one more question, since you brought it up several times during the discussion. It’s basically… we always try to question the idea of this empowered user, because this what’s basically is sold to us by the, you know, app providers, by the tech industry that says. you know, we provide, or they provide all the information and the user must decide for himself or herself, what does she want and what she doesn’t, or he doesn’t want, right? It’s also in the GDPR. So, the GDPR is focusing on the personal capabilities of a user to be responsible for his or her data. And, you know to worry about it and to have tools that help her or him, you know, use the data, delete the data, remove the data, check the data and so forward. But on the other hand, as you mentioned, the users, they’re not all super users, right? They’re not going to be very invested in this investigative work, reverse algorithm, type of dealings. So, do you think we need more emphasis on collective protection? So, you’ve mentioned privacy activists as a possible solution, also an establishment of, let’s say a government body like the Information Commissioner or the Data Protection Authority to focus particularly or specifically on the algorithms on the decision-making systems that are running, you know, behind the behind the curtains to sort of, you know, speed up the process of addressing these issues in real time. 00:36:22 Claudio Agosti You talk about workers or every algorithm? 00:36:24 Domen Savič / Citizen D Generally, but yeah, we can focus on workers, yes. 00:36:28 Claudio Agosti Because for workers, in theory the body that protects the situation that protects workers, interest and rights should be the trade union and the problems that they are not updated on the new challenges perhaps? That shows if you want the institution that were established when Internet arrive have not caught up and this, we know it because we saw how certain kind of privacy regulation arrived 15 years after the beginning of the surveillance capitalism. But uh, but some institutions are even, uh, tougher to change and to update themselves and the Labour Union seems to be one of the most traditional ones. And I would hope that these algorithms or these logics should be transparently communicated to the worker and to the Union so they can oversee and agree on what is in their best interest for everybody. We are far from it and regulation may go in this direction, but it needs to be mostly a cultural shift. You need to have a worker asking for it, and you need to also to understand what is better for you. Because I think this example, if I just made a delivery and I have to bike for 30 minutes, would be fairer for me if I stopped for 5 minutes to take breath or because I’m on my working shift I should optimize and do as much as delivery I can? So what is the logic? The algorithm implementation is also something that is thinking how to better optimize the time of the worker and how to better offer the services. But what is better for the rider? Does the rider have ever had the opportunity to ask for it, or do the rider have a special interface that say, OK, today I’m working, but I don’t feel that well, so please, slow pace or today I’m working and I want to make as much money as possible, so please give me all the difficult deliveries. So those are not the options that have been ever discussed but that’s if happen is a sign of our society that actually understands the new power dynamics, understands how they’re implemented and has all the actors that equally speak and ask their demands. We are far from it, but that is what I hope to see. 00:39:34 Domen Savič / Citizen D Fingers crossed. Thank you, Claudio or I guess the final call to action would be to update the trade unions that that would be. That would be a good start. Thank you so much, Claudio, for sitting down with us. Best of luck with your future endeavors. This was a Citizen D podcast episode, we publish an episode every month, so we’ll see you next time. Citizen D advice: Workers rights need an update related to the information society Privacy is not a commodity The globality of surveillance capitalism needs to be tackled locally More information: Exercising workers rights in algorithmic management systems – Lessons learned from the Glovo-Foodinho digital labour platform case – report AI talks @ ETUI: Algorithmic management – event AlgorithmWatch in Italy – reports About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!
Nov 15, 2023 · 31 min
093 Sarah Lamdan and data cartels
We sat down with Sarah Lamdan, Professor of Law with a Master’s Degree in Library Science and Legal Information Management. Professor Lamdan works with immigration groups on government surveillance issues, with library advocacy organizations on open access and researcher privacy projects, and with open government advocates on federal records preservation and access initiatives. She recently wrote a book called Data Cartels: The Companies That Control and Monopolize Our Information that is focusing on the issues of internet feudalism, data monopolies and solutions to these issues. You guessed it – this will be the topic of our conversation. Transcript of the episode: Expand the transcript 00:00:10 Domen Savič / Citizen D Welcome everybody. It is the 25th of September 2023, but you are listening to this podcast of Citizen D on the 15th of November 2023. With us today is Sarah Lamdan, professor of law with a master’s degree in library science and legal information management. Professor Lamdan works with immigration groups on government surveillance issues, library advocacy organizations on Open Access and research and privacy projects, and with open government advocates on federal records preservation and access initiatives. She recently authored a book called Data Cartels, the companies that control and monopolize our information and you guessed it, this will be the topic of our conversation, Professor lent and welcome. Welcome to the show. 00:00:56 Sarah Lamdan Hi, it’s great to be here. 00:00:59 Domen Savič / Citizen D It’s such an interesting read and it’s such a great topic to discuss because you start with the issue of open data, of data cartels, of companies that control and monopolize our information from a very, I should even say, personal experience. So I would like to know, to get us going, what was the reason you developed an interest in this topic and how did you go about how did you go about researching it. 00:01:28 Sarah Lamdan Those are two really good questions. So to answer the first question, I kind of fell into this topic by chance. In a way, I’m kind of the ideal person to write about it, cause I didn’t come in with any sort of agenda. I was a law professor and also I’m a librarian, so I deal a lot with information access and informational resources. And I actually saw in the news that a lot of our research providers in the United States were vying to work with our Immigrations and Customs Enforcement Agency, so ICE. And I’m not sure how it’s viewed across the world, but in the United States, especially in 2017, when I first started digging into this issue, ICE was problematic. Immigrations and Customs Enforcement in the United States was known as committing human rights abuses. Basically, separating families at the border, putting children in cage like enclosures and just doing all sorts of really icky gross things. So working with ICE wasn’t positive, right? It wasn’t about helping immigration agency reunite families or help people ascertain citizenship, it was really about human rights issues. So I became very interested in why our research providers were working with ICE and what they were giving ICE and I started asking, you know, I started asking our research vendors at the school that I worked what is your company doing with ice? And I asked in my professional organization, the American Association of Law Libraries what exactly is Lexus, our main research provider and Westlaw, our main research provider. How exactly are their companies working with ICE? And instead of getting the answers, I got kind of censored, the American Association of Law Libraries wouldn’t allow me to ask that question on their web pages and at my law school, my vendors became very agitated when I asked them about about their work with ICE. I wasn’t getting answers and that’s really what started the research process behind data cartels and kind of all the research I’ve uncovered then. And really what came after that even-though I’m an academic, the research process for digging up those connections that I described in the book, it was almost journalistic. It was almost more of an investigative reporting type of research where I was trying to look at corporate filings and advertising and what work that other journalists had done to connect the dots between our research products and government data brokering and surveillance, which is what I uncovered ultimately. 00:04:44 Domen Savič / Citizen D Do you have any reasons… you mentioned in the book and you just mentioned the journalistic way of researching the topic, it seems that on one side there’s let’s say a fair amount of information on these systems in the public, you just have to find a way to get it, at the same time or on the opposite side people, as you’ve mentioned just now, are very are being very obtuse about it in terms of, you know, they’re not giving you answers and stuff. How do you reason those two extremes? Why do you think there’s such a such a code of silence related to to these systems in, in, in academia and in a broader society? 00:05:31 Sarah Lamdan That’s a good question. So I think most of it is obtuse for kind of public relations reasons. I don’t think that Reed Elsevier, Lexus Nexis or Thomson Reuters or any of the other entities that are doing government surveillance… I don’t think… let me turn the negative into a positive. I do think that these companies recognize that the public generally doesn’t like to be surveilled by the government and that the work it’s doing with government agencies probably doesn’t have a good public relations base. It doesn’t look good. So the companies themselves are purposely obtuse about the work that they’re doing with the government, in fact. I work with a bunch of organizations that do legal work around these issues, and they’ve actually in the United States found that American agencies actually have non-disclosure provisions in their contracts with Lexis Nexis that prohibit government agencies from discussing their contracts so they recognize it. They they know that the public doesn’t think it’s great that they’re working with ICE or that they’re sharing information with the FBI or the, you know, or other government agencies. It doesn’t look good and I think one of the reasons in academia that we feel a lot of discomfort around this topic… I think there are two reasons. I think the first is that we recognize how much we depend on Elsevier in science, direct and you know all of the other products that these companies provide us, we need them, right? We need to have good working relationships with Elsevier in order to get contracts in order to make sure that our academics get their work in the right journals, right? So it’s important for us and for all librarians and for our administrative staff and to work with these. And also the other reason. The other reason is that we don’t know a lot about what is going on in these companies, right. And more information about that is starting is to come out. And when it does, people like me or brands or everybody else who kind of does work in this area, we tend to get a lot of push-back from the companies. We get nasty letters from them in the mail. They call our deans and our and people in our academic institutions to to tell them that we are spreading rumors or false information. So it can be kind of scary for an academic who stand up to these companies without a lot of backing from their institutions or their professional groups. 00:08:37 Domen Savič / Citizen D So in your book and in the debate around data cartels.. you have, let’s call them two sides, right? You have government agencies, government institutions, public institutions that are buying or that are using these data sets that are working with these private companies, essentially, that are putting together these data sets. So who do you think started this, should we say the ball rolling? Is it the government coming to individual or independent companies, saying OK, we need that. Can you provide that or was it the other way around? Was it the public private companies putting together these data sets and asking or, you know, pitching this to the government saying “I’m sure you could find a way to use this?” 00:09:32 Sarah Lamdan That’s also a good question. I’m going to say that after every question cause they’re all good questions, but the that question just got an entire book answering it, so one of my favorite journalists who writes about data analytics companies, his name is Mackenzie Funk, his book is coming out, I think in the next few weeks. It’s called the Hank Show and it’s actually about the man who created these data analytics systems and it’s it’s a really fascinating story, I urge everybody to go out and get The Hank show, it is from Saint Martin’s press, it’s coming out I believe at the end of this month or maybe next month, so keep your eye out for it. But in the book he describes how the data analytics systems were created actually in the private sector. They were created in Florida by this man, jh created a company called Citizen and Citizen became Matrix and after September 11th, the man who created these systems, named Hank Asher, actually decided to market terrorism prediction products or products that would predict who is more likely of committing a “terrorist act”, he took that product to the White House, to the government and sold it to them so I think that that is kind of the system that these companies have been working on ever since. They create these predictive policing systems, these predictive fraud systems, these predictive you know, they predict all sorts of different things, who might default on a loan, who might be a bad tenant, who might be a good employee, right, who might be an insurance risk. So they they create all these systems and then they market them to health insurance providers, to auto insurance providers, to ICE, to the FBI, to, you know, any government agency, Social Security, Administration, IRS. All of these are examples of agencies that use predictive systems that that companies like these built. So I think that these systems are created in the private sector and then marketed to the public sector as easy solutions, and then they’re very appetizing to a public sector that can use algorithmic solutions and hiring more investigators and hiring more people, which is more expensive, more time, you know, it takes more time for a human to do investigatory work than for an IRS agent to just press a button and pull up a hot list of, you know, the 10 most likely fraudsters or what have you. Fraudsters is an adjective that Lexis/Nexis uses, so I like to use because I think it’s just like… they they find fraudsters. But the government finds it appealing to use these, you know, easy algorithmic digital seemingly miraculous solutions instead of hiring more agents and, you know, doing more expensive things and more time consuming things. 00:12:53 Domen Savič / Citizen D If we go step back and we leave the issue of the government buying these systems, how will the government effectually regulate them? That’s that’s a spoiler for later on but let’s start at the beginning. So why are data cartels bad for consumers or in citizens cause cause usually when you ask the government or you ask these companies, they go exactly the way you went. So they’re talking about, you know, effectiveness improving the work process, the finding, you know, a needle in a haystack. So they don’t talk about about the negative things while your book is basically just, you know negative and I would like to know why. Why is there such a discrepancy and what are some of the bad consequences that that derive from from the these these data cartels that are that are all over the place. 00:13:53 Sarah Lamdan Right. Yeah. It’s funny. People assume that I’m, you know, a Luddite or that I hate technology based on the things I describe in Data cartels. But I don’t. I I wish that these products were as efficient and miraculous as they reported. Unfortunately in their current form they are not and that’s kind of that’s what I described in my book. So a lot of the problems are problems around, you know, algorithmic bias and biased data sets, but I’m not an algorithm expert. I’m not a technologist. I’m a librarian, so I always direct people to other authors and other books about algorithmic integrity and algorithmic biases. You could read any work by Ruha Benjamin, by Safiya Umoja Noble, Kathy O’Neill… there’s a lot of really great work out there and research out there that describes why it is problematic for government entities to rely on algorithmic solutions for ranking people and sorting how risky people are, right? And there are also a lot of books out there about the problems with our data sets, you know, like Victoria Eubanks wrote a book called, I think it’s called Algorithms of inequality… I’m looking back at my bookshelf trying to find it, but about you know how law enforcement data sets tend to disproportionately have certain types of demographic data in them and certain communities are over-represented in in law enforcement data sets, right? So both algorithms and data sets tend to be biased. What my book focuses on and what I feel safe discussing as a librarian without falling into unknown territory for myself is the fact that there are a few companies that have swallowed up and now control a lot of our informational resources right? So what I focus on is why it is harmful that Lexis/Nexis for example which is part of the company, Reed Elsevier, Lexis/Nexis. Why this one company or how this one company came to dominate so many of our informational markets and why that’s a problem. And I use Reed Elsevier, Lexis/Nexis, as my example not to pick on the company, but because they are so uniquely massive. So usually if I’m doing a presentation I have this slide where I show that Reed Elsevier Lexis/Nexis in the United States dominates the legal information market, right? In the United States if you want to do legal information research, research or you want to look at legal information you have to subscribe to either Thompson orders Westlaw or Lexis/Nexis Lexus research platform. Those are the only two really only two competitive games in town. Those are the gold standard of legal research in the United States. If you’re an academic and you either want to publish or do research, you have to have access to Elsevier and you have to have access to Elsevier Science direct platform, right? And if you want to assess how you’re doing as a researcher, how impactful you are as a researcher, you have to use Elsevier Scopus or you know your data somehow has to feed through Elsevier academic data analytics systems or you have to use Clarivate, which is the really the only major competitor to Elsevier in the academic data analytics sector, Lexis/Nexis, boasts the one of the largest news archives in the world, so media and news information is another market where Lexis/Nexis Reed dominates. Financial information, so information called for from public sector like SEC fillings, financial fillings that companies are required to submit to the government and also news about financial institutions… Lexis/Nexis has kind of a…is part of an oligopoly of companies that that control that information. And I feel like I’m forgetting one of the markets here…oh, and also this is the most creepy one, right? It turns out that Lexis/Nexis is also one of the biggest personal information providers, one of the biggest personal information holders and personal information data brokers in the world. Right, so they have all of our personal data too, and what the company does is it doesn’t just stay in those unique sectors. Academic information, legal information, financial information, personal data. Lexis/Nexis has found a way in multiple ways to use all of those different datasets and combine them to make new informational assets, to make new “data analytics products” that mix and match those types of data – our personal data with data about our academic success, our personal data with news information, our personal data with legal information and all of that data is kind of mixed and mashed and “crunched”. It’s put through data analytics systems, machine learning systems, you know various algorithms to create new unique information types that then Lexis/Nexis can sell for even more money, right? That’s where our predictive policing products come, our predictive insurance products, our legal analytics and academic analytics products come from. 00:19:51 Domen Savič / Citizen D, Hmm. But, you know, speaking as a devil’s advocate, I would go and say, you know, if you did nothing wrong, if you have nothing to hide, then these data sets that are then brought up by ICE, by police, by other other agencies, they don’t concern you, right? So what’s wrong with police having all of this data at their fingertips to sort of, you know, fight crime and other illegal activities? 00:20:23 Sarah Lamdan Absolutely. That’s a really good question, right? And I think in an ideal world where there is no algorithmic or data error or bias, I think that that’s a really good question to ask. Like how how amazing would it be to have a digital data fueled product that would make policing perfect, and that would maybe eliminate bias in academia by really using proven data to to determine which types of research are the most important and which types of research should be funded, and then in a perfect world, these products might really be miraculous, right? In fact, one of the things that Mackenzie Funk heard about in the beginning of the Hank Show is how Hank Ashers matrix product actually identified 5 of the people who planned the September 11th attacks before law enforcement even knew who they were. That’s miraculous, that is amazing in a moment where there has been some sort of horrible crime committed, to know immediately who committed the crime. Fantastic. The problem, unfortunately, is in our current system, we still have the problems that that the algorithmic bias problems and the data bias problems that Virginia Eubanks and Safiya Umoja Noble and Ruhab Benjamin and all of the other kind of critics of our current AI systems have unearthed those problems. Whether we want to acknowledge them or not, they are present, right now, our algorithms are imperfect. We don’t exactly know how they work and we do know that they tend to be biased and we actually… one of the things that my book focuses on a lot because although I’m not, like I said, I’m not an algorithmic expert, I do know a lot about, you know, information assets and information. A lot of the information that Lexis/Nexis and companies like Lexis/Nexis use, are erroneous right now, right. One of the articles that I point to a lot, is an expose in Newsweek called, I think it’s something about how Lexis/Nexis, how, when Lexis/Nexis makes a mistake, it hurts you … and the article is basically a bunch of interviews with people who have been harmed because Lexis/Nexis has incorrect data about them in their systems. There is an example of a woman who gets locked out of her own bank account because her sister is having credit problems, but in Lexis/Nexis’s data set her sister’s data is combined with her data, and so she gets locked out of her bank account. There’s a story of a man who has the same name as a completely different person, the other person has insurance issues, this man doesn’t, but because this man’s data is conflated with that other man’s in Lexis/Nexis, he can’t get auto insurance. So you run into problems like this again and again and again, and the more when an insurance company uses the wrong data about your or it has erroneous data about you. That’s annoying, right? You can’t get the insurance you want. But when your landlord, when your when your landlords tenant screening product, this is Lexis/Nexis, you might not be able to get housing, right. There’s a whole expose in the I think it’s the Texas Observer about how people get blacklisted getting apartment from getting a roof over their head because Lexis/Nexis has erroneous data about them, right? And then that becomes even more harmful when law enforcement uses that data, right, you can get arrested because your name is the same as somebody else’s name, right? Or because Lexis/Nexis another person drivers license conflated with yours in their system and that becomes really problematic so until we can ensure that the data about us is correct in these systems and that the algorithms aren’t biased in these systems I still have a lot of concern about us using these systems. 00:24:47 Domen Savič / Citizen D Yeah, and do you see that happening in the future? Like when you usually debate around these topics and you have like representatives from an industry or decision makers, they usually go “Well yeah you know, no system is perfect. These are glitches. We’re ironing them out.” Do you see that happening or do you see us as a species, to be a bit dramatic, coming into a situation where we have, you know, the perfect data set, the perfect algorithm, the perfect automated decision making system and the perfect result in the end? 00:25:23 Sarah Lamdan That’s a good question and I mean, I will… I wanna before I even answer that hypothetical, I will also point out, like humans are innately biased, right? The reason that there’s so much data bias in law enforcement data is because for you know, for over a century, our law enforcement agency has been doing things that are are racist and otherwise, you know biased and prejudice, right? So humans have their own… I don’t, I don’t want to pretend that like the perfect system is already in place and that is the human system. There are problems with the human systems as well, so I just wanted to put that in the world, because that’s an important thing to recognize. Our systems are also problematic and I do mean, yes, if we treated data systems with the care that they warrant, with the care that they deserve, I do, I could envision a future where we have these systems in place and they aren’t biased, but I think a couple things that we’re missing right now that we’d have to put into place are… OK, so one of the main things we would have to put into place, it is more transparency. Right now I can’t see what my data dossier looks like in Lexis/Nexis, even if because I live in New York State. So even if New York State and my local law enforcement agency is using Lexis/Nexis to determine whether I might commit a crime or whether I have committed a crime, I can’t see what data they’re using about me. I can’t know if it’s correct or if it’s erroneous and I can’t correct it right? Like I can’t say “Ohh I never lived in that address” or “Actually that wasn’t me who was at that place at that night” or “I don’t drive that car”. I can’t say those things, I have no power, so I think in order to move to a place where maybe we could use these systems and we could use them well and we could use them effectively, things would have to be a lot more transparent for consumers and for the public, we’d have to be able to view our own dossiers, and we’d have to be able to correct them, right? We’d have to be able to ensure and feel comfortable that when NYPD or you know any other law enforcement or other type of agency was using the data. I want to feel comfortable knowing that what they were using about me was information that I feel OK sharing and that I know is correct. I think that would have to happen, and that’s a big undertaking. I don’t think it’s impossible, and I don’t even necessarily think that it’s a bad idea, but right now, that type of transparency and that type of corrective measure isn’t in place so that I think first and foremost, those are two really basic things that would have to happen. 00:28:28 Domen Savič / Citizen D Would you say lack of transparency is also one of or maybe the reason that data cartels are so hard to regulate… you can’t even see them or you can’t even see what you’re regulating. And this is something that, yeah, impacts the regulatory frameworks or would you say something else is going on that these data systems are being almost, you know, untouched and free to do what they do? 00:28:59 Sarah Lamdan Yeah, I think you’re… I think what you’re getting at is absolutely correct. The lack of transparency is a huge problem, right? Lack of transparency to the public to even know that the government is using these types of systems or know that their insurance agent or their landlord or any other entity that’s making decisions about their lives. Usually the public doesn’t even know that these algorithmic systems are being used behind the scenes, right? You don’t know when you get hired that your job application was run through some sort of employment screening system that was algorithmically powered, right? So that type of transparency is absent. And on a deeper level, we don’t, let’s say you do know, let’s say you do get. You live in a place where you’re required to get a notice that your job application has been run through one of these systems, you don’t know what the data about you that they ran through the system includes. You know what the data inside the system that your applications being checked against includes and this is key. You don’t know what the algorithm is assessing and you don’t know how the algorithm is working to make that assessment. The major problem with that is these algorithms that are being developed at Lexis/Nexis and at other types of of data analytics firms. We don’t know how the algorithms work and most of the people who design the algorithms don’t know how they work because the algorithms work so quickly and do so many things with so many data sets that algorithmic transparency is not only unavailable, it might be impossible in the current system, right there are. There are a lot of entities and advocates that that are trying to demand algorithmic transparency, which is an important and noble aspiration, but a lot of times, even the people who developed the algorithm in like a Lexis/Nexis or an Experian or other system…they don’t know how the algorithm they developed works, so how are we supposed to understand? How it works? 00:31:10 Domen Savič / Citizen D You mentioned that in in the conclusion of your book and I love it because it’s such a refreshing take that differs very much if you compare it to these calls to, you know, active citizen that that just needs to be educated about certain topics, saying everything will be alright, because we’ll know what’s going on and we’ll know which buttons to push to to get what we want. And you basically called for the right blend of of governance, oversight and support to resolve the issues of of privatized data collections and treating the essential information as a public resource. So if we take a walk down the memory lane and look at the history of the net. How feasible do you think it is your call that’s A… and B, where did we go so wrong that we ended up in this privatized two companies in the world type of digital sphere that started off as a part of the expression, like a like a hippie commune, right? 00:32:22 Sarah Lamdan OK, I’ll admit that I tend to gravitate towards the hippie commune ideal so… Yeah. OK. First and foremost, yeah, I think right now we are in kind of a hyper capitalist system when it comes to tech especially, and I’m not sure if that’s because it’s so kind of Silicon Valley based and that’s very like American and we just love our Uber capitalists, you know. I want to call it fantasy, but really we are playing it out in reality so our Uber capitalist tech reality. And yes, I’ll admit… I do put my ideals, kind of over the other extreme, but really what I… in my ideal… It’s not about choosing between capitalism and like some sort of hippie Co-op for data, but it’s about putting the onus where it belongs. I think the problem with our current hyper capitalist system is that what we’ve let the companies do is make consumers think that it’s their fault and that it’s their problem to solve like “Ohh, I’m sorry, do you feel like data companies are too invasive? You need to change the privacy settings on your phone. Or you need to not use social media. We put it all on the individual to try to keep their data out of these systems, or even to request their own data dossiers. Right? Like now, our kind of new ideal is passing these laws that allow individuals to get their own data dossiers. Like, what’s the benefit of that? Let’s say I live in California. Yeah, well, I’m legally entitled to my own data dossier, so I live in California. I request my own data dossier from Lexis/Nexis. Great. Now I have a PDF full of data about myself. Some of it is right. Some of it is wrong. That doesn’t really empower me to do much right. I could try to call all the companies that I think have wrong incorrect data about me and beg them to fix it, but that that would take hundreds of hours that I do not have right. So right now, even in our current best scenario legal framework, where we can see our own data dossiers, the responsibilities placed on individuals and then individuals have very little power. I am urging us to rethink this system so that instead of the onus being on the individuals that way it is now on the companies, the companies who create these systems should be responsible for making sure the data is correct, giving members of the public easy recourse to correct their data or to erase their data, right? That should be the responsibility of the companies who are making billions of dollars building these systems. And really those companies are the only entities that have the real power to make our digital lives better. So what I’m trying to do in a way that I’m not sure anybody could succeed. That is, I’m trying to take that boulder of responsibility and just like, dump it across the ground from individuals back to the powerful companies that could actually make the data world better because like I said, I’m not a complete Luddite. It’s not even a question of whether these companies should have a right to exist. They exist. This is reality. This is the system. Some of the ideas that the companies have are very, very cool how can they execute those ideas in a way that doesn’t harm us or put us at risk. 00:36:23 Domen Savič / Citizen D Before we wrap up, I just want to mention something. You apologized sort of in advance several times saying that you don’t hate technology. I have a same same feeling working on issues like like privacy and security and open access… you always have this need of apologizing in advance if you’re not totally on board with you know the complete version of surveillance surveillance capitalism. And I wonder, do you think the issue starts at just basic language structures around these issues, like saying OK, “I’m not sure this is the best way to go” equals you know, “We need to tear everything down and just, you know, build those to the to the ground”. So. So do you think we’re sort of like cultivated in this this techno solutionistic approach that basically prevents any normal discussion happening… not, you know, not to start with with solutions, but just, you know, have an honest conversations about these issues. 00:37:39 Sarah Lamdan Absolutely, I think that the discussion around these issues is so polarized that if you say anything less than “Ohh I love you!” or “I love TikTok. It’s favorite thing the whole world!”… If you say anything less than that then you hate technology and you want to throw your computer in the river, right. And I think we really haven’t been able to have nuanced discussions about practical solutions. I’m not sure why that is, I do know that since my book has been published, people who don’t like what I have to say and people who don’t agree with what I have to say, which is completely fair and I expect like. Yes, please, let’s have these discussions. But people who don’t like it really do paint me as somebody who just hates technology or hates data analytics and it’s a shame, because I think that that really diminishes the ability to have real fruitful discussions, right. And I do wonder if that’s because of the way it’s being framed commercially and politically because one of the things that happens is whenever the government, any government works to incrementally place responsibilities on the companies that develop these types of tools, whenever, especially you know in the US or in the EU when we look to clamp down on the data collection that, like a Facebook or a, you know, Lexis/Nexis is doing… there are lobbyists who visit the politicians office and say “This would destroy the Internet, we can’t do this!” and now also pundits who come out on the Internet and on talk shows and say you know “These people just wanna ruin the Internet”, or “They don’t care about law enforcement”, right? But I think it’s possible to care about law enforcement or to not be an absolute, just about law enforcement but to be concerned about law enforcement using, you know facial recognition apps or predictive policing apps. I feel like sometimes it becomes kind of a scare tactic used by tech companies to make critics look silly, non professional, you know. So it diminishes detractors and critics and it also kind of silences people by making them think that if they critique tech platforms, tech companies, then they might destroy them or ruin something that they like or that they enjoy. Because I do think there’s a way for us to say, buy something on Amazon and get the helpful “More like this” option because I’ll admit… if I read a book and I enjoy it, I want to see more books like that book. That is a really cool tool and that’s a data-driven tool, data-driven by my data, the data of other readers and users. Yeah, that’s a data-driven tool, but is there a way that we can implement that and also protect my data privacy and be transparent about how those recommendations are being developed right? But I think that that would put a lot more expense and work in Amazon’s products, right. They have to do a lot more work and engage in a lot more transparency and I think that a lot of companies don’t want to pay those extra costs and don’t want to do that extra work. So it’s a push and pull game where tech companies have a PR advantage and they can make us look like we are foolish or that we’re not real experts or that we don’t really have real solutions. 00:41:56 Domen Savič / Citizen D And just one more question before we wrap up… Let’s compare notes on the countering tactics or on the tactics countering that narrative, what do you think works in sort of persuading people that, yes, you’re not throwing laptops into the river and smashing light bulbs and yeah, writing your stuff with ink and pen. 00:42:19 Sarah Lamdan Right. Yeah. No, that’s a good question. One thing that is surprising, like a few months ago, I was asked if I’d be willing to be on a panel at a conference with somebody from Lexis/Nexis. And my answer is yes, absolutely. I would be really glad and excited for the opportunity to talk about, you know what our concerns are as academics, as consumers and then you know, have that discussion about what are some real solutions like if we sat down and thought of three things that we could work on together. What would those three things be? I think that would be really, really cool. But notice, I’m a willing panelist, it’s not me who’s not willing to show up on that panel. I guess that’s what I’ll say, right? So I think just to show a continued willingness to have that discussion and also to not be cowed by push-back you get from other companies, cause I’ve got I’ve gotten push-back from representatives at Lexis/Nexis and people who work at other data analytics firms, right? They’ve panned by books, they’ve said really nasty things about my motives, right? But I’m completely willing to have a discussion about why do you feel that way? And is there a place where we agree, right? Because I do think that there are places where we agree, we just need to find those places together and that that has to be a discussion that both sides are willing to have and everybody I want to say sides but like that, everybody in the community creating this work and then using this work is willing to have. 00:44:20 Domen Savič / Citizen D Professor Landon, thank you so much for dropping by. This has been Citizen D podcast. We publish an episode every month, so see you next time. Thank you again and best of luck. 00:44:31 Sarah Lamdan Thank you. Citizen D advice: Algorithm and data transparency is the first step towards corporate responsibility Personal responsiblity will only get you so far Technosolutionistic language is affecting the debate around the social impact of technology More information: McKenzie Funk, The Hank Show: How a House Painting, Drug Running DEA Informant Built the Machine that Rules Our Lives – book Andrew Guthrie Ferguson, The Rise of Big Data Policing: Surveillance, Race, and the Future of Law Enforcement – book Who’s Behind ICE? The Tech and Data Companies Fueling Deportation – analysis [PDF] Alice Holbrook, When LexisNexis Makes a Mistake, You Pay For It (Newsweek Magazine, 2019) – article About the podcast: Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!

Showing 1–20 of 20 episodes