3.1. First pitfall: Failure to create informed consent

When implementing any new urban innovation, informed consent is an important factor in guaranteeing the protection of citizens. As a practice, informed consent prioritises the digital rights of citizens, making them an active part of the process. Within the Quayside project, Sidewalk Labs treated citizen feedback as an additive feature, rather than a central aspect of the project’s development. The first pitfall I will examine is Sidewalk Labs’ failure to adhere to a proper informed consent model. This failure resulted in minimal citizen engagement, making it difficult to achieve public trust.

Within Toronto, there are three pieces of legislation that are especially important in terms of public–private partnerships and data collection: Municipal Freedom of Information and Protection of Privacy Act (MFIPPA 1990), Freedom of Information and Privacy Protection Act (FIPPA, 1990) and Personal Information Protection and Electronic Documents Act (PIPEDA, 2000). These three acts work in conjunction to provide strict privacy protections regarding the collection of personal information, with PIPEDA working at the federal level, FIPPA at the provincial level, and MFIPPA at the municipal level. While the three legislations differ in scope—PIPEDA covering private entities and FIPPA/MFIPPA covering public institutions—all three highlight the importance of consent and the ability to “opt out” as essential to privacy protection. PIPEDA creates the conditions in which corporations are deterred from using unethical data practices through ten principles to safeguard the collection of personal data: accountability; identifying purposes; consent; limiting collection; limiting use, disclosure, and retention; accuracy; safeguards; openness; individual access; challenging compliance (PIPEDA Fair Information Principles, 2019). FIPPA and MFIPPA complement this practice, as they apply to public institutions and aim to protect an individual’s right to privacy while ensuring that data is being collected for specific purposes such as administration or law enforcement (Beamish, Reference Beamish2019, 2). In proposing “urban data” as a new category, Sidewalk Labs was able to sidestep these regulations that only apply to personal data, subsequently neglecting the need for consent and participation.

In failing to adhere to the requirements of existing legislation, Sidewalk Labs violated Canadians’ personal and collective privacy rights, as informed consent was impossible to achieve within smart city infrastructure. (CCLA, 2019, 4). Smart neighbourhoods present an unacknowledged risk of coercion for civilians, as often individuals living, working, or passing through these areas have no alternative to the “smart” services existing within the neighbourhood, creating (Beamish, Reference Beamish2019). As citizens would be unable to meaningfully opt out of data collection within the space, the potential for surveillance and commoditization presented a contradiction to the precedent established by federal privacy legislation, in addition to the rights to Freedoms of Assembly and Association, Life, Liberty and Security of the Person, and Unreasonable Search or Seizure as outlined in the Canadian Charter of Rights and Freedoms (CCLA, 2019, 9). As Ellen Goodman makes clear in her Affidavit regarding informed consent in the Quayside neighbourhood, “consent is insufficiently “knowing” when the user does not understand the technology being agreed to or the practical consequences of agreeing. Consent is insufficiently “free” when a person must choose between consent and the loss of an important function or asset. The project poses both of these dangers to a significant degree” (Goodman, Reference Goodman2020, 13). Sidewalk Labs aimed to impose infrastructure that would have strong impacts on citizens’ daily lives, and as such, should have treated informed consent with precarity and caution. If people do not understand the implications of mass data capture, they will be unable to hold corporate actors and governments accountable. This becomes especially essential when technology is integrated into urban spaces, as simply telling people to not enter a neighbourhood is not always a possibility.

Yet, these issues are not solely representative of the Quayside project. Rather, the current global structure of data regulation itself relies on an interlocking value system, where the ideas of notice and consent are at the forefront. Canada’s PIPEDA, as well as the UK’s Data Protection Act (2018), and both the EU and UK General Data Protection Regulation (GDPR, 2016, Information Commissioner of the UK, 2021) share a commitment to a regulatory data framework in which individuals can both know when their information is being processed, and consent to its collection. Similar to PIPEDA’s Fair Information Principles, the GDPR outlines seven key principles with which personal information is to be governed: Lawfulness, fairness, and transparency; Purpose limitation; Data minimisation; Accuracy; Storage limitation; Integrity and confidentiality; and Accountability (GDPR, 2016, 5.1). These sets of principles outline the values behind these policy decisions, encouraging practices of transparency and acceptance.

On deeper reflection, these principles are far from effective in promoting safe data practices, as they all place an overwhelming onus of responsibility on the individual to find, understand, and then challenge or accept the collection of their data. Here, the desire for pragmatic solutions to complex problems sees transparency as a direct path to accountability. However, these legislative principles fail to account for the full problem at hand—a lack of digital literacy and a feeling of powerlessness when face-to-face with Big Data (Obar, Reference Obar2015). Data protection in its current form takes seeing over understanding, where an increase in transparency is equated with an increase in knowledge, and subsequently an increase in control (Ananny and Crawford, Reference Ananny and Crawford2018). Yet, being given information does not mean that meaningful change will occur. Nor does it mean that those who are supposed to be holding a system accountable will be able to achieve those goals. Rather, a sole focus on transparency fails to reach a critical stage, which recognizes the deployment of smart infrastructure as reflective of collective harms and meaningful engagement with the construction of community-based systems. The implementation of intelligent decision-making and data collection cannot be regulated with a sole focus on individuality. Instead, data governance needs to develop clear policies aligned with feminist protocols of challenging power to provide residents and citizens with the tools to critically evaluate this information (Kern, Reference Kern2019). Informed (and participatory) consent within digital policy is an incredibly important practice for ensuring ethical technological intervention. Integrating a new type of data alongside no clear legislative protections fails to ensure that legitimate consent is possible. As the Quayside project highlights, developing new standards built solely on the perception of transparency fails to address the need for collective support and engagement.

3.2. Second pitfall: promotion of corporate governance

To regulate this new category of data, Sidewalk Labs created the Urban Data Trust—an external regulatory body responsible for making decisions surrounding data in Quayside. The roles of the Urban Data involved the “approval and management of data collection devices placed in the public realm, as well as addressing the challenges and opportunities arising from data use, particularly those involving algorithmic decision-making” (Sidewalk Labs, 2019a, p.420). The data-sharing agreements were to be reminiscent of data licence agreements and enforceable in court (Sidewalk Labs, 2019a). However, Sidewalk Labs also stipulated that this trust was not a legal trust, and failed to identify what these court proceedings would look like or how data breaches could be legally enforceable. Further, the cases themselves would be evaluated not according to Canadian legislation, but Sidewalk Labs’ own Responsible Data Use Guidelines” (Sidewalk Labs, 2019b). The establishment of their own criteria for responsible data use, combined with the creation of an entirely new form of data, essentially makes Sidewalk Labs the new expert on the ethical use of data in smart cities. Rather than turning to the government for perspectives, a corporate actor sought to create the new rules of the game.

The issue at hand involves the notion of corporate governance as being essential to the implementation of the Quayside project. The increasing prevalence of corporate governance can be seen in the Trust’s ability to turn residents’ data into assets. The process of creating an asset-based understanding of data is outlined by Artyushina (Artyushina, Reference Artyushina2022) as constitutive of five interconnected processes: “introducing new legal definitions for the data, retrofitting city infrastructure with data-tracking devices, creating a self-certification regime for data collectors, accumulating the data collected in the smart city in one physical location, and establishing IP-intensive data sharing agreements” (Artyushina, Reference Artyushina2022, 8). In turning residents’ data into assets for evaluation, the data trust therefore aimed to establish data as an infrastructural element of urban life, rather than as part of a transactional relationship, redefining the rules of data governance according to Sidewalk Labs’ own procedure.

Further, there was a distinct lack of government consultation, ultimately resulting in harsh consequences and a failure to protect the rights of residents. Unlike its previous operating practices, Waterfront Toronto did not adequately consult with any levels of government regarding the Sidewalk Labs project (Auditor General, 2019, 650). The scope of the project, from self-driving vehicles to data collection, falls under multiple provincial and federal ministries and city departments, but Waterfront Toronto did not adequately consult with any of them before signing an initial agreement (AG, 2019, 650). In 2002, Waterfront Toronto established an Intergovernmental Steering Committee made up of representatives from the municipal, federal, and provincial governments (Eidelman, Reference Eidelman2013). Yet, Sidewalk Labs and Waterfront Toronto failed to adequately use the committee’s expertise and knowledge to its advantage. The steering committee was consistently in the dark regarding the decisions surrounding Quayside, as the Auditor General of Ontario found that the committee was only made aware of the name of the successful bidder 5 days before the public announcement (Auditor General, 2019, 649). Upon further investigation, the Auditor General found that the steering committee itself was not provided with a framework or guide to support its decision-making process (Auditor General, 2019, 681).

Following the proposal of the Urban Data Trust, Waterfront Toronto’s Digital Strategy Advisory Panel (DSAP) rejected the notion of urban data entirely (DSAP, 2020). Instead, they asserted that data collected through Quayside would be understood within the existing Canadian legal structure. Yet, given the limited control and input the Intergovernmental Steering Committee was given in regard to the Quayside project, the ways in which these laws would impact the governance processes still appeared vague. These concerns about government participation and the establishment of new trusts and categories of data governance present an important consideration moving forward, as there needs to be a rigorous process to protect the rights of citizens so that it is not corporations establishing the rules and trades of the game.

Public–private partnerships present complexities in governance, especially with regard to data. When viewed through a feminist lens that aims to see the “not-seen” (Smith, Reference Smith, Dubrofsky and Magnet2015), these complexities represent deeper systemic issues regarding mass surveillance and human rights. While these proposals are marketed under the guise of public benefit, enabling corporate governance provides more drastic consequences for civilians. As Sidewalk Labs aimed to promote an environment where technology solves all issues, the Quayside project ignored the risks that a hyper-surveilled space poses to certain populations. Creating seemingly innovative solutions without considering their full socio-political consequences creates the potential for further marginalization of people who are already experiencing direct harm from these technologies at multiple intersectional levels. Where Sidewalk Labs may promote increased surveillance as an efficiency or safety mechanism, it poses risks to people of colour, women, and gender minorities who are misrecognised by this technology more frequently than others (Buolamwini and Gebru, Reference Buolamwini and Gebru2018; Marshall, Reference Marshall2019). When data is mobilized as a source of power, collecting more of it presents the risk of upholding hierarchies of discrimination. Technology does not impact everyone equally, and the assumption that employing algorithms would “solve” the problems of a city vastly ignores the issues that technology cannot fix. In proposing to integrate an entire neighbourhood with surveillance technology, Sidewalk Labs subsequently isolated marginalized residents from safely accessing their own city.

If technology is to be integrated into cities, it needs to occur in a process where civilians are able to meaningfully engage with the development procedures, while knowing that their rights are thoroughly protected under the law. Within Toronto, the proposal to build a city “from the internet up” (Sidewalk Labs, 2019a) came alongside a series of different infrastructural changes. Yet, Sidewalk Labs is not the only corporation, nor the first one, to attempt to integrate corporate governance into the inner workings of a city.Footnote ³ Rather, the push for corporate governance within the Quayside project reflects a larger practice of corporate capture in the city. As corporate actors are on the cutting edge of innovation, research and development, their technology becomes integrated into these urban environments through an increasing reliance on public–private partnerships. As ICTs make data collection easier than ever, data are emerging as a key resource that—if not properly protected—can be exploited in search of profit (Sadowski, Reference Sadowski2019). The challenge to public–private partnerships in urban “smart” environments highlights that for corporate actors, there is an increased incentive to collect data en masse within the city. Here, the initial investment is not the sole source of profit. Rather, the ability to constantly collect and distribute key informational data about a community promises a stronghold on a valuable resource in the 21st century (Hollands, Reference Hollands2015). As corporate actors retain their position as the experts in the field of smart infrastructure, they can promote their own forms of universalist application and self-regulation without an acknowledgement of the economic benefits this data collection provides to corporate actors. In favouring universalism over a localized understanding of urban issues, smart cities ignore context and systemic issues in search of increased profit.

4.1 Participation in design

While urban bureaucracy often faces a common complaint of slowness, I argue for a leaning into the idea of slowness. From a feminist perspective, moving slowly involves a critical analysis of who is “not-seen” and subject to a hyper in/visibility in decision-making (Smith, Reference Smith, Dubrofsky and Magnet2015). This process integrates collaboration from the start, working to develop technological solutions in tandem with the communities they are meant to serve, rather than imposing corporate technology as a “universal” benefit. It also involves concrete accountability measures, where the public can understand what happens to their data. The slowness experienced by urban bureaucracy is often accompanied by confusion and frustration about being “out of the loop” with regard to the process. My articulation here emphasises the need for residents to be viewed as direct collaborators in the process of urban development, rather than as an additive feature later on in the process. Taking this approach to designing a smart city means putting the experiences and potential consequences for those impacted front and centre. Drawing from the key principles of an open smart city as articulated by Open North’s open smart cities project (Lauriault et al., Reference Lauriault, Bloom and Landry2018), I emphasise that a feminist perspective sees slowness not as a hindrance, but as a potential to mobilize technology and data when warranted according to ethical, accessible, and transparent standards.Footnote ⁴ This approach therefore builds upon calls to recognise that data and technology are not the solution to many of the systemic issues cities face, nor are there always quick fixes (Mattern, Reference Mattern2021; Powell, Reference Powell2021).

Where Sidewalk Labs sought to integrate public opinion into their project as an additive feature, an open smart city would build itself with public interest and contribution as initial stepping stones. A design justice framework would help organise citizen involvement. In this approach, designers are required to look to the community for guidance. Sasha Costanza-Chock (Reference Costanza-Chock2020) highlights designers engaging in this process as practitioners who aim to incorporate the ways in which the members of the community are already working together to face a challenge. Taking a design justice framework involves not only considering the perspectives of community members but genuinely involving them in the design process itself. This important work is already being done by groups publishing the Critical Engineering Manifesto (Reference Oliver, Savičić and Vasiliev2021), Feminist Data Manifest-No (Reference Cifor, Garcia, Cowan, Rault, Sutherland, Chan, Rode, Hoffmann, Salehi and Nakamura2019), and the More than Code Report (Reference Costanza-Chock, Wagoner, Taye, Rivas, Schweidler and Bulllen2018), which each highlight the importance of design as a process that involves meaningful engagement with those most impacted by technology. To embed a city with just design practices and open smart city frameworks, there is no need to reinvent the wheel, but there is a need to reflect on these critical interventions in current design theory.

4.2 Localized governance

However, it must be understood that design cannot solve everything. Design as a process goes hand-in-hand with developing clear policy initiatives to uphold the rights of residents. Currently, there is a call to redefine Canadian privacy legislation to reflect the necessary conditions of the digital age. As of early 2024, Bill C27—The Digital Charter Implementation Act—has yet to be passed. The proposals for Bill C27 involve integrating a comprehensive AI legislation based on a risk-assessment framework, while also replacing PIPEDA with an act that addresses the need for Canadians to be able to access, erase, correct, and transfer data about themselves (Bill C27, 2022) While this new approach to data governance addresses the need to develop strategies for the digital age, Bill C27 has already received criticism for its inability to address collective rights (Duncan and Wong, Reference Duncan and Wong2022), lack of a focus on shared prosperity and community engagement (Brandusescu and Sieber, Reference Brandusescu and Sieber2023), and its failure to integrate an intersectional, human rights-based framework (Kim and Thomasen, Reference Kim and Thomasen2023). Governance in the age of artificial intelligence requires meaningful citizen engagement to address the multifaceted ways in which technology used in urban spaces has intersectional, far-reaching impacts. Therefore, a critical policy intervention involves moving away from the self-regulatory precedent, and ensuring that the human rights of residents are not sacrificed in the development of these policies.

This articulation builds from the current push within critical AI studies to approach artificial intelligence development from a localized, context-based approach. This draws from Katz and Nowak’s (Reference Katz and Nowak2017) articulation of new localism in which “local jurisdictions are increasingly taking it upon themselves to address a broad range of environmental, economic, and social challenges, and the domain of technology is no exception” (Verhulst, Reference Verhulst, Brandusescu and Reia2022, 80). Within artificial intelligence communities, there has been an increase in specifically localized approaches to governance as a way to enable public participation in artificial intelligence development. Recent examples of AI localism include San Francisco’s ban on AI-powered facial recognition technology (Conger et al., Reference Conger, Fausset and Kovaleski2019), New York City’s rules on automated decision-making in hiring (Francis, Reference Francis2023), Helsinki and Amsterdam’s public registries of AI systems used in local government (Team AI Regulation, 2020), Montreal’s City Council motion against facial recognition (Serebin, Reference Serebin2020), and Barcelona’s citizen watch initiatives (March and Ribera-Fumaz, Reference March, Ribera-Fumaz, Karvonen, Cugurullo and Caprotti2018). A key element of this appeal to localized governance highlights that it works as a decentralised, rather than fragmented approach. Here, a decentralised approach aims to learn from the other local regulatory strategies to build a specific and contextual regulatory response, rather than applying them universally. While there are certain inalienable rights—particularly to privacy—this approach builds upon these rights to encourage a collaborative policy approach, while utilising the skills that other municipalities have developed to find what fits best with a community’s goals and needs.

Beyond ensuring governance principles in abstract, the notion of accountability in artificial intelligence policy involves creating a culture of trust. To this extent, Brudvig highlights the importance of consent as an ongoing project involving “a) revoking tools or systems that are not democratically agreed upon; b) opening up spaces for local creation of new tools and systems; c) decentralised governance or custodianship of data and knowledge systems; d) inclusion of local leaders in decision-making; and e) interdisciplinary collaboration in spaces of technology creation, policy and decision-making” (Brudvig, Reference Brudvig, Brandusescu and Reia2022, 29). Each of these acts reflects a commitment to creating a transparent and accountable city that prioritises the needs of its residents. Gaining meaningful transparency itself is a process that engages not only the public but also stakeholders, governing bodies, and corporations. Ensuring the interpretability of transparency, therefore, is necessary to develop this culture of trust, as accountability can only be enacted when civilians can understand how and why a system needs to change (Brauneis & Goodman, Reference Brauneis and Goodman2018, 132). Here I second Robert Brauneis and Ellen Goodman’s call for a shift away from the pull model of transparency in algorithmic governance—where individuals have to pull information that they wish to know—and instead call for a push model—in which those making decisions make their information publicly accessible and interpretable. Implementing accessible policy therefore goes hand-in-hand with ensuring participatory methods that engage the public at each stage of development, to ensure that all interests are recognised. In prioritising the development of a localized approach to policy, smart cities can be developed with the interests of residents at the forefront.