1.2.1 AI Systems: Concept and Definition

In the European legislation, whether in force or pending adoption, references to automation appear scattered and with disparate terminology. Both in the General Data Protection Regulation (GDPR),Footnote ³⁰ or in the Digital Services Act references to automated individual decisions, algorithmic decisions, algorithmic content recommendation or moderation systems, algorithmic prioritisation, or the use of automatic or automated means for various purposes can be spotted. But there is no definition or explicit reference to ‘AI’ in the said texts. It is the future, and still evolving, AI Act that expressly defines ‘AI systems’, for the purposes of the regulation, in order to delimit its material scope of application.

The initial definition of AI system for the purposes of the proposed instrument in the European Commission’s proposal was as follows: artificial intelligence system (AI system) means ‘software that is developed using one or more of the techniques and approaches listed in Annex I and can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations or decisions that influence the environments with which it interacts’ (Art. 3.1 AI Act).

With this definition, AI systems are defined on the basis of two components. First, the qualification as learning systems and thus separated from more traditional computational systems. That is, the fact that they employ or are developed using ‘AI’ techniques, which the AI Act would define in an Annex I, subject to further extension or modification, and which currently includes: machine learning strategies, including supervised, unsupervised, and reinforcement learning, employing a wide variety of methods, including deep learning; logic and knowledge-based strategies, especially knowledge representation, inductive (logic) programming, knowledge bases, inference and deduction engines, expert systems and (symbolic) reasoning; statistical strategies, Bayesian estimation, search and optimisation methods. Second, the influence on the environment with which they interact, generating outcomes such as predictions, recommendations, content, or actual decisions. Behind this definition lies the assumption that it is precisely the ‘learning’ capabilities of these systems that largely determine their disruptive effectsFootnote ³¹ (opacity, vulnerability, complexity, data dependence, autonomy) and hence the need to reconsider the adequacy of traditional rules. This is, in fact, the reasoning that leads to rethink the rules of liability and thus assess their adequacy in the face of the distinctive features of AI as proposed in the report published on 21 November 2019, titled Report on Liability for Artificial Intelligence and other emerging technologies.Footnote ³² And it was issued by the Expert GroupFootnote ³³ on New Technologies and Liability advising the European Commission.Footnote ³⁴

Along the same lines, the Commission adopted two related proposals in 2022: proposal for a directive on adapting non-contractual civil liability rules to artificial intelligenceFootnote ³⁵ and proposal for a directive of the European Parliament and of the Council on liability for defective products.Footnote ³⁶

However, the wording of this definition for AI systems in the Commission’s proposal has been subject to significant reconsideration and might still evolve into its final wording. The compromise text submitted at the end of November 2021 by the Slovenian Presidency of the European Council (Council of the European Union, Presidency compromise text, 29 November 2021, 2021/0106(COD), hereafter simply Joint Undertaking) proposed some changes to this definition.Footnote ³⁷ The text, in its preamble, explains that the changes make an explicit reference to the fact that the AI system should be able to determine how to achieve a given set of pre-defined human objectives through learning, reasoning, or modelling, in order to distinguish them more clearly and unambiguously from more traditional software systems, which should not fall within the scope of the proposed Regulation. But also with this proposal, the definition of an AI system is stylised and structurally reflects the three basic building blocks: inputs, processes, and outputs.

Yet, a subsequent version version of the compromise textFootnote ³⁸ of the AI Act offers another definition that refines the previous drafting and provides sufficiently clear criteria for distinguishing AI from simpler software systems. Thus, AI system means a system that is designed to operate with a certain level of autonomy and that, based on machine and/or human-provided data and inputs, infers how to achieve a given set of human-defined objectives using machine learning and/or logic- and knowledge-based approaches, and produces system-generated outputs such as content (generative AI systems), predictions, recommendations, or decisions influencing the environments with which the AI system interacts. Some key elements of the initial definition are preserved or recovered in this recent version that finally narrows down the description of ‘learning systems’ to the definition to systems developed through machine learning approaches and logic- and knowledge-based approaches.

The definition is still evolving. In the latest compromise textFootnote ³⁹ the new definition of AI system is ‘a machine-based system that is designed to operate with varying levels of autonomy and that can, for explicit or implicit objectives, generate outputs such as predictions, recommendations, or decisions, that influence physical or virtual environments’.

Also, the European Parliament Resolution on liability for the operation of artificial intelligence systemsFootnote ⁴⁰ referred expressly to AI systems and formulated its own definition (Art. 3.a).Footnote ⁴¹ This Resolution contains a set of recommendations for a Regulation of the European Parliament and of the Council on civil liability for damage caused by the operation of AI systems. The proposal has not been adopted. Instead, the Commission proposed the abovementioned tandem of draft Directives, that follow a substantially different approach aimed to revise the Defective Product Liability rules so as to accommodate AI-driven products and to alleviate the burden of proof in fault-based liability scenarios on damages caused by AI systems.

The rest of the regulatory texts do not explicitly refer to AI, although they contain rules on algorithms, algorithmic systems of various types, automation or automatic decision-making. Thus, as mentioned above, the GDPR, the DSA, the DMA or, among others, the P2B RegulationFootnote ⁴² refer to algorithmic rating, algorithmic decision-making, algorithmic recommendation systems, algorithmic content moderation, algorithmic structures, automated profiling, or a variety of activities and actions performed by automated means. They include rules related to algorithms, such as disclosure, risk assessment, accountability and transparency audits, on-site inspections, obtaining consent, etc. As the definition of AI systems proposed by the AI Act reveals, recommendations, decisions, predictions, or other digital content of any kind, as well as actions resulting from the system in or in relation to the environment, are natural and frequent outputs of AI systems. Consequently, regulatory provisions that in some way regulate algorithmic processes and decision-making by automated means in a variety of scenarios and for a variety of purposes are also relevant for the construction of the regulatory framework for AI in the European Union.

Provided that the AI system falls under the scope of application of the proposed AI Act, an AI system may be subject to the AI Act as well as to other rules depending on the specific purpose, the purpose for which it is intended or the specific action. As an illustration, if the system is intended to produce recommendations by a very large banking platform, the DSA (Art. 27) – applicable to any online platform – applies, or if the system is intended for profiling, the GDPR (Art. 22) would be relevant.

In conclusion, understanding the complementarity between the various legal texts that directly or indirectly address the use of AI systems for a variety of purposes and from a range of legal perspectives is fundamental to composing the current and future regulatory framework for AI, as discussed below.

1.2.2 Current and Potential Uses of AI in the Financial Sector

With varying degrees of intensity, AI systems are used transversally in the banking sector along the entire front-line-mid-office-back-office value chain. For customer service and interaction, AI systems offer extraordinary possibilities for personalisation, recommendation and profiling, account management, trading and financial advice (robo advisers), continuous service via chatbots and virtual assistants, and sophisticated Know Your Customer (KYC) solutions.Footnote ⁴³ In the internal management of operations, AI solutions are applied in the automation of corporate, administrative and transactional processes, in the optimisation of various activities, or compliance management. For risk management, AI solutions are projected to improve fraud prevention mechanisms, early warning and cybersecurity systems, as well as being incorporated in predictive models for recruitment and promotion. Another interesting useFootnote ⁴⁴ of advanced analysis models with machine learning is the calculation and determination of regulatory capital. Significant cost savings are estimatedFootnote ⁴⁵ if these models are used to calculate risk-weighted assets.

Acknowledging this transversal and multipurpose use allows to anticipate some considerations of interest and relevance for legal analysis. It can be seen that automation has an impact on decision-making processes, actions, or operations of a diverse nature, which will be decisive in determining at least three elements.

First, the applicable regulatory regime – for example, whether it is used to automate compliance with reporting rules, to prevent fraud, to personalise customer offers, or to handle complaints via a chatbot. Second, the possible liability scenarios – for example, whether algorithmic biases and data obtained from social media for the credit scoring and creditworthiness assessment system could lead to systematic discriminatory actions. Third, the transactional context in which it is used – for example, in consumer relations with retail customers, in relations with the supervisor, or in internal relations with employees or partners.

The benefits deriving from the use of automation and AI and the expected gains from systematic and extensive application are numerous.Footnote ⁴⁶ Algorithm-driven systems provide speed, simplicity, and efficiency in solving a multitude of problems. Automation drastically reduces transaction costs, enabling services that would otherwise be unprofitable, unaffordable, or unviable to be provided on reasonable and competitive terms. Cost reduction explains, for example, the burgeoning sector of robo-advisersFootnote ⁴⁷ that have expanded the market beyond traditional financial advisers with appreciable benefits for consumers by diversifying supply, increasing competition, and improving financial inclusion.Footnote ⁴⁸ Such expansion has facilitated financial advice to small investment and low-income investors on market terms.

ADM systems can therefore perform automated tasks and make mass decisions efficiently (high-frequency algorithmic trading, search engines, facial recognition, personal assistants, machine translation, predictive algorithms, and recommender systems). The use of automated means is critical for the large-scale provision of critical services in our society that would otherwise be impossible or highly inefficient (search, sorting, filtering, rating, and ranking).

However, the expansive and growing use of algorithms in our society can also be a source of new risks, can lead to unintended outcomes, have unintended consequences, or raise legal concerns and social challenges of many different kinds. ADM may be biased or discriminatoryFootnote ⁴⁹ as a result of prejudiced preconditions, based on stereotypes or aimed at exploiting user vulnerabilities, inadequate algorithm design, or an insufficient or inaccurate training and learning data set.Footnote ⁵⁰ The automation of ADM makes bias massive, amplified and distorted, and easily gain virality. In a densely connected society such as ours, virality acts as an amplifier of the harmful effects of any action. Negative impacts spread rapidly, the magnitude of the damage increases, and the reversibility of the effects becomes less likely and increasingly impractical. The incorporation of decision and learning techniques into increasingly sophisticated AI systems adds to the growing unpredictability of future response. This leads to greater unpredictability and unstoppable complexity that is not always consistent with traditional rules and formulas for attribution of legal effects and allocations of risk and liability (infra 3.1.2.2 and 3.1.2.3).

1.3.1 The Expected Application of the Future AI Law to the Uses of AI in the Financial Sector

The (future) AI Act is based on a risk-based classification of AI uses, applications, and practices, to which a specific legal regime is attached: prohibition, requirements for high-risk systems, transparency, and other obligations for certain low-risk systems. The classification of AI systems is not done on the basis of the employed technology but in conformity with the (intended, actual, reasonably expected) specific uses or applications. This means that there is no explicit sectoral selection, but certain practices can be identified with preferred sectoral uses such as creditworthiness assessment, and automated credit rating determination.

1.3.2 Principles and Rules for the Use of AI Systems in Decision-Making

However, the eventual application of the AI Act does not exhaust the regulatory framework of reference for the use of AI systems in the financial sector, nor, in fact, does it resolve a good number of questions that the implementation and subsequent operation of such systems in the course of their activity will generate. To this end, and for this reason, it is essential to explore other regulatory instruments and to discover legal avenues to answer a number of important questions. First, to what extent can automated systems be used with full functional equivalence for any activity, decision, or process without prior legal authorisation? Second, to what extent are decisions taken or assisted by AI systems attributed to the financial institution operating the system? Third, who bears the risks and liability for damage caused by the AI systems used?

2.2.1 Robo-Advisers

Robo-advisersFootnote ³⁰ provide ‘automated financial product advice using algorithms and technology and without the direct involvement of a human adviser’.Footnote ³¹ In principle, robo-advice might cover automated advice about any topic relevant to financial management, such as budgeting, borrowing, investing, superannuation, retirement planning, and insurance. Currently, most robo-advisers provide automated investment advice and portfolio management.Footnote ³²

Typically, robo-advice services begin with consumers answering a questionnaire about their goals, expectations, and aptitude for risk. An investment profile for consumers is derived from this information, based on their goals and capacity to bear risk. An algorithm matches consumers’ profiles with an investment portfolio available through the advisory firm to produce an investment recommendation.Footnote ³³ Should a consumer choose to follow the advice and invest in that portfolio, many robo-advisers will also manage the portfolio on an ongoing basis, keeping it within the parameters recommended for the consumer. Consumers generally pay a fee for the service provided by the robo-adviser, often a percentage of the amount invested, with minimum investment amounts required to access the service.

Robo-advice is sometimes described as ‘trading with AI’.Footnote ³⁴ This language might be thought to suggest specialised insights into the stock market uniquely tailored to consumers’ needs and arrived at through sophisticated machine learning models. The practice is more straightforward. At the time of writing, robo-advisers do not rely on state-of-the-art AI technology, such as using neural networks to process data points and make predictions about stock market moves, or link individual profiles to unique investment strategies. As Baker and Dellaert explain, the matching process will be based on ‘a model of how to optimise the fit between the attributes of the financial products available to the consumer and the attributes of the consumers who are using the robo-advisor’.Footnote ³⁵ The robo-adviser will typically build the consumer profile based on the entry questionnaire and match this with an investment strategy established using financial modelling techniques and based on the investment packages already offered by firm. The process will usually have been automated through some form of expert system – a hand coded application of binary rule identified by humans. Ongoing management of the consumer’s portfolio will be done on a similar basis, often using exchange-traded funds (ETFs) that ‘require no or less active portfolio management’.Footnote ³⁶

Unlike human financial advisers, robo-advice tools typically do not provide budgeting or financial management advice to consumers.Footnote ³⁷ Their recommendations are limited to the kinds of investment that will match consumers’ investment profiles. Robo investment advisers do not provide advice on matters of tax, superannuation, asset management, or savings, and they do not yet have the capacity to provide this more nuanced advice.Footnote ³⁸ Sometimes robo-advice tools are used in conjunction with human financial advisers who will provide a broader suite of advice. Automated budgeting tools are also increasingly available on the market.

2.2.2 Budgeting Tools

Budgeting tools allow consumers to keep track of their spending by categorising expenses and providing dashboard-style visualisations of spending and saving.Footnote ³⁹ Some banks offer budgeting tools to clients, and there are many independent service providers. Some neo-banks have, additionally, consolidated their brand around their in-built budgeting tools.Footnote ⁴⁰

As with robo-advisers, automated budgeting tools collect information about consumers through an online questionnaire. Budgeting tools also typically require consumers to provide access to their bank accounts, in order to scrape transaction data from that account,Footnote ⁴¹ or alternatively rely on data-sharing arrangements.Footnote ⁴² Based on this information, the services provided by budgeting tools include categorising and keeping track of spending; providing recommendations about budgeting; and monitoring savings.Footnote ⁴³ In some cases, the tools will transfer funds matching consumers’ savings goals to a specific account, provide bill reminders, make bill payments, monitor information about credit scores, suggest potential savings through various cost-cutting measures or identifying alternative service providers.Footnote ⁴⁴ Additionally, automated budgeting tools may provide articles and opinion pieces about financial matters, such as crypto, non-fungible tokens (NFTs), or budgeting.Footnote ⁴⁵ Some budgeting tools have a credit card option,Footnote ⁴⁶ and at least one is linked to a ‘buy now-pay later’ provider.Footnote ⁴⁷

Automated budgeting tools often describe their service as relying on AI.Footnote ⁴⁸ Again, however, they do not, as might have been expected from this terminology, typically provide a personalised plan for saving derived from insights from multiple data points relating to consumers. They may use some form of natural language processing to identify spending items. Primarily, somewhat like robo-advisers, they rely on predetermined, human-coded rules for categorising spending and presenting savings. Most budgeting tools are free, although some charge for a premium service. This means that the tools are funded in other, more indirect ways, including through selling targeted advertisements on the app, fees for referrals, commissions for third-party products sold on the app, the sale of data (usually aggregated), and in some cases a percentage of the savings where a lower cost service or provider is identified for consumers.Footnote ⁴⁹

2.3.1 Quality of Performance

One of the notable features of automated financial advice is that consumers are unlikely to be able to scrutinise the quality of the service provided. Consumers will typically turn to automated advice tools because they lack skills in the relevant area, be it investing or budgeting. This lack of expertise makes it difficult for them to assess the quality of the advice they receive.Footnote ⁵¹ There is not a lot of information for consumers in selecting between different tools, as compared to standard consumer goods. While some rankings of automated financial advice tools have emerged, these often focus on ease of use – the interface, syncing with bank data, fees charged – rather than the quality of the advice provided,Footnote ⁵² and some ranking reviews include sponsored content.Footnote ⁵³ Accordingly, at least at this point in time, automated financial advice tools may be very much a credence good – for which assertions of quality are all that is available to consumers. Unless the advice provided by the tools is patently bad, it may not be apparent that the poor quality of the automated process is to blame, as opposed to other external factors. Indeed, without a point of comparison, which is effectively excluded by the personalised nature of the service, it may be difficult for consumers to identify poor quality advice at all.

There is currently little academic research on the extent to which consumers are well-served by automated financial advice tools, particularly when weighted against possible costs in terms of data-sharing.Footnote ⁵⁴ There have been a number of concerns raised in the literature about how well the tools may function. Although robo-advisers may operate in a manner that is more objective and consistent than human financial advisers,Footnote ⁵⁵ this does not mean they operate free from the influence of commissions, which may be coded into their advisory process. It is unclear to what extent the recommendations provided by automated financial advice tools are personalised to consumers, as opposed to being generic or based on broad target groupings. Additionally, concerns have been raised about the relatively small number of investment options actually held by robo-investment advisers.Footnote ⁵⁶ While automated budgeting tools may assist consumers by providing an accessible, straightforward, and visual way of monitoring spending,Footnote ⁵⁷ this does not necessarily translate into long-term savingsFootnote ⁵⁸ or improved financial literacy.Footnote ⁵⁹ It is further possible that one of the main functions of at least some budgeting apps is to obtain consumers’ attention in order to market other financial services, such as credit cards, as well as the opportunity for the providers to profit from the use or sale of consumer data for marketing and data analytics.Footnote ⁶⁰

In consumer transactions – particularly those that are complex, hard for consumers to monitor, or which carry the risk of high impact harms – reliance is usually placed on regulators to take ‘ex ante’ measures for ensuring that the products supplied to consumers are acceptably safe and reliable. Financial services regulators in jurisdictions such as Australia, the United Kingdom, the European Union, and the United States of America have responded to the rise of robo-advisers by affirming that the existing regulatory regime applies to this form of advice.Footnote ⁶¹ Financial services providers are typically subject to an array of statutory conduct obligations, which overlap, albeit imperfectly, with their fiduciary duties arising under general law.Footnote ⁶² These statutory duties require firms to manage conflicts of interest,Footnote ⁶³ act in their clients, best interests,Footnote ⁶⁴ ensure the suitability of the advice provided,Footnote ⁶⁵ and take reasonable care in proving the advice.Footnote ⁶⁶ These obligations should, in principle, assist in addressing concerns about the quality of the service provided by robo-advisers.Footnote ⁶⁷ Nonetheless, some uncertainties remain, including, for example, whether the category-based approach deployed by robo-advisers fits with statutory requirements for personalised advice that is suitable for the individual consumer.Footnote ⁶⁸

Regulators have additionally stated they expect firms providing robo-advice to have a ‘human in the loop’, in the sense of a person with ‘an understanding of the technology and algorithms used to provide digital advice’ and who are ‘able to review the digital advice generated by algorithms’.Footnote ⁶⁹ Recommendations for a human overseeing the automated advice leave open the question of what that human should be monitoring – is it merely compliance with existing law applying to the giving of advice, or should there be other considerations taken into account, arising from the automated character of the advice?

In terms of the issue of automation, regulators have focused on the informational aspects of the process. They have emphasised that firms providing automated advice should give consideration to the way in which the information on which the advice is based is collected from consumers so as to ensure it is accurate and relevant, especially because there is no human intermediary to pick up possible discrepancies or errors. Regulators have also advised firms to take care in the way the advice is framed and explained, given the potential for misunderstanding and error in an automated process.Footnote ⁷⁰ Issues of information gathering and reporting are important but they are only part of the challenge presented by automation for consumer protection law and policy. Moreover, they tend to represent a very individualised response to the risks of harm to consumers relying on automated financial advice, focusing on what consumers need to provide and understand, as opposed to the substance of the process through which advice is provided.

Notably, there is typically no specific law or regulatory guidance that applies to automated budgeting tools, which do not involve financial services. These tools will be subject to general consumer protection regimes, which typically prohibit misleading conduct, and mandate reasonable care and skill in the provision of services.Footnote ⁷¹ Uncertainties about the application of existing law to automated advice give rise to the question of whether other kinds of regulatory mechanisms are required to complement sector-specific or general consumer protection law in order to address the risks of harms that are specific to the use of AI and related digital technologies. In answering this question, we suggest that, at minimum, the risks around data collection and bias need to be considered.

2.3.2 The Data/Service Trade-Off

Automated financial advice tools operate on the core premise that consumers necessarily hand over data to obtain the service. A firm may be using consumer data for the dual purposes of providing advice and making a return for itself, such as through promoting other products for a commission on sales, up-selling add-on products for a fee, or on-selling the data for profit.Footnote ⁷² This behaviour is particularly apparent in the case of budgeting apps, which are typically free. As already noted, these services earn income through in-app advertising, fees, and commissions for referrals and potentially through selling aggregated consumer data, as well as targeted advertising. Notably, the privacy terms of automated budgeting tools commonly allow the collection of a wide range of consumer data and the use of that data for a number of purposes, including improving the service and related company group services, marketing, and, in aggregated form, sharing with third parties.Footnote ⁷³

Data protection and privacy law impose obligations on the collection and processing of data.Footnote ⁷⁴ However, the key requirements of notice and consent typically found under these regimes may easily be met in automated advice contexts because the exchange is at the heart of the transaction. Consumers provide their data in order to obtain the advice they need. While consumers may be unaware of how much information they are handing over, there is some evidence that consumers, particularly younger consumers, are prepared to trade data for cheaper, more efficient financial services.Footnote ⁷⁵ However, to the extent consumers are ill or under-informed about the quality of the service being provided by automated advice tools, the data-for-service bargain may look thinner than they might have at first thought.Footnote ⁷⁶ Under the fintech service model, consumers provide personal data to obtain a personalised and cost-effective service but have few objective measures as to the quality of what is actually being provided.

2.3.3 Bias and Exclusion

In discussing legal and regulatory responses to the growing influence of AI and related technologies, much attention has rightly been given to their role in amplifying surveillance, bias and discrimination.Footnote ⁷⁷ The technologies may use personal data to profile consumers, which in turn allows firms to differentiate between different consumers and groups with a high degree of precision, leading to risks of harmful manifestations of targeted advertising, or differential pricing.Footnote ⁷⁸ Bias and error are particular concerns in firms’ use of AI technologies for decision-making, including in decisions about lending,Footnote ⁷⁹ credit,Footnote ⁸⁰ or insurance.Footnote ⁸¹ Automated lending decisions and credit scoring might be more objective than human-made decisions and might benefit cohorts that have previously been disadvantaged by human prejudice.Footnote ⁸² But there is no guarantee this is the case, and indeed the outcomes may be worse for these groups. Differential treatment of already disadvantaged groups – such as minoritiy or low-income cohorts – may already be embedded in the practices and processes of the institution. To the extent this data is used in credit-scoring models or to inform automated decisions, historical unequal treatment may be amplifiedFootnote ⁸³ or distorted.Footnote ⁸⁴ Unequal treatment may, moreover, be difficult to identify or address where it is based, not directly on protected attributes, but on proxies for those attributes found in the training data.Footnote ⁸⁵

Bias may also be embedded in automated advice tools used by consumers. For example, a robo-advice tool might exhibit bias by treating a person who takes time off work for childrearing as going through a period of precarious employment or being unable to hold down steady employment. An automated budgeting tool might exhibit bias by characterising products for menstruation as discretionary spending, instead of essentials. There are complex technical and policy decisions to be made in identifying and responding to the risks of unacceptable bias in automated financial advice tools.Footnote ⁸⁶ Consumer protection and financial services law have not traditionally have not been central to this process, which is primarily the domain of human rights law. However, decisions based on historical prejudice may be unconscionable or unfair, contrary to consumer protection law. Certainly, in the United States, the Federal Trade Commission has indicated that discriminatory algorithms would fall foul of its jurisdiction to respond to unfair business practices.Footnote ⁸⁷

A related issue concerns financial exclusion. Fintech innovators and government initiatives to encourage innovation often refer to an aspiration of promoting inclusion and overcoming exclusion.Footnote ⁸⁸ There are few findings on the extent to which this aspiration is achievable. There are plausible reasons why automated advice tools may fail to assist, or assist adequately, consumers already excluded from mainstream financial or banking services, or consumers who have had less engagement with the mainstream banking system, such as where they are ‘not accessing or using financial services in a mainstream market in a way that is appropriate to their needs’.Footnote ⁸⁹ Financially excluded consumers might not be offered meaningfully relevant advice tools because there is no relevant or useful data about them or because they are unlikely to be sufficiently profitable for financial services providers to develop products suited to them. These consumers may also find that the models on which the advisory tools are based are inaccurate when applied to their circumstances.

For example, investment tools may be of little value to consumers struggling to make ends meet and with no savings to invest. The models used by automated budgeting tools may have a poor fit with consumers living on very low incomes and for whom cutting back on discretionary spending is not an option available. In these circumstances, the tools will do little to improve equity, leaving unrepresented groups without advice, or relevantly personalised advice. Moreover, there may be a real risk of harm. Inept recommendations may subject consumers to harms of financial over-commitment or lull inexperienced consumers into a false sense of financial security. At a more systematic level, the availability of automated advice tools for improving financial well-being may feed into longstanding liberal rhetoric about the value of individual responsibility, as opposed to government initiatives for improving overall financial well-being.

It is possible to envisage services that would be useful to financially excluded consumers or consumers experiencing financial harshi, such as for example, advice on affordable loans and other services.Footnote ⁹⁰ Emma Leong and Jodi Gardner point to proposed uses of Open Banking in the United Kingdom to provide tools that assist with better managing fluctuating incomes.Footnote ⁹¹ The United Kingdom Financial Conduct Authority notes there are some apps on the market providing legal aid and welfare support advice.Footnote ⁹² These kinds of initiatives are likely to require a deliberate policy decision to initiate rather than arising ‘naturally’ in the market.Footnote ⁹³ This is because there would seem to be little commercial incentive for firms to invest in tools specifically tailored to low-income or otherwise marginalised consumers from whom there is little likelihood of ongoing lucrative return to the firm, without government support.

2.4.1 Transparency and Explanations

Principles of ethical AI typically require the use of such technologies to be transparent.Footnote ¹⁰⁶ A starting place for transparency is to inform consumers when AI is being used in an interaction with them. Applied to automated financial advice tools, transparency must mean more than informing consumers that AI is being used to provide advice. Consumers choosing to turn to a robo-adviser or budgeting app will usually be aware of the automated character of the advice. Consumers also require transparency in the kind of technology being used to provide that advice: i.e. is it based in machine learning or a hand coded expert system. Additionally, a principle of transparency would require firms to inform consumers clearly about the scope of the service that is being provided, including the limitations of the technology in terms of personalised or expert advice.Footnote ¹⁰⁷ If the advice provided is generalised to broadly defined categories of consumers, then this should be made clear, to counter consumers’ expectations of a unique and personal experience.

To the extent that consumers overestimate the capacities of fintech, transparency in way the advice is produced is important to ground expectations and allow scrutiny of the veracity of claims made about it. For regulators, transparency is key to overseeing the performance of the tools. Transparency is key to allowing bias or distortions in the scope of advice to be identified, scrutinised and, in some instances, rectified. Regulation can support the imperative for firms to take these ethical demands seriously, including by treating them as necessary elements of statutory obligations of suitability or best interests, and essential to ensuring that claims about the operation of the product are not misleading. For example, the process of automation, and its claims to objectivity and consistency, may make consumers overconfident about the advice and more likely to act on it.Footnote ¹⁰⁸ This might suggest an obligation on firms to be scrupulously clear on the limits of what is able to provided by automated advice tools, and of the insights that can be derived from the technology being utilised.Footnote ¹⁰⁹

Transparency in ethical AI is closely associated with initiatives in AI ‘explanations’ or ‘explainability’.Footnote ¹¹⁰ Explanations in this sense do not lie in the details of the code. Rather, explainable AI considers the kind and degree of information that should be provided in assisting the various stakeholders in the decision or recommendation process to understand why decisions were taken or the factors that were significant in reaching a recommendation.Footnote ¹¹¹ Explainable AI aims to provide greater transparency into the basis for automated decisions, predictions, and recommendations.Footnote ¹¹² There are different ways in which explanations may be provided, and indeed the field of study in computer science is still developing.Footnote ¹¹³ Possibilities include the use of counterfactuals, feature disclosure scores, weightings of influential factors, or a preference for simpler models where high levels of accuracy are not as imperative.Footnote ¹¹⁴ Overall, however, a requirement for explanations would assist in scrutinising the basis of the recommendations produced through automated financial advice tools.

For lawyers, suggesting that a core element in the regulation of automated financial advice tools should focus on requirements related to transparency/explanations may seem a surprising aspiration.Footnote ¹¹⁵ Disclosure as a consumer protection strategy has increasingly fallen out of favour, particularly in the regulation of financial services and credit. The insights into decision-making from behavioural psychology have shown that mere information disclosure does not lead to better decisions by consumers. Consumers are subject to bounded rationality which means they rely on rules of thumb, heuristics, and behavioural bias rather than information.Footnote ¹¹⁶ In this light, it may be thought that any demand for greater transparency in automated financial advice tools may be of marginal utility. However, in a consumer protection context, consumers’ interests are substantially protected by regulators, and therefore transparency and explanations are relevant to both consumers seeking to protect their interests, and regulators charged with overseeing the market. Explanations should be provided in a form that is meaningful to the recipient.Footnote ¹¹⁷ This means that the detail and technicality of the information provided may need to differ between consumers and regulators.Footnote ¹¹⁸ In other words, the requirements should be scaled according to who is receiving the explanation.

2.4.2 Accountability

Principles of AI ethics typically require mechanisms for ensuring firms are accountable for the operation of the technologies.Footnote ¹¹⁹ To have impact, accountability will require more than allocating responsibility for supervising the AI to a person. There is little worth in having a ‘human in the loop’ in circumstances where the design of the AI or automated tool means it is difficult for that person genuinely to oversee, interrogate or control the tool.Footnote ¹²⁰ Accountability for automated financial advice tools should therefore require a firm to implement systematic processes for reviewing the operations and performance of the tools.Footnote ¹²¹ A commitment to accountability may therefore require firms to have processes for scrutinising the data on which the AI is trained, its ongoing use, and its outputs.Footnote ¹²² A model for the kind of robust approach required might be found in the audits increasingly recommended for AI used in public sector decision-making.Footnote ¹²³ Such processes should aim to ensure the veracity of the tools and are a critical element in addressing and redressing concerns about bias, equity, and inclusion.Footnote ¹²⁴

2.4.3 Contestability

There is little utility in requiring transparency and accountability in AI systems if there is no mechanism available to those affected by an AI or automated decision for acting to challenge an outcome that is erroneous, discriminatory, or otherwise flawed. Some formulations of AI ethical principles respond to this issue by requiring processes for contesting adverse outcomes.Footnote ¹²⁵ While accountability processes should aim to be proactive in preventing these kinds of problems, contestability is a mechanism for individuals, advocates, or regulators to respond to harms that do occur.

Lyons et al. make the point that little is currently known about ‘what contestability in relation to algorithmic decisions entails, and whether the same processes used to contest human decisions … are suitable for algorithmic decision-making’.Footnote ¹²⁶ Contestability for automated decisions may not be able simply to follow existing mechanisms for dealing with individual complaints or concerns. The models informing AI may be complex and opaque, thus creating challenges for review by subject domain experts who may nonetheless be unfamiliar with the technology. Additionally, scale creates a challenge. This is because one of the benefits of automated decision-making is that it can operate on a scale that is not possible for human decision-makers or advisers, and yet this makes processes for individual review potentially unmanageable.

The inquiry into what contestability requires may be different in the context of automated financial advice tools, as opposed to public sector use of automated decision-making. Consumers using automated advice tools will not be challenging a decision made about their rights to access public resources or benefits. Rather they will be challenging the advice given to them, the consistency of this advice with any representations about the tool, or compliance with any applicable regulatory regimes. Nonetheless, complexity and scale remain significant challenges. It is possible that the field of consumer protection law may have insights given its focus on both legal rights and structural mechanisms for protecting consumers’ interests in circumstances where there are considerable imbalances in power, resources, and information, which in some ways mirrors concerns around AI contestability. For example, in this context of automated financial advice tools, contestability for poor outcomes may come through the oversight provided by ombudsmen and regulators, rather than traditional litigation. These inquiries have the capacity to look at systemic errors, thus bringing expertise and capacity to review processes through which advice or recommendations are provided, rather than necessarily reopening every decision.

3.5.1 Advantages

New technologies are key to improving the management of regulatory risks. Banks have begun exploring the use of AI to assist analysts in what has traditionally been a manually intensive task to improve the performance of AML processes.Footnote ⁵⁶ In 2018, US government agencies issued a joint statement encouraging banks to use innovative methods, including AI, to further efforts to protect the integrity of the financial system against illicit financial activity.Footnote ⁵⁷ The United Kingdom Financial Conduct Authority has supported a series of public workshops aimed at encouraging banks to experiment with novel technologies to improve the detection of financial crimes.Footnote ⁵⁸ AUSTRAC has invested in data analysis and advanced analytics to assist in the investigation of suspicious activity.Footnote ⁵⁹ Indeed, developments in AI offer an opportunity to fundamentally transform the operations of banks, equipping them to combat modern threats to the integrity of the financial system.Footnote ⁶⁰ And, where AI reaches the same conclusions as traditional analytical models, this can confirm the accuracy of such assessments, ultimately increasing the safeguards available to supervisors.Footnote ⁶¹ Although machine learning remains relatively underutilised in the area of AML, it offers the potential to greatly enhance the efficiency and effectiveness of existing systems.Footnote ⁶²

3.5.2 Challenges

Despite the growing recognition of the potential for AI to improve the accuracy, speed, and cost-effectiveness of AML processes, banks remain slow to adopt these technologies due to the regulatory and operational challenges involved.Footnote ⁷⁶ Significant hurdles to wider adoption persist and these may continue to stifle innovations in AML compliance.

3.5.3 Consideration

AI and machine learning have the potential to provide banks with effective tools to improve risk management and compliance with regard to AML. However, if these new technologies are not introduced with care and diligence, they could adversely affect AML systems by introducing greater burdens and risks. Some of the challenges presented by AI are similar to those posed by other technology-based solutions aimed at identifying and preventing money laundering. Machine learning, however, offers a relatively new and unique method of classifying information based on a feedback loop that enables the technology to ‘learn’ through determinations of probability.Footnote ¹³⁸ Banks can thus analyse and classify information through learned anomaly detection algorithms, a technique that is more effective than traditionally programmed rule-based systems.Footnote ¹³⁹ At the same time, the utilisation of AI can exacerbate the complexity and severity of the challenges inherent in AML compliance, particularly in relation to interpretation and explanation.Footnote ¹⁴⁰ As discussed above, machine learning algorithms usually do not provide a rationale or reasoning for the outcomes they produce, making it difficult for compliance experts to validate the results and deliver clear reports to regulators.Footnote ¹⁴¹ This is particularly concerning for banks, where trust, transparency, and verifiability are of great importance to ensure satisfaction and regulatory confidence.Footnote ¹⁴² Nonetheless, in the current regulatory climate, it seems almost inevitable that banks will continue to leverage AI for AML compliance.

4.2.1 Opacity of Credit Scoring and the (Lack of) Explanation of Financial Decisions

Despite their widespread use in the financial industry, credit scores are difficult for consumers to understand or interpret. A person’s credit risk has traditionally been calculated based on ‘three C’s’: collateral, capacity, and character.Footnote ¹⁹ Due to the rise of AI and ADM tools in the financial industry, the ‘three C’s’ are increasingly being supplemented and replaced by diverse categories of data.Footnote ²⁰ An interesting example can be found through FICO scores, which are arguably the first large-scale process in which automated computer models replaced human decision-making.Footnote ²¹ FICO, one of the best-known credit scoring companies,Footnote ²² explains that their scores are calculated according to five categories: ‘payment history (35%), amounts owed (30%), length of credit history (15%), new credit (10%), and credit mix (10%)’.Footnote ²³ These percentage scores are determined by the company to give consumers an understanding of how different pieces of information are weighted in the calculation of a score, and the ratios identified within FICO scores will not necessarily reflect the weightings used by other scoring companies. Further, while FICO provides a degree of transparency, the ways in which a category such as ‘payment history’ is calculated remains opaque: consumers are not privy to what is considered a ‘good’ or a ‘bad’ behaviour, as represented by data points in their transaction records.Footnote ²⁴

Globally, many credit scoring systems (both public and private) produce three-digit numbers within a specified range to determine a consumer’s creditworthiness. For example, privately operated Equifax and Trans Union Empirica score consumers in Canada between 300 and 900,Footnote ²⁵ whereas credit bureaus in Brazil score consumers between 1 and 1,000.Footnote ²⁶ In an Australian context, scores range between 0 and 1,000, or 1,200, depending on the credit reporting agency.Footnote ²⁷ By contrast, other jurisdictions use letter-based ratings, such as Singapore’s HH to AA scale which corresponds with a score range of 1,000–2,000,Footnote ²⁸ or blacklists, such as Sweden’s payment default records.Footnote ²⁹

Credit scoring, it turns out, is surprisingly accurate in predicting financial breakdowns or future loan delinquency,Footnote ³⁰ but the way different data points are combined by models is not something even the model designer can understand using just intuition.Footnote ³¹ Automated scoring processes become even more complex as credit scoring companies increasingly rely on alternative data sources to assess consumers’ creditworthiness, including ‘predictions about a consumer’s friends, neighbors, and people with similar interests, income levels, and backgrounds’.Footnote ³² And a person’s credit score is just one of the elements lenders, Automated Banks, feed into their models to determine a consumer’s risk score. It has been reported that college grades, and the time of day an individual applies for a loan have been used to determine a person’s access to credit.Footnote ³³ These types of data constitute ‘extrinsic data’ sources, which consumers are unknowingly sharing.Footnote ³⁴

The use of alternative data sources is purported as a way of expanding consumers’ access to credit in instances where there is a lack of quality data (such as previous loan repayment history) to support the underwriting of consumers’ loan.Footnote ³⁵ Applicants are often faced with a ‘Catch-22 dilemma: to qualify for a loan, one must have a credit history, but to have a credit history one must have had loans’.Footnote ³⁶ This shows how ADM tools offer more than just new means to analyse greater than ever quantities of data: they also offer a convenient excuse for Automated Banks to effectively use more data.

Of course, increasing reliance on automated risk scoring is not the origin of unlawful discrimination in financial contexts. However, it is certainly not eliminating discriminatory practices either: greater availability of more granular data, even when facially neutral, leads to reinforcing of existing inequalities.Footnote ³⁷ Automated Banks have been also shown to use alternative data to target more vulnerable consumers, who they were not able to reach or identify when only using traditional data on existing customers.Footnote ³⁸ The quality change that AI tools promise to bring is to ‘make the data talk’: all data is credit data, if we have the right automated tools to analyse them.Footnote ³⁹

Collection, aggregation, and use of such high volumes of data, including ‘extrinsic data’, also make it more difficult, if not impossible, for consumers to challenge financial decisions affecting them. While laws relating to consumer lending (or consumer financial products in general) in most jurisdictions provide that some form of explanation of a financial decision needs to be made available to consumers,Footnote ⁴⁰ these rules will rarely be useful in the context of ADM and AI tools used in processes such as risk scoring.

This is because AI tools operate on big data. Too many features of a person are potentially taken into account for any feedback to be meaningful. The fact that risk scores and lending decisions are personalised make it even more complicated for consumers to compare their offer with anyone else’s. This can be illustrated by the case of Apple credit card,Footnote ⁴¹ which has shown the complexity of investigation necessary for people to be able to access potential redress: when applying for personalised financial products, consumers cannot immediately know what features are being taken into account by financial firms assessing their risk, and subsequent investigation by regulators or courts may be required.Footnote ⁴² The lack of a right to meaningful explanation of credit scores and lending decisions based on the scores makes consumers facing Automated Banks and the automated credit scoring system quite literally powerless.Footnote ⁴³

4.2.2 Trade Secrets and ADM Tools in Credit Scoring

The opacity of credit scoring, or risk scoring more generally, and other automated assessment of clients that Automated Banks engage in, is enabled by ADM tools which ‘are highly valuable, closely guarded intellectual property’.Footnote ⁴⁴ Complementing the limited duty to provide explanation of financial decisions to consumers, trade secrets laws allow for even more effective shielding of the ADM tools from scrutiny, including regulators’ and researchers’ scrutiny.

While trade secrets rules differ between jurisdictions, the origin and general principles that underpin these rules are common across all the legal systems: trade secrets evolved as a mechanism to protect diverse pieces of commercial information, such as formulas, devices, or patterns from competitors.Footnote ⁴⁵ These rules fill the gap where classic intellectual property law, such as copyright and patent law, fails – and it notably fails in relation to AI systems, since algorithms are specifically excluded from its protection.Footnote ⁴⁶ Recent legal developments, for example the European Union Trade Secrets Directive,Footnote ⁴⁷ or the US Supreme Court case of Alice Corp. v CLS Bank,Footnote ⁴⁸ mean that to protect their proprietary technologies, companies are now turning to trade secrets.Footnote ⁴⁹ In practice, this greatly reduces the transparency of the ADM tools used: if these cannot be protected through patent rights, they need to be kept secret.Footnote ⁵⁰

The application of trade secrets rules leads to a situation in which financial entities, for example lenders or insurers, who apply third party automated tools to assess creditworthiness of their prospective clients might not be able to access the models and data they use. Using third party tools is a common practice, and the proprietary nature of the tools and data used to develop and train the models will mean financial entities using these tools may be forced to rely on the supplier’s specifications in relation to their fairness as they may not be able to access the code themselves.Footnote ⁵¹

Secrecy of ADM tools of course has implications for end users, who will be prevented from challenging credit models, and is also a barrier for enforcement and research.Footnote ⁵² Trade secret protections apply not only to risk scoring models, but often extend also to data sets and inferences generated from information provided by individuals.Footnote ⁵³ Commercial entities openly admit they ‘invest significant amounts of time, money and resources’ to draw inferences about individuals ‘using […] proprietary data analysis tools’, a process ‘only made possible because of the [companies’] technical capabilities and value add’.Footnote ⁵⁴ This, they argue, makes the data sets containing inferred information a company’s intellectual property.Footnote ⁵⁵

The application of trade secrets rules to credit scoring in a way that affects the transparency of the financial system is not exactly new: ‘[t]he trade secrecy surrounding credit scoring risk models, and the misuse of the models coupled with the lack of governmental control concerning their use, contributed to a financial industry wide recession (2007–2008)’.Footnote ⁵⁶

In addition to trade secrets laws, a sui generis protection of source code of algorithms is being introduced in international trade law through free trade agreements,Footnote ⁵⁷ which limit governments from mandating access to the source code. The members of the World Trade Organization (WTO) are currently negotiating a new E-commerce trade agreement, which may potentially include a prohibition on government-mandated access to software source code.Footnote ⁵⁸ WTO members, including Canada, the EU, Japan, South Korea, Singapore, Ukraine, and the United States support such a prohibition,Footnote ⁵⁹ which in practice will mean a limited ability for states to adopt laws that would require independent audits of AI and ADM systems.Footnote ⁶⁰ It is argued that adoption of the WTO trade agreement could thwart the adoption of the EU’s AI Act,Footnote ⁶¹ demonstrating how free trade agreements can impose another layer of rules enhancing the opacity of AI and ADM tools.

4.2.3 ‘Depersonalising’ Information to Avoid Data and Privacy Protection Laws: Anonymisation, De-identification, and Inferences

Automated Banks’ opacity is enabled by the express exclusion of ‘anonymised’ or ‘de-identified’ data from the scope of data and privacy protection laws such as the GDPR.Footnote ⁶² In its Recital 26, the GDPR defines anonymised information as not relating to ‘an identified or identifiable natural person’ or as ‘data rendered anonymous in such a manner that the data subject is not or no longer identifiable’. This allows firms to engage in various data practices, which purport to use anonymised data.Footnote ⁶³ They argue they do not collect or process ‘personal information’, thus avoiding the application of the rules, and regulatory enforcement.Footnote ⁶⁴ Also, consumers to whom privacy policies are addressed believe that practices focusing on information that does not directly identify them have no impact on their privacy.Footnote ⁶⁵ This in turn may mean privacy policies are misrepresenting data practices to consumers, which could potentially invalidate their consent.Footnote ⁶⁶

There is an inherent inconsistency between privacy and data protection rules and the uses and benefits that ADM tools using big data analytics promise. Principles of purpose limitation and data minimisationFootnote ⁶⁷ require entities to delimit, quite strictly and in advance, how the data collected are going to be used, and prevent them from collecting and processing more data than necessary for that specific purpose. However, this is not how big data analytics, which fuels ADM and AI models, works.Footnote ⁶⁸ Big data means that ‘all data is credit data’, incentivising the Automated Banks to collect as much data as possible, for any possible future purpose, potentially not known yet.Footnote ⁶⁹ The exclusion of anonymised or de-identified data from the scope of the protection frameworks opens doors for firms to take advantage of enhanced analytics powered by new technologies. The contentious question is at which point information becomes, or ceases to be, personal information. If firms purchase, collect, and aggregate streams of data, producing inferences allowing them to describe someone in great detail, including their age, preferences, dislikes, size of clothes they wear and health issues they suffer from, their household size and income level,Footnote ⁷⁰ but do not link this profile to the person’s name, email, physical address, or IP address – would it be personal information? Such a profile, it could be argued, represents a theoretical, ‘model’ person or consumer, built for commercial purposes through aggregation of demographic and other information available.Footnote ⁷¹

De-identified data may still allow a financial firm to achieve more detailed segmentation and profiling of their clients. There are risks of harms in terms of ‘loss of privacy, equality, fairness and due process’ even when anonymised data is used.Footnote ⁷² Consumers are left unprotected against profiling harms due to such ‘narrow interpretation of the right to privacy as the right to anonymity’.Footnote ⁷³

There is also discussion as to the status of inferences under data and privacy protection laws. Credit scoring processes are often based on inferences, where a model predicts someone’s features (and ultimately their riskiness or value as a client) on the basis of other characteristics that they share with others deemed risky by the model.Footnote ⁷⁴ AI models may thus penalise individuals for ‘shopping at low-end stores’, membership in particular communities or families, and affiliations with certain political, religious, and other groups.Footnote ⁷⁵ While AI-powered predictions about people’s characteristics are often claimed to be more accurate than those made by humans,Footnote ⁷⁶ they may also be inaccurate.Footnote ⁷⁷ The question is if such inferences are considered personal information protected by privacy and data laws.

Entities using consumers’ data, such as technology companies, are resisting against expressly including inferred information in the scope of data and privacy protections. For example, Facebook openly admitted that ‘[t]o protect the investment made in generating inferred information and to protect the inferred information from inappropriate interference, inferred information should not be subject to all of the same aspects of the [Australian Privacy Act] as personal information’.Footnote ⁷⁸ The ‘inappropriate interference’ they mention refers to extending data correction and erasure rights to inferred information.

Second, there is an inherent clash between the operation of privacy and data protection rules and the inference processes AI tools are capable of carrying out. Any information, including sensitive information, may be effectively used by an ADM system, even though it only materialises as an internal encoding of the model and is not recorded in a human understandable way. The lack of explicit inclusion of inferred information, and its use, within the privacy and data protection frameworks provides another layer of opacity shielding financial firms (as well as other entities) from scrutiny of their ADM tools.

When information is ‘depersonalised’ in some way: de-identified on purpose through the elimination of strictly personal identifiers,Footnote ⁷⁹ through use of anonymous ‘demographic’ data, through ‘pseudonymisation’ practices, or because it is inferred from data held (either personal or already de-identified), the result is the same – privacy and data protection rules do not apply. The firms take advantage of that exclusion, sometimes balancing on the thin line between legal and illegal data processing, making their data practices non-transparent to avoid scrutiny by consumers and regulators.

As a US judge in a recent ruling put it: ‘[i]t is well established that there is an undeniable link between race and poverty, and any policy that discriminates based on credit worthiness correspondingly results in a disparate impact on communities of color’.Footnote ⁸⁰ The data used in large-scale AI and ADM models is often de-identified or anonymised, but it inherently mirrors historical inequalities and biases, thus allowing the Automated Banks to claim impartiality and avoid responsibility for the unfairness of data used.

The reason why privacy and data protection rules lack clear consideration of certain data practices and processes enabled by AI may be due to these tools and processes being relatively new and poorly understood phenomena.Footnote ⁸¹ This status quo is however very convenient for the companies, who will often raise the argument that ‘innovation’ will suffer if more stringent regulation is introduced.Footnote ⁸²

4.3.1 Financial Products Governance Rules

Financial firms have always been concerned with collecting and using data about their consumers, to differentiate between more and less valuable customers. For example, insurance firms, even before AI profiling tools were invented (or at least before they were applied at a greater scale) were known to engage in practices referred to as ‘cherry-picking’ and ‘lemon-dropping’, setting up firms’ offices at higher floors in buildings with no lifts, so that it would be harder for disabled (potential) clients to reach them.Footnote ⁸⁴ There is a risk that the widespread data profiling and use of AI tools may exacerbate issues relating to consumers’ access to financial products and services. AI tools may introduce new or replicate historical biases present in data,Footnote ⁸⁵ doing so more efficiently, in a way that is more difficult to discover, and at a greater scale than was possible previously.Footnote ⁸⁶

An additional disadvantage resulting from opaque risk scoring systems is that consumers may miss out on the opportunity to improve their score (for example, through the provision of counterfactual explanations, or the use of techniques including ‘nearby possible worlds’).Footnote ⁸⁷ In instances where potential customers who would have no trouble paying back loans are given low risk scores, two key issues arise: first, the bank misses out on valuable customers, and second, there is a risk that these customers’ rejections, if used as input data to train the selection algorithm, will reinforce existing biases.Footnote ⁸⁸

Guaranteeing suitability of financial services is a notoriously complicated task for policymakers and regulators. With disclosure duties alone proving largely unsuccessful in addressing the issue of consumers being offered financial products that are unfit for purpose, policymakers in a number of jurisdictions, such as the EU and its Member States, the United Kingdom, Hong Kong, Australia, and Singapore, have started turning to product governance regimes.Footnote ⁸⁹ An important component of these financial product governance regimes is an obligation placed on financial firms, which issue and distribute financial products, to ensure their products are fitness-for-purpose and to adopt a consumer-centric approach in design and distribution of the products. In particular, a number of jurisdictions require financial firms to delimit the target market for their financial products directed at retail customers, and ensure the distribution of the products within this target market. Such target market is a group of consumers of a certain financial product who are defined by some general characteristics.Footnote ⁹⁰

Guides issued by regulators, such as the European Securities and Markets AuthorityFootnote ⁹¹ and the Australian Securities and Investment Commission,Footnote ⁹² indicate which consumers’ characteristics are to be taken into account by financial firms. The consumers for whom the product is intended are to be identified according to their ‘likely objectives, financial situation, and needs’,Footnote ⁹³ or five ‘categories’: the type of client, their knowledge and experience, financial situation, risk tolerance, and objective and needs.Footnote ⁹⁴ For issuers or manufacturers of financial products these considerations are mostly theoretical: as they might not have direct contact with clients, they need to prepare a potential target market, aiming at theoretical consumers and their likely needs and characteristics.Footnote ⁹⁵ Both issuers and distributors need to take reasonable steps to ensure that products are distributed within the target market, which then translates to the identification of real consumers with specific needs and characteristics that should be compatible with the potential target markets identified. Distributors have to hold sufficient information about their end clients to be able to assess if they can be included in the target market,Footnote ⁹⁶ including:

Financial products governance frameworks invite financial firms to collect data on consumers’ vulnerabilities. For example in Australia, financial firms need to consider vulnerabilities consumers may have, such as those resulting from ‘personal or social characteristics that can affect a person’s ability to manage financial interactions’,Footnote ⁹⁸ as well as those brought about by ‘specific life events or temporary difficulties’,Footnote ⁹⁹ in addition to vulnerabilities stemming from the product design or market actions.

The rationale of product governance rules is to protect financial consumers, including vulnerable consumers,Footnote ¹⁰⁰ yet the same vulnerable consumers may be disproportionately affected by data profiling, thus inhibiting their access to financial products. Financial law is actively asking firms to collect even more data about their current, prospective, and past customers, as well as the general public. It provides more than a convenient excuse to carry out digital profiling and collect data for even more precise risk scoring – it actually mandates this.

4.3.2 How ‘Open Banking’ Increases Opacity

Use of AI and ADM tools, together with ever-increasing data collection feeding the data hungry models,Footnote ¹⁰¹ is promoted as beneficial to consumers and markets, and endorsed by companies and governments. Data collection is thus held out as a necessary component of fostering AI innovation. Companies boast how AI insights allow them to offer personalised services, ‘tailored’ to individual consumer’s needs. McKinsey consulting firm hails ‘harnessing the power of external data’ noting how ‘few organizations take full advantage of data generated outside their walls. A well-structured plan for using external data can provide a competitive edge’.Footnote ¹⁰²

Policymakers use the same rhetoric of promoting ‘innovation’ and encourage data collection through schemes such as open banking.Footnote ¹⁰³ The aim of open banking is to give consumers the ability to direct companies that hold financial data about themselves to make it available to financial (or other) companies of the consumer’s choice. Thus, it makes it possible for organisations to get access to consumers’ information they could never get from a consumer directly, such as for example their transaction data for the past ten years.

Jurisdictions such as the EU, United Kingdom, Australia, and Hong Kong have recently adopted regulation promoting open banking, or ‘open finance’ more generally.Footnote ¹⁰⁴ The frameworks are praised by the industry as ‘encourag[ing] the development of innovative products and services that help consumers better engage with their finances, make empowered decisions and access tailored products and services’.Footnote ¹⁰⁵

While open banking is making it possible for financial firms to develop new products for consumers, the jury is still out as to the scheme’s universally positive implications for consumers and markets.Footnote ¹⁰⁶ One thing that is clear, however, is that because of its very nature, open banking contributes to information and power asymmetry between consumers and Automated Banks.

Traditionally, in order to receive a financial product, such as a loan or an insurance product, consumers would have to actively provide relevant data, answering questions or prompts, in relation to their income, spending, age, history of loan repayments, and so on. Open banking – or open finance more broadly – means that consumers can access financial products without answering any questions. But these questions provided a level of transparency to consumers: they knew what they were being asked, and were likely to understand why they were being asked such questions. But when an individual shares their ‘bulk’ data, such as their banking transaction history, through the open banking scheme, do they really know what a financial firm is looking for and how it is being used? At the same time, in such a setting, consumers are deprived of control over which data to share (for example, they cannot just hide transaction data on payments they made to merchants such as liquor stores or pharmacies). The transparency for financial firms when data is shared is therefore significantly higher than in ‘traditional’ settings – but for consumers the process becomes more opaque.Footnote ¹⁰⁷

4.4.1 Preventing Automated Banks from Designing Harmful AI Systems

International and national bodies in multiple jurisdictions have recently adopted, or are currently debating, various measures with an overarching aim of protecting consumers from harm. For example, the US Federal Trade Commission has provided guidance to businesses using AI, explaining that discriminatory outcomes resulting from the use of AI would contravene federal law.Footnote ¹¹⁰ The most comprehensive approach to limiting the use of particular AI tools can be found in the EU’s proposed Artificial Intelligence Act. Its Recital 37 specifically recommends that ‘AI systems used to evaluate the credit score or creditworthiness of natural persons should be classified as high-risk AI systems’. This proposal is a step towards overcoming some opaque practices, through the provision of ‘clear and adequate information to the user’ along with other protections that enable authorities to scrutinise elements of ADM tools in high-risk contexts.Footnote ¹¹¹ Early criticisms of the proposed Act note that while a regulatory approach informed by the context in which ADM is used has some merit, it does not cover potentially harmful practices such as emotion recognition and remote biometric identification,Footnote ¹¹² which could be used across a range of contexts, generating data sets that may later be used in other markets such as financial services.

An alternative approach to regulating AI systems before they are used in markets is to limit the sources of information that can be used by ADM tools, or restrict the ways in which information can be processed. In addition to privacy protections, some jurisdictions have placed limitations on the kinds of information that can be used to calculate a credit score. For example, in Denmark, the financial services sector can use consumers’ social media data for marketing purposes but is explicitly prohibited from using this information to determine creditworthiness.Footnote ¹¹³ Similarly, the EU is considering a Directive preventing the use of personal social media and health data (including cancer data) in the determination of creditworthiness.Footnote ¹¹⁴ Such prohibitions are, however, a rather tricky solution: it may be difficult for the regulation to keep up with a growing list of data that should be excluded from analysis.Footnote ¹¹⁵ One way of overcoming this challenge would be to avoid focusing on restricted data sources, and instead create a list of acceptable data sources, which is a solution applied for example in some types of health insurance.Footnote ¹¹⁶

Imposing limits on how long scores can be kept and/or relied on by Automated Banks is another important consideration. In Australia, credit providers are bound by limits that stipulate the length of time that different pieces of information are held on a consumer’s file: credit providers may only keep financial hardship information for twelve months from the date the monthly payment was made under a financial hardship arrangement, whereas court judgements may be kept on record for five years after the date of the decision.Footnote ¹¹⁷ In Denmark, where the credit reporting system operates as a ‘blacklist’ of people deemed more likely to default, a negative record (for instance, an unpaid debt) is deleted after five years, regardless of whether or not the debt has been paid.Footnote ¹¹⁸ A challenge with these approaches is that the amount of time particular categories of data may be kept may not account for proxy data, purchased data sets, and/or proprietary scoring and profiling systems that group consumers according to complex predictions that are impossible to decode.

4.4.2 Aiding Consumers Understand When ADM Systems Are Used in Financial Services

Despite the development of many principles-based regulatory initiatives by governments, corporates, and think tanks,Footnote ¹¹⁹ few jurisdictions have legislated protections that require consumers to be notified if and when they have been assessed by an automated system.Footnote ¹²⁰ In instances where consumers are notified, they may be unable to receive an understandable explanation of the decision-making process, or to seek redress through timely and accessible avenues.

Consumers face a number of challenges in navigating financial markets, such as understanding credit card repayment requirementsFootnote ¹²¹ and failing to accurately assess their credit.Footnote ¹²² For individuals, it is crucial to understand how they are being scored, as this will make it possible for them to be able to identify inaccuracies,Footnote ¹²³ and question decisions made about them. Credit scoring is notoriously opaque and difficult to understand, so consumers are likely to benefit from requirements for agencies to simplify and harmonise how scores are presented.Footnote ¹²⁴An example of a single scoring system can be found in Sri Lanka, where credit ratings, or ‘CRIB Scores’ are provided by the Central Information Bureau of Sri Lanka, a public-private partnership between the nation’s Central Bank and a number of financial institutions that hold equity in the Bureau. The Bureau issues CRIB Score reports to consumers in a consistent manner, utilising an algorithm to produce a three-digit number ranging from 250 to 900.Footnote ¹²⁵ In Sri Lanka’s case, consumers are provided with a singular rating from a central agency, and although this rating is subject to change over time, there is no possibility of consumers receiving two different credit scores from separate providers.

Providing consumers with the opportunity to access their credit scores is another (and in many ways complementary) regulatory intervention. A number of jurisdictions provide consumers with the option to check their credit report and/or credit score online. For example, consumers in CanadaFootnote ¹²⁶ and AustraliaFootnote ¹²⁷ are able to access free copies of their credit reports by requesting this information directly from major credit bureaus. In Australia, consumers are able to receive a free copy of their credit report once every three months.Footnote ¹²⁸

However, such approaches have important limitations. Credit ratings are just one of many automated processes within the financial services industry. Automated Banks, with access to enough data, can create their own tools going outside the well-established credit rating systems. Also, it is consumers who are forced to carry the burden of correcting inaccurate information which is used to make consequential decisions about them, while often being required to pay for the opportunity to do so.Footnote ¹²⁹

In addition, explainability challenges are faced in every sector that uses AI, and there is considerable investigation ahead to determine the most effective ways of explaining automated decisions in financial markets. It has been suggested that a good explanation is provided when the receiver ‘can no longer keep asking why’.Footnote ¹³⁰ The recent EU Digital Services ActFootnote ¹³¹ emphasises such approach by noting that recipients of online advertisements should have access to ‘meaningful explanations of the logic used’ for ‘determining that specific advertisement is to be displayed to them’.Footnote ¹³²

Consumer experience of an AI system will depend on a number of parameters, including format of explanations (visual, rule-based, or highlighted key features), their complexity and specificity, application context, and variations suiting users’ cognitive styles (for example, providing some users with more complex information, and others with less).Footnote ¹³³ The development of consumer-facing explainable AI tools is an emerging area of research and practice.Footnote ¹³⁴

A requirement of providing meaningful feedback to consumers, for example, through counterfactual demonstrations,Footnote ¹³⁵ would make it possible for individuals to understand what factors they might need to change to receive a different decision. It would also be an incentive for Automated Banks to be more transparent.

4.4.3 Facilitating Regulator Monitoring and Enforcement of ADM Harms in Financial Services

The third category of potential measures relies on empowering regulators, thus shifting the burden away from consumers. For example, regulators need to be able to ‘look under the hood’ of any ADM tools, including these of proprietary character.Footnote ¹³⁶ This could be in a form of using explainable AI tools, access to raw code, or ability to use dummy data to test the model. A certification scheme, such as quality standards, is another option, the problem however is the risk of ‘set and forget approach’. Another approach to providing regulators insight into industry practices is the establishment of regulatory sandboxes, which nevertheless have limitations.Footnote ¹³⁷

Financial institutions could also be required to prove a causal link between the data that they use to generate consumer scores, and likely risk. Such approach would likely reduce the use of certain categories of data, where correlations between data points would not be supported by a valid causal relationship. For example, Android phone users are reportedly safer drivers than iPhone users,Footnote ¹³⁸ but such rule would prevent insurers from taking this into account when offering a quote on car insurance (while we do not suggest they are currently doing so, in many legal systems they could). In practice, some regulators are looking at this solution. For example, while not going as far as requiring direct causal link, the New York State financial regulator requires a ‘valid explanation or rationale’ for underwriting of life insurance, where external data or external predictive models are used.Footnote ¹³⁹ However, such approach could result in encouraging financial services providers to collect more data, just to be able to prove the causal link,Footnote ¹⁴⁰ which may again further disadvantage consumers and introduce more, not less, opacity.