Responsible AI Engineering

20 min readJan 7, 2022

This chapter introduces the last part of the book, which covers different facets of how to responsibly develop ML-enabled systems. It covers material from the lectures “Intro to Ethics + Fairness” and “Explainability & Interpretability” of our Machine Learning in Production course. For other chapters, see the table of content.

Software systems deployed in the real world can be a source of many forms of harm, small and large, obvious and subtle, easy to anticipate and surprising. Even without machine learning, we have plenty of examples of how faults in software systems have led to severe problems, such as massive radiation overdosing (Theract-25), disastrous crashes of spacecrafts (Ariane 5), losing $460 million in automatic trading (Knight Capital Group), and wrongly accusing and convicting 540 postal workers of fraud (Horizon). With the introduction of machine learning components, learned from data and without specifications and guarantees, there are even more challenges and concerns, as we will discuss in this and following chapters, including the amplification of bias, leaking and abuse of private data or creating deep fakes, exploiting cognitive weaknesses and manipulating humans, and many others.

We didn’t need machine learning for software faults to cause disasters. Ariane 5 Crash. CC BY 2.0, ESA, S. Corvaja

This raises questions for developers of ML-enabled systems about what constitutes responsible and ethical engineering practices. This involves many interrelated issues, such as ethics, fairness, justice, discrimination, safety, privacy, security, transparency, and accountability. So, to what degree should developers and data scientists feel responsible for fairness, safety, and security of their systems? And if they do, what steps can they take to build systems responsibly? This last question is what the remainder of this book explores.

Legal and Ethical Responsibilities

The exact responsibility that software engineers and data scientists have for their products is contested. Software engineers have long gotten away (arguably as one of very few professions) to claim no responsibility for their software through clauses in license agreements, such as this all-caps statement from the open source MIT license, which is mirrored in some form in most other commercial and open source software licenses:

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

In general, such a stance and these licenses have been successful in mostly keeping software companies and individual developers from being held liable for bugs and security vulnerabilities. With such licenses, it has even been difficult to adjudicate claims of negligence. Liability discussions are more common around products that include software, such as medical devices, cars, and planes, but rarely around software itself.

There is some government regulation to hold software systems accountable, including those using machine learning, though many of these stem from broader regulation. For example, as we will discuss there are various anti-discrimination laws that also apply to software. Emerging privacy laws have strong implications of what can and cannot be done with software. In certain safety-critical domains including aviation and medical devices, regulation requires certain quality assurance practices and upfront certification processes before (software-supported) products are sold. However, regulation usually only affects selected aspects of software and is often restricted to specific domains and jurisdictions.

Martin Shkreli testifying before the House Committee on Oversight and Government Reform, 2016

Even though developers may not be held legally responsible for the effects of their software, there is a good argument of ethical responsibilities. There are many actions that may be technically not illegal, but that are widely considered to be unethical. A typical example to illustrate the difference is the decision of pharma CEO Martin Shkreli to buy the license for producing the 60-year-old drug Daraprim to then raise the price from $13 to $750 per pill. All actions were technically legal and Shkreli stood by his decisions, but the 5000% price increase was largely perceived by the public as unethical and Shkreli was vilified.

While the terminology is not used consistently across fields, one can typically distinguish legality, (professional) ethics, and morality:

Legality relates to regulations codified in law and enforced through the power of the state. Professionals should know the relevant laws and are required to abide by them. Violating legal constraints can lead to lawsuits and penalties.
Ethics is a branch of moral philosophy that guides people regarding basic human conduct, typically identifying guidelines to help decide what is considered right and wrong (“What should I do?”). Ethics can guide what is considered responsible behavior. Ethics are not binding to individuals and violations are not punished, beyond possible public outcry and shaming. In severe cases, regulators may write new laws to codify that certain behavior shall be illegal in the future. The terms ethics and morality are often used interchangeably, or to refer to either group or individual views of right and wrong.
Professional ethics govern professional conduct in a discipline, such as engineering, law, or medicine. Professional ethics are described in standards of conduct adopted more or less formally by a profession, often coordinated through an organization representing the profession. Some professions like law and medicine have more clearly codified professional ethics standards. For software engineers and computer scientists, professional organizations like the ACM have developed codes of ethics. Professional organizations may define procedures and penalties for violations, but they are usually only binding within that organization and do not carry civil or criminal penalties.

So, we may individually or as a group consider that (a) raising the price of a drug or (b) shipping software without proper quality control is bad (unethical), but there is no law against it (legal). Professional ethics like the ACM code of ethics may require to “avoid harm (1.2)” and “strive to achieve high quality in both the processes and products of professional work (2.1)” and will provide guidance for ethical behavior, but professional ethics usually have little teeth in terms of enforcement. In the end, ethical behavior is often driven by individuals personally striving to be a good person or to be seen as a good person. High ethical standards may also yield long-term benefits to individuals and organizations through better reputation or motivating and retaining staff.

Even when not legally required, we hope that our readers are interested in behaving responsibly and ethically. When it comes to software development and machine learning there are lots of issues to consider with regards to ethics and responsible engineering. Many guidelines can be derived from professional codes of ethics of the engineering profession, but also of ethical considerations in science more broadly and of the field of technology ethics.

Why Responsible Engineering Matters for ML-Enabled Systems

Almost daily, we can find new examples of machine-learning projects gone wrong and causing harm. Reading about so many examples of projects going wrong can be outright depressing. In the following, we discuss only a few examples to motivate why responsible engineers should worry about issues like quality assurance, versioning, safety, security, fairness, interpretability, and transparency.

With a few lines of code…

First of all, software engineers and data scientists can have massive impacts on individuals, groups of people, and society as a whole, possibly without realizing the scope of those impacts, and often without training in moral philosophy, social science, or systems thinking.

Simple implementation decisions like (1) tweaking a loss function in a model, (2) how to collect or clean data, or (3) how to present results to users can have profound effects on how the system impacts users and the environment at large. Let’s consider two examples: A social media platform uses machine learning to rank content that users most likely will click on: To what degree should the data scientists focused on a specific prediction problem in one part of a large system be worried about whether the given loss function will indirectly foster polarization and teen depression? A restaurant review site uses machine learning to identify fraudulent reviews: To what degree should the developer of the fraud detection system be responsible for how the system may influence recommendations and may unintentionally disadvantage some minority-owned businesses?

The key point here is that everyday decisions in the design of a software system, which are represented by only a few lines of code, can have vast consequences that may not be immediately obvious from the local engineering decision. On top of that, there is always the risk of making mistakes that are not caught before deployment. A responsible engineer should likely stand back from time to time to consider the potential impact of local decisions and possible mistakes in a system, especially for systems operated at scale.

Safety

We may not be worried too much about a Terminator-style robot uprising that may end humanity (though some serious researchers see unaligned artificial intelligence as a serious extinction risk, e.g., Toby Ord in The Precipice estimates the risk at 10% over the next 100 years). Yet software systems with and without machine learning can cause (and have caused) serious harms, recent examples include autonomous vehicles crashing (Uber), malfunctioning smart home devices shutting off heating during freezing outside temperatures (Netatmo), and autonomous delivery robots blocking wheelchair users from leaving a street crossing (Starship Robots). What responsibilities do software engineers and data scientists have in building such systems? What degree of risk analysis and quality assurance is needed to act responsibly?

Manipulation and addiction

Machine learning is great at optimizing for a goal and learning from subtle feedback. The system goal set by the organization and encoded by developers does not have to align with the goals of its users. For example, users of social media systems typically do not seek to maximize their time on the site and see as many ads as possible. Systems with machine learning components, especially those that continuously learn based on telemetry data, can often exploit known shortcomings and biases in human reasoning — humans are not perfectly rational actors, are bad at statistics, and often easily influenced and insecure. Machine learning is great at figuring out how to exploit such weaknesses from data, for example when building systems that maximally seek our attention (attention engineering). For example, as already mentioned in the introduction, YouTube’s recommendation algorithm was long recommending conspiracy theory videos over proportionally, because it learned that users who start watching one such video will often go down a rabbit hole and watch many more (thus increasing screen time and ad revenue). Similarly exploiting weaknesses in human cognition, a shopping app may learn to send us reminders and discounts at just the right time and with the smallest incentives to get us to buy their products when we rationally would not. Various dark patterns and gamification can lead to behavior manipulation and addiction in many domains, including games and even stock trading. Bad actors can use the same techniques to spread misinformation, generate fakes, and try to influence public opinion and behavior. What is the responsibility of developers to anticipate and mitigate such problems? How can a system be designed to detect unanticipated side effects early?

Polarization and mental health

Social media companies have been criticized for fostering polarization and depression as side effects of algorithmically amplifying content that fosters engagement (i.e., more clicks, staying longer on the site). It turns out that extreme content and content that enrages users is more likely to keep those users engaged. Hence, machine-learning algorithms learn to recommend such content, which then may skew the users’ perceptions of news or popular opinions. Personalization of content with machine learning can further contribute to filter bubbles, where users see content with which they already agree with but not opposing views — possibly endangering balanced democratic engagement in favor of more extreme views. Also amplification of unrealistic expectations for beauty and success has been shown to relate to maximizing engagement, and has been shown to associate with mental health issues, especially prevalent among teenage girls. So how does one build systems responsibly to avoid negative personal and societal side effects?

Healthline article raising concerns about the association of social media with mental health issues.

Job loss and deskilling

As machine learning can now outperform humans in many tasks, we see increasing automation of many jobs. Previously this affected mostly repetitive jobs with low skill requirements, but the scope of automation is increasing and projected to possibly soon displace vast numbers of jobs, including travel agents, machine operators, cashiers and bank tellers, insurance agents, truck drivers, and many physicians. The positive vision is that humans will work together with machines, focus on more enjoyable and creative work, and generally work less. Many fear though that humans will have less autonomy and will do primarily low-skilled jobs like picking items from a shelf as guided by automation, with only few high-skilled people developing and maintaining the automation systems. This raises many concerns about inequality, human dignity, and the future of work. To what degree should developers engage with such questions while focused on a specific short-term development project?

Weapons and Surveillance

Machine learning powers autonomous weapon systems and has been a powerful tool for surveillance. While currently most weapon systems require human oversight, some consider autonomous weapon systems making life and death decisions (“killer robots”) inevitable because human decisions are too slow against other automated systems and because drones and robots may operate in areas without reliable or fast enough network connections. In parallel, big data and machine learning promises to scale analysis of data to a degree that was not possible by human analysts, combing through all kinds of digital traces from social media, to cell phone location data, to video footage from surveillance cameras to identify behavior patterns that others may not even realize they have or that they want to keep private. Data can further be aggregated into social credit systems with the goal of steering the behavior of entire populations. Such surveillance technologies can easily make mistakes and be abused for suppressing certain populations, not only in authoritarian regimes. To what degree is it ethical to contribute to such systems? Are they inevitable? Could one build such systems responsibly, reducing their risks and side effects?

Discrimination

While it was always possible to encode discriminatory rules in software code (intentional or not) or to underserve people from certain demographics by not recognizing or ignoring their requirements, with the increasing use of machine learning, concerns for algorithmic discrimination has been gaining attention. As machine learning learns decision rules from data, bias in the data (for example, from historic discriminatory practices) is learned as decision rules, such as in cases where automated resume screening algorithms would reject most applications from female applicants. Hence, use of machine learning can encode and even amplify existing bias, all while presenting decisions as objectively made by a neutral machine instead of made by a biased human. So how proactive should responsible developers be in screening their system for bias?

Facets of Responsible ML Engineering

There is no agreed rubric of responsible or ethical ML engineering entails and different organizations, researchers, and practitioners make different lists. For example, Microsoft lists principles as (1) fairness, (2) reliability and safety, (3) privacy and security, (4) inclusiveness, (5) transparency, and (6) accountability. Google lists objectives as (1) be socially beneficial, (2) avoid unfair bias, (3) safety, (4) accountability, (5) privacy, (6) scientific excellence, (7) responsible deployment. The nonprofit AlgorithmWatch catalogued 167 ethics guidelines at the time of this writing, summarizing that all of them include similar principles on transparency, equality/non-discrimination, accountability, and safety, while some additionally demand societal benefits and protecting human rights.

Microsoft’s framing of their AI principles for responsible engineering.

In the remainder of the book, we will selectively cover responsible ML engineering topics. This will include two pieces of technical infrastructure that are important building blocks in many responsible engineering activities and four areas of concern that crosscut the entire development lifecycle.

Additional technical infrastructure for responsible engineering:

Versioning, provenance, reproducibility: Being able to reproduce models and predictions, as well as to track which specific model made a certain prediction and how that model was trained, can be essential for trusting a system and is an important building block in responsible engineering.
Interpretability and explainability: Considering to what degree developers and users can understand internals of a model or derive explanations about the model or its prediction are important tools for responsible engineers when designing and auditing trustworthy systems.

Covered areas of concern:

Fairness: Bias can easily sneak into machine-learned models making important decisions in software systems. Responsible engineers should understand the possible harms of discrimination, the possible sources of biases, the different notions of fairness, and develop a plan to consider fairness throughout the entire development process, both at the model and at the system level.
Safety: Even when they may not directly kill humans, many software systems with machine learning components pose safety concerns. Responsible engineers should take safety seriously and take steps throughout the entire lifecycle, from requirements engineering and risk analysis to system design and quality assurance.
Security and privacy: Systems with machine-learning components can be attacked in multiple novel ways and their heavy reliance on data raises many privacy concerns. Responsible engineers should evaluate their systems for possible risks and take mitigating steps.
Transparency and accountability: For users to trust a software system with machine-learned models, they should be aware of the model and have some insights into how the model works, as well as having mechanisms to hold people accountable for the system.

Regulation is Coming

Ethical issues in software systems with machine-learning components have received extensive media and research attention in recent years, triggered by cases of discrimination and high-profile accidents. At the same time, technical capabilities are evolving quickly and may outpace regulation. There is an ongoing debate about the role of AI ethics and to what degree responsible practices should be encoded in laws and regulation. In this context, regulation refers to rules imposed by governments, whether directly through enacting laws or through empowering an agency to set rules; regulations are usually enforceable either by imposing penalties for violation or opening a path for legal action.

Regulation and Self-Regulation

For many years now, there have been calls for government regulation specifically targeted at the use of machine learning, with very little actual regulation emerging. Of course, existing non-ML-specific regulations still apply, such as anti-discrimination statutes, privacy rules, pre-market approval of medical devices, and safety standards for software in cars and planes. However, those often do not match the changed engineering practices when using machine learning, especially as some assumptions break with the lack of specifications and the increased importance of data.

There have been many working groups and whitepapers from various government bodies that discuss AI ethics, but little has resulted in concrete regulation so far. For example, in 2019, the president of the United States issued an executive order “Accelerating America’s Leadership in Artificial Intelligence,” which in tone suggested that innovation is more important than regulation. A subsequent 2020 White House white paper drafted guidance for future regulation for private sector AI, outlining many concerns, such as public trust in AI, public participation, risk management, and safety, but generally favored non-regulatory approaches. The aforementioned 2019 Ethics Guidelines for Trustworthy AI in Europe and the 2022 Blueprint for an AI Bill of Rights in the US outline principles and goals but are equally nonbinding, while actual regulation is debated.

At the time of finalizing this book in late 2023 the closest to actual serious regulation is the European Union’s Artificial Intelligence Act. The EU AI Act was first proposed by the European Commission in 2021 and was approved by the European Parliament in 2023, is expected to become law after more discussions and changes in 2024, and would come into effect about two years after that. The EU AI Act entirely outlaws some applications of machine learning considered to have unacceptable risks, such as social scoring, cognitive manipulation, and real-time biometric surveillance. In addition, it defines foundation models and machine-learning use in eight application areas, including education, hiring, and law enforcement, as high-risk. For those high-risk applications and models, the AI Act requires companies to register the system in a public database and imposes requirements for (ongoing) risk assessment, data governance, monitoring and incident reporting, documentation, transparency to users, human oversight, and assurances for robustness, security, and accuracy. For applications outside these high-risk domains, obligations are much lower and relate primarily to transparency. All other systems outside these domains are considered limited-risk or minimal-risk and have at most some transparency obligations to disclose the use of a model. The AI Act provides an overall framework, but the specific implementation in practice remains to be determined — for example, what specific practices are needed and what forms of evidence are needed to demonstrate compliance.

Another significant recent step that may lead to some regulation is the White House’s October 2023 Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. This executive order directs various agencies in the US to develop regulations or standards for various aspects of AI systems, including developing quality assurance standards and standards for marking AI-generated content. For very large models, it proposes reporting requirements where developers need to inform the government about the model and quality assurance steps taken. In addition, several agencies are instructed to develop guidelines for specific chemical, biological, radiological, nuclear, and cybersecurity risks. In general, the executive order is expansive and covers many ethics and responsible-engineering concerns. But rather than setting explicit enforceable rules, the executive order instructs other agencies to collect information, form committees, develop guidance, issue reports, or invest in research.

In the meantime, many big tech companies have publicly adopted policies and guidelines around “AI ethics” or “responsible AI” on their websites, and some have established AI ethics councils, fund internal research groups on the topic, or support academic research initiatives. They argue that the industry can self-regulate, by identifying ethical and responsible practices and adopting them. Companies work with each other and with nonprofit and government organizations to develop guidelines and recommendations. Company representatives and many think pieces argue that companies have more expertise to do the right thing and are more agile than bureaucrats defining stifling and unrealistic rules. Companies often set ambitious goals and principles around AI ethics, develop training, and may adopt some practices as part of their processes.

In contrast to government regulation, there is no enforcement mechanism for self-regulation outside the organization. The organization can decide what ethics goals and principles to pursue and has discretion of how to implement them.

Ethics Bashing and Ethics Washing

While many developers will be truly interested in being more responsible in their development practices, the current discussions on ethics and safety in machine learning, especially when framed through self-regulation, have their critics.

Some perceive the public discussions of AI ethics, the setting of goals and declaring principles and self-enforced policies, and the funding of AI ethics research of big tech companies as ethics washing — an attempt to provide an acceptable facade to justify de-regulation and self-regulation in the market. The argument of these critics is that companies instrumentalize the language of ethics, but eventually pay little attention to actually effective practices, especially when they do not align with business goals. They may point to long-term existential risks, such as a widely shared open letter calling for a six-month pause in the development of more powerful large language models in early 2023, while ignoring immediate real-world harms caused by existing systems. Such a self-regulation strategy might be a distractions or might primarily address symptoms rather than causes of problems. It has little teeth for enforcing actual change. Journalists and researchers have written many articles about how companies are trying to take over the narrative on AI ethics to avoid regulation.

Some companies actually push for some regulation, but here critics are concerned about regulatory capture, the idea that companies might shape regulation such that it aligns with the company’s practices but at the same time raises the cost of business for all (by requiring costly compliance to regulation) and thus inhibiting competition from small and new organizations. In addition, some organizations seem to use public statements about AI ethics and safety as a mechanisms to advertise their own products as so powerful that we should we worried whether they are too powerful (“criti-hype”).

At the same time, in what is termed ethics bashing, some critics can go as far as dismissing the entire ethics discussion, because they see it only as a marketing tool or, worse, as a way to cover up unethical behavior. These critics consider ethics as an intellectual “ivory tower” activity with little practical contributions to real system building. Hence, some may dismiss ethical discussions entirely.

It is important to maintain a realistic view of ethical and responsible engineering practices. There are deep and challenging questions, such as what notion of fairness should be considered for any given system or who should be held accountable for harm done by a system. There are lots of ways in which developers can significantly reduce the risk from systems by following responsible engineering practices, such as hazard analysis to identify system risks, threat modeling for security analysis, providing explanations to audit models, and requiring human supervision. Even if not perfect, these can make significant contributions to improve safety, security, and fairness and to give humans more agency and dignity.

Do Not Wait for Regulation

It is widely expected that there will be more regulation around AI ethics and responsible engineering in the future. The US has a tendency to adopt regulation after particularly bad events, whereas Europe tends to be more proactive with the AI Act. We may see more targeted regulation for specific areas such as autonomous vehicles, bio-medical research, or government-sector systems. Some regulatory bodies may clarify how they intend to enforce existing regulation — for example in April 2021 the US’s Federal Trade Commission publicly posted that they interpret the Section 5 of the FTC Act enacted in 1914, which prohibits unfair and deceptive practices, to prohibit the sale or use of racially biased algorithms as well. In addition, industry groups might develop their own standards and, over time, not using them may be considered negligence.

However, we argue that responsible engineers should not wait for regulation but get informed about possible problems and responsible engineering practices to avoid or mitigate such problems before they lead to harm, regulation or not.

Summary

Software with and without machine learning has the potential to cause significant harm when deployed as part of a system. Machine learning has the potential to amplify many concerns including safety, manipulation, polarization, job loss, weapons, and discrimination. With a few lines of code, developers can have outsized power to affect individuals and societies, and they may not even realize it. While current regulation is sparse and software engineers have traditionally been often successful in avoiding liability for their code, there are plenty of reasons to strive to behave ethically and to develop software responsibly. Ethical AI and what exactly responsible development entails is broadly discussed and often includes concerns about fairness, safety, security, and transparency, which we will explore in subsequent chapters.