The World and the Machine and Responsible Machine Learning

Separating World and Machine

Requirements, Assumptions, Specifications, and Implementation

  • Requirements (REQ) describe how the system should operate, expressed entirely in terms of the concepts in the world. For example, the self-driving car should never exceed the speed limit.
  • Assumptions (ENV) express the relationship of real-world concepts to software inputs and outputs. For example, we assume that the GPS correctly represents the car’s speed, that the manually entered target address correctly represents the user’s intention, and that the car actually honors the system’s break commands and will slow down according to an expected pattern (as expected from the physics of the situation).
  • Specifications (SPEC) describe the expected behavior of the software system in terms of input and outputs. For example, we expect the system to never issue an acceleration command (output) if the speed (input from GPS) is larger than the speed limit (input from map) in the current location (input from GPS).
  • Implementation (IMPL) provide the actual behavior of the software system that is supposed to align with the specification (SPEC), usually given with some code or an executable model. A mismatch between implementation and specification may be detected say with testing and is considered a bug. For example, a buffer overflow in the implementation leads to acceleration commands (output) if the car is in a certain unusual location (input).
  • The requirements REQ are flat out wrong. For example, the car actually should be able to exceed the speed limit in emergency situations.
  • The assumptions ENV are incorrect. For example, the GPS sensor provides wrong information about the speed or the car’s breaks do not act as quickly as expected.
  • The system’s specification SPEC is wrong. For example, the specification incorrectly sets a default top speed if no map is available.
  • Any one of these parts can be internally inconsistent or inconsistent with each other. For example, the specification (SPEC) together with the assumptions (ENV) are not sufficient to guarantee the requirements (REQ) if the specified breaking logic in the software (SPEC) does not account for sufficient breaking time (ENV) to avoid going over the speed limit (REQ). Even two requirements may not be consistent, which is actually a common problem already in non-ML systems, known as feature interactions.
  • The system is implemented (IMPL) incorrectly, differing from the specified behavior (SPEC), for example, a buffer overflow bug in the implementation causes the software to issue wrong acceleration commands that violate the specification.

Lufthansa 2904 Runway Crash

Wreckage of Flight 2904 on 15 September 1993
Illustration of time elapsed between touchdown of the first main strut, the second, and engagement of brakes. CC BY-SA 3.0 Anynobody

Questioning Assumptions in Machine Learning

What now?




associate professor @ Carnegie Mellon; software engineering, configurations, open source, SE4AI, juggling

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Organization Structure and AI : What it means to you

What trends will dominate the fintech services market in 2022?

First processor designed using Artificial Intelligence.

What is Conversation Designing? — The Basics

The user asks Alexa, “Do you believe in God, Alexa?”

A.I. Isn’t as Advanced as You Think

Humans are Hooked, Machines are Learning

Could AI Be the Future of Blog Graphics?

The Power of Artificial Intelligence During a Crisis

Artificial intelligence in healthcare during COVID-19

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Christian Kästner

Christian Kästner

associate professor @ Carnegie Mellon; software engineering, configurations, open source, SE4AI, juggling

More from Medium

Flavors of model evaluation

Model evaluation in ML lifecycle

ML upgrading ML

Tackling ML Risk Management and Mitigation

The Secret Weapon in the War Against Deepfakes