At the Pacific Northwest Software Quality Conference in October 2024, I gave a keynote presentation titled “What Are We Thinking in the Age of AI?“
There’s a lot to think about, and for testers, there’s a lot to do. For one, we need to understand the basis for the “AI” claim. Any kind of software can be marketed as “AI”, since it’s doing something that (presumably) a human could do, at least in theory, given time and resources. So we must consider the scope and risk associated with the AI claim, as an extra element in the general scope and risk for the product.
AI comes with special problems: algorithmic obscurity; radical fragility; claims that are even more wishful than usual; social intrusiveness; from executives and managers with FOMO, corporate defensiveness; and from the non-critical AI fanboys, social aggressiveness. The notes provide more details.
One of the most important points is that — as with all software — we cannot treat a demonstration of a happy outcome as evidence of reliability and validity. A demonstration is not a test. In the notes, there are examples of how James Bach and I have tested various LLM-based products.
There’s also a page — soon to be expanded to a blog post — on circumstances in which AI might be safely and reasonably useful.
Enjoy!
Thanks for the presentation. Interesting perspective.
GREAT POST!!
Extent on this in this way I would say. Because this feels like the right way to discuss about A.i.!
(skeptic about a.i., but non-exclusionary a.i.)
/°/°
Algorithmic obscurity
refers to the phenomenon where algorithms, especially complex or proprietary ones, become difficult for people to understand, inspect, or explain. This can happen for several reasons, including:
Technical Complexity: The underlying logic and computations are intricate, often involving machine learning models or other advanced techniques that are not easily interpretable.
Proprietary Code and Black-Box Models: Companies may protect algorithms as trade secrets, making it impossible for outsiders to scrutinize the code or understand the decision-making process.
Opaque Data Processing: Algorithms might use vast datasets with intricate processing steps, making it hard to trace how input data is transformed into output decisions or predictions.
Adaptive or Evolving Systems: Some algorithms continually learn from new data and adjust their behavior over time, which can lead to unpredictability in how they make decisions.
Lack of Transparency in Intent: Sometimes, even when an algorithm is technically understandable, its objectives or weighting of different criteria may be obscured, leaving users unclear on why certain outcomes occur.
Algorithmic obscurity can lead to issues of accountability, fairness, and trust, particularly in systems that impact people’s lives, like credit scoring, hiring, or legal judgments.
°/°/°
Radical fragility
refers to a state of extreme vulnerability or instability, where a system, entity, or structure is highly susceptible to disruption from even minor changes or shocks. Unlike ordinary fragility, radical fragility implies that the system is so delicate that it can break down quickly and unpredictably with little provocation, often in ways that are difficult to anticipate or control.
In social, economic, or technological contexts, radical fragility might describe:
Highly Interconnected Systems: Systems where a small disturbance in one area can lead to a cascade of failures across the whole, such as in global financial markets or complex supply chains.
Lack of Redundancy or Resilience: Systems without safeguards, buffers, or alternative pathways are especially prone to radical fragility because they lack mechanisms to absorb shocks.
Extreme Specialization: When a system or organization relies on very specific conditions or inputs, it becomes fragile in the face of any shift away from those narrow requirements.
Overreliance on Technology or Automation: Radical fragility can arise when systems depend heavily on technology or algorithms without enough human oversight or contingency plans.
Radical fragility highlights the importance of building resilience, flexibility, and adaptability into systems, as these qualities can mitigate the impacts of shocks and reduce the risk of sudden, catastrophic breakdowns.
This doesn’t look like your usual style, Zorro. But it does look a lot like output from a chatbot.