Mohammad Jazim

QA Isn’t Breaking. Our Definition of QA Is.

3–5 minutes

For years, Quality Assurance has been treated as a checkpoint.

A final gate before release.

A function responsible for validating what engineering has already built.

Test cases.
Automation scripts.
Regression cycles.

All designed around one foundational assumption:

Systems are stable enough to validate.

That assumption no longer holds.

The stakes are especially clear in healthcare products, where testing must address clinical risk alongside software behavior. I explore that distinction in why a healthcare MVP must also be clinically safe.


The Ground Has Shifted

Traditional software behaved predictably.

Given the same inputs, it produced the same outputs.
Flows were deterministic.
Releases were structured.

QA fit neatly into this world.

You could define expected behavior.
You could test against it.
You could sign off with confidence.

But AI systems don’t behave like that.

They don’t “settle.”
They evolve.

Their behavior changes based on:

  • Data
  • Prompts
  • Context
  • User interaction patterns

Which creates a fundamental tension:

You can’t validate a system that keeps changing.

So what exactly are we testing?

Not just outputs.
Not just flows.

We are testing behavior under uncertainty.


The Problem Isn’t QA Execution. It’s the QA Model.

When engineering velocity increases, QA is usually the first thing that breaks.

Not because QA is weak—but because it was designed for a different paradigm.

A world of:

  • Predictable releases
  • Stable interfaces
  • Deterministic logic

In that world, scaling QA meant:

  • Writing more test cases
  • Expanding automation
  • Increasing regression coverage

In AI systems, that approach hits a wall.

Because:

Running the same tests on a system that behaves differently each time doesn’t increase confidence—it creates false assurance.

The bottleneck isn’t effort.

It’s the model of QA itself.


A Shift in Perspective: QA as an Environment

A more useful way to think about QA in AI systems is not as a function—but as an environment.

Not something that sits at the end of the pipeline.

But something that operates in parallel to development and staging.

An independent layer designed not to validate the system—

but to challenge it.


From Test Cases to Agents

In this new model, QA doesn’t primarily write scripts.

It deploys systems.

Imagine a QA layer powered by agents that continuously probe the product:

  • One agent breaks interfaces
  • One overloads APIs
  • One generates pathological data
  • One simulates chaotic user behavior
  • One probes for security vulnerabilities

These agents don’t wait for releases.

They run continuously.

Across builds.
Across environments.
Across scenarios no human would systematically cover.

This isn’t testing in the traditional sense.

It’s systematic pressure.


The Real Lever: Control of the Environment

In deterministic systems, coverage was the goal.

In AI systems, exposure is what matters.

Because:

  • Who controls test data → controls coverage
  • Who controls prompts → controls behavior
  • Who controls environments → controls outcomes

This shifts QA closer to:

  • Data pipelines
  • Prompt engineering
  • System design
  • Runtime behavior

QA is no longer just observing the system.

It is actively shaping it.


From Verification to Orchestration

This leads to a fundamental role shift.

QA moves from:

  • Verifying correctness
    → to orchestrating behavior

From:

  • Writing scripts
    → to designing systems

From:

  • Executing tests
    → to controlling environments

The goal is no longer to confirm that the system works as expected.

The goal is to understand:

How the system behaves under stress, ambiguity, and scale.


Continuous Challenge, Not Periodic Validation

Traditional QA is episodic.

It happens:

  • Before release
  • After major changes
  • During defined cycles

AI-native QA is continuous.

It operates:

  • During development
  • During runtime
  • Across evolving conditions

It doesn’t ask:

“Did this pass?”

It asks:

  • Where does this break?
  • How does this degrade?
  • What patterns emerge under pressure?

QA as Infrastructure

This is the most important shift.

QA doesn’t disappear.

It moves down the stack.

Becoming infrastructure.

A persistent layer that:

  • Generates adversarial conditions
  • Simulates real-world unpredictability
  • Continuously explores edge cases
  • Surfaces system weaknesses early

And crucially—

keeps running even when no one is watching.


The Compounding Effect

Traditional QA scales linearly:

More features → more tests → more effort

AI-native QA compounds:

More agents → more scenarios → more emergent insights

The system becomes better at testing itself over time.

Not because humans wrote every case—

But because the environment continuously evolves.


The Bottom Line

QA isn’t breaking.

It’s outgrowing the model we’ve constrained it to.

In a world of AI-driven systems:

  • Stability is no longer guaranteed
  • Behavior is no longer fixed
  • Validation is no longer sufficient

The future isn’t better test cases.

It’s systems that continuously test systems.

And in that future—

QA isn’t a phase.
It’s infrastructure.

Share this article

Explore more of my work

Read original research and practical product notes, or explore the products and systems I have worked on.

Comments

Leave a Reply