DeepMark vs PerTh: Which audio watermark actually holds up

We tested DeepMark’s zero-bit model and Resemble AI's Perth across 40 audio attacks × 250 files = 10,000 attacked-file cases per model. DeepMark’s zero-bit model maintained 100% true-positive detection on 36 of 40 attacks. Perth did so on 31 of 40.

In false-positive tests where a positive actually represents a mistake, DeepMark’s zero-bit model stayed at 0% FP across every attack. Perth produced non-zero false positives under three attacks: 6.53% with high-pass filtering, 2.86% with STFT quantization, and 0.82% with speech tokenization.

Then we ran a much more targeted stress test: a 200 Hz-wide bandstop filter swept across 30+ frequency bands. DeepMark’s zero-bit model stayed at 100% TP / 0% FP across the entire sweep. PerTh had bands where real watermark detection fell all the way to 0%, and other bands where clean audio was falsely detected as watermarked 100% of the time. What makes this especially notable is that the attack remained highly effective while preserving very high speech quality, meaning the watermark could be disrupted without destroying the underlying audio. That's what makes the comparison useful.

Is DeepMark a direct replacement for PerTh?

For production zero-bit audio watermarking, in many cases yes.

Both systems are solving essentially the same problem:

Embed an inaudible watermark into audio, then later answer: "Is my watermark still there?"

On untouched audio, both work perfectly.

That's the easy part.

The interesting question is what happens after the audio leaves your server.

Real audio gets:

• compressed;
• resampled;
• filtered;
• denoised;
• sent through codecs;
• pitch shifted;
• deliberately modified by somebody trying to remove the watermark.
  

That's where two detectors that look similar on clean audio can behave very differently.

Across our main benchmark. fully documented in our official peer-reviewed paper, including the methodology, attack definitions, evaluation setup, and complete results available here on IEEE Xplore

Regarding the true-positive (TP) detection, PerTh works very well on many transformations. DeepMark’s zero-bit model maintains successful detection across more of them.

Accuracy is only one side of the story

A single accuracy score is useful, but it doesn't tell you how a watermark detector is succeeding or failing.

For watermark detection, two numbers give a much clearer picture:

True-positive rate: does the watermark survive?

If we watermark audio and then attack it, the detector should still say:

"Yes, the watermark is there."

That's the true-positive rate, or TP rate.

100% TP = every tested watermarked sample was still detected.
75% TP = one in four was missed.
50% TP = half of the tested watermarked samples are missed.
0% TP = the watermark is effectively gone for that transformation.

Simple enough.

False-positive rate: does the detector start seeing ghosts?

Now take audio that should not trigger the detector.

If the detector says it contains the watermark anyway, that's a false positive.

For a production system, this can be worse than missing an occasional watermark.

Imagine scanning 1 million files.

A 1% false-positive rate can mean roughly 10,000 false alarms.

At 6.5%, you're potentially dealing with tens of thousands of bad detections.

So when a detector says:

"Yes, this is mine."

you really want that answer to mean something.

Which detector cries wolf?

DeepMark’s zero-bit model: zero FP failures

Across the attacks in this benchmark:

DeepMark’s zero-bit model remained at 0% FP.

No false-positive attack condition appeared.

PerTh: three attacks produce false positives

PerTh has three:

The one that stands out is the high-pass filter.

PerTh still achieves 100% TP, so it continues detecting genuine watermarks.

But its false-positive rate rises to 6.53%.

That's an uncomfortable combination.

The detector hasn't stopped working.

It has stopped being trustworthy.

If a positive result can represent either a real watermark or a clean sample accidentally pushed across the detection threshold, every positive becomes harder to automate.

And that's exactly what you don't want from a provenance detector.

The biggest gap in the 10,000-case benchmark: bandstop filtering

The standard bandstop attack produces one of the largest differences between the models.

In this test, we removed a narrow frequency range between 1000 and 1200 Hz from the audio and then checked whether each model could still detect the watermark.

That's 100% vs 0.82%.

Not 98% vs 96%.

Not something you'd need statistical gymnastics to make interesting.

After removing just the 1000–1200 Hz frequency band, PerTh detects fewer than 1% of the watermarked samples under this attack condition.

DeepMark’s zero-bit model detects all of them.

In plain English:

Removing a narrow 200 Hz slice of the spectrum almost completely removes PerTh's usable detection signal. DeepMark’s zero-bit model doesn't move.

And importantly, this is not an attack that simply destroys the audio to destroy the watermark. Speech quality remains very high after the filtering, making the result much more relevant to a realistic watermark-removal scenario.

But that raised an even more interesting question:

Was 1000–1200 Hz just an unlucky range for PerTh, or are there other frequencies where the same thing happens?

So we tested that separately across 30+ frequency bands.

What happens when we sweep the bandstop filter across both models?

This is the experiment that exposes the difference much more clearly.

Instead of applying one generic bandstop configuration, we swept a narrow notch through the wider spectrum.

And the results aren't subtle.

This is a separate targeted stress test from the main 10,000-case benchmark.

And it tells us something the aggregate number can't.

DeepMark’s zero-bit model: 33 notches, no change

The DeepMark result is almost boring:

TP = 100%. FP = 0%. Every frequency.

Remove 1000–1200 Hz?

Still detects everything.

Remove 1600–1800 Hz?

Still detects everything.

Go all the way through the sweep?

Still 100% TP / 0% FP.

That's exactly what you want to see from a watermark designed to survive spectral manipulation.

No individual 200 Hz slice appears to contain enough critical information that removing it destroys detection.

And equally, or even more important:

None of those spectral cuts makes DeepMark’s zero-bit model hallucinate a watermark in clean audio.

PerTh has two different problems, depending on which frequency you remove

This is where the sweep gets interesting.

PerTh doesn't fail in one consistent way.

It has two opposite failure modes.

Problem #1: some frequency cuts erase the watermark

The most dramatic result is the 1600–1800 Hz notch:

PerTh TP: 0%.

The watermark effectively disappears from the detector.

And look at the quality numbers for the resulting audio:

PESQ: 4.290
STOI: 0.9939

In other words, the audio remains extremely high quality.

You don't need to destroy the recording to destroy the detection result.


Problem #2: other frequency cuts make Perth see watermarks that aren't there

This is arguably even stranger.

At three notch positions, Perth's false-positive rate goes to:

100%.

Think about what that means operationally.

A relatively ordinary spectral change doesn't just make Perth miss a watermark.

Depending on the frequency, it can make the detector do the opposite:

insist that clean audio is watermarked.


The plain-English bandstop result

If you're not an audio engineer, here's the whole experiment in one paragraph:

We repeatedly removed one narrow slice of frequencies from the audio and asked both detectors what they saw. DeepMark’s zero-bit model gave the same correct answer across all 33 slices. PerTh had frequency slices where the real watermark disappeared completely, and other slices where clean audio was falsely classified as watermarked every single time.

That's a much bigger deal than a few percentage points of benchmark accuracy.

Because a simple filter is not an exotic attack.

It's basic audio processing.

The real finding isn't the filter; it's what the filter reveals,

The bandstop sweep is the symptom. The design flaw is the diagnosis.

If a 200 Hz notch erases a watermark, the watermark was living in that 200 Hz.

That means the embedding is content-independent: it writes into roughly the same spectral neighborhood no matter what the audio actually is.

A fixed location is an address. And once an address exists:

• an attacker finds it once, and it works on every file you have ever watermarked;
• ordinary processing can hit it by accident, with no attacker involved;
• the detector's own reference point can be knocked loose, which is why cutting the wrong band makes clean audio read as watermarked.

Here is why that matters commercially:

It makes the rest of the robustness table unsustainable.

A model with one spectral point of failure can still post 100% TP on 31 of 40 attacks, as long as none of those attacks happens to touch its band. That isn't robustness. It's an untested coincidence.

Averages hide single points of failure. This sweep was built to expose them.

Content alignment is the fix, and it isn't a tuning fix.

A content-aligned watermark is placed as a function of the signal itself. It moves with the audio, so there is no fixed band to find, and none to remove. Every file's watermark sits somewhere different, because every file is different.

You don't get there with more attack augmentation. It's a different design, and it's the harder half of this problem.

That's the level DeepMark is built at. It's also why the sweep is boring for us.


DeepMark’s zero-bit watermark is not "unremovable."

Neither is PerTh's.

Any company claiming a watermark literally cannot be removed is setting the wrong expectation.

The useful question is not:
"Can somebody remove it?"

Given sufficiently destructive reconstruction, the answer is usually yes.

The useful question is:

"How much manipulation does it take before the detector stops working?"

Does DeepMark’s zero-bit model sound better?

Slightly.

We measured widely used PESQ and STOI values:

Higher is better.

DeepMark’s zero-bit model wins both measurements.

But look at STOI:

0.99484 vs 0.99407.

That is a very small difference.

The realistic conclusion is:

Both models preserve speech extremely well. DeepMark’s zero-bit model has the numerical quality advantage, but audio transparency is not where the biggest difference lies.

The major differences appear under attack.

DeepMark’s zero-bit model is also much smaller

Beyond robustness, the models differ substantially in footprint.

That's roughly:

• 8× smaller on disk;
• 9× fewer parameters;
• and more than 2× faster embedding at every measured percentile.
  

For one audio clip, the latency difference probably doesn't matter.

At large scale, it can.

Model size can also matter for deployment environments where memory, startup cost, or distribution size matters.

Is PerTh still worth using?

Absolutely.

PerTh has one advantage that matters a lot:
It's open-source and free.

DeepMark is commercial/proprietary.

That makes PerTh very attractive for:

• academic research;
• prototypes;
• internal experiments;
• hobby projects;
• zero-budget applications;
• teams that require an open-source
• implementation;
• use cases where occasional false alarms aren't expensive.

If you're trying audio watermarking for the first time and your budget is zero, PerTh makes a lot of sense.

If you're making high-stakes automated decisions based on the detector's answer, the priorities change.

A free detector becomes expensive very quickly if humans have to review thousands of false alarms.

What the benchmark really tells us

A good benchmark shouldn't prove that your model is magical.

It should expose where it breaks.

Across 40 attacks × 250 files = 10,000 attacked-file cases per model, most transformations don't dramatically separate PerTh from DeepMark’s zero-bit model.

Both handle a lot of them perfectly.

The interesting part is the exceptions.

And some exceptions are huge.

The goal isn't to build a watermark that wins a benchmark table.

The goal is to build one that can still be trusted after the audio leaves the lab.