Goodbye, PESQ. Meet ANNI.

When we started working on audio watermarking, one of the first challenges we encountered was to make our watermark inaudible. That turned out to be harder than it sounds, because "inaudible" isn't a yes-or-no property. It depends on the audio, the listener and how carefully they listen. And you can't settle it by listening alone. Ears get tired, two people rarely agree, and a reliable result needs many listeners to hear and rate a large number of clips.

We needed an objective way to put a number on it, one that anyone could reproduce, both to compare our models and to report the results in our papers. The only quality measure we knew back then was SNR. It took us about a day to see that it wasn't going to cut it. A watermark tucked under loud speech can have a terrible SNR and be completely inaudible, while a faint hiss in a pause has a lovely SNR and is perfectly audible.

So we googled “best audio quality metric,” (back when people still googled things) and dug through a few threads on the late, great Reddit and Stack Exchange, and ended up with PESQ. PESQ went into our first research papers, and over time (whether thanks to us or because it really was the best option around) it became something of a standard in audio watermarking.

We were never fully happy with it, though. Like any statistic, its numbers sometimes hid what was really going on. There’s one experiment we still talk about. The results looked phenomenal. PESQ above 4.5. We were ready to pop the champagne. But then, we actually listened to the clips. Most of the audio sounded fine, but in one short stretch, it croaked like a frog. PESQ didn't mind at all.

That frog taught us a lesson we never forgot. A recording that is perfect for 99 % of its length and croaks for half a second isn't 99 % good. It's a broken recording. Whatever measure we trusted would have to catch the frog.

From listening tests to ANNI

Over the years, we tried ViSQOL, NISQA, SI-SNR, STOI, and others. Again and again, what our ears heard and what the numbers said just didn't line up.

To be fair to these metrics, they were never built for our problem. PESQ was designed for telecommunications and judging how bad a phone call sounds after codecs and network trouble. On top of that, it ignores everything above 8 kHz, where younger listeners hear differences with ease. ViSQOL was built to predict an opinion score for speech over VoIP and for compressed music. None of them were designed for differentiating between near-identical audio pairs.

So we built our own web tool for listening tests. Honestly, it was a lot of fun. We ran competitions over who has the best ears in the company. We also used it to torture friends and family by making them listen to dozens of almost identical audio clips.

It helped. We learned things about our models that we wouldn’t have learned from the scores alone. But, it doesn’t scale when you’re training and comparing new watermarking models every week, and at DeepMark that's pretty much what happens. We needed a way to make those everyday comparisons without assembling a listening panel each time.

Then, some time ago, I read a paper by Qin Liu, Strahinja Nikolić and Miloš Cernak called CochlearMark: Audio Watermarking via Phase Locked Auditory Nerve Representations. Their approach uses an auditory nerve representation, called neurogram, for watermarking. I didn’t know much about auditory nerve models at the time. I spent a fair amount of time asking Claude and Astra questions about cochleas, hair cells and nerve fibres while trying to understand how this works.

In a nutshell, it works like this. Your inner ear (the cochlea) splits incoming sound into frequencies. Thousands of nerve fibres, each tuned to its own frequency, then fire signals to the brain. The louder the sound in their frequency region, the faster they fire. A neurogram is a picture of that activity. Frequency on one axis, time on the other, and colour showing how fast each fibre fires, the darker the faster. In other words, it's roughly what your brain actually receives. To simulate it, the CochlearMark authors used an open-source auditory-nerve model by Mark Saddler.

So we got an idea. If this model shows what the ear sends to the brain, why not compare two audio files there? If the original and the watermarked file produce practically the same nerve activity, nobody should be able to hear a difference. The only way to find out whether that idea would work was to build it.

One weekend and a lot of help from Claude later, we had a first version. We called it ANNI: Auditory Nerve Noticeability Index. The code and examples can be found here: github.com/deepmark/anni.

Building something like this used to take smart people months of reading, coding and calibrating. In our case, the generous pile of Claude credits that came with joining Y Combinator gave me the luxury of trying it on a weekend. If OpenAI can go after Millennium Problems, why couldn't we build one quality measure for ourselves?

How it works

This section gets a bit more technical. If you'd rather just use ANNI, feel free to skip ahead to Try it.

ANNI takes a reference recording and a modified recording, runs both through an auditory model, and compares the resulting nerve responses. It works in six steps, the same six as in the figure above.

Step 1. Both files are played at a realistic level. They aren’t normalised independently, so a change in volume still counts as a difference.

Step 2. They're lined up in time, so a tiny delay doesn't look like a big change.

Step 3. A filter models the outer and middle ear. This accounts for how sound is filtered before it reaches the cochlea.

Step 4. Saddler’s model produces a neurogram for each recording, using 128 simulated nerve fibres, each tuned to a different frequency between 80 Hz and 16 kHz.

Step 5. The two neurograms are compared. ANNI looks for two kinds of difference, slow lasting changes and fast flicker, and weighs each one against how noisy the nerve fibres naturally are.

Step 6. The differences become a score. They are added up over short stretches of 200 ms, and ANNI reports the score of the most noticeable stretch and of a median/typical one.

Comparing two neurograms

Most quality measures calculate how much two recordings differ by counting every difference the same, wherever and whenever it happens.

ANNI measures it a bit differently. Think of a whisper. In a quiet library, everyone hears it. At a rock concert, nobody does. The whisper is the same, but the background decides whether it stands out. The ear works the same way. When a nerve fibre is busy carrying loud speech, its firing is also noisier, and a small change gets lost in it. In a quiet pause, the same change stands out clearly. This is called masking, and it's why a watermark can hide under speech but not in silence.

The obvious way to compare two neurograms would be to subtract them and add up the differences, a plain Euclidean distance. A toy example shows why that isn't enough. Here is a tiny neurogram with 3 nerve fibres and 4 moments. The numbers are firing rates in spikes per second. Someone speaks during the first two moments, and then there's a pause.

Now make the same small change in two different places.

Version A. During speech, the high fibre fires 160 instead of 150 at moment t1.

Version B. During the pause, the high fibre fires 12 instead of 2 at moment t3.

Euclidean distance sees the same thing twice. One cell changed by 10 spikes per second, so both versions get a distance of 10. To it, A and B are equally different from the original.

ANNI first estimates how noisy that fibre is at that moment, and divides the squared difference by that noise. The spikes in nerve fibres behave like random events. A fibre that fires 100 times per second gives about 10 spikes in a tenth of a second, but sometimes 7 and sometimes 13. The busier the fibre, the bigger this wobble, so the noise grows with the firing rate. On top of that, ANNI adds a constant of 67 spikes per second (rounded from 66.6) for the ear's own background noise, so that even a silent fibre isn't treated as perfectly noise-free. That constant is tuned so that ANNI reproduces classic listening experiments. The noise is therefore the firing rate plus 67.

Version A. 10² ÷ (155 + 67) = 100 ÷ 222 ≈ 0.45, where 155 is the average of 150 and 160, the fibre's rate in the two versions.

Version B. 10² ÷ (7 + 67) = 100 ÷ 74 ≈ 1.35, where 7 is the average of 2 and 12.

Same change, three times as much evidence in the pause. That is masking, in numbers.

Two observers

The toy example compared how fast each fibre fires. ANNI actually looks for two kinds of change, each with its own observer. Both follow the same recipe. They find a difference, weigh it against the noise, and turn it into a piece of evidence. The difference between them is speed. The pattern observer handles slow, lasting changes, and the texture observer handles fast, flickering ones.

Imagine a row of lamps. Each lamp represents a nerve fibre. The pattern observer checks whether some lamps have been turned brighter or dimmer. The texture observer checks whether any of them started to flicker.

There are two observers because the ear itself seems to handle these two kinds of change separately. It keeps track of how much sound reaches each frequency region, and separately of how that sound fluctuates. One observer couldn't do both well, because judging steady changes needs smoothing over time, and smoothing wipes out exactly the brief clicks and bursts that the second kind of listening catches.

The pattern observer looks for frequency regions that got louder or quieter. It smooths the difference between the two neurograms over the last few hundred milliseconds before comparing. Here is what that does to two different changes on one fibre. In this toy version, each moment is simply averaged with the three before it.

The lasting change survives untouched, while the click gets spread out and flattened. After weighing by the noise, A keeps all of its evidence, while the click is left with only about a quarter of its evidence. In the real ANNI, the smoothing spans far longer than a click, so almost nothing of a click is left. That's on purpose.

The texture observer picks up exactly what the smoothing removed, the fast ups and downs of the firing. To do that, it splits each fibre's ups and downs by speed, much like the cochlea splits sound into frequencies. It runs the firing through a set of filters, each of which lets through only fluctuations of one speed. The first passes slow wobbles of about four per second, roughly the rhythm of syllables. The next passes about eight per second, and so on, doubling each time up to a fast buzz of 128 per second. What comes out of each filter shows how much the firing flickers at that speed. In each speed band, the texture observer measures how strongly the firing fluctuates, relative to how busy the fibre is, so a flicker on a busy fibre counts less. Then it compares the original and the test as a ratio, so a doubling and a halving count the same. The ear also judges such changes in proportion. A big ratio becomes a big piece of evidence. The click in row B above is exactly this kind of change.

Here is a toy example of that splitting. During speech, a fibre's firing wobbles slowly with the syllables, and a watermark adds a faint, fast flutter on top. In this example, the filters are perfect and split the firing cleanly into a slow part and a fast part.

The slow filter sees the same syllable wobble in both versions, so it finds no change. The fast filter was silent in the original and now shows the flutter. Going from nothing to something is strong evidence. Without this splitting by speed, the watermarked version would flicker only about 3 % more than the original, and the flutter would be lost behind the speech. For the technically curious, the real filters aren't perfect. They are standard Butterworth filters (six one-octave band-pass filters here, and one low-pass filter for the pattern observer's smoothing), and the code documentation walks through every step.

A second toy example shows masking, with the same clicks on one fibre, once in a pause and once during speech. In this toy version, the flicker is simply how far the firing typically strays from its own average. The real ANNI does this separately for each speed band, as in the example above.

Step 1. Measure the flicker as the standard deviation of each row.

Step 2. Make it relative to how busy the fibre is, by dividing the flicker by the average plus 67, the same noise as before.

Step 3. Compare the clicks with the original as a ratio.

In the pause, the original fibre is perfectly calm, so going from no flicker to clear flicker is an enormous ratio. (ANNI adds a tiny floor so the ratio never becomes infinite.) During speech, the fibre already fluctuates on its own and it's busy, so the same clicks only about double its relative flicker. Turned into evidence, the pause gives about fifteen times more than the speech. That's masking again. The clicks stand out in the pauses and hide under the speech, which is also what you'll hear in our example further down.

Besides clicks, the texture observer catches short bursts, crackle, and a rough or buzzy sound. All of these make the firing flicker differently without making a frequency region louder or quieter on average.

Here is how the evidence splits between the two observers for the examples you can listen to below. For each example, we took the 200 ms window where the change is most noticeable, the one that gives D_peak, and checked how much of its evidence came from each observer.

Lasting changes in tone or loudness go to the pattern observer, and short or flickering ones to the texture observer. Noise-like changes such as the hiss use both, because they raise the average firing and make it jitter at the same time. With only one observer, ANNI would miss half of these.

From evidence to a score

The two observers leave behind a pile of small pieces of evidence, one for every fibre and every moment. All pieces of evidence from both observers and all 128 fibres inside a 200 ms window are added together. The 200 ms comes from hearing research. It is roughly how long the ear keeps collecting a sound before it decides whether it heard something. The square root of the sum is the score for that window. Four changes worth 0.5 each make a score of 1, because their squares, 0.25 each, add up to 1.

A few constants set the scale so that a change people just notice, such as a 1 kHz tone made 1 dB louder, scores 1. Finally, the window slides along the recording 1.9 ms at a time, so every moment gets its own score. The 1.9 ms is a practical choice rather than a fact about hearing. It's short enough to follow the fastest flicker the texture observer looks for, and long enough to keep the computation light.

This kind of score is called d′, or "d-prime", a measure of detectability from psychology. It represents the size of a change compared to how much the signal naturally wobbles. Zero means no difference. A score of about 1 marks a just-noticeable change. Which means that in a test where a listener has to pick the changed version out of two, they'd get it right about three times out of four.

From all the window scores, ANNI reports two numbers.

D_med is the median, the typical moment. Half of the windows score lower and half score higher.

D_peak is the largest, the most noticeable moment.

The peak is how we avoid the frog problem. A recording can have a persistent low-level change, one brief ugly moment, or both, and the median alone would hide the brief one. For example, we added a single 10 ms noise burst during a pause. ANNI’s median was 0.00. Most of the recording was unchanged, so that made sense. Its peak was 15.7. This time, the frog didn't get away.

With this in place, for the first time, we have a number that ranks our watermarking models in the same order we ranked them by ear, and with clear margins.

Try it

Install it with:

pip install deepmark-anni

Then compare two audios with the same sampling rate:

import soundfile as sf
from anni import ANNI

reference, sr = sf.read("original.wav")
test, _ = sf.read("watermarked.wav")

anni = ANNI()
print(anni(reference, test, sr=sr))
# e.g. {'D_peak': 1.49, 'D_med': 0.81}

Note: ANNI simulates 128 nerve fibres, so it's slower than PESQ, at about a second per 5 s clip on a laptop. It's written in PyTorch, and in the next version we plan to turn it into a training loss, so models can learn to be inaudible directly.

Listen for yourself

We made a few changes to a speech recording and ran the results through PESQ, ViSQOL and ANNI. For PESQ and ViSQOL, higher scores mean better quality. For ANNI, higher scores mean a more noticeable difference, so the numbers run in the opposite direction and aren’t directly comparable.

Try listening to the original and the modified versions, ideally on headphones, before looking too closely at the scores.

The interesting part for us was how little some scores changed between the original and versions where we could hear a clear difference. PESQ gave the original, the high-frequency hiss and the brighter version the same score. ViSQOL’s audio mode gave the short noise burst the same score as the original.

These examples don’t cover everything an audio metric needs to handle. They do show the kinds of changes that have caused trouble in our own work.

Results beyond watermarking

We also wanted to know whether ANNI is useful outside watermarking, so we tested it on public datasets where many people rated the quality of degraded audio.

ODAQ

ODAQ, the Open Dataset of Audio Quality from Fraunhofer IIS and Netflix, collects listening tests in which expert listeners rated music, film sound and speech with all kinds of degradations, from audio-coding artefacts and filtering to stereo artefacts and speech from neural codecs. Its creator, Matteo Torcoli, recently turned it into an open benchmark that compares 38 quality metrics against the listeners' ratings, and he added ANNI to it.

Overall, ANNI's D_med ranks the recordings more like the listeners than any other metric in the benchmark. Its Spearman correlation is 0.91, ahead of SCOREQ (0.85) and the 2f-model (0.84), and its Kendall correlation is 0.93, ahead of 0.88 for the next best. That's a lead of about 7 % in Spearman and 5 % in Kendall over the runner-up. To put it in perspective, the gap between ANNI and second place is bigger than the gap between second and tenth place. ANNI's D_peak also makes the top five. In Matteo's words, "Impressive indeed to see that, overall, ANNI-Dmed beats all other metrics from SOTA in terms of rank correlations (Spearman and Kendall) - well done!"

Source: ODAQ benchmark.

The plot shows how far ANNI-Dmed stands out from the rest of the field. The metrics it beats build on decades of research, from NMR and PEAQ, which go back to the late 1980s and the 1990s, and PESQ, the long-time standard for speech quality, to recent machine-learning models such as SCOREQ.

TCD-VoIP

TCD-VoIP, a dataset of degraded phone-call speech rated by 24 listeners, is PESQ's home turf, and overall PESQ is clearly better here (0.92 against ANNI's 0.72). Looking at individual effects, ANNI matched or slightly beat PESQ on background noise, echo and competing speakers, and tied on clipping. Where it struggled was choppy audio from lost packets. ANNI expects the two recordings to stay aligned after its initial alignment step, and handling cuts and timing jumps is on our list for future versions.

What comes next

ANNI is open source, and this is version 0.1.0. It is already helping us compare watermarking models, but we have a long list of things to add and improve: the training loss, support for audio with cuts and timing drift, stereo, longer recordings, etc.

If you work on audio quality, hearing models or watermarking, we’d love to hear what you think. Try it on your own recordings. Tell us where it agrees with your ears and, especially, where it doesn’t.