The SoundStage! Network began publishing in 1995. By 1999, we had started expanding, adding more sites to the network, and we inked a deal with Canada’s National Research Council (NRC) to measure speakers in their anechoic chamber and publish the results. The NRC’s anechoic chamber and associated lab facilities were world-famous then—and still are today—because that’s where Dr. Floyd Toole initially conducted his groundbreaking loudspeaker research (in the 1970s and into the 1980s) that correlated listening impressions with objective measurements. Toole’s research is considered by many to be the most comprehensive body of work ever produced on the subject and was so influential that it helped to build and shape the Canadian loudspeaker industry.
Measuring speakers in the chamber was a watershed moment for us because none of our online competitors were publishing speaker measurements at that time. Some print magazines were, but none of them had access to a world-class facility like the NRC’s. A few years after launching our speaker measurement reports, we began working with famed amplifier designer and reviewer Bascom H. King, of BHK Labs, to measure amps and preamps.
In the National Research Council’s chamber
Publishing measurements on the equipment we reviewed was a natural step for us because it’s what serious hi-fi publications did at the time. Print magazines such as Audio, Stereophile, and Hi-Fi News & Record Review were all founded years before us and included measurements in most of their product reviews. And for good reason—a proper set of objective measurements provides more information to readers, particularly technically savvy ones, because for decades we’ve had a good understanding of the correlation between measurements and listening impressions. Measurements also do a great job of telling you if a product is competently designed or not. They also tell you if a manufacturer’s performance claims are honest or exaggerated—something that’s difficult or impossible to determine purely by listening. There is no good reason not to include them alongside reviews—provided you’re able to do them.
Unfortunately, these days you’ll find a surprising number of audio reviewers who will tell you that measurements are meaningless. Some of them run their own websites or write for print or online magazines, while others have YouTube channels. But they’re doing more than disparaging good measurement practices—they also seem to be trying to undo much of the painstaking work that designers, engineers, and scientists have done to develop these measurements and correlate them with what we hear.
I won’t call out these reviewers by name, but if you Google “why speaker measurements don’t matter,” “do audio measurements matter,” “why I hate audio measurements,” or something similar, you’ll figure out who many of them are.
But why are they damning measurements? And do they really believe what they’re saying and writing? Is there something else going on? Over the years, I’ve watched their videos and read their articles and formed my own opinion about the reasons why they’re going on like they do—reasons that might surprise you because they have little or nothing to do with the validity of measurements. Instead, what most of them are doing is misleading—and self-serving. I think there are four main reasons why they’re taking this route.
1) They use anti-measurement rants to get attention
It doesn’t matter whether you’re a YouTuber or producing content for a website or print magazine, getting eyeballs on your material means making money. And since an unhinged rant attracts attention like a car crash, some reviewers apparently use this strategy to get people looking at what they’re doing, whether they believe what they’re saying or not. In Internet jargon, these anti-measurement rants, written or recorded on video, are nothing more than clickbait.
Why are people attracted to these rants? They’re often entertaining, but they also make a seductive pitch. Instead of having to learn a little about what measurements mean, the viewer or reader can simply “trust their ears,” like the anti-measurement reviewer is supposedly doing—although, since it is doubtful they’ve heard whatever piece of equipment the reviewer’s discussing, they’ll probably have to trust the reviewer’s ears instead. And they can also feel like members of an aesthetically sensitive and savvy elite, superior to mere scientists and engineers with a “tin ear” who “don’t know how to listen to music.” (Those quotes, by the way, are from the late Harry Pearson, founder of The Absolute Sound, a publication that does not use measurements.)
2) They don’t understand what the measurements mean
I’ve yet to see an anti-measurement reviewer demonstrate even a marginal knowledge of how measurements are done and what they mean. Instead, every anti-measurement reviewer I’ve encountered has come at it from the side of ignorance, which they’re not shy to share publicly.
For example, a magazine writer who regularly disparages measurements posted on Facebook last year, criticizing the way digital-to-analog converters (DACs) are measured—or at least how he thinks they are measured. He’d been listening to a very old DAC that he felt wasn’t adept at reproducing low-level signals and then proceeded to declare online that no one who measures DACs seems concerned with this issue. I almost felt embarrassed for him when a chorus of responses pointed out that objective tests for noise, distortion, and low-level linearity have been standard practice in DAC measurements for decades, and provide a better indication of the response to low-level signals than any human ear. He didn’t seem to have a clue—yet he’d taken a stance that measurements aren’t informative. And because he’s a hi-fi reviewer, some people actually think he’s an expert. Nothing could be further from the truth.
Sample Audio Precision test for DAC linearity at all signal levels
Around the same time, another reviewer posted a frequency-response chart of a loudspeaker’s output to his Facebook news feed—a chart I imagine he’d lifted from a website that does measurements, because he certainly didn’t have the wherewithal to produce the chart himself. Alongside the image he typed a caption that insinuated to his followers that the chart couldn’t tell you anything about how a speaker might sound. By doing that, he was unknowingly proclaiming to the world that he has no understanding of speaker frequency-response measurements.
While one or even a few charts like the one he posted won’t tell you everything about the way a loudspeaker will sound in your home, they will each tell you something if you know how to interpret the data—and with a whole bunch more of them taken at different points around the speaker you can combine all that data to get a good idea of how a speaker will sound. That post showed me that he had no idea what he was looking at, and ultimately he merely provided yet another example of a know-nothing reviewer masquerading as an expert.
I could go on and on about the ludicrous things I’ve read or heard from the anti-measurement reviewing crowd, but it’s easier to sum it up with this: when they’re railing against measurements, they’re railing against something they don’t understand and haven’t taken the trouble to learn about. And if someone looks to them for advice, simply because they’re reviewers, it becomes an example of the blind leading the blind.
3) They aren’t willing to invest the time and money to produce their own measurements
If you’re going to produce measurements, you either have to have the equipment and skill to do it yourself, or you have to pay an outside party to conduct the tests for you. Both cost money, and that’s something we at the SoundStage! Network know all about.
When I mentioned the deal we made in 1999 with Canada’s NRC to measure loudspeakers, I left out one important fact—we had to pay for those measurements, because we were effectively renting their facility and employing their staff to do the work for us. It was the same thing when we had BHK Labs measure amps and preamps for us—along with the sheaf of measurements for a product was a bill for payment. Several years ago, we brought on Brent Butterworth to write headphone-related reviews and provide associated measurements—and of course we pay him for his work, too. All of these measurements of loudspeakers, amplifiers, preamplifiers, and headphones have cost us many tens of thousands of dollars over the years, but we’ve always felt it was worth it because of the benefits for our readers.
We still pay the NRC and Brent, but nowadays we measure amps, preamps, and other electronics in-house—last year, we purchased our own state-of-the-art Audio Precision APx555 audio analyzer to do the job. But the APx555 plus the necessary load resistors and other things required to do proper measurements cost us about $35,000 (in USD). That’s a substantial cash outlay just for the gear. And we also needed to employ someone with the skills to use it, so we now have Diego Estan, our main measurements guru, who gets paid for the measurements he produces on the APx555. Ka-ching!
Diego Estan with the Audio Precision APx555
Would these YouTubers and audio reviewers I’ve been talking about be willing to dish out the cash like we do to produce proper measurements? Not as far as I can see. Would they invest the hundreds of hours it takes to become proficient at any one of the dozens of audio measurements we report? No.
4) They’re entrenched in their beliefs
The first three reasons why I think some reviewers become measurement deniers suggest that they have cynical and unsavory motivations—and I suspect, for many, this is true. But I realize that some of them genuinely and sincerely believe what they say is right, and simply can’t process facts that conflict with their beliefs. This is a common human flaw—one we can see all over the Internet on websites and YouTube videos dealing with politics, religion, and wellness, to cite just a few examples.
Many reviewers are audio enthusiasts who got interested in the field by reading the popular writings of, for example, Harry Pearson and the writers he influenced. They adopted those writings as their belief system—so when they later encounter the scientific work of Dr. Toole and others, which proves the weakness of a purely subjective, superficial approach to audio product evaluation, their brains simply cannot accept the facts, no matter how clear they may be. Or just as likely, they haven’t even read Toole’s work, and won’t because they know it argues against their beliefs.
The APx555 comes with an 800-page manual!
There’s nothing reprehensible about such reviewers—they’re only human, after all—but their advice on selecting audio gear is of limited use. Perhaps their comments about the styling or ergonomics of a product are worthy of attention. But they don’t know the science of audio well enough to understand the weaknesses and vicissitudes of their ears and brains, and how these might affect their perception and judgment of a product. This is something they’d learn very quickly if they started measuring the products they review, which we’ve been doing for more than 20 years and others have been doing for much longer.
The anti-measurement ruse
When I see someone decrying well-established measurement protocols and the decades of good work people have done correlating what they measure with what we hear, I don’t get upset about it—at least not anymore. Instead, I try to determine what their real reasons might be—attention-seeking, ignorance, lack of skill, money, and beliefs are all factors. Therefore, if you’re watching or reading one of these measurement deniers, pay less attention to what they’re saying and much more attention to why they’re saying it. Their true motivation might surprise you.
. . . Doug Schneider
das@soundstagenetwork.com
