An experiment for comprehensible headphone tests

My headphone tests have always been subjective and the results cannot be transferred one-to-one to other people. A pair of headphones that is comfortable for me may be unsuitable for another head shape. It is even more difficult with sound: it is impossible to describe its characteristics in such a way that another person understands exactly what I mean. If I write that the headphones sound bass-heavy, that's not precise enough. And besides, you'll just have to take my word for it, you can't check it yourself.

I have been dissatisfied with this situation for a long time, but have not yet found a solution. We are not equipped for laboratory measurements, nor do we have the necessary specialised knowledge. Moreover, laboratory measurements are often out of touch with reality: you measure things that are irrelevant in everyday life. It is better to have subjective findings based on real experiences.

Is the way out a binaural microphone?

A few weeks ago I came across the 3Dio FS II Pro. This is a professional binaural microphone. Headphones can be mounted on the device. There are microphones in both ears that record the sound as a human ear would hear it - at least approximately.

Of course, you will not hear the recorded sound as it originally comes out of the headphones. This is because the playback is shaped by your own headphones or speakers. But it is still possible to compare different models. The following video shows that this works: No matter which headphones you use, you can hear that the Apple Airpods Max sound different from the Sony XM5.

First test the principle

The 3Dio FS II Pro costs 2000 dollars. Plus about $70 shipping and handling. The Neumann KU-100 binaural microphone is even around 8000 Swiss francs, dollars or euros. There is also a cheaper model from 3Dio, but it does not record low tones below 60 hertz. That would not make sense for a headphone test.

That's quite a lot of money for something of which it is not clear whether it serves the purpose. Therefore, I first want to test the procedure with simpler means. Some time ago, my colleague Thomas Kunz made his own binaural microphone to record room sound better and to delight us with cuddly, creepy ASMR. I use this microphone for my first steps.

So I mount Tom's binaural microphone on a camera and plug it into the microphone input. The camera serves as a holder for the microphone. Recording directly to the computer also requires a suitable audio interface.

First I put my Sennheiser Momentum 2 on the microphone.

The earpieces do not rest on a surface next to the ears, but hang freely in the air. This is certainly not ideal, because the ear cups have a great influence on the sound image. Nevertheless, I run a few test pieces through the headphones and record the sound with the camera. Then I repeat the same thing with two other headphones I have with me at the moment: the Sennheiser HD449 and the Skullcandy Crusher ANC. These are not the most interesting headphones now, but they all sound very different and are therefore suitable for a first test.

Positively surprised

I didn't expect too much from this experiment and I am pleasantly surprised: I hear a clear difference in the sound of all three headphones. The characteristics remain the same over several recordings and reflect the actual sound characteristics. The HD449 sounds very mid-range, almost a bit hollow and more open than the other two. The Skullcandy Crusher mainly emphasises the treble when the bizarre bass vibration is switched off. The Sennheiser Momentum 2 sounds balanced and powerful.

But listen for yourself.

In a second version, I have also included the original sound file; however, it sounds so different that it distracts from the differences between the headphone recordings.

Still needs improvement

For real tests, however, some things would have to be optimised. This starts with the choice of test pieces. Only recordings that are not protected by copyright should be considered, otherwise Youtube will block the video. Here I have chosen a self-made recording, which is of course not optimal. Sorry for the poor production and the errors in timing.

A reliable test would also have to have the headphones resting on a surface so that the sides are closed.

But the biggest problem remains that you never hear the recording as it sounded in the original. Not even when you listen to it with the tested headphones. On the contrary: the special features of the headphones are amplified, so that the recording sounds extremely unbalanced. On the Skullcandy Crusher, it sounds extremely treble-heavy with far too few mids. On the HD449, on the other hand, it sounds acceptable, while the HD449 sounds far too mid-range. This means: If something does not sound good, it is not necessarily due to the quality of the headphones, but most likely to an unfavourable combination of test device and playback device.

Through a graphic representation of the sound spectrum, I could possibly create more objectivity here. This would always be the same, regardless of the loudspeaker used. But I haven't got that far yet.

Testing in-ear headphones in this way would be even more difficult. With Tom's ear microphone, the microphones are now located in the ear cup, roughly where an in-ear would be placed. So I would have to make an artificial ear canal behind the auricle and place the microphones there. A difficult undertaking.

My conclusion

Presenting a recording of a single pair of headphones is completely pointless. The sound does not correspond to what you hear on your playback device. However, the method has potential when it comes to directly comparing several headphones. Especially if the comparison can be displayed graphically. The fact that one pair of headphones sounds flatter than the other can be determined regardless of the playback device.

What do you think? Is it worth pursuing this path and investing more money in a binaural microphone?