Testing audio gear – especially headphones – is a minefield to navigate. Audio is so subjective that the old adage of one man’s food being another man’s poison actually holds true here. Personal preferences overwhelmingly dictate what is considered good or bad. Funnily enough the same does not apply in other product categories.
To give you an example of how unique audio is, consider the case where we’re testing LCD displays. A particular display happens to have a slightly warm hue. Any such deviation from a neutral colour temperature would be thought of as an inaccuracy and even penalised – no matter how ‘pleasing’ that inaccuracy would appear to the reviewer. Doing so in categories like Displays is generally accepted as kosher, the same in audio raises innumerable heckles.
This phenomenon of whatever is ‘pleasing’ to the ear getting better reviews, has pushed some audio manufacturers to tune their gear to produce those particular sound signatures. But the fact remains that such headphones are technically ‘inaccurate’ and might not work well with certain genres of music or tracks that are not mixed properly. While testing audio, reviewer bias or subjectivity comes in as a result of some people preferring say bass over treble (in the most simplistic form). Instead in an ideal scenario, headphones should be judged by how “true-to-source” their audio output actually is. That is, does the headphone reproduce a particular audio track the way the sound engineer mixing it intended it to sound. How can one know that?
Instruments that output frequency response curves is one way, but the trouble with those is unless the equipment is top notch such tests produce very little variance for it to be meaningful at all. In this case a “reference” speaker or headphone helps assign a benchmark against which to map deviations. This reference device should be one that’s generally known or proven to be flat response. Flat response is a simple concept to understand: it means that the frequency curve produced by the speaker or headphone is flat (or near flat). That is, no frequency range – lows (bass) or highs (treble) – is favoured over the other.
Minimising subjectivity is the holy grail of audio testing but as you can imagine this is a difficult task because as they say, ultimately the proof of the pudding is in the eating. Currently, listening to an audio device is the only way evaluate it. To put our in-ear headphones through their paces properly, we recently revamped our test process. A few more test tracks were added to the mix in order to properly represent multiple genres. The tracks had a sampling rate of at least 96 kHz (some even 192 kHz when available) and a bit depth of 24 bits (as opposed to the standard bit depth of 16 bits). A few legacy tracks were still deliberately kept in in the mix for reproducing real world scenarios but even those necessarily had a minimum bitrate of 320kbps.
As for track selection I generally use a series of tracks (some of which I’ve been using for years now) to test specific attributes such as stereo separation, vocal tonality, detail (when instrument density is high), dynamic range etc. As most of you already know, we have a penchant for guitar driven music here at Digit, but in this test process revamp we’ve deliberately diversified our genres. There’s even Ed Sheeran’s latest chartbuster thrown in (gasp!).
Apart from these performance tests the build, comfort and features are rated. Longer, woven cables, for instance, get a better score. Attributes such as isolation and subjective ones such as soundstaging and aesthetics are also scroed; although the last two don’t get too much weightage in the overall score.
For testing IEMs we either use our trusty old ASUS Xonar STX and or go with whichever trendy, portable new high resolution digital audio player is available – for example the FiiO – the X5 Gen 3. It’s got some of the best op amp chips in the industry today and helps isolate against any random buzz, static, leak or interference that might creep in because of faulty wiring in the AC supply. Such DAPs are easily able to drive headphones with high impedance and lastly it’s just convenient to quickly be able to change headphones on the fly.
In any of our headphone reviews you’ll keep finding mentions to a few reference headphones such as the RHA S500i, M6Pro, Bose QC20, M50, and the UE 900. These headphones are used for comparison but not specifically thrown into the scoring mix. They make for a good reference point with which to assess these latest models to enter the audio arena.
Very well, now that you know all about our test process, here’s another important question: How should you, as a reader and prospective buyer, interpret these scores? Firstly think of the scores assigned as ordinal numbers rather than cardinal ones. Usually the highest performance scorer is the headphone that’s best adept at producing a wide range of genres. However, some genres and types of music thrive only when a particular frequency range has been boosted.
Lastly a small note on nomenclature: the terms in-ears, IEMs and headphones will be used interchangeably in our reviews. Purists hold your peace and the rest of you – enjoy.