DXOMark: "The Way We Test Is Always The Same For Whatever Device"

Amongst the various benchmark tools we have for testing a smartphone’s capabilities, DxOMark is a crucial one. Established in 2008, the company went from testing digital cameras to testing smartphone cameras and soon became the name that people look up to for camera ratings and reviews.

DxOMark recently expanded its portfolio of testing tools, and we got a chance to talk to them more about the features and a lot more. Hence, read on to learn more about DxOMark.

Before we move ahead, if you need to know about the new camera testing tools and the audio tool DxOMark has introduced, you can head over to the links attached.

DxOMark Interview

Q1. DxOMark recently revealed its low-light photography test. Can you elaborate on the “test scenes” you take into consideration while testing?

A1. The test scenes are based on typical low light use cases that most consumers experience, such as, cityscape, architecture shots, portraits, extreme low light. Night Mode is activated differently on different phones; there is a dedicated option in some, and there isn’t in others.

We ensure that the smartphones we test have the night mode. However, it doesn’t mean that the ones that have a good night mode will always score highly; the night mode is one of the criteria. Other factors (to evaluate low-light photography) include flash on, flash auto, and more.

Hence, if a smartphone has a dedicated night mode, it will still be evaluated based on all the factors, which will ensure all smartphones are tested in an unbiased manner.

Q2. In DxOMark’s recent wide-angle test, what exactly do you mean by light levels ranging from around 1000 to 50 lux?

A2. The lighting conditions are different for different places. The indoors have a low lux level while the outdoors tend to have a high lux level. We have a range of lighting systems for various lab scenes. For the wide-angle, the lux range is between 50 to 1000 lux.

For an end-user, there are different use cases (in different settings). As there are different light conditions, the wide-angle is tested in different ranges to ensure that the tests are performed accurately.

To put things into perspective, the wide-angle test includes different lighting conditions that have different lux range. This includes both 1000 lux (for more light) and 50 lux (for less light.)

Q3. Can you simplify the process of testing smartphone cameras for us?

A3. Simply put, the first step in the process is to capture pictures and the videos necessary for the tests. The images and videos (1600 photos and 2 hours of videos) we take are comprehensive. The images and videos are taken as per the pre-defined protocols we have, and they remain the same for all the smartphones we have to test. Following this, the photos and videos are analyzed to obtain objective and perceptual analysis test scores.

Each factor that is considered for testing (exposure, color, autofocus, texture, noise, artifacts, night, zoom, bokeh) gets qualitative and quantitative scores. The scores are then combined to form one image score and one video score. They are further combined to form one overall score, which acts as the final score for the smartphone.

The scores are not always combined on a simple average. There are many attributes we consider, for which there are a number of sub-attributes that are evaluated. For each of the sub-attributes, there are sub-scores, which are, further, used in various use cases. Finally, a final score is decided with the use of our cloud of data helps in coming up with a score.

Additionally, our ratings for smartphones are not out of 100, and there is no maximum limit. If you observe the devices that have been launched in the last 15 and 20 years, there is a significant difference, and we can never find the best one for more than five years as technology changes every year. Hence, we don’t have an upper limit, and based on how technology evolves, we test based on what is relevant to the users.

Q4. DxOMark has forayed into testing the device’s audio quality. Can you explain how the process works?

A4. We select various types of audio samples such as various types of music, voice recordings, and gaming audio and have them tested by our experts for playback and recording quality in our audio lab. Different audio tracks are played, and different attributes are tested. The attributes include Timbre, Spatial, Dynamics, Volume, Artifacts, and Background.

The same audio piece is heard several times, back and forth, so that each attribute is easily tested. To test the various audio scenarios, audio experts remain in the testing labs for around 20 hours. Yet again, there are various use cases (multimedia recording and multimedia playback) considered to finally rate a particular smartphone.

Q5. When calculating the DxOMark Camera scores for a smartphone, will you now consider the audio recording results for the video recording part? Have you have previously considered the same?

A5. For now, video scoring and audio scoring are independent testing tools; we test both aspects separately. And of course, we haven’t done it previously. However, in the future, we are planning to add new features to the DxOMark portfolio, and we are considering the option to test the video and audio together, so we can add this option for our next release, preferably in 2020.

While testing audio and video together is a must, the overall video quality relies on various other video factors. Hence, it doesn’t really hinder the video quality results we give out.

Q6. Do you think a person should buy a smartphone based on the cumulative DxOMark camera score? (Given there are various aspects of a smartphone camera, and not everyone would read your detailed analysis.)

A6. The decision to buy a smartphone, based on our camera scores, is subjective. There are people who will go through the entire review we publish and then buy a smartphone if they consider the camera performance of a device. However, people like my father or anyone else, they might not go through the ratings and just go for one.

Professional photographers would like to go through the sub-tests and tests to buy one digital camera or a smartphone.

Q7. DxOMark (before testing a smartphone’s camera) acts as a consultant to tech companies so that they perform better in the test. This somehow creates a wrong impression about the tests. What is your take on this?

A7. We develop our scoring system for smartphones before we extend our consultancy services to companies. “The way we test smartphones is always the same for whatever device.” There is no compromise on the method. We have a defined methodology and scoring, and this is the way it’s done every time.

For our website, we have a dedicated editorial team, which works with our experts and forms the editorial content. We separate the publication and the consulting part. Our methodologies are quite robust and help drive the progress of the teams of the smartphone manufacturers we work with.

We can purposely direct companies to improve their smartphones for better image/video and audio quality, which consumers also want: they need good-quality smartphones.

Q8. With the megapixel war brewing between the smartphone makers’, has DxOmark geared up for it? We’ve seen that Nokia’s smartphone with the Penta-camera setup failed to live up to expectations, so what’s DxOMark’s take on the number of lenses and growing megapixel numbers?

A8. We have seen that for smartphone cameras, there is a problem with space, and you can’t increase the size of the sensor. This leads to more pixels and a greater number of camera sensors. This might result in two situations: you may do well with multi-camera sensors, you may not do well with the same.

We don’t care about how many cameras a smartphone has, whether they have one, two, or 12. We care about the output of the image quality. On a smartphone, the image quality depends on the software, the hardware, and how they work together.

It matters if we have more cameras on a smartphone, but only if we know how to treat them right, which involves good computing and ISP. Hence, more pixels don’t necessarily mean that the image quality will be good too.

Q9. Does DxOMark have any other testing protocols in the pipeline?

A9. We can’t confirm our product roadmap as of now, but, yes, we would like to keep up with technology and would love to experiment with it. We do have something in the pipeline that will most likely launch in 2020. Hence, you have to stay tuned to us for that.

With Inputs From Anmol and Aditya.