The year has started on an AI note for smartphones. Users didn’t expect it but Samsung has put AI left, right and centre on its new flagship Galaxy S24 series, making Galaxy S24 Ultra, S24+ and the S24 among the best Android devices you can buy at the moment.

Having AI means the company can easily upgrade software features to take advantage of high-end hardware, especially on the camera front. It means these phones will not get boring. In fact, Samsung is offering seven years of OS and security upgrades, ensuring your phones stay up-to-date for a long time. Of course, the battery may need replacement at some point but the features will always be up-to-date.

All the features on the new phones are the effort of engineers across Samsung’s teams spread out globally, be it in South Korea, the US or India. Recently, we had an opportunity to ask Joshua Cho, executive vice-president, head of visual solution team, Mobile eXperience Business, Samsung Electronics, in San Jose. Over a freewheeling conversation that lasted almost one hour, he touched on several interesting points. “We have the main office in South Korea and R&D offices in many places, like in the US and India. It’s difficult to specify the team that worked on a particular feature. We have a large number of engineers in Korea and a similar number of engineers in India. We work as one team and round the clock. We have teams all over the world.

He feels hardware and software will always get the same priority at Samsung. “It’s a question without an easy answer because there is a lot of R&D work going on in both sides, both software and hardware. It has to be a combination of both. We need to see what we can come up with at the right time. It’s something that’s going to unfold in the future,” said Joshua. Samsung has been using AI for camera to improve the overall quality, besides taking advantage of increased processing power and performance.

AI meets hardware

Let’s talk about the regular photo modes in case you are buying the Samsung Galaxy S24 Ultra, the dude of dudes in the Android world. When you take photos, you can start with 0.6 ultra-wide 12MP photos. There is a nice depth to ultra-wide photos unlike on any other Android device. Then comes the 1x shot, which is about 23mm full-frame equivalent, f/1.7, 200MP sensor. Next, 3x shots (10MP) and when you use this you will get f/2.4 aperture. The next lens is the big change — 5x (50MP). This is a classic lens and I have been using this randomly. It’s fantastic. You can also shoot at 10x but it crops the centre of a frame and retains all the clarity you would demand from a Samsung phone.

Joshua Cho, head of visual solution team, Mobile eXperience Business, Samsung Electronics

But all eyes are on the AI features. There is both an editing feature and an information button. The information button now has more to offer than just EXIF data. AI analyses the picture and makes suggestions, like detecting and removing reflections. Or taking care of unwanted shadows.

Next, there is the editing button. AI helps to straighten photos and fill out the edges. So far we have seen something like this happening on a few websites offering generative AI tech. Here, you have it on your phone. Hold down on any object or subject, you’ll notice it being cut out from the background. You can either completely erase it from your image or drag it elsewhere. Click Generate and boom, magic happens. Samsung AI analyses the photo and then fills in blank areas.

Samsung has been offering AI on its phones before AI became fashionable. If you go back to the Galaxy S10, which came quite a few years ago, it was the first time the company used a dedicated Neural Processing Unit or NPU. The number of AI models Samsung had used back then was few. But now that number has grown massively. “For the best AI model we need a large amount of data and a high-performance NPU,” said Joshua. “When you press the shutter button, the phone starts working on the AI models in the background and they deliver the best picture or video quality. When we measure the NPU performance, we use the terminology TOPS or tera operations per second. In Galaxy S10, the NPU performance had fewer TOPS but that has increased many times over in the last few years.”

The 5x game

The camera tuning on the Samsung Galaxy S24 Ultra has been made to capture true-to-life colours

Partnerships too play an important role in bringing about user experience on the S24 series. For example, so far we have noticed that if you record a video using your phone camera and then upload it to Instagram, the output is quite good but that’s not the case when you use Instagram’s internal camera on most Android phones. With Instagram, Samsung users can now enjoy the “same image quality as on the device”. Every photo and video you take, or view, will be shown in its full range of colour and contrast, from the moment you snap the content, to the moment you post.

Another big change mobile phone photography enthusiasts will notice is the display: It can tackle reflection far better than on any other phone.

One of the things I was most excited about when I used the Galaxy S23 Ultra was the 10x telephoto lens. This year, you get a 5x lens mixed with a bigger, higher-res sensor. With clever algorithm, the new version uses crop zoom to get to 10x, and Joshua says that the image quality is just as good as the 10x optical zoom on the previous version. It’s true and all the details remain the same. In fact, the massive improvement on the 5x zoom helps because the S23 Ultra was using digital zoom at that focal length. The 5x focal length has more practical uses and is used more than the 10x.

A night shot taken on the Galaxy S24 Ultra

Joshua is executive vice-president of the Visual Solution team for Samsung Electronics’ Mobile eXperience Business. His job involves overseeing the development of camera software and image quality solutions as well as the AI and AR solutions, including Nightography. Since joining Samsung in 2004 as a senior researcher, he has led the development of various camera differentiation solutions, including panorama shot, wide selfie, sound and shot, and photo/video editor as well as receiving HDR 10+ certification for the smartphone display. In 2011, his team developed mDNIe, a solution to improve the smartphone display quality, which won them the Jang Young-shil Award from the Korea Industrial Technology Association.

One important aspects of Joshua’s job is to accommodate users with different levels of expertise. He said: “We have a two-track approach, so on the one hand we try to make things easy easier for general users by putting in a lot of things in the auto mode and then for experts, there are applications for them to download as well as special features. Instant mode helps a lot of consumers and when they go through the gallery, suggestions for edits are made. We make things easier for everyone.”

Recently, we have been noticing that Samsung is adding a lot of photography features to its on-device app as well as through apps like Galaxy Enhance X. Will there be a point when everything converges? “As far as combination of features goes, we are accommodating user preferences. Some like to keep it separate and some like to combine. We will continue to listen to our concerns before we take any step,” he said.

Users who want to take full advantage of the camera setup should use the Expert RAW mode more often. You can take full control over the camera settings and these RAW photos pack a lot more data for processing and editing and you can do all of that right on the phone.

HDR as well as colour performance on the Galaxy S24 series have been improved a lot over the company's phones from the previous generation

There are moments when you will shoot an off-the-cuff moment with the Galaxy S24 Ultra and then with a pro camera, only to find that the phone matches the quality so well that only different depth of field can give away the S24 Ultra… that too to a professional set of eyes.

The Galaxy S24 Ultra is a step in the right direction for mobile photography. We are not expecting it to replace professional cameras but it’s a phone that will make more and more people interested in photography.