Scalable AI: Nano-Edge to Mega-Cloud
Scalable AI: Nano-Edge to Mega-Cloud
Q&A with MediaTek
After the last few years in the industry, I've noticed that we've been speaking about AI on the edge - and there has been a concerted effort to bring the best models onto our handheld devices. However, there are limitations. Memory capacity is the big one, especially for generative AI, but these models work best when the silicon supports built-in acceleration. There are AI accelerators being built into everything, from edge to TV to automotive to smartphone to PC.
Time to first token and how many tokens per second are becoming the de facto standard metrics in generative AI. However, the AI and machine learning space is more than just generative. Not only do we have large language models and stable diffusion, but we also have machine learning to do many things on many devices. Even something as simple as power management these days is optimised by machine learning. This is why you need stuff on the device to help with battery life, to help with performance.
This is a transcript of an interview with Finbarr Moynihan and Amy Guesner from MediaTek. Finbarr is GM of MediaTek’s Global Marketing but a veteran at the company for 15 years with a PhD in electrical engineering, and Amy is the VP of Strategic Partnerships in Compute, relatively new to MediaTek has 15 years experience in Altera and Intel PSG with a master's in EEE as well.
Here’s the video version for people who would like to watch.
IC: My audience is fairly technical. They've heard of MediaTek before, but If I speak to my parents or friends, they probably use your devices, but they've never heard of you. So, how big is MediaTek?
AG: So, in 2023, we were just over $14 billion USD in revenue, and I’d say you're correct that in different geographies, we’re more well-known than in others and across different market segments, such as smartphones. But if your parents have a smart TV in their home or an Amazon Echo or other components like that, they're already using MediaTek devices; they just don't know that MediaTek is inside.
IC: So, is this the foundation of what is to become your AI strategy, the fact that you're in everything and this (AI) is the buzzword of the day?
AG: It definitely is. I mean, the AI technology that we utilise in our smartphones today, we're carrying that technology, and our network processing unit that we use inside that device, our premium segment into other components, such as laptops, gaming modules, other household appliances, your TVs, kitchen appliances, pretty much anything today can actually utilise AI capabilities.
IC: I think AI is still very much the buzzword for many people, and it's getting to a point where we're hearing it more. It used to be IoT. If you remember a few years ago people were talking about 50 billion IoT devices! Why are we seeing this massive interest in AI in what could arguably be the simplest things with a chip in?
FM: I think you've got to kind of separate things out - we've had AI built into our chips for years, but it was probably what we would call ‘traditional AI’ that maybe wasn't getting as much buzz as it's getting in recent years. But that's been doing a lot of things like enhancing images, making computational photography better, helping with voice assistants, making picture quality better on our TVs. These are things that just improve the overall experience for the consumer. Obviously, the recent buzz is with generative AI, which, at least to me, is still sort of magical technology, right? It has this wonder factor, and it's pretty compelling stuff. We're at the very early stages of that deployment on edge devices. But it's clear that it has enormous potential, and we, like everybody else, are trying to figure that out. It's already landed in smartphones, and the same technology is going to get deployed into solutions. As Amy was saying, the same thing for a whole host of other products across our portfolio.
IC: So, is that one architecture that's going to go end to end? If I program something for a smartphone, could it work in something as small as a watch?
FM: Yep. The performance may scale, but it's the same architecture, yes.
IC: So, given that you are a silicon company, and yes, you build platforms, and some customers may take the turnkey platform or design their own thing, does that make talking about AI to a wider audience difficult?
FM: Yeah, good question. Obviously, we're more like the ingredient that powers the end devices that the consumers experience. But I think for us, when we come to talk about AI, there are a couple of things.
One, obviously, we build the silicon with the integrated neural processing engine, so the NPU, obviously there the focus is on delivering the best possible performance at the lowest possible power consumption with the architecture, the implementation, all the stuff we're known for.
On top of that, of course, we deliver a software SDK for our customers to use if they're building in-house applications or for developers to use if they're building applications. That's about abstracting the hardware, giving them the tools that can port the models, optimise the models, prune and quantize, and all of that kind of stuff.
And I guess the third prong of that now with the generative AI work is actually work that we have to do as well in optimising the foundation models, the large language models, etc, for our hardware, right? So that when developers come to write to those models, it's already sort of “pre-tuned” or “pre-developed”. So we're kind of building maybe the foundation for others to build the experience on top of.
IC: So, are you fully invested in the open-source models, or do you end up having to create your own?
AG: We're definitely using the open-source models. We're leveraging the large language models being developed by Google, Apple, Meta, etc., to be utilized on our devices. I think what you'll find is, as these language models are further optimized and really reduced in size so that they can be deployed on the edge, that you’ll be able to deploy generative AI and all sorts of consumer electronics that maybe you weren't able to previously and do most of the compute on the edge and local to the device rather than having to go all the way back to the cloud like you do today.
FM: But to clarify, our focus is not so much on building the models like others are investing in that space. Ours is about deploying or enabling people to use the models on our hardware.
IC: So when we discuss the latest Dimensity chipset using LLAMA2 7B, that's more of a proof of concept for you, which your customers then have to optimise.
FM: Yeah. Someone's got to build an application on top of that. Exactly.
IC: A lot of people are talking about what the killer app for this (AI) is? Please tell me you have an answer! This is potentially a trillion-dollar question.
AG: I think there are going to be a lot of killer apps. Today, one of the apps I love on the smartphone is the magic eraser, which allows me to remove something I don't like in pictures or people like in pictures. I think productivity is going to be a large segment for generative AI. If I'm putting together a paper, and I want to leverage multiple documents, being able to consolidate that or summarise it, also if I have large Excel spreadsheets and then I want to create charts or truly analyse the data in different methods, being able to do that. There are so many applications. I'm not sure that there's a singular killer application for AI, but I think there will be many advantages to using Generative AI.
IC: Do you see stuff like that actually coming to be running locally?
AG: I do, especially the ones I mentioned being able to take a document and consolidate it or Excel spreadsheet and put together charts. That can all be done locally. I would say if it's information that's not contained either on your particular device or in your organisation's network, then yes, you would have to either have a hybrid environment or go directly to the cloud.
FM: Maybe we can build on what Amy said. Our view is that it's such fundamental technology that it isn't going to get pigeonholed into one thing, right? But I think we can sort of see some early trends. Like, the productivity thing is clearly going to be there. Probably in computing devices, tablets, Chromebooks, you know, some degree on phones, perhaps. I think the phone experience is going to be a more personal experience. Some people are going to refer to it as a smart AI agent. But something that really understands your uses, applications, history, and what you do every day, and proactively and autonomously kind of get in and help with those. This will get better over time, right? It also then has the advantage of keeping everything personal to you on the device. Of course, there are times when it's still going to the cloud, as Amy says, for, you know, the latest information, the applications, and the actions that it needs to take. But I think the bones of that capability are already there today. We're just not quite seeing it yet.
IC: I'm one of the rare people who owns a Dimensity 9300 device. I paid my own money to get the phone, but the downside is that it's a Chinese domestic market device. So, all the AI features are in Chinese, but we see a lot of MediaTek flagship devices in China. Is that where you're seeing most of MediaTek's AI growth today?
FM: I keep having to remind everybody that Dimensity 9300 was launched in November. That was the first chip that we launched with on-chip generative AI capabilities. But yes, it's in the mobile space; it's in the flagship part of the mobile space. The majority of our business in that space today is in China. Some of those OEMs are now expanding into other markets, including Southeast Asia, India, and Europe. But the volume, I would say, is still in China. And that probably then explains the ecosystem of AI, developers, etc. that sort of developed around that.
IC: Given what you guys presented today at your analyst summit, one of the other areas I was interested in is your ASIC business, which builds custom silicon for people who want custom chips. How much does the machine learning side influence that part of the business?
AG: It's definitely influencing it, especially as the hyperscalers, when they develop out their data centres, are looking to optimise on their platforms for their infrastructure and for their architecture. In order to do that, they need custom silicon. So, from inference capabilities or training capabilities to networking capabilities, all of those are being done through custom silicon. They could utilise off-the-shelf, but then that doesn't give them the opportunity to differentiate, optimise for power, or optimise for performance. So yes, custom silicon is being heavily used and you're seeing large resources that are going into and hiring engineers to do custom silicon at each of these companies.
IC: But, I guess, is that silicon positioned to either be the compute behind any machine learning workloads or is it more for, say, the infrastructure to enable machine learning workloads? Where are you seeing your business?
AG: I think it can be both. But today, it would be more on the infrastructure side.
IC: Okay. That's where MediaTek traditionally plays. So how do you see that shifting over time?
AG: Custom models are the secret sauce of these companies - they're not necessarily sharing that with us. They will “black box” it and send it to us in order for us to then add connectivity around it or memory. So those types of IP blocks we do add, but I would say as far as the large language models or the AI capabilities for training and inference, they're doing themselves and they're keeping it in house. That's their differentiation. That's their secret sauce.
IC: So right now, from a MediaTek portfolio perspective, most of the machine learning capability is in the smartphone, but it's going to be coming end-to-end (in the product stack). What are the next verticals to be attacked with this feature?
FM: Smartphones I think, will very quickly adopt this; you'll see it in tablets and Chromebooks and other types of computing devices. With our Kompanio line that leverages a lot of the mobile core IP. A lot of that architecture scales very nicely into IoT platforms, and again, we're seeing a lot of demand for AI capability for a whole host of applications. Differently, but just as important, is high-performance computing trend in automotive platforms, which we've launched now with our partnership with NVIDIA, a whole bunch of smart cockpit solutions. Those are obviously going to bring huge capability for AI into the cockpit inside of the inside of the car for a whole host of use cases.
IC: That's for the in-car entertainment, isn't it?
FM: In-car entertainment, interaction with the driver, you know, large language models, voice interactions while you're driving, all of the in-cabin car experiences. Obviously, ADAS is something different. We're talking about the cockpit solutions. And that's a partnership with NVIDIA.
IC: You were showcasing that partnership with NVIDIA at GTC. How has that come about? NVIDIA does AI day in and day out, right? So why would they partner with MediaTek? Why wouldn't they do it themselves? What does MediaTek bring to the table?
FM: When you look at the automotive market holistically, I think we complement each other very well. Nobody needs to explain Nvidia's chops in AI or in GPU/graphics. However. MediaTek has a lot of capability in low-power compute; in terms of driving displays, when you think about our business in TVs, think about our business in tablets, Chromebooks, etc, we drive a lot of displays out there, So in terms of picture quality, video, display management, we have a lot of capability. Then you sort of take back a little further. You know, telematics and WiFi connectivity are again a huge part of the overall automotive portfolio going forward. Again, that's something we bring to the table. So when we looked at it, I think the two companies' CEOs looked at what they saw and noticed that actually, there's an awful lot more synergy here than actual overlap, right? We will now drive the smart cockpit roadmap forward with Dimensity Auto Cockpit. Of course, NVIDIA is focused on the ADAS solutions up and down the stack, and they have phenomenal technology for that. They bring, you know, DriveOS software and a whole host of other assets to the partnership. But I think the companies feel that together we can deliver a better overall solution to the car makers and to the car industry than we could as separate companies.
IC: You guys had a keynote at Computex - and all the other companies that had major keynotes spoke at length about AIPC. Will you be doing similar?
AG: Maybe you should show up and find out!
IC: Can you speak a little bit about how developers come on board? I speak with AI hardware companies often about how they're implementing a developer strategy. Some of them do it open source. Some of them rely on traditional frameworks, such as PyTorch, TensorFlow, and caffe. We got a brief hint of it at the summit, but can you go into how you're enabling developers? Are you just doing it for customers, or are you enabling everyone?
FM: You've got to kind of go back in time. I would say it’s fairly concentrated on our mobile business. and maybe by definition it is a little bit more concentrated in the Chinese market today. But I think it'll scale. But it uses standard interfaces - all the ones you mentioned, such as caffe, PyTorch, etc - we support all of those through the NeuroPilot software development environment. People can write using standard interfaces. Remember, a lot of these developers started back in the day when AI was used for cameras and AI for other capabilities. So again, they're used to the tools; they've been deploying them for years.
Today, I think it's a combination of the phone OEMs themselves building applications in-house using our chips with the SDK and also working with developers and third-party partners to bring other capabilities using the tool. So, a fairly vibrant ecosystem has already been developed. It's of course, exploding right now with the Generative AI activity. Bringing all new capabilities, how to target these models for new applications, etc. We're still in the very early days, but I would say it's a very standard interface-type approach, nothing magical about it. We're not doing anything proprietary there. I mean, the tools are proprietary, but the interfaces are trying to be as standard and open as possible and then it's about just running as fast as we can to enable it as fast as we can.
IC: Is there any appetite for creating a hardware development kit - having a kit that people can just go out and buy and use and build tools on?
AG: We offer developer kits in the IoT space using our Genio product of SoCs, which also has the NPU embedded into the device. So yes, we have that capability. And then we also have an open source software that's available to our customers that they can use SDKs such as Ubuntu, Linux, Android and then of course you can, you can add these models on top of it.
IC: So NeuroPilot, I did have a small chance to look at the website. It's the first time I heard about it today, and it was described to me by one of your senior colleagues as an optimisation platform.
FM: So optimisation, I guess, is how people would take models that they've developed, prune, quantise, etc. We've got some memory compression capabilities for large models, so more precisely, we have memory decompression on the chip. So the software environment compresses the model we can so we can do all the usual stuff that everybody does pruning, and quantization, but this gets you an extra savings of memory, and then the hardware decompresses it when it comes back in. So all that's done in the tools environment as well. So It is a tools environment as well as an SDK.
IC: Is the goal for MediaTek in this to be market leaders? The ones carving the way forward or, you know, fast followers? You're number one in a lot of things and a lot of markets and what we've spoken about today. So I kind of want it to be, you're the ones leading. You're the ones enabling a million developers.
AG: We definitely want to lead across these markets. We have strong market share in many of these markets. So we're in a great position to do so.
FM: Yeah, I would add, I mean, you know, when you look at any of the connectivity technologies now, like 5G, Satellite, RedCap (reduced capacity, think IoT), Wi-Fi 7, we're leading all of those, right? You know I would say we’re increasing our in-house capability for high-performance computing. You see some of that already in the mobile space, but we're talking about going much further when you look at what we're targeting for automotive, ASIC, enterprise, cloud kind of business; that's definitely an area of focus that we are investing heavily in to close those technology gaps. But yeah, absolutely. The goal is to be global technology leader across these spaces.
IC: I know you guys have said growth, especially in your Southeast Asia and Europe, and eventually you'll be coming to the US en masse in the smartphone side; you're already here in IoT. Does the fact that MediaTek is a Taiwanese company, even with a strong “rest of world” contingent, make any of that growth difficult?
AG: I don't think it does. We're not seeing that as a challenge today. And we're investing in supply resiliency, right? And making sure that we support dual fab strategies, dual OSAT strategies that will be required moving forward. Once we have that in place, our customers are satisfied.
FM: Yeah, I mean our engineering resources are global, right? So, Taiwan, Singapore, China, India, Europe, US. We have large R&D sites. You know, we're certainly adding senior management strategically in the US, senior technical people too, but I haven't seen it as an issue either. You know, maybe even it's an advantage.
IC: You did mention today that you're a strong partner with Intel on manufacturing/Intel 16. TSMC is your main partner for the high-performance silicon, and then you're also leveraging a mix of TSMC and Intel and other OSATs on the packaging. Is there a future where you may use Intel for the leading edge? Samsung was mentioned today as a manufacturing partner. I think a lot of people are interested to hear where the interest is, some of the areas of tightness, right? CoWoS at TSMC, especially, um, so you've got Intel coming up with EMIB and the fact that you guys are a named person as part of foundry, I think people are interested to hear about how's it going, and is the future bright?
AG: Across the board, we evaluate all of our fab partners to see what technologies work best with which applications. It really depends on the device and the requirements that we're trying to meet with ourselves.
IC: Is that mostly like the ASIC business when you have a client that requires fully onboard US soil manufacturing and packaging, or do you do any government work?
AG: We do some government work, but I would say mostly, we're talking about maybe automotive, medical, or other applications where they truly just want something local.
IC: So we'll be speaking a lot today at the Analyst summit. You guys gave us a breadth of what the business is like, and a lot of it is machine learning. If everybody in your audience were to take over one thing from our discussion, what would it be?
FM: I think that MediaTek is at the centre of this AI wave that we're on. We will be an enabler of AI adoption across all of the client-edge devices we talk about, from smartphones to computing devices to automotive. But I think what people may not know is our position in enabling this to be adopted in the cloud as well.
AG: I would say our strength at the edge and in connectivity and making sure that all the significant amount of data that's at the edge that's being transported back to the cloud is important. So we have everything at the edge, we have the connectivity to the cloud, and then in the cloud, having the custom silicon capability and the offering that we're providing there really completes the path from edge to cloud, all with MediaTek. And I think we're one of the only suppliers who can provide that.