Billion-parameter brains in pocket-sized chips: The local AI revolution - We talk IoT #75

In this episode, we explore how engineers are embedding powerful AI directly into hardware – no cloud connection required. Michaël Uyttersprot from Avnet Silica and Cedric Vincent from Tria Technologies reveal how they run ChatGPT-quality language models on resource-constrained embedded devices. What once demanded data centre infrastructure now fits onto chips with just 2GB of RAM. The conversation covers the technical challenges of cramming billion-parameter models into embedded systems, real-world applications from conference rooms to industrial robotics, and the three compelling reasons driving this shift: data privacy, power efficiency, and cost control. Michaël and Cedric discuss hardware platforms from AMD, NXP, and Qualcomm, explain techniques like quantisation and mixture of experts, and demonstrate applications including a vintage telephone box that lets you call avatars from different time periods. Tune in to learn why the future of AI might not be in the cloud at all – and what that means for industries from manufacturing to healthcare. #AI #LLM #embeddedsystems #IoT #privacy #wetalkiot   Summary of this week's episode: 02:48 What makes large language models special 05:27 Why run LLMs locally on embedded devices 07:42 Real-world applications: Vision LLMs and OCR 11:12 Technical deep dive: How to fit billions of parameters into tiny devices 18:52 Understanding temperature: Making AI creative or accurate 22:41 Industries moving fastest: OCR, security, and robotics 24:52 Future applications: Robotic arms and time series analysis 28:00 The biggest technical hurdle: Power consumption 30:55 Advice for engineers: Start with llama.cpp   Show notes: Michaël Uyttersprot: https://www.linkedin.com/in/micha%C3%ABl-uyttersprot-aaa971211/ Cedric Vincent: https://www.linkedin.com/in/cedric-vincent-19222910/ Tria Technologies: https://www.tria-technologies.com/   Generative AI at the Edge: https://my.avnet.com/silica/solutions/technologies/artificial-intelligence/generative-ai/ The podcast episode where the generative AI examples where discussed: https://www.podbean.eu/ep/pb-9juiy-d4dec4 How to enhance embedded systems with Generative AI and Local LLMs | Michael Uyttersprot at HWPMAX25: https://www.youtube.com/watch?v=wL9g2wJ1a7c   Listen to the "We Talk IoT" Soundtrack on: Spotify: https://open.spotify.com/playlist/05MOV4OV2MH2in2txsAGtG?si=ad08112cb8d443f4 YouTube: https://www.youtube.com/watch?v=D-NvQ6VJYtE&list=PLLqgVFfZhDRVYmpEqbgajzDvGL4kACRDp The Llama song: https://youtu.be/JavZh3y1ue0   About Avnet Silica: This podcast is brought to you by Avnet Silica — the Engineers of Evolution. Subscribe to our newsletters here: https://my.avnet.com/silica/resources/newsletter/ You can connect with us on LinkedIn: https://www.linkedin.com/company/silica-an-avnet-company/. Or find us at www.avnet-silica.com.