09 Inference Optimization

Summary

Today we have Philip Kiely from Baseten on the show. Baseten is a Series B startup focused on providing infrastructure for AI ... This is part 1 of Ted's review of a tutorial from the Amazon AWS team on making your LLMs run faster. This runtime performance is ... ... training cost so why do we focus on the Download the source code from here: onepagecode.substack.com/ Open-source LLMs are great for conversational applications, but they can be difficult to scale in production and deliver latency ... In the "AI From Scratch" study group, we are reading Chip Huyen's book AI Engineering. This is the recording of our first study ... Part 2 of 5 in the “5 Essential LLM Philip Kiely, Head of Developer Relations at Baseten, presents the “Golden Triangle” of Video 1 of 6 | Mastering LLM Techniques: Two GPU kernels can compute the exact same attention, on the same chip, with identical inputs and identical outputs, and one still ... An LLM serves tokens on $40000 GPUs, and the bottleneck is almost never the math. It is memory and scheduling. This is LLM ... Tri Dao, Chief Scientist at Together AI and Princeton professor who created Flash Attention and Mamba, discusses how