Puneet Mangla

Table of Contents Building a Multimodal Chatbot with Qwen3-VL Instruct and Thinking Models Qwen3-VL Vision-Language Model: Architecture, Training, and Capabilities Qwen3-VL Architecture Overview: SigLIP2 Vision Encoder and Multimodal Transformer Design Training Pipeline: Multimodal Pretraining with Image-Text and Video-Text Data Performance…

Table of Contents Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models Qwen3 Model Details Qwen3 Model Architecture: Dense Models, MoE Variants, and Dual-Mode Reasoning Qwen3 Training Data and Post-Training Pipeline for Reasoning and Instruction Following Qwen3 Benchmark Performance:…

Table of Contents Building and Training a Kimi-K2 Model Using DeepSeek-V3 Components Kimi-K2 vs DeepSeek-V3: Key Architecture Differences in LLM Design Mixture of Experts Scaling in Kimi-K2: Model Size, Sparsity, and Efficiency Attention Head Optimization in Kimi-K2 for Efficient Long-Context…

Table of Contents Autoregressive Model Limits and Multi-Token Prediction in DeepSeek-V3 Why Next-Token Prediction Limits DeepSeek-V3 Multi-Token Prediction in DeepSeek-V3: Predicting Multiple Tokens Ahead DeepSeek-V3 Architecture: Multi-Token Prediction Heads Explained Gradient Insights for Multi-Token Prediction in DeepSeek-V3 DeepSeek-V3 Training vs.…

Table of Contents DeepSeek-V3 from Scratch: Mixture of Experts (MoE) The Scaling Challenge in Neural Networks Mixture of Experts (MoE): Mathematical Foundation and Routing Mechanism SwiGLU Activation in DeepSeek-V3: Improving MoE Non-Linearity Shared Expert in DeepSeek-V3: Universal Processing in MoE…

Table of Contents Build DeepSeek-V3: Multi-Head Latent Attention (MLA) Architecture The KV Cache Memory Problem in DeepSeek-V3 Multi-Head Latent Attention (MLA): KV Cache Compression with Low-Rank Projections Query Compression and Rotary Positional Embeddings (RoPE) Integration Attention Computation with Multi-Head Latent…

Table of Contents DeepSeek-V3 Model: Theory, Config, and Rotary Positional Embeddings Introduction to the DeepSeek-V3 Model The Four Pillars of DeepSeek-V3 What You Will Build Prerequisites and Setup for Building the DeepSeek-V3 Model Implementing DeepSeek-V3 Model Configuration and RoPE DeepSeek-V3…

Table of Contents KV Cache Optimization via Tensor Product Attention Challenges with Grouped Query and Multi-Head Latent Attention Multi-Head Attention (MHA) Grouped Query Attention (GQA) Multi-Head Latent Attention (MLA) Tensor Product Attention (TPA) TPA: Tensor Decomposition of Q, K, V…

Table of Contents KV Cache Optimization via Multi-Head Latent Attention Recap of KV Cache The Need for KV Cache Optimization Multi-Head Latent Attention (MLA) Low-Rank KV Projection Up-Projection Decoupled Rotary Position Embeddings (RoPE) RoPE in Standard MHA Challenges in MLA:…