LLM Inference & Infra Interview: 15 In-Depth Questions

③ Quantified high-score answer

TTFT and TPOT represent opposing hardware constraints. The Prefill phase computes all prompt tokens concurrently, making it compute-bound: optimization requires FlashAttention kernels maximizing Tensor Core occupancy and Chunked Prefill to interleave long prompts into iteration batches. Conversely, the Decode phase generates single tokens autoregressively, making it memory-bandwidth bound (constrained by HBM throughput). Optimization focuses on enlarging batch sizes to amortize weight transfers, running speculative decoding, or deploying FP8 quantization to reduce memory traffic.

③ Quantified high-score answer

Standard Attention materializes the complete N-by-N attention matrix in high-latency HBM, causing severe memory bandwidth bottlenecks. FlashAttention introduces tiling and online softmax: Q, K, and V matrices are loaded in small tiles into high-speed on-chip SRAM (delivering 10TB/s+ bandwidth). By maintaining running maximums and normalization factors mathematically, softmax computes incrementally without writing intermediate attention matrices back to HBM. FlashAttention-2 and 3 further optimize warp scheduling and leverage Hopper TMA asynchronous memory transfers.

③ Quantified high-score answer

Speculative decoding trades inexpensive draft compute for parallelized target verification. The serving engine deploys an efficient draft model alongside a target model: the draft model generates K candidate tokens rapidly. The target model evaluates the context and all K candidates in a single forward pass utilizing tree attention masks. Applying rejection sampling based on probability ratios confirms M valid tokens per step. Because verification matches the latency of single-token decoding, maintaining acceptance rates above 70% accelerates throughput by two-to-threefold.

③ Quantified high-score answer

Quantization degradation stems from high-magnitude activation outlier channels. GPTQ executes second-order error compensation via inverse Hessian matrices, updating unquantized weights row-by-row for W4A16 weight-only storage. AWQ observes that weights are not equally salient: protecting the top 1% of salient weights based on activation magnitude retains FP16 precision, quantizing only residual weights. SmoothQuant enables true W8A8 matrix multiplications by mathematically migrating outlier scaling factors from activations into weights, unlocking hardware INT8 Tensor Core acceleration.

③ Quantified high-score answer

TP and PP address distinct communication and memory boundaries. Tensor Parallelism partitions matrix multiplications within individual layers: in MLP blocks, first-layer weights are split column-wise and second-layer weights row-wise, requiring a single All-Reduce synchronization per layer. Because communication occurs per token, TP demands ultra-high-speed NVLink interconnects within single server chassis. Pipeline Parallelism splits layers across GPUs, introducing idle bubbles; engines mitigate bubbles during inference using micro-batching. Standard topology applies TP=8 intra-node, scaling PP or Data Parallelism inter-node.

③ Quantified high-score answer

Coupling prefill and decode phases inside identical nodes creates severe resource contention: compute-heavy prompt processing stalls ongoing autoregressive streams. PD Disaggregation decouples clusters into specialized node pools: Prefill pools run large-batch prompt processing at near-100% compute saturation. Upon completion, intermediate KV Caches transfer zero-copy over high-speed RDMA/RoCEv2 networks directly into Decode pools. Decode nodes run isolated autoregressive steps with stable TPOT. This disaggregation doubles aggregate cluster throughput while reducing P99 latencies by 60%.

③ Quantified high-score answer

Prefix caching eliminates redundant prefill compute across agent workflows and few-shot prompts. In engines like SGLang and vLLM, a Radix Tree indexes historical token sequences against physical KV Cache blocks. When requests arrive, the engine executes longest-prefix matching: matching prefix blocks reuse existing VRAM allocations directly, computing prefill solely for novel suffix tokens. Pair this with Cache-Aware Routing at the gateway layer, dispatching requests sharing system prompts to identical physical GPU instances, slashing cluster prefill compute costs by over 70%.

③ Quantified high-score answer

Distributed LLM serving demands hardware-aware scheduling. We deploy Kubernetes schedulers (Volcano or Kueue) with Node Feature Discovery: scheduling pods requiring Tensor Parallelism exclusively across GPUs co-located on identical NUMA nodes interconnected via full-mesh NVLink, eliminating PCIe cross-switch degradation. For autoscaling, KEDA monitors real-time request queue depths and KV Cache saturation. To accelerate pod cold starts, we implement peer-to-peer container image distribution and parallel safetensor direct memory mapping, compressing spin-up durations from ten minutes to under 40 seconds.

③ Quantified high-score answer

Conquering 128K+ contexts requires combining architectural sparsification with sequence parallelism. Algorithmically, we deploy sparse attention mechanisms (such as StreamingLLM retaining initial attention sinks and sliding local windows) to break quadratic complexity barriers. Infrastructure-wise, we implement Sequence Parallelism via RingAttention or DeepSpeed-Ulysses: partitioning input token sequences along the sequence dimension across ring-connected GPUs. Each card computes localized attention blocks while asynchronously passing KV tensors in a ring, fitting 1M token contexts across multi-GPU clusters alongside NTK-aware RoPE position scaling.

③ Quantified high-score answer

LLM APM requires capturing specialized generation telemetry across three layers: At the hardware layer, we record GPU power draw, SM utilization, memory bandwidth, and NVLink saturation. At the inference engine tier, we expose real-time runtime metrics: KV Cache allocation percentages, waiting queue dwell times, TTFT distributions, TPOT percentiles (P50/P90/P99), and prefill preemption counts. At the API gateway tier, we map token counts to financial cost metrics, enabling immediate triage to isolate whether performance degradation stems from compute saturation, memory bandwidth limits, or network transport latency.

③ Quantified high-score answer

Mitigating systemic GPU OOM crashes requires immediate load shedding followed by phased recovery. Step one is perimeter containment: the ingress gateway trips circuit breakers, returning HTTP 429 backpressure to incoming requests and shedding traffic to protect surviving nodes. Step two stabilizes the serving engine: schedulers reject new prefill jobs while completing in-flight decode generations, preventing work-in-progress state loss. Step three identifies root causes: tracing payloads reveals whether malicious prompt bombs bypassed input validation limits. After tightening gateway input guards, traffic is gradually restored.

③ Quantified high-score answer

Resolving stakeholder dissatisfaction demands empirical transparency and targeted performance engineering. Rather than arguing policy, I benchmark real production traffic: running automated scripts evaluating proprietary serving against commercial APIs across TTFT, TPOT, and uptime to identify concrete bottlenecks. I launched a one-week optimization sprint: upgrading to modern vLLM engines, enabling Chunked Prefill, and deploying FP8 quantization, cutting P90 latency by 50% to surpass cloud baselines. Delivering an analysis highlighting data privacy, custom fine-tuning, and 80% lower cost per token restored stakeholder trust.

③ Quantified high-score answer

AI infrastructure bridges research innovation with production serving velocity. When evaluating non-standard operators, I partner with research scientists to understand the architectural intent: quantifying the algorithmic accuracy gains. If the gains are substantial, I author custom Triton or CUDA kernels to implement the operator natively, compiling them into TensorRT-LLM or vLLM custom plugin registries. We run precision regression suites to verify numerical accuracy within 1e-3 tolerance while benchmarking throughput, preserving algorithmic breakthroughs without compromising production serving SLAs.

③ Quantified high-score answer

Communicating with executive leadership requires translating technical acceleration into commercial financial metrics. I instituted our corporate LLM Cost and Efficiency Compass, tracking cost per million tokens alongside throughput capacity per GPU card. During quarterly reviews, I demonstrated that by deploying speculative decoding, PagedAttention, and dynamic prefix caching, our infrastructure absorbed a 3x surge in corporate LLM usage without procuring additional GPU hardware. This avoided millions in cloud leasing and capital expenditures, proving infrastructure engineering as a strategic profit-enabler.