Prompt Attention Analysis
Layer-by-layer analysis of Gemma 3 4B attention for 25 prompt pairs in five modification categories, separating local (sliding-window) and global layers, with entropy / sparsity metrics, heatmaps and Inseq attribution. Reproducible Hydra + uv pipeline with 71 tests.
Overview
An explainable-AI study of how small changes in a prompt reshape the attention of a language model. The pipeline runs Gemma 3 4B with attention outputs enabled, pools heads with grouped-query-attention awareness, splits the 34 layers into local sliding-window layers and five global layers, and compares base and modified prompts with attention entropy, sparsity, L1/L2 difference and mean attention per token category. Results: every modification increased attention entropy (mean +0.34, 25 of 25 pairs), framing prompts changed attention most and paraphrases least, and local layers reacted 1.5-2x more strongly than global layers. The instruction-versus-content question stayed inconclusive because base prompts are very short and token categories are keyword-based, and the report documents why the base-vs-modified diff metric is unreliable when sequence lengths differ.
Key Highlights
- ✓Attention extraction from all 34 layers with GQA-aware head pooling
- ✓Separate analysis of local (sliding-window) and global layers
- ✓25 prompt pairs, 5 categories, manual token-category annotations
- ✓Entropy, sparsity, L1/L2 diff and per-category metrics exported to CSV
- ✓Base, modified and diff heatmaps plus a self-contained HTML report
- ✓Inseq attention attribution and Weights & Biases tracking
- ✓Hydra config, uv lockfile, make targets, 71 tests without GPU or network
Technical Deep Dive
Problem & Goal
Rewording a prompt can change a model's answer. The study asks whether instruction tokens get more attention than content tokens (RQ1), whether different modification types produce different attention patterns (RQ2), and whether local and global layers react equally (RQ3).
Findings
RQ1 is inconclusive: instruction tokens get attention in early local layers, but base prompts have only 4-8 tokens, so the effect is largely a length artefact. RQ2: entropy increase is largest for framing (+0.55) and style (+0.43) and smallest for reformulation (+0.09). RQ3: local layers show 1.5-2.0x larger entropy change than global layers in every category. Attention weights are treated as observations, not causal explanations.
My Role & Scope
Team project of three students. My commits cover the metrics module (entropy, sparsity, diff, per-category means), token-category annotations and categorisation, the diff heatmaps and category bar chart, the expanded 25-pair dataset, config validation, the pytest suite (including integration tests), the Makefile, and the English documentation and results plots.
Screenshots & Images
