On Nov 6, arXiv's cs.LG section published two large-model studies on the same day, focused respectively on boosting model capability and on safety alignment. D²-OPD proposes an on-policy distillation method that decouples direction from distance, beating the strongest baseline by up to 9.45 points on math reasoning benchmarks; CMaG reveals that recognizing and refusing compositional harm in vision-language models are decoupled, and proposes a training-free auditing method.
D²-OPD: Decoupling the Direction and Distance of Distillation
On-policy distillation (OPD) trains a student model to match a teacher's next-token distributions on student-generated prefixes. Extrapolative OPD further extends the teacher's log-probability shift relative to a reference policy to construct stronger learning targets. But existing methods couple extrapolation direction and distance, so the magnitude of the teacher's shift determines how far the target moves.
D²-OPD decouples the two factors: a hard KL constraint on each prefix limits the target's distance from the teacher, while an adaptive combination of the teacher's post-training shift and the initial student-teacher gap determines the extrapolation direction. The team optimizes the student with forward KL, which provides a learning signal even for low-probability tokens.
Experiments cover three settings: scenarios where standard OPD performs well, scenarios where OPD struggles under base-model initialization, and scenarios without a teacher reference. On math reasoning benchmarks, D²-OPD scores highest in all three settings, beating the strongest baseline by up to 9.45 points. Ablation experiments confirm the contribution of each component.
CMaG: Decoupled Recognition and Refusal, and Training-Free Auditing
Large vision-language models (LVLMs) are vulnerable to compositional harm: an image and a prompt that are benign in isolation can become harmful through the model's cross-modal reasoning. Recent alignment methods report strong scores on compositional safety benchmarks, but an LLM judge only measures observed behavior and cannot reveal whether safety is robustly integrated into the model. The researchers argue that un-alignment offers a complementary audit of alignment robustness.
The study first shows that minimal supervised fine-tuning can match the LLM-judge scores of multi-stage reinforcement learning pipelines, so similar benchmark scores alone do not prove robustness. Second, recognizing and refusing compositional harm are decoupled: nine standard direction-finding methods detect compositional harm with AUC up to 0.99, but the directions recovered by ablation either leave refusal unchanged or break model coherence.
The team therefore proposes CMaG, a training-free un-alignment method. CMaG measures refusal intent via the log-probability margin between refusal and compliance, takes gradients of intermediate-layer activations, and contrasts them against partially benign pairs to isolate the cross-modal interaction term. A single backpropagation over 80 samples yields the direction, and steering along it monotonically flips refusal into compliance. On Qwen2.5-VL-7B and LLaVA-1.5-7B, CMaG reduces compositional harm.
What to Watch
The two studies map onto two main threads: boosting model capability and safety alignment. Whether D²-OPD's decoupling approach can extend to more distillation scenarios, and whether CMaG's training-free audit can become a complementary tool for LVLM safety evaluation, are worth watching.