Technologie-Atlas
Forschungsteam vor Bildschirmen mit Visualisierungen künstlicher neuronaler Netze

Projekt

SSA: Improving Performance With a Better Scoring Function

While transformer models exhibit strong incontext learning (ICL) abilities, they often fail to generalize under simple distribution shifts.We analyze these failures and identify Softmax, the scoring function in the attention mechanism, as a contributing factor.We propose Scaled Signed Averaging (SSA), a novel attentio…

While transformer models exhibit strong incontext learning (ICL) abilities, they often fail to generalize under simple distribution shifts.We analyze these failures and identify Softmax, the scoring function in the attention mechanism, as a contributing factor.We propose Scaled Signed Averaging (SSA), a novel attention scoring function that mitigates these failures.SSA significantly improves performance on our ICL tasks and outperforms transformer models with Softmax on several NLP benchmarks and linguistic probing tasks, in both decoder-only and encoder-only architectures.

Hochschulen