---
title: "Discussion on \"Transformer & Self-Attention: I Built an Encoder From Scratch, Then Caught It Overfitting\" | Hashnode"
description: "Discussion on \"Transformer & Self-Attention: I Built an Encoder From Scratch, Then Caught It Overfitting\". A Transformer Encoder built from scratch for intent classification, caught overfitting, then fixed via cross-validation and transfer learning."
url: "https://hashnode.com/posts/transformer-selfattention-overfitting/6aa437fb1c546bafd4e79105"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Transformer & Self-Attention: I Built an Encoder From Scratch, Then Caught It Overfitting](https://shaka-ai.hashnode.dev/transformer-selfattention-overfitting?utm_source=hashnode&utm_medium=feed)

A Transformer Encoder built from scratch for intent classification, caught overfitting, then fixed via cross-validation and transfer learning.

Sep 11 · 7 min read · shaka-ai.hashnode.dev  
Read the article

-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#nlp](https://hashnode.com/tag/nlp)
-   [#pytorch](https://hashnode.com/tag/pytorch)
-   [#deeplearning](https://hashnode.com/tag/deeplearning)
-   [#deep-learning](https://hashnode.com/tag/deep-learning)
-   [#bert](https://hashnode.com/tag/bert)
-   [#transformers](https://hashnode.com/tag/transformers)

## Written by

[Muhammad Ariel Shakaramiro](https://hashnode.com/@arielshakaramiro)

@arielshakaramiro

  
[View profile](https://hashnode.com/@arielshakaramiro)

Newbie for AI Learning | Builder | Developer | AI Enthusiast

## Responses 1

Markdown supported · ⌘↵ to post

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Sep 21](https://hashnode.com/posts/transformer-selfattention-overfitting/6aa437fb1c546bafd4e79105/comment/6ab0cb9d59c4f1597952bf21)

Chasing down why it overfit being the most interesting part is the right instinct, and intent classification is a domain where it happens almost immediately. A transformer encoder has far more capacity than a few thousand short utterances justify, so it memorises phrasing rather than learning intent.

The tell worth checking is whether the errors concentrate on the rare classes. With an uneven intent distribution a model can post a respectable overall score while being useless on everything outside the top few.

At that dataset size the boring fixes usually beat architectural ones: fewer layers, aggressive dropout, and starting from a pretrained multilingual encoder instead of random init.