---
title: "Discussion on \"What are special tokens in Tokenization?\" | Hashnode"
description: "Discussion on \"What are special tokens in Tokenization?\". Special tokens are additional tokens added during the tokenization process to serve specific purposes in natural language processing tasks. These tokens are not derived from the original text but are inserted to provide additional information or perf..."
url: "https://hashnode.com/posts/what-are-special-tokens-in-tokenization/661f558f2cea5d8a6be898e4"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [What are special tokens in Tokenization?](https://farhanbytemaster.hashnode.dev/what-are-special-tokens-in-tokenization?utm_source=hashnode&utm_medium=feed)

Special tokens are additional tokens added during the tokenization process to serve specific purposes in natural language processing tasks. These tokens are not derived from the original text but are inserted to provide additional information or perf...

Apr 17, 2024 · 3 min read · farhanbytemaster.hashnode.dev  
Read the article

-   [#ai](https://hashnode.com/tag/ai)
-   [#genai](https://hashnode.com/tag/genai)
-   [#tokenization](https://hashnode.com/tag/tokenization)

## Written by

[Farhan Naqvi](https://hashnode.com/@farhan16)

@farhan16 · Stuttgart, Germany

  
[View profile](https://hashnode.com/@farhan16)

MSc Computer Science @ University of Stuttgart | Ex-Cohesity | Ex-Veritas | Open to SWE / ML Engineering Roles in Europe

I build distributed systems that stay reliable under pressure, and I bring ML intuition to every layer of the stack. With 2+ years shipping production-grade C++/gRPC services at Cohesity (formerly Veritas), I've owned everything from anomaly detection pipelines to cyber-resiliency features end-to-end. Now pursuing my MSc at Stuttgart, I'm combining systems depth with autonomous intelligence research to work on problems that actually matter at scale. At Cohesity, I contributed to Stargate - an enterprise-scale distributed file services layer - where I designed garbage collection logic, built high-performance concurrent file services with thread safety guarantees, and implemented stress-testing frameworks that validated system robustness under high-load conditions. I also independently drove a cyber-resiliency feature from architecture proposal through to production delivery. At Veritas, I led anomaly detection for structured workloads (Oracle, MySQL) using unsupervised ML (K-Means, DBSCAN, Isolation Forest) and built the full ELK data pipeline - Logstash, Elasticsearch, Kibana - containerized with Docker and orchestrated with Kubernetes.

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
