---
title: "Discussion on \"What is Tokenization?\" | Hashnode"
description: "Discussion on \"What is Tokenization?\". Tokenization is the process of breaking down a sequence of text into smaller units, called \"tokens\". These tokens could be words, subwords, characters, or even phrases, depending on the specific tokenization strategy used. Tokenization is a fundament..."
url: "https://hashnode.com/posts/what-is-tokenization/661cae9245f8a7f5dff4c75a"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [What is Tokenization?](https://farhanbytemaster.hashnode.dev/what-is-tokenization?utm_source=hashnode&utm_medium=feed)

Tokenization is the process of breaking down a sequence of text into smaller units, called "tokens". These tokens could be words, subwords, characters, or even phrases, depending on the specific tokenization strategy used. Tokenization is a fundament...

Apr 15, 2024 · 4 min read · farhanbytemaster.hashnode.dev  
Read the article

-   [#tokens](https://hashnode.com/tag/tokens)
-   [#tokenization](https://hashnode.com/tag/tokenization)
-   [#nlp](https://hashnode.com/tag/nlp)
-   [#nlp-transformers](https://hashnode.com/tag/nlp-transformers)
-   [#natural-language-processing](https://hashnode.com/tag/natural-language-processing)
-   [#ai](https://hashnode.com/tag/ai)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#ml](https://hashnode.com/tag/ml)
-   [#llm](https://hashnode.com/tag/llm)
-   [#large-language-models](https://hashnode.com/tag/large-language-models)
-   [#genai](https://hashnode.com/tag/genai)
-   [#genai-applications](https://hashnode.com/tag/genai-applications)
-   [#genai-technology](https://hashnode.com/tag/genai-technology)

## Written by

[Farhan Naqvi](https://hashnode.com/@farhan16)

@farhan16 · Stuttgart, Germany

  
[View profile](https://hashnode.com/@farhan16)

MSc Computer Science @ University of Stuttgart | Ex-Cohesity | Ex-Veritas | Open to SWE / ML Engineering Roles in Europe

I build distributed systems that stay reliable under pressure, and I bring ML intuition to every layer of the stack. With 2+ years shipping production-grade C++/gRPC services at Cohesity (formerly Veritas), I've owned everything from anomaly detection pipelines to cyber-resiliency features end-to-end. Now pursuing my MSc at Stuttgart, I'm combining systems depth with autonomous intelligence research to work on problems that actually matter at scale. At Cohesity, I contributed to Stargate - an enterprise-scale distributed file services layer - where I designed garbage collection logic, built high-performance concurrent file services with thread safety guarantees, and implemented stress-testing frameworks that validated system robustness under high-load conditions. I also independently drove a cyber-resiliency feature from architecture proposal through to production delivery. At Veritas, I led anomaly detection for structured workloads (Oracle, MySQL) using unsupervised ML (K-Means, DBSCAN, Isolation Forest) and built the full ELK data pipeline - Logstash, Elasticsearch, Kibana - containerized with Docker and orchestrated with Kubernetes.

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
