---
title: "Discussion on \"How are tokens processed by ML models?\" | Hashnode"
description: "Discussion on \"How are tokens processed by ML models?\". After you perform tokenization, the numerical identifiers representing each token are typically converted into embeddings before being processed by ML models. Embeddings are dense, lower-dimensional vector representations of tokens that capture seman..."
url: "https://hashnode.com/posts/how-are-tokens-processed-by-ml-models/662e964e71272644a5a4b0e7"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [How are tokens processed by ML models?](https://farhanbytemaster.hashnode.dev/how-are-tokens-processed-by-ml-models?utm_source=hashnode&utm_medium=feed)

After you perform tokenization, the numerical identifiers representing each token are typically converted into embeddings before being processed by ML models. Embeddings are dense, lower-dimensional vector representations of tokens that capture seman...

Apr 28, 2024 · 2 min read · farhanbytemaster.hashnode.dev  
Read the article

## Written by

[Farhan Naqvi](https://hashnode.com/@farhan16)

@farhan16 · Stuttgart, Germany

  
[View profile](https://hashnode.com/@farhan16)

MSc Computer Science @ University of Stuttgart | Ex-Cohesity | Ex-Veritas | Open to SWE / ML Engineering Roles in Europe

I build distributed systems that stay reliable under pressure, and I bring ML intuition to every layer of the stack. With 2+ years shipping production-grade C++/gRPC services at Cohesity (formerly Veritas), I've owned everything from anomaly detection pipelines to cyber-resiliency features end-to-end. Now pursuing my MSc at Stuttgart, I'm combining systems depth with autonomous intelligence research to work on problems that actually matter at scale. At Cohesity, I contributed to Stargate - an enterprise-scale distributed file services layer - where I designed garbage collection logic, built high-performance concurrent file services with thread safety guarantees, and implemented stress-testing frameworks that validated system robustness under high-load conditions. I also independently drove a cyber-resiliency feature from architecture proposal through to production delivery. At Veritas, I led anomaly detection for structured workloads (Oracle, MySQL) using unsupervised ML (K-Means, DBSCAN, Isolation Forest) and built the full ELK data pipeline - Logstash, Elasticsearch, Kibana - containerized with Docker and orchestrated with Kubernetes.

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
