Nothing here yet.
Nothing here yet.
Great point on memory per active request under realistic concurrency! A setup that comfortably holds 15 sessions at the start might only hold a handful an hour later. Agree on quantization too: dropping from 16-bit to 4-bit weights is about making the model fit, and at the same time about buying room for more concurrency. KV cache quantization also helps in this case.
It all started with graphic designing. When I was in class 10, I had my first computer. I used to practice making basic designs in photoshop. After SLC, I took a graphic designing course from an institute. Then I was more into the design field. In +2, I took computer as an optional subject. In class 12, there was C programming. After that, I was interested in programming. After completing +2, I took PHP course from the same institute I completed the graphic designing course. And then studied BIM (Bachelors in Information Management) hoping to learn more about tech.
