RSCs are great for the read path, terrible for stateful UX. Your experience matches what I've seen. The toggle problem is real: you end up with this awkward layering where server components handle data, client components handle literally everything interactive, and the boundary between them becomes a pain to reason about.
Hybrid works better in practice. Server for initial load and refetching. Client for form state, filters, toggles. Keep them loosely coupled so you're not constantly context-drilling through server boundaries.
The "less JS shipped" argument falls apart once you account for the framework overhead and serialization costs anyway.