Thanks! Faced a very similar challenge on a different project, this time involving a Chinese government portal.
What looked like a straightforward data extraction task became much more challenging once we started analyzing how the application was structured. We were able to identify several downstream APIs, but understanding how the search workflow generated and passed identifiers through the system was a different challenge altogether.
The combination of a Vue.js frontend, dynamically generated request parameters, session-dependent validation, and browser-context requirements made it clear that discovering an endpoint is often only a small part of understanding the full architecture.
That's why your point about surfacing applications more intentionally resonates. As modern web applications become increasingly dynamic, understanding existing workflows can sometimes be more difficult than extracting the data itself.
Inshort : Although the child APIs were clearly connected to the parent search mechanism, the actual parent API remained difficult to isolate.
Shall post a detailed breakdown soon! Thanks for the response once again!
Sara
This is a great breakdown, especially the part about how modern frontend architectures make scraping much harder than it initially seems. It really shows how different today’s web apps are compared to more static sites. One interesting side effect of this is that discovering tools or extracting useful workflows from existing apps becomes less straightforward as everything gets more dynamic and protected. I’ve seen some early attempts to approach this from a different angle, like https://unstore.io, where web apps are surfaced more intentionally instead of being discovered indirectly. Feels like as the web evolves, we might need better ways to navigate what’s already built rather than relying on workarounds like scraping.