Discussion on "Building a web scraper in Typescript"

Caelin Sutch · 2020-08-16T20:31:29.117Z

Web scraping refers to the process of gather information from a website through automated scripts. With web scraping, one can gather large amounts of data from websites where no official API exists. Web scraping can be broken down into two simple ste...

Awesome!

Haven't got round to writing this myself or using cheerio for scrapping external. What will happen when they change their divs? Will it fail or return a error?

Great question. It’ll return null if the divs change. In production, I have it throw an error if it gets null, which signals me that I have to update my parsing!

Caelin Sutch That's amazing, another question that popped up is React/Vue kind of applications, they often use dynamic classes and id's build runtime, is that something you experiences already?

I mean for instance the price div would look like this in a runtime build app:

<span id="price_#1221">$34</span> where the ID would change all the time, and a class would not be needed since it is a styled component.

Daily Dev Tips so it gets a little tricky with React and other web apps that use JS to create HTML. Libraries like puppeteer (which run a headless chrome browser) will get the rendered HTML/CSS like your browser would.

In terms of dynamic tags, I don’t have direct experience, but I’m sure using Regex or some sort of logic you could get the correct data, it’d just be a little more difficult.

Search Hashnode

Building a web scraper in Typescript

Responses1