What are some good libraries for scraping a web page in PHP/JS?

Do anyone have experience with this technology. I need to be able to scrape a web page using PHP or JS while logged in. Maybe using CURL request to log in and then store the token and use it later.

Could you recommend me some library for this or a right way.

Thank you in advance!

I think PHPSimpleDOM is enough to scrape the web. Its pretty straight and uses css based selectors.

github.com/FriendsOfPHP/Goutte this could be used but in general you still need to define the patterns.

I would recommend using selenium because that way you don't have the problem with heavy javascript pages and there are browser extensions where you can actually easily export certain behaviours as a base template :)

Emil Moe wouldn't you use simplexml for that ?

otherwise i would go for symfony.com/doc/current/components/dom_crawler.ht…

it's pretty neat.

Vladimir Kyuchukov this could interest you as well.

So I managed to do some stuff, but It appears I need scraper for dynamically created page. Pagination is dynamic. There is no query params which I can use. The website is written in react and everything is JS.

Selenium is a good tool for this case, but can be used with anything different than Python.

Vladimir Kyuchukov selenium is java. it's the common BDD brower tool for most applications. I think it uses an XML interface so it should be able to interface with any language.

you might want to look at Behat a PHP BDD tool which interfaces with it.

Thread

What are some good libraries for scraping a web page in PHP/JS?

Responses(2)

Recent threads

Search Hashnode

What are some good libraries for scraping a web page in PHP/JS?

Responses(2)

Recent threads