Great code, I like how you solved this issue gracefully! But in this situation, you might want to look into https://webscraping.ai , quite handy to use, for solving similar issues.
Thank you Jack Lonkli!! Really appreciate your feedback ♥.
And yes, you are correct, WebScraping AI and other webscraping tools are really awesome tools that do what I did and even complex stuff much better and efficiently.
This was just a simple intro to Web Scraping, so I tended to go with BS4. Hoping to write more about web scraping and using these tools/APIs in the future.