Home
Feed

Extend

Hashnode Pro
Hashnode for Agents

Resources

The foreword
Hashnode gql skill

Buildful - AI coding agent that delivers frontier-level results at a fraction of the cost Bug0 - The AI-native e2e QA regression testing Hackathons Changelog Brand @hashnode on X Hashnode on LinkedIn Support - hello+support@hashnode.com Code of Conduct Terms Privacy Sitemap

Search Hashnode

Search posts, tags, users, and pages

Discussion on "Scrapy - your neighborhood spider!" | Hashnode

Feed/

Discussion

Krish

Mar 11, 2021

Scrapy - your neighborhood spider!

Spider? You are kidding :) Well, spiders are all over the internet. It crawls websites and retrieves information readily available on the wild web. From the Wikipedia: A Web crawler, sometimes called a spider or spiderbot and often shortened to cra...

krish.hashnode.dev7 min read

#side-project #python #python3 #web-scraping #scraping

Responses2

Markdown supported · ⌘↵ to post

Sholanki Biswas

I am a software developer

Mar 15, 2021

Interesting read Krish. I will try this one out to see what I come up with. Thank you for explaining what you are actually doing in code. I am wondering if this is also one part of lighthouse searches. Quite trippy!

What do you mean by lighthouse searches?

Krish

Mar 16, 2021

Neha kumari

I am a Salesforce Developer, Notion expert and I am on my journey to explore all things new and interesting in tech.

Mar 11, 2021

Very interesting read. I'm curious to give "scrapy" a try. Your article prompted me with an idea. If I could automate the python script to see where the search result of my personal website ranks every week, it could help me with the SEO for my website if need be.

Krish

Mar 11, 2021

Great thought! So, to know your seo rankings I'm sure you'd be looking it up on some website. Isn't it? You could probably scrape those websites to retrieve the information you are looking for.

Neha kumari

I am a Salesforce Developer, Notion expert and I am on my journey to explore all things new and interesting in tech.

Mar 11, 2021

Krishanu : Yes, you're right. It usually is based on the Google search result. So, I guess I can scrape the result from Google. I will give this a try. Of course, there may be OOB tools available for this. Google Analytics being the simplest one of them. But, I'm curious to try this as a use case. Thanks for sharing your knowledge.