---
title: "Discussion on \"Scrapy - your neighborhood spider!\" | Hashnode"
description: "Discussion on \"Scrapy - your neighborhood spider!\". Spider? You are kidding :)\nWell, spiders are all over the internet. It crawls websites and retrieves information readily available on the wild web. \nFrom the Wikipedia:\n\nA Web crawler, sometimes called a spider or spiderbot and often shortened to cra..."
url: "https://hashnode.com/posts/scrapy-your-neighborhood-spider/6049b638258b6d5f4339f677"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Scrapy - your neighborhood spider!](https://krish.hashnode.dev/scrapy-your-neighborhood-spider?utm_source=hashnode&utm_medium=feed)

Spider? You are kidding :) Well, spiders are all over the internet. It crawls websites and retrieves information readily available on the wild web. From the Wikipedia: A Web crawler, sometimes called a spider or spiderbot and often shortened to cra...

Mar 11, 2021 · 7 min read · krish.hashnode.dev  
Read the article

-   [#side-project](https://hashnode.com/tag/side-project)
-   [#python](https://hashnode.com/tag/python)
-   [#python3](https://hashnode.com/tag/python3)
-   [#web-scraping](https://hashnode.com/tag/web-scraping)
-   [#scraping](https://hashnode.com/tag/scraping)

## Written by

[Krish](https://hashnode.com/@lowcodekrish)

@lowcodekrish

  
[View profile](https://hashnode.com/@lowcodekrish)

I build products in Web2 &amp; Web3. Develop custom APIs &amp; integrate disparate services. Also, a cyber security enthusiast in my free time.

## Responses 2

Markdown supported · ⌘↵ to post

[Sholanki Biswas](https://hashnode.com/@ShoBiz)

[Mar 15, 2021](https://hashnode.com/posts/scrapy-your-neighborhood-spider/6049b638258b6d5f4339f677/comment/604f082137d84907f1329eca)

Interesting read Krish. I will try this one out to see what I come up with. Thank you for explaining what you are actually doing in code. I am wondering if this is also one part of lighthouse searches. Quite trippy!

[Krish](https://hashnode.com/@lowcodekrish)

[Mar 16, 2021](https://hashnode.com/posts/scrapy-your-neighborhood-spider/6049b638258b6d5f4339f677/comment/604f082137d84907f1329eca#reply-6050f1c5416f221f065b713f)

What do you mean by lighthouse searches?

[Neha kumari](https://hashnode.com/@Nehakumari)

[Mar 11, 2021](https://hashnode.com/posts/scrapy-your-neighborhood-spider/6049b638258b6d5f4339f677/comment/6049d8e3258b6d5f4339f8e8)

Very interesting read. I'm curious to give "scrapy" a try. Your article prompted me with an idea. If I could automate the python script to see where the search result of my personal website ranks every week, it could help me with the SEO for my website if need be.

[Krish](https://hashnode.com/@lowcodekrish)

[Mar 11, 2021](https://hashnode.com/posts/scrapy-your-neighborhood-spider/6049b638258b6d5f4339f677/comment/6049d8e3258b6d5f4339f8e8#reply-6049dbe82f49f75f68d3ba19)

Great thought! So, to know your seo rankings I'm sure you'd be looking it up on some website. Isn't it? You could probably scrape those websites to retrieve the information you are looking for.

[Neha kumari](https://hashnode.com/@Nehakumari)

[Mar 11, 2021](https://hashnode.com/posts/scrapy-your-neighborhood-spider/6049b638258b6d5f4339f677/comment/6049d8e3258b6d5f4339f8e8#reply-6049dd70258b6d5f4339f9a1)

[Krishanu](https://hashnode.com/@krishanu22) : Yes, you're right. It usually is based on the Google search result. So, I guess I can scrape the result from Google. I will give this a try. Of course, there may be OOB tools available for this. Google Analytics being the simplest one of them. But, I'm curious to try this as a use case. Thanks for sharing your knowledge.
