---
title: "Discussion on \"Greenfinger 2.0: One url in, a searchable archive out\" | Hashnode"
description: "Discussion on \"Greenfinger 2.0: One url in, a searchable archive out\". A distributed web crawler for the JVM. One pass writes files, a full text index and vectors. Search by words, by meaning, or by describing a picture."
url: "https://hashnode.com/posts/greenfinger-2-0-one-url-in-a-searchable-archive-out/6ab8f55a943a730c5bfca3a1"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Greenfinger 2.0: One url in, a searchable archive out](https://fredfeng.hashnode.dev/greenfinger-2-0-one-url-in-a-searchable-archive-out?utm_source=hashnode&utm_medium=feed)

A distributed web crawler for the JVM. One pass writes files, a full text index and vectors. Search by words, by meaning, or by describing a picture.

Sep 27 · 15 min read · fredfeng.hashnode.dev  
Updated Oct 1  
Read the article

-   [#webscraping](https://hashnode.com/tag/webscraping)
-   [#distributed-systems](https://hashnode.com/tag/distributed-systems)
-   [#nio](https://hashnode.com/tag/nio)
-   [#springboot](https://hashnode.com/tag/springboot)
-   [#elasticsearch](https://hashnode.com/tag/elasticsearch)
-   [#vector-database](https://hashnode.com/tag/vector-database)
-   [#playwright](https://hashnode.com/tag/playwright)
-   [#angular](https://hashnode.com/tag/angular)

## Written by

[Fred Feng](https://hashnode.com/@paganini2008)

@paganini2008

  
[View profile](https://hashnode.com/@paganini2008)

Java

## Responses 1

Markdown supported · ⌘↵ to post

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[Oct 1](https://hashnode.com/posts/greenfinger-2-0-one-url-in-a-searchable-archive-out/6ab8f55a943a730c5bfca3a1/comment/6abe611a8f44060ae8f5ec8c)

search never reads the database, cause with files as the source of truth replay should survive a backend swap or an analyzer rewrite. just double check that replay really pulls nothing from the metadata table. the scaling math looks off to me, since the books.toscrape.com run did 1000 pages plus 3204 images in 252s, which is about 16.7 fetches per second on one node, so images clearly skip the 1s pacing, and the config table says interval while the prose says fetchInterval the frontier looks like per node rocksDB, so a dead node strands its urls from the survivors, and resume only covers a process kill where the disk survives, which that leaked runningState will probably make messier.
