---
title: "Discussion on \"No method has been shown to predict what software to build. So I'm backtesting mine on three years of store data\" | Hashnode"
description: "Discussion on \"No method has been shown to predict what software to build. So I'm backtesting mine on three years of store data\". Update, Oct 9: I wrote this on Sept 29. The first backtest finished on Oct 1. One signal first looked 13x better than random; compared against needs of the same size, it was about 2x; with finer size"
url: "https://hashnode.com/posts/no-method-has-been-shown-to-predict-what-software-to-build-so-i-m-backtesting-mine-on-three-years-of-store-data/6ac83de63ea584ea9edc7f40"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [No method has been shown to predict what software to build. So I'm backtesting mine on three years of store data](https://reidstonji.hashnode.dev/no-method-has-been-shown-to-predict-what-software-to-build-so-i-m-backtesting-mine-on-three-years-of-store-data?utm_source=hashnode&utm_medium=feed)

Update, Oct 9: I wrote this on Sept 29. The first backtest finished on Oct 1. One signal first looked 13x better than random; compared against needs of the same size, it was about 2x; with finer size

Oct 9 · 7 min read · reidstonji.hashnode.dev  
Read the article

-   [#ai](https://hashnode.com/tag/ai)
-   [#chrome-cj73auo4o0012c3wted1yb7a1](https://hashnode.com/tag/chrome-cj73auo4o0012c3wted1yb7a1)
-   [#buildinpublic](https://hashnode.com/tag/buildinpublic)

## Written by

[Reid Ji](https://hashnode.com/@reidstonji)

@reidstonji · Hangzhou, China

  
[View profile](https://hashnode.com/@reidstonji)

Building is cheap now. Knowing what to build isn't. Data picks, AI builds, every result goes public — misses included.

I'm a founder building with AI, and I run the "what to build" question as a public experiment. An AI system reads the market (30k+ Chrome extensions with real users and ~1M user reviews) to find demand that isn't being served, rules ideas in or out, and predicts how each one will do. Then AI builds them, and I publish the results: installs, revenue and the misses. Weekly notes: https://reidstonji.substack.com

## Responses 2

Markdown supported · ⌘↵ to post

[StackShortlist](https://hashnode.com/@veristria)

[Oct 17](https://hashnode.com/posts/no-method-has-been-shown-to-predict-what-software-to-build-so-i-m-backtesting-mine-on-three-years-of-store-data/6ac83de63ea584ea9edc7f40/comment/6ad2fed9874abf8e984c785b)

The update is the most useful part: a signal that first looked 13x better than random drops to about 2x against needs of the same size, then 1.4 to 1.6x with finer bands, and still in-sample. Publishing the misses alongside the wins is what makes the experiment worth following.

[Reid Ji](https://hashnode.com/@reidstonji)

[Oct 17](https://hashnode.com/posts/no-method-has-been-shown-to-predict-what-software-to-build-so-i-m-backtesting-mine-on-three-years-of-store-data/6ac83de63ea584ea9edc7f40/comment/6ad2fed9874abf8e984c785b#reply-6ad333fbbbe835398831fb2d)

Right — in-sample is the honest caveat. The finding is the drop itself: a raw signal looks great until you control for need size. So it's a hypothesis to test out-of-sample next — and I'll publish whichever way it goes.