This is interesting! Lately I've been experimenting with running some smaller models locally, and have done a lot of reading and playing around. I wonder if you might be able to use a controlnet to specify parts of the image that should contain a dinosaur sprite? I also wonder if going even smaller than 512x512 could be beneficial in your case? Smaller images can generate quite a bit faster, so you could use them to more quickly generate the new parts of the map that scroll onto screen, just giving them a bit of overlap with the existing image instead of passing the whole thing. Though, most models aren't trained below 512x512. You CAN make one generate at lower resolutions, but if it's not trained at that scale, it can get a bit odd. I think there may be a 256x256 (or a 512x512 that had additional training done at 256x256?) floating around somewhere iirc? Any smaller might be too niche for pre-existing solutions, though.