AI/ML Google Developer Expert, AI/ML Cloud Champion, Angular GDE
I help developers succeed in Artificial Intelligence and Web3; Former AWS Amplify Developer Advocate. I am very excited about the future of the Web and JavaScript. Always happy Computer Science Engineer and humble Google Developer Expert. I love sharing my knowledge by speaking, training and writing about cool technologies. I love running communities and meetups such as Web3 London, GraphQL London, GraphQL San Francisco, mentoring students and giving back to the community.
Markdown supported · ⌘↵ to post
Zbigniew Łukasiak
I agree that In Context Learning is not really learning (in fact I would just call it reasoning by analogy).
But:
"In practice, they describe automation of prompt engineering: feeding outputs back as inputs, applying heuristic scoring functions, selecting better completions through sampling.
This is useful! It genuinely improves performance on many benchmarks. But it doesn’t transcend the pretraining distribution. The model is still generating next-token predictions, just with more elaborate scaffolding around the generation process."
How do you define the "pretraining distribution" that it is not changed by changing the generator? How it could not change but still give us better results?