Question: Did you allow "public access from all networks" on your key vault's networking config?
Or did you use self-hosted agents, where the agent is in a virtual network that have access to the key vault?
I'm thinking those are the only 2 ways our pipeline can have access to our keyvault but I might be missing something.