There are quite some things that can cause strange behaviour, I'm not familiar enough with Swift/Metal to know if they apply...
Could you have allocated memory that wasn't initialized? I don't know that exists for Swift, but some compilers have flags to initialize all memory. You could initialize all floats as NaN to more quickly catch use of uninitialized memory.
Could there be dangling references (accessing memory that is no longer used)? There are also compiler flags for that in some languages (mudflap). Maybe a problem with Swift's reference counting and a pointer still existing that isn't being counted?
Is there any concurrency? That's a common cause of non-determinism. If it can be temporarily disabled, that'd be a good way to test if it's happening. If it's happening, I hear Go has a race detector, but I don't think that's commonplace...
Or it could just be a bug where things are being modified that you didn't think were being modified.
Some languages are better at preventing these problems than others (and prevention really is preferable for this), but I doubt you have the choice of rewriting in Rust or Haskell or something...