Performance is something to optimize after all of the basics are in place, including (but not limited to) accessibility. It seems premature to optimize something without basic keyboard and screen reader support. There are some improvements and optimizations that can be made for accessibility early on, however, such as using native HTML elements to limit the amount of code that gets sent over the wire. My research and conference talks might be relevant as I go into what can impact both keyboard and screen reader users:
marcysutton.com/accessibility-and-performance
https://www.youtube.com/watch?v=RkbyP-OJRgw