I prefer writing my own. For me passport was always blocking me.
Simple example. I tried two-factor authentication with passport. Needed like 5 hours and it still didn´t work. Completely removed passport and implemented two-factor by myself, after 30 minutes it was done working.