There are arguments supporting both schools of thought but consider this... That fact that you have to "try to make it comfortable for the mobile users" should throw a red flag. Anytime you have to force the content or the UI to work, you can pretty much bet the users are going to suffer for it. Start to move away from looking at the process of building web based projects as responsive / adaptive, vs. static. View the process more as the modern way to build such projects going forward. Planning out your content flow is key. Understanding the hierarchy of the site and it's content structure will allow you to design better flow patterns for the various devices out there.
I recently had this discussion with my Wife who was working on a redesign project. I advised her to first analyse the content and start with mobile. That way she didn't have to try and force content to fit into the smaller form factor thus allowing her to design a better experience for the user. She was a bit reluctant at first because she, like yourself, likes to start with desktop and then try and solve the issues that arise with tablet and mobile. She found this approach easier since it allowed her to "expand" the site and the content as she moved from mobile, to tablet, to desktop.
In addition to this, whenever I design and build a site, because of the various breakpoints, I use fewer rules and media queries when I start with mobile. All the major UI styling remains consistent through out devices only changing when the content needs to be presented differently.
I hope this shines some light on the subject for you and good luck with your projects.