Just like @apertureless, I'll assume you mean a full-fledged JavaScript front-end framework, and not one that will mostly only give you a head start with respect to styling and layout (like Bootstrap).
I guess the “big enough” point is a symbolic one where the cost of learning, setting up, and adhering to the chosen framework becomes less than implementing and maintaining the necessary features yourself. However, I think it's often easy to underestimate how much it will actually cost to do all the small details yourself. For example, most of these frameworks, in contrast to your own solution, will already have been tested and fixed by “thousands” of users and developers. It is really naïve to belive that you won't need to fix at least a fraction of all those small wrinkles and bugs that they had to fix; and that cost can become huge. And, given that the majority of the cost of developing a software system is maintenance (see for example The Mythical Man Month, by Fred Brooks Jr.), using a well-maintained, open source, front-end framework can save you years of effort, in the long run.
On the other hand, the initial cost of learning, setting up, and adhering to the models and methods of the chosen framework can seem very high before you've made the walk. But, once you've made the investment, the cost for those activities will be much lower for your next project, because you've already gained so much experience from the first project you used it.
Based on these considerations, I would suggest that if the spec for the front-end features is very small and is not expected to change much, then you could consider developing the front-end without the use of third-party frameworks. However, if you chose not to use a framework, you obviously don't get any of the pros of using one, and you surely won't gain any of the experience that would have made it much cheaper to use one in the next project where you'll need to decide whether or not to use one.