For the tasks that my Pi computers do, I have never noticed a difference. I run some pretty intensive processes (airplane radar, arcade games, wifi monitoring, etc.). If your Pi is absolutely starved of memory, it might make a difference of course. But it is a tradeoff I am willing to make.