No but....
It doesn't inherently make anything slow. The code you write using jQuery might make the site slow.
Minified and gzipped, its about 35kb or (1 jpeg). So downloading it probably not, but parsing it (especially on older mobiles) maybe does.
You might not need jQuery depending on your browser support or what you are trying to achieve, but then you lose the sugar syntax that makes JQ amazing and popular. I think JQ 2.0 still has 88 fixes for cross browser weirdness in "modern" browsers. So it might make writing the code for a site slower.
I woul;d totally use it not worry, but vanilla JS is super cool too.