I don't have a specific answer to your question, but I do wonder whether the fact you are seeing increased load times is a product of insufficient caching or something, since jQuery isn't a particularly big library (~32k or thereabouts minified and gzipped).