In a nutshell:
- Optimize your images
- Reduce HTTP requests (combine CSS and JS files)
- Minify CSS and JS (and HTML if you want too)
- Use caching (Memcache, Varnish etc.)
- Links to CSS at the top, Links to JS at the bottom of the HTML
And basically everything that Steve Souders said:
stevesouders.com/hpws/rules.php