My longest job-stint was with a local company that was 100% self hosted.
Our server room was about 40ft x 20ft - it's own air conditioning units; fire suppression system, etc... the building had a Diesel backup generator and we did use it from time to time over the 8 years I was there.
We ran it all - from Microsoft Exchange to Windows AD / print servers; our own PBX, a mix of Dell Power Edge Servers / HP switches and some off the shelf PCs for smaller tasks. When I left it was at about 6 42U racks full of servers (the PBX consumed 4 of them - about 700 internal phone lines)
The website was a traditional Lamp server that accessed the IBM AS400 mainframe over ODBC.
We did our own backups to LTO3 then LTO5 tapes; rotated them off site in case of disaster.
It was a lot of work and when stuff hits the fan, you only have yourself to fix it or blame but I got a LOT of education out of it. Not even the cleaning crew was allowed in there; we swept (no vacuums / static charge == big no no)
To answer the question - what was it like - Really, no different then hosting a website on any other host - just now, you need to check the servers and make sure there aren't any hard drive failures; make sure backup runs; etc... An internet connection going down is the same as AWS or Rackspace going down; it's just now up to you to figure out if it's you (the site) or the telecom company (often, it's them)
I still on occasion recommend self hosting - recommended it to my current day job to save some money but we don't have the room for a generator or the room for a proper server room.
It's as small or as complex as you want it to be. It can be 1 tower server behind lock and key or it can be an entire room under scheduled badge access with fire suppression, security cameras, generators, etc...