That's a very technical POV. Accessibility for your websites, for example, may be unnecessary from this POV, because it slows down your developers. However, I think we all agree that it is our duty to not exlucde people with disabilities. Same goes for supporting old browsers. Take mobile browsers, for instance. There are a lot of manufactures that don't ship OS upgrades for their devices. Should we exclude these users and require them to buy new devices, creating a lot of unnecessary electronical waste?
For me, the critical line is crossed when the manufacturer stops shipping security updates for the software. Than we should also refuse to support these legacy browsers, because they are a risk to the user's security.