I can accept the loss of the Temple building as a simple matter of economics (and facadisms were not in vogue in 1970). Had Toronto decided to focus its new office towers elsewhere, we would have a lot more of it left (unlike say, Montreal). But when it's replaced with something equally as good, it tends to dull the pain. We lament over the Temple Building, because while Queen-Bay Centre is an improvement in terms of income for its owner, it's not an improvement from an architectural point of view.
We don't lament so much for Carrère and Hastings Bank of Toronto building, because we got Mies's TD Centre in return.