To oversimplify, Toronto post-1945 demolished most of the financial district, Regent Park, Moss Park and St. James Town. Everything else is basically intact.
That's more than an oversimplification, it's an over-reach.
The St. Lawrence/Old Town Neighbourhood is more new than old. If we're looking at Yonge to Parliament and the railway to Richmond or Queen....
It's not just the Board of Trade/TTC building at Yonge/Front we lost, a straight walk out Front shows an area that should be our heritage centre piece (outside of the signature buildings we lost in the Financial District).....
is littered with stuff like the St. Lawrence Centre for the arts, the Condos on the east side of Jarvis opposite the St. Lawrence Market, the Time and Space nonsense from Pemberton and so much more.
Saying it's intact simply isn't right. Now, some of the replacement architecture along the Esplanade, or at Market Square could reasonably be said to be context sensitive and generally somewhere between appreciated and inoffensive, but it
certainly isn't the history that was once there. (let me acknowledge here, that many of the current buildings replaced parking lots, and that better quality heritage was often lost decades earlier)
I noted above, I'm not particularly keen on the 'heritage' here, and don't really have an inherent problem w/losing it at this point, save and except for the current proposal being a clunker.
That, however, doesn't change the reality that what most people want is to see the best of 'vibe' they like about an area preserved. Most aren't tied to the interiors of buildings, unless they were particularly grand, and well preserved.
The desire is to keep the area similar in 'feel' as they walk along the street. Simple use of scale and material at the podium level addresses much of this, it could be pastiche, but it needn't be. The Ace Hotel being a great example.
In general if you can address the podium; then you can produce a fairly modern tower in behind as long as you can tuck it back enough that it doesn't visually intrude at street level; alternatively you can try to build a tower with higher quality materials, I might off The Selby as a good choice.
That doesn't mean there isn't a place for modern architecture. Just not on every block or in every neighbourhood.
Let me blow up a section of the map above. Remember green/blue is contemporary architecture.