The value isn't that limited.........
A base reference would be the purchase price CF paid for it, which was 650M .
But current values per buildable ft2 are trading lower than they were then.
Right now, with the office market down and the condo market dire, the value is likely somewhat depressed below the earlier valuation, for which CF may have been guilty of overpaying. (They really wanted to add Sak's at TEC; an idea that probably wasn't worth paying for...)
The Bay's lease on the premise runs through 2039, so absent a buy-out, nothing is happening near term anyways.
The tower just got re-clad (molested)
The store's subbasement levels are currently occupied by Metrolinx doing work on the O/L
That pretty much puts a hold on any major re-think until they're in wrap up mode, in six years, maybe.
***
Add to the above, your legitimate points concerning heritage preservation, shadowing of NPS, and the prospect of working on a site whose value would only be maximized by removing an existing office tower, and then having to work next to two operating subway lines..... and a station to which you would have to preserve public access during construction.