You're making it seem more inconsequential than it is. I'll put it more simply, let's say if an LRV approached an intersection where there was no signal priority activated. If a light was about to change to red and the LRV was just a couple dozen metres out, the light would change to red and the LRV would be forced to hold for the next green signal. Multiply that over the widely spaced traffic lights, that will get less widely spaced out as the Golden Mile stretch becomes more urbanized, and that results in longer travel times than what you stated above.
And that's just one instance, because the lack of signal priority comes into play in more scenarios than what I illustrated above.
First of all, hitting each and every intersection just when it turns red will be a very rare misfortune. Let' say for simplicity that the chance of hitting red at each intersection is 1 in 2. Then the chance of hitting 6 reds in a row will be 1 in 64, or less than 2%.
Furthermore, fully grade separated does not equate a completely sterile environment. Signal problems, crew changes, alighting/boarding delays, fire at the track level (or detectors giving alerts even if there is no actual fire); all sort of things happen and cause delays unrelated to surface traffic signals.
If the subways typically clock ~32 kph travel speed, and buses clock ~18 kph, then the estimate of ~23 kph for surface level dedicated-lane LRTs is pretty reasonable. Even in the denser environment, LRTs will have larger stop intervals than buses, and will not be blocked by general traffic while running between the traffic lights. It would be strange if those two factors didn't give LRTs any speed advantage over buses.
In fact, I suspect that the LRT speed through the Golden Mile will be higher initially, perhaps 25-26 kph, and then will drop to ~23 kph as more traffic lights are added.
In summary: yes it is a factor, street-median will be slower than tunnel or guideway, but this is not a factor that totally alters the quality of service. Most of the riders will hardly notice, they will just pad their travel plans by 5-6 min compared to what they could have if the line had a fully exclusive route for its whole length.