It's really all about perception though, isn't it? Doesn't matter what the actual stats are, people choose where they want to be based on their own biased perceptions. And how do we change that????? That's the big question.
I have the suspicion that there is also more to it; using 'crime' as a factor is more like a scapegoat or the final blow to a decision that gets all the attention. I think people have a list of things they don't like about downtown, they've already created a negative opinion. Then they hear of one mugging and its like... 'yup, nope, never going downtown for anything'.
For me personally, I don't feel unsafe but I am disgusted with passing by people shooting up or smoking glass pipes in broad daylight, not even hiding it. I don't want to see it, I don't want to be near it, and it puts me in a negative mindset. So I can see why people don't want to go downtown for work or events.