The haredi "gadol hador" (leader of this generation) does not want haredim to rent property to secular Jews.
This has nothing to do with how those secular Jews might effect the character of haredi neighborhoods. He opposes such rentals on the grounds that they might…
…be considered in halakha (Jewish law) to be helping secular Jews do "things" against halakha:
It is better when possible to not rent out apartments to people who do not keep Torah and Mitzvos, because it might be considered assisting him in doing things against the Torah while in the apartment. However, in this situation, where there is no chance a Haredi person would want to rent the apartment (because of it being located in a secular neighborhood), he has what to rely on and is allowed, bdieved [after the fact, a Jewish legal term that allows an action otherwise disallowed], to rent it out to someone who does not keep Torah and mitzvos.
What might those "things" be?
Cooking meat with milk in the apartment kitchen might be one. Sleeping with your wife when your wife doesn't use a mikva (ritual bath) would be another.
I suppose paying taxes that go disproportionately to feed, house and clothe haredim who refuse to work would be another, as would serving in the army to defend haredim who themselves refuse to serve.
I think secular Jews should refuse to rent to haredim. Not in retaliation, mind you – in self defense.