The problem with the idea of white privilege might be a marketing problem. Here's why:
If I'm a white guy, I can't just use the justification that I have free speech and good intentions enough that I should be able to use the 'N' word in certain context. The fact that the 'N' word is offensive to most black people is enough for us as a society to decide not to use it or condone it.
So, in that instance, it is not what the white guy thinks is important, it is the idea that word is offensive to black people that overrides in the decision about it's permissibility.
Mostly liberals want to make the case that systemic inequities based in race should be addressed to make society better. That's an idea that is plausible and should be able to get enough support and attention to make improvements.
The problem is how the idea is being communicated. Whether liberals understand it or not, the term 'white privilege' is derogatory to white conservatives. Privilege is being used to suggest entitlement, unearned status, and unfair advantage. Those are negative attributes that diminish any accomplishment of the people that are being generalized by it. And because it's based on race - it is also bigoted. You would not, for example, use the term 'black' followed by something derogatory. That would be bigoted.
Furthermore, whether it's intended of not, the term 'white privilege' is perceived by white conservatives as a microagression. It is a way for liberals or people of color to denigrate them under the guise they are promoting social consciousness.
Again, just like the use of the 'N' word, it doesn't really matter how the people that want to use it feel about it or want to justify it. The idea that the term is offensive to white conservatives should be enough to discontinue it's use, it the left truly wants it considered.
That does not suggest the idea should be abandoned. My suggestion is to reframe it so it's less divisive and more open to discussion.
Instead of focusing on the privilege of white people (which is divisive and bigoted) focus on the difficulty faced by people that liberals feel are disadvantaged.
'Black struggle'? 'People of Challenge'? I don't know what term would be acceptable for liberals that want to make the case. I do know that continuing to sell this message in divisive and bigoted terms will not work.