The problem I see with your logic, is that people who are making judgements have a pretty good grasp of the facts.
You are correct in that not everything is black and white.
However, many people see this as a strictly ethical issue, on on that basis, feel it is ok to judge another.
In regards to your last statement, it is what a person DOES that determines his value, not who he IS.
You are entitled to your opinion, and to act accordingly, but there are consequences.