You’re either equating your comfort with freedom or someone else’s freedom with comfort- you can’t have it both ways. There is no truly free society- we all give up some amount of personal freedom for the functioning of a healthy & safe community. (Social Contract Theory.) Each of our personal freedoms exist only to the extent they don’t place undue burden on those around us or society at large. Yes, there’s a greater discussion to be had on this specific point, but in general, your perception of being a free actor is not the ultimate factor in how our conflicting interests are balanced.