When those words become a threat, someone else must now have their rights protected. This is mostly where the limitations on “Freedom of Speech” come from.

A good expression I’ve heard along these lines is “your freedom to swing your fist ends where my nose begins.”