A penis does not dictate what you wear and how you act, same with vaginas. All they control is how you piss and how you climax. Sure, there are particular undergarments that are designed for either sex, but what you wear over your body should not be dictated by your particular appendages.
This means the only people who should be concerned about a person’s biological sex are their parents, doctors and sexual partners. So, unless you are one of these people, there’s no need to know if that person over there is male or female.