That's kind of funny because no it's not at all about semantics. This is a discussion about of pragmatics.

Semantics is concerned with context free meaning, but that's specifically (and ironically exactly) what we are not talking about. What we are talking about is context dependent meaning, that is pragmatics.

That is the entire point, no matter if it is words or clothes, context affects meaning.