As long as it isn't a Texas necktie then it's fine to wear them with nice clothes. I wear them on occasions but they always feel like they're choking me. It's not weird at all.
In my personal opinion, I think it looks rather classy to be dressed to impress! If you enjoy the style, then, by all means, keep it.
I wore ties since I was six years old. School, military, business. Ties, ties, ties. After retiring the first time I swore I would never wear a tie, except for my youngest daughter's wedding.
Getting back into the land of the employed it was initially difficult without a tie but eventually I was accepted. On occasion when questioned by the powers-to-be as to why I was not appropriately dressed, suit and all but no tie, I would quietly mention it was against my religion (not true) and they would grunt and leave it at that never to annoy me again. (LOL)
It probably depends how you wear it.
I once saw a TV show in which a man had been refused service at a restaurant because he wasn't wearing a tie. He returned wearing a tie, but nothing else.
Along the same lines, in 1961 British Minister of War, John Profumo, was giving a party at his flat and answered the door wearing nothing but a ribbon. And, before you ask, Yes, that is exactly where he was wearing it.
So wearing a tie isn't weird, but it would look more natural if you wear appropriate clothing with it.