Everyone has had an experience in elementary school where someone was left out. Left out of a birthday invitation or left out of a game on the playground. It sucked as a child and it’s even more hurtful and damaging as an adult.
Continued exclusion of your co-workers creates a hostile work environment for everyone.
It wasn’t fair to exclude people as a child. If your parents were anything like mine, you probably got scolded for excluding them, regardless of your reasons. It is even more unfair and beyond that, disrespectful to exclude people as an adult.
If you are going to invite the whole office to a party or get together, make sure you invite the whole office. Even the people you don’t necessarily like. Not only will it go toward creating a goodwill bridge with that person, but also you might find you like them a lot better in the non-office setting.