When you find yourself in a situation where you have to decide what happens next, you must be completely honest, with everyone concerned, including yourself. When that decision is possibly not what others expect or want, you must be compassionate when you announce it.
Of course you can hurt others if when your decision is contrary to theirs, but you will hurt them more by delaying or by going along with them, just to go with the flow. Eventually, your feelings will show and the result will only be more angst for all.
So when you feel that you have to make a decision that will be difficult for others to accept, make it quickly, announce it gently but firmly, and reduce the pain, to you and those affected, to the minimum.
Having made the decision, you must stick to it. There can be no going back, and doing so only makes it worse for everyone concerned.