No one speculates that anything good is going on. This is namely because we each judge ourselves according to our intentions and we judge others according to their behavior. And the more authority a person has the more the "magnification effect" is at play, making each simple or harmless behavior take on new (and often sinister) meaning. It's human nature.
Be bigger than your human nature. Choose to be generous with your explanations and clarify or confirm with people directly. It will save you hours of speculation and add years to your life.