And then it becomes my problem if my feelings are hurt because it's just meant as a joke. And if I'm not amused it means that I can't take a joke and no one wants to be that person. So I smile and say, "Of course," and allow the perpetrator to carry on guilt free.
I'm inclined to think this whole "just kidding" business is actually a mild form of emotional abuse... and depending on the circumstances maybe even not so mild. Under the guise of humor, a quick assault is launched and executed. It's conscious and it's cruel, and yet, it slips by. It is allowed and even condoned, because of the two words that follow the jab- because of the "just kidding." Let's strike them from our usage. We can have fun and be silly, but let's not make a joke out of ourselves or anyone else.
If I am really kidding, people will know it, and if I am not, then saying "just kidding" is a sorry excuse for a stinging barb. I choose to be kind instead.