What do I have to feel guilty about?

For non-believers, guilt is a poor strategy.
Unless someone gives some value towards the words of the bible or the church, then guilt won't work.

Why?

Because I don't feel guilty about staying in on Saturday and not going to synagogue.
Nor do I feel bad about skipping Islamic prayer times.

But I'm not an observant Jew or Muslim.

So why would someone who's a non-Christian feel bad about missing church or having sex before marriage?