You might want to look at the capabilities set up for "Authenticated User" too. I'm not certain but I think that a guest counts as being logged in, which means they get any capabilities assigned to "Authenticated User" as well as guest capabilities.

If that doesn't work, can you give more details on exactly which things you don't want them to see?

Simon