I think it's more the fact that to those that don't believe in prayer, you are perceived to be doing nothing. Getting involved is appreciated. Donate. Give blood. Etc.
Or perhaps just discussion about truth? I recommend calling The Atheist Experience between 4:30pm and 6pm CST on Sundays. More information can be found here:
http://www.atheist-experience.com
Have a good one!
Carly