Yea, I think it was the ignition switch. I got it changed out and haven't had a problem since, but I have only started it a few times since the new switch was in. But, it makes since that sometimes the switch was sticking and causing my problem.
I got the new switch put in today, but man it was a pain to hunt down a security torx bit to take out the old ignition with. Finally found one at a hardware store and the job was pretty easy from there out. Guess I got a new starter now too, cause I am not crawling under there and swapping them back out
. Thanks for all your replies!