You have a problem in your fuel system. The pressure doesn't build up fast enough (weak pump), or you have a leak some place that is preventing the pressure from remaining after the pump is turned off (leaky injectors, cracked hose near the pump inside the tank). You need a fuel pressure gauge to test the system, and to determine where the problem lies. Next time you try to start it, do this:
Turn the key ON, but do not try to start it. Wait 2 seconds, and turn the key off. Wait 2 seconds, and turn the key back on (but don't try to start it).
Wait 2 seconds, and turn the key off.
Wait 2 seconds, and then try to start it. If it fires up quicker after doing this, it is definately a problem in your fuel system.
Take care,
~Chris