Suri Cruise turned down Tom Cruise after being abandoned, to enjoy a simple dinner with Katie Holmes