I'm A Celebrity's Danny Miller sets record straight on leaving 3 week old son to do show