7. Say “I have an early morning tomorrow, so I need to get some sleep” and specify your reasoning to him honestly
“This is very understandable, so I won’t feel rejected.” You will avoid hurting him once you clearly lay out your reasons. When you turn him down, make sure you give a credible explanation.