"Because I'm distracting you? We're out of time here, there will be work and duties and projects enough to keep us busy on the other side of the door,"

Yes, they probably should go back as Erik is rested now but he doesn't want to. He doesn't know when Erik will next trust enough to enjoy himself or choose to leave.