“For me, it just didn’t work out schedule-wise,” Brock Nelson said of his decision to skip the U.S. men’s team’s White House visit this week.