It's a tricky system; it never quite feels intuitive! But you're absolutely right; the later games may not make sense if you haven't played the side games, but they're going to make just as little sense if you have.