To be fair, Ryuji was always a bit inconsistent when it suited the plot. I think it only kicks off in earnest once his issues with Kamoshida are no longer part of the dramatic plot, but it peeks out even earlier. For example, when he's all ANN IS TOTALLY KAMOSHIDA'S PIECE one day and then points out—the very next day!—that she wouldn't do that, he's known her long enough to be sure what sort of person she is. Like, dude, it can't be both.

That said, I do agree with both your points. I think the story is strongest during the Kamoshida arc and after Haru shows up because there isn't time for "comedic" nonsense.