Is there anyway I can fight this guy in Chapter 5 without trailing around Spira looking for cactuars?
Nope. Once you’re in chapter 5 you can’t fight him any more until you finish your cactuar quest.
To lighten the load a little, keep in mind that you don’t even have to try to win the little mini-battles against the Cactaurs if you don’t want to. But you still have to find them; look up the locations if you want to get this quest out of the way quickly.