The amount/type of diving you do has a direct impact on your flying limitations immediately after the dive. There is no set time frame for flying after diving, and no fine line either. For example, if you do 4 dives in one day, each of them being deep dives to about 100ft or so, at about 20-30 mins each time, then by the end of the day you will be pretty loaded up on nitrogen. In that case you wouldn’t want to fly at all for 12-24 hours. On the other hand, if you do 1 dive to a mere 15-20 feet, for only 15-20 mins, then you are probably pretty safe to fly within a couple hours. Point being, the effects that going to high altitude after diving has on your body are a direct result of the type of diving you do, as well as, of course, your physical conditioning, and how well your body itself handles the nitrogen at high altitude (as has been said, everyone’s different). I myself, for my body type, have found that if I follow the no-fly times on my computer I am perfectly fine (my computer is relatively aggressive and I’ve been fine). I’m in very good shape and have never had an issue. Diving with nitrox helps to not load you as much on nitrogen as well. For the record, I dove regularly for a year while living in an area such that I had to drive over a 1,200 foot mountain pass to get down to/from the ocean/my house and never had an issue. I think we would all agree here that the best thing to do, to play it safe is to follow tables and stay within your limits, since there is no hard line rule for when you should/shouldn’t fly.
So all in all, your idea of flying from dive site to dive site sounds fun! Maybe couple it in with some overnight camping at the first site to bleed off some nitrogen before flying to the next site?