We just returned yesterday from a 6 week trip west which included Rocky Mountain and Grand Tetons. If you want to be sure to avoid crowds, plan to get up early. (sunrise). If you have to choose, my personal preference was Rocky Mountain. There is more variety - hike options, scenic drives, overlooks, wildlife and in mid/late Sept, you have the bonus of fall aspen colours and the Elk bugling.
RMNP: The west side of Rocky Mountain is less crowded, but less to see/do. Most of the must do hikes are off Bear Lake road, which requires a timed entry. There are also many other hikes elsewhere in the park. Camping options are somewhat limited. We stayed 4 nights at the Jellystone near Estes Park. Convenient location, but campground is on a hill, which has its challenges.
GT: My observation was, with the exception of Jenny Lake area, GT seemed less crowed than RMNP. I did prefer the 'must-do' hikes in Jenny Lake (Inspiration Point/Cascade Canyon) more than RMNP Bear Lake must-do hikes (Emerald, Gem lake etc). We stayed 4 nights at Colter Bay RV (book as early as possible), which is conveniently located. With this campground. the plus side is you are less than an hour from the west side of Yellowstone. (We toured Yellowstone separately, staying at Fishing Bridge in the park).
You can't go wrong with either choice