The patch optimizes this case by converting the cached reference from a sequence number to a pointer to the highest packet in the queue that has been SACKed in the past.
However, those larger average Windows also mean the SACK receiving code needs to do a lot more work for every packet received as the queue to be scanned contains more packets.