He was working with Jack Buck, and CBS boss Bob Wussler thought the two commentators sounded too much alike.
NPR: Super Voice Gone: Pat Summerall Dead At 82
2.
Summerall told Wussler that if a change was going to be made that he'd like to do play-by-play, and the following Sunday that's what Summerall was doing.