The main issue is calories, so if you load starchy foods with fat -sour cream and butter on a baked potato, for instance -or eat them in large quantities, the caloric load can add up.