But the Robot works with screen coordinates instead of Swing component references, so using it directly in tests will make the tests fragile, meaning any change in layout will break them.

With the right tools, the right motivation, and often with someone helping you out directly, one-on-one, you can change something you want to change.

There are many reasons why exposing public fields directly is a bad habit, the most important of which is that it leaves you with no options should something in your implementation change.