When Thoreau wrote that line, he was thinking of the Walden Pond he knew as a boy.

You say happiness is a warm puppy; I say happiness is living alone in a cabin at Walden Pond; etc.

Instead, the Harvard man seemed to spend most of his time loafing in the woods near his hometown over near Walden Pond.