Mr. Sorsby figures the Fed's withdrawal will prompt at least a one-percentage-point rise in mortgage rates, which he fears could squash recent glimmers of more demand for homes.

WSJ: Fed Plan to Stop Buying Mortgages Feeds Recovery Worries

If the Fed stops buying, "it would be the beginning of a crisis again, and we haven't emerged from the last one, " said Larry Sorsby, chief financial officer at home builder Hovnanian Enterprises Inc.

WSJ: Fed Plan to Stop Buying Mortgages Feeds Recovery Worries