I've always thought he was talking about both himself and Marianne here. When the colonel says this:

"Does your sister make no distinction in her objections against a second attachment? or is it equally criminal in everybody? Are those who have been disappointed in their first choice, whether from the inconstancy of its object, or the perverseness of circumstances, to be equally indifferent during the rest of their lives?"