The shock departure follows months of rumoured talks with servicer Vertex which was understood to be the sole bidder for the Skipton-owned firm. It is understood that the talks have since fallen through.
Andrew Jones, currently chief operating officer will act as interim chief executive.
Richard Twigg, group finance director of Skipton and chairman of HML, said: "We would like to thank Brian and Neil for the enormous contribution they have made to HML over the last three and a half years and wish them every success for the future.”
HML confirmed it will be recruiting to replace both Warman and Brodie.
Jones said: "We will enter 2012 with a stable and sustainable operational and financial base, giving us the right platform to develop and grow, backed by Skipton Building Society."
Earlier this year HML lost two high profile accounts as both GMAC-RFC and Nationwide took their £3.6bn and £2bn mortgage books in house.
And after the Financial Services Authority fined Kensington £1.225m over its arrears handling in April 2010, it took its special servicing back in-house.
HML has cut roughly 550 jobs since March 2010 bringing its total workforce to just shy of 1,500.
It currently services a £43bn mortgage book on behalf of other lenders but doesn’t own a stake in these assets. At the height of the boom its assets under management were valued in excess of £60bn.
