The cleaner was accused of treason when the wife found out about the affair