A fruit farmer who claims his fortune may have been hit hard by Brexit is fighting his ex-wife over her £12.2 million divorce payout.