A top Chinese quantitative hedge fund was ordered to halt new product launches after poor behavior by employees against its peers, as regulators tighten oversight amid intensifying competition.