Because of time-reversal symmetry, two-dimensional topological insulators support counterpropagating helical edge modes. Here we show that, unlike the infinitely sharp edge potential utilized in traditional calculations, an experimentally more realistic smooth edge potential gives rise to edge reconstruction and, consequently, spontaneous time-reversal symmetry breaking. Such edge reconstruction may lead to breaking of the expected perfect conductance quantization, to a finite Hall resistance at zero magnetic field, and to a spin current. This calculation underpins the fragility of the topological protection in realistic systems, which is of crucial importance in proposed applications.

• Received 29 September 2016

DOI:https://doi.org/10.1103/PhysRevLett.118.046801