First, there is a clear difference between Labour and Tories immigration stance and policies.
Secondly, about a third of Labour voters in 2019 supported leave, a big chunk of former labour voters switched to voting Tories or Reform over brexit - and immigration was significant reason behind that.
If Starmer, a remainer, is moving to the right with his rhetoric, it is just as much likely because on issues like immigration, a significant chunk of labour voters have already moved that way.