Some of the harshest criticism of high immigration is coming from people who work for the exact organisations that have supported high immigration for decades.