I could see the robes being the darb of the Masters, and the sleeker ESB uniform that of the Jedi Knights. Still doesn’t fill the holes your theory puts into the prequels though.