I'm not sure of the exact mechanics going on here, but I'd say its partly due to inaccuracies due to timestep errors since you're having to run it kind of fast to keep the outer moons moving at a decent pace (At least I assume so, correct me if I'm wrong) and partly due to the gravitational pull of the other moons making the inner one's orbit unstable in that they pull on it from so many directions that it gets pulled into a polar orbit because I suppose thats whats most stable considering the timestep errors...Not sure though, just my theory for this weird phenomenon...