Ahh... I re-read your question and I think I understand the problem now.
If you set two planets to have a binary orbit of one another, they're not given the necessary velocity to orbit the star.
Try this in a simulation with two planets very near each other and a star:
1 - Pause the simulation.
2 - Select just the two planets.
3 - Click on binary orbit
4 - Click on create barycenter.
5 - Now just select the barycenter and the star.
6 - Click on binary orbit
Let me know if that works for you.
No, this works!