I've got what is essentially a binary planetary system, although done as a very large moon orbiting the primary planet rather than using the binary orbit tool.
However I can't manage add any other moons that will maintain an orbit.
I go to the add tool with my main planet selected, select moons, and select any of the default moons [generally ones around 300-3000km in diameter].
When I place the mouse around the planet it displays the moon with the short trail indicating where it will travel, the arc looks correct for a circular orbit around my planet at the given distance. But after placing the moon, when I hit play they aren't even close to orbiting. They either quickly degrade and get suck into a collision or near collision with the planet, or take a weird path of a very light arc with repeated humps of approx 45-80 degrees.
If I select them and click auto-orbit, it adjusts their parameters but they still do pretty much the same thing and don't orbit the parent planet.
Time step is the default 2 minutes. My main planet is about 1.4 Earths mass, and the large moon is about 1.02 Earths in a close (53,000km) 21.6 hour orbit. It's orbit is completely stable.
komatii