the moons need to be close. Select them and click binary orbit.
Thanks Mudkipz.
I think I found out what went wrong. For some reason you have to place the binary moons really far away from the earth. A single moon, or 2 single moons can be set to orbit earth really close or relatively far.
Binary moons need to be relatively very far from earth cz otherwise they get drawn by gravity.
EDIT: No, it can't be done, eventually the binary moons fall on earth no matter how far you place them. It seems that we can not set an orbit for binary moons around a planet.