While I was trying to make Earth have a REALLY close encounter with Jupiter (by having it in orbit of Jupiter) in a highly eccentric orbit, I noticed both the rotation increasing (tidal forces I guess) and the SMA changing. So, I decided to make a new one to see what was happening.
My setup is Jupiter with Earth in an orbit at 220 day orbit with 0 eccentricity and I upped the timestep and saw that the orbit was going outwards, but when I got up to a timestep on the scale of days, it stopped changing. I found the limit to be at about 15.9.
Also, I'm no expert on this, but it seems like it's changing too fast? Then again, Jupiter.
I already submitted an in-program bug report.
Edit: I just tried it with the moon MUCH closer to the Earth and it shows on timesteps of days and up, so maybe it's a Jupiter or gas giant thing?