Kol people.
You already tried portuguese? The masculine/feminine distinction is far bigger than in english.
Like, english sounds almost neutral in this aspect to me
english: "the ball"/"the balls"
portuguese: "a bola"/"as bolas"/"o bola"/"os bolas"
In english, "a bola" or "o bola" would sound as the same thing! Here, "a bola" is a ''normal ball", while "o bola" sounds more like a nick for a fat guy
Trying to do a language even more neutral sounds like craziness to me! I mean, languages should be more simple and easy to learn and spell, but is the difficulty and variety that gives "a good taste" to them, at least in my vision.
I AM NOT SAYING TO NOT REMOVE THE GENDERS, but yes that, if you remove, you should add bits in other things to give it a more "natural" look.
A "natural look", in this case, would mean a look of a language that passed through a process of evolution, and its full of fails and errors, and unecessary things.