This is an ongoing issue I have with Google - both as a website owner and as a regular user of their search. Too often the search result is a click or two away from what I feel is the obvious, best page for the search I made.

Just this week, I've been seeing some shifting in this area - usually whemn there are indented results. In the past, the root domain would very often be on top but the indented result was the best page. Now sometimes I do see the internal page on top. I hope this means they're working on it.

As for what you can do - well, look at everything you know of that optimizes a page, I guess. Just one direct backlink from another domain might help a lot. But definitely tread gently with onn-page changes. If (when) the algorithm shifts again, you may find a new situation where your original content would have done just fine, but the tweaks you made now bring on a penalty.