If I understand this you want your website to rank for widgets but Google is ranking you for sprockets.

What specifically have you done to change the topic of your website?
What did you do with your old content?
What have you done for new content?
What did you do with your old backlinks & their anchor text?
What have you done to develop new backlinks & different anchor text?
Have you checked the competition levels of the new keyword? Maybe your site is only strong enough to rank for widgets and the sprockets industry is just more competitive.

The website is CMS based and was initially launched on topic A which has niche appeal. However the site has grown over the years to also encompass topics B and C which are much bigger and more prominent on the internet today than topic A ever was. For as long as I can remember topic B has attracted 75% of all the organic traffic the site gets from google.

Starting a couple of years ago I changed all the meta data of the site to better reflect my new desired focus on topic C and stopped looking for links to it from sites on topics A or B. I also focused heavily on creating content on topic C and relevant back links from sites related to it with appropriate anchor text.

1 year into this process I decided it was appropriate to rename the domain name from which the web site is served from a name including keywords X and Y which related predominantly to topic B to a name including (the same) keywords X, Y and (a new keyword) Z which changed it's relevance very heavily towards topic C. I also 301 redirected all the old URLs from the old domain to the new, and informed google of the change as discussed in the following thread:

[webmasterworld.com...]

Another 8 months later seeing maybe only a 10% change in favor of topic B. I moved all the content on that topic to a whole new web site and domain name (using the same old keywords X and Y but a different TLD extension), 301 redirecting all the old URLs on topic B from from web site 1 to web site 2. Website's 1 and 2 are linked to each other however and hosted on the same host and IP address.

Regarding back links I 301 redirected all those related to topic B to web site 2 and focused on legitimately generating as many as possible new links towards web site 1. I also did my best to have as much anchor text updated around the change as possible.

However 6 months on again, organic referrals are showing maybe another 15% to 20% increase in favor of web site 2 only and no change at all to web site 1 which is still attracting traffic mostly related to topic B.

Competitiveness is an issue. Topic B is not in a commercial sector while topic C is however web site 1 is definetely sizable and relevant enough to rank at least partially for topic C...

Are you an authority in these topics? I mean, a real standout, no question about it authority?

Definitely. Yes.

That's part 1. Part 2 is whether your site and the user behavior on that site reflects that. I don't mean to disparage your site or doubt what you say. But over the last fifteen years or so, I have asked my clients why they think they deserve to rank for particular search phrases, and they answer OF COURSE they are the authority, and in many cases they are - but the site doesn't necessarily send those signals.

Sorry, I don't have answers but I do have some stuff to consider.

Although it may not be the way Google looks at this, I think it's valid to be concerned about the fact that you are trying to move from a non-commercial set of referals to commercial ones.

Google clearly sees your site as trustworthy to some degree. However, part of your problem is that Google has been gamed so often by people (not saying you are) who alter the focus of sites that you probably need a great deal more trust from Google now to "get away" with a move into commercial territory.

If a 15 year old site that was #1 for "free widgets" (and had good reason to be) had a sudden change to targeting "premium widgets" do you think Google would trust the site as much for the new term? (I know your change has not been sudden).

If your site is a true authority in the commercial sphere then you could be just finding out how different life is building search traffic when there's money riding on the results.

Are you being outranked by other decent sites or are you behind sites that are genuinely poorer than top 10?

Do you know how much traffic to realistically expect for the commercial terms? Is there enough of it to become the majority of your visitors?

I have a site with a similar issue, but in my case it's because I didn't build it around a Google vertical: I built it around the related-yet-varied interests of a target demographic, in traditional marketing style. Google doesn't always know what to do with that, and seems to latch onto one particular aspect of your site without really "understanding" the whole. It's kind of like you're running a site about DIY car repairs, and thinking Google should send you traffic for "DIY auto repair" and related terms, but all you get is "DIY air filter change".

Not exactly what you're talking about, but maybe another example of how the algorithm doesn't always "understand" every site the way humans would.

I literally and truly built that site for humans, knowing it didn't fit any particular keyphrase or vertical. I relied mainly on social media for traffic, not Google. I believe the problem is that the algo simply does still need some guidance from "optimization", despite Google's hate for optimization. It's not really calibrated for sites that aren't calibrated for Google.

I'm fighting this to a degree on a site right now. The site is about widgets but one particular article is about what you can do with this widget and Google thinks my site is about that activity almost exclusively. If I post content about things not related to that activity they mire in traffic mediocrity but if I relate it to this one activity watch out, the floodgates open.

The problem, there are many activities related to this widget which people actively search for and I feel boxed in with just this one. Psst, Google, my site is about much more than this one activity!

It's not really calibrated for sites that aren't calibrated for Google.

yep. so much for the "build for the user not for google" mantra.

with my sites, i know that if they were more optimized for google like some of my competitors do, they'd rank higher. but i refuse this slimy approach in exchange to personal freedom to do what i deem right on my web property. the price is high: more often than not, if you really focus on the user experience without search engine aspects in mind, google seems to have no way to recognize the value of a website in a timely fashion.

for years, i had two websites - one content, one directory - that were interlinked with keywords. google sent all the visitors to the directory that contained the keyword links - not to the site with the actual content. result: more hassle for the user, disappointment, higher bounce rate. it took years that google finally recognized it should send more users directly to the content site - with content creation meanwhile being 95% of my working time instead of 5% for maintenance of the site with the linking list.

very frustrating to see this happening. googles' algo is very conservative, downright anxious. they take much too long (years) to detect and react on quality or topic changes. same applies to earning trust for new websites. what's good for the old establisherd webmaster who doesn't change a thing is bad for the active webmaster who likes to experiment and modernize.

Going back to my original question. What role do we think the human reviewers at google play in this kind of situation?

Little; probably none. That type of thing wouldn't scale, and Google is all about scale.

But might it take a human review to change vertical?

It might, Hugh. That, or an algo change that makes it "smarter" re: your niche. But the odds of that are long.

How smart is Google about niches? Does it classify sites in very broad niches, or narrow ones, or both: widgets vs gadgets, but not domestic widgets vs industrial widgets?

As far as I'm aware, the people doing the human reviewing don't have that kind of capability.

Hugh / Sgt,
Been exactly there and done that. Track my posts in one thread on here around mid-July 2013. As I realized then, your problem is more than likely NOT that they think you are about another niche subject, so much as they WILL NOT ALLOW YOU TO RANK FOR KEYWORDS IN/INCLUDING YOUR PRIMARY NICHE. Thus it looks like they have you in the wrong niche. Like Sgt mentioned, they pick up on all the secondary topics, which in our case, just like his, included tangential topics. I call it an "anti-authority penalty", where your site is so concentrated on the one topic that maybe they think you are trying to game them and thus lock you out of it (even though you may have BEEN the original authority for many years before G existed). If you look in that thread you'll see I was not the only one who came out of it back in July (fortunately). As far as how or why I can't shed much light on it beyond what was mentioned in the thread.
I did notice you also did some 301 redirs which we also had to do. Our site was split across 3 different domains for load balancing reasons (in the old days 90's that was the cheapest way unless you wanted to pay 10x as much to simply double your bandwidth) and we slowly moved a few pages at a time to the 1 primary page. Perhaps this makes them think you are trying to transfer PR from bought sites (even if you owned them all to begin with).
I spent 2 years of my life, that could probably have been much better spent bettering humanity and the website, trying to unravel it. Don't know if anything I did in that time made a lick of difference. But look at your site in that light and at least it may make more sense. I know it is extremely frustrating and they should be more sensitive to fixing the problem.

Thanks all, I really appreciate the feedback and suggestions! And I take some comfort in not being alone in this but must say that, despite this problem with organic referrals, I'm really pleased overall with the way things have gone after so many big changes. I guess will just have to keep running the site for my users, creating genuinely unique and useful content, and boosting my web sites' USPs until Google accepts the change.

I just found some not very high quality user generated content on topic b I didn't know about that was ranking way off into the long tail, now deleted and redirected! Thank you for clutching at straws google. Lets see what happens now...

Thought I should share that my fortunes have changed dramatically with website 1 seeing a 500% increase in organic search referrals on topics other than A and B 1 1/2 months after I picked up on the fact that irrespective of having moved and redirected 110000 URLs of quality content those didn't matter to google. What did matter were 30 to 40 remaing URLs which were still ranking in the long tail despite being shallow, poor quality and left over from the move. As soon as those were gone from our rankings things started to change for the better...