Thanks bill

Keeps returning message "The URL does not exist. Please enter a valid URL."

Most likely because the tool uses a method or UA I block.

Still, I'd like to know the outcome so when I have time to check my logs, I'll see how much effort it is to allow an exception.

The LXR URL? I tried it from outside Japan from a few different countries and it seems to load. (Thought they might have Geo-restricted access) However, it won't fully load on my normal setup. Had to run it in a wide open browser due to all of the JavaScript trickery.

Most likely because the tool uses a method or UA I block.

Ya think?

:: wandering off to check, with preliminary detour to G### Translate because I couldn't find the /en/ button ::

[Tue Dec 19 22:04:11 2017] [error] [client 183.90.250.abc] client denied by server configuration: /full/url/path/here

and

2017-12-19:22:04:12
URL: /redacted
IP: 183.90.250.abc
Connection: close
Host: example.com

And that was all she wrote.

Idle query: What's the use of a test that relies on a robotic visitor that everyone would block?

:: wandering off to check, with preliminary detour to G### Translate because I couldn't find the /en/ button ::

Welcome to my world... ;)
The last thing most Japanese companies think about is the world outside their borders.

That IP looks like it's from Japan. Probably the NTT server farm. At least they don't seem to be skimping on the infrastructure. Are you blocking the UA, or Japan IPs?

The LXR URL? I tried it from outside Japan from a few different countries and it seems to load

No, the tool page loads. When I test one of my pages it says the URL is invalid. I block most known server farms, allowing some exceptions access.

The tool page does load, but I am getting the same message: URL site does not exist. Please enter the correct URL again.

183.90.250.0/24
183.90.224.0 - 183.90.255.255 XSERVER Inc JP

I do not block this IP range, so she can have at me.

Tried a friend's site and it worked.

Are you blocking the UA

Well, for a given definition of “UA”. I did say I was quoting the headers in full, but you thought I was exaggerating. Other absent or defective headers may get holes poked for them depending on UA and/or IP, but when the UA itself is absent ...

TorontoBoy, I think your friend's site needs some work on its access controls ;)

Anyone in this thread read Japanese? I'm wondering if it really says “this site doesn’t exist” or if it’s mistranslated from something like “the site couldn’t be reached” or “no data available for this site”.

My Japanese is not so good, so I google "translate japanese to english"

Yes, my friend's site is pretty much wide open, unless you continually scrape his site for a couple of hours. He says it's too much bother to monitor his access log.

Yup, checking logs shows the UA field empty, hence why the requests were blocked.

Since my pages will displays very differently depending on what is requesting them, a tool that doesn't declare this doesn't really have much use.

[fix typo]

[edited by: keyplyr at 8:08 pm (utc) on Dec 20, 2017]

Ok, if the UA is empty I do block.

if the UA is empty I do block

Yes, that's why this thread is reporting one “site doesn’t exist” after another.

Now, it’s true that there are differences in behavior between webmasters and ordinary humans. But this is about a tool that’s specifically targeted at webmasters, so it seems like somebody dropped the ball. (Keyplyr, I didn't even think of looking at access logs, but proceeded directly to headers, assuming there would be some minor deficiency. Logs, as it turns out, would have provided the same information.)

sharp increase in Chinese visitors to Japan amid Beijing's tightening internet censorship

Does this mean that people from China are increasingly looking at sites in Japan, because China's censorship applies primarily to sites hosted in China? I thought they also restricted access in the other direction: China is no North Korea, but you still can't view everything.

If a Japanese-language site has really dense, grownup-oriented content, would a Chinese visitor be able to pick out the kanji and guess at what the text is saying? (Paradoxically, a site aimed at children would be incomprehensible.) Or has there been a boom in Chinese-language sites hosted in Japan?

I can't speak very well for 1.4B people, but here it goes. China's firewall keeps bad Western influence out, regardless of country. There is no firewall within China, but then again China has great control over any and everything within. English is by far the most popular foreign language, but some people have learned Russian and Japanese

For Chinese people to reach banned sites outside they need to use a VPN, so the government banned VPNs, with the exception of gov't sponsored ones. VPN use, as with many other Chinese laws, are applied only when advantageous. Otherwise people just use them.

Historically Chinese people have never forgiven Japan for past military transgressions, and Japan has never admitted guilt. I doubt that Japanese sites are very popular with the majority of Chinese people. Every single year there are protests concerning these incidents, so time has not healed all wounds.

Man jailed 5½ years, fined US$76,000 for selling VPN in southern China [scmp.com]

It is increasingly difficult to maintain VPN service in China.

I see this tool as Baidu coming on out. In that regard, Japan, long a manufacturing high tech Asian leader, is a logical first step. Yes, there are serious long standing grievances. And a longer history, measured in millennia, than just about any other two countries extant. Despite all that China is Japan's second most important export partner (after the US) and it's first for imports (double second place US).

The US has long had isolationist tendencies while at the same time expansionist ones. China much the same since 1949. Except that they are starting from farther back industrially/technologically. With a much greater population engine. China vs the US economically is much like the US vs the UK a hundred years ago: just about everything being built shiny new state of the art vs just about everything built fifty plus years before carrying enormous upgrade/replacement costs, momentum vs. inertia.

Chinese government leaders and officials have publicly discussed how manufacturing growth is slowing and have, consistently, in each 5-year plan, increased the importance of advanced tech in their social economic initiatives. This past July they issued a State Council AI development planning notice that in effect states that they want their own AI development efforts to match that of the west by 2020, to be making 'major breakthroughs' by 2025, and to be the envy of the world by 2030. The money is flowing and the recruiting of Chinese researching, working in AI in the west is already underway. Western media doesn't cover Chinese tech well or often so that what already exists and the scale of the ramp up is not easily available but Chinese ML/AI companies and research facilities have raised, invested ~100 billion yuan (~15 billion USD) since July. At least, for I just note the larger announcements.

Where commentators in the west generally see tech replacing workers, increasing unemployment because of that, and the need to phase in some form of GAIN (guaranteed annual income) in response, in China they see ML/AI as an employment generator and a social 'all boats rising' panacea; a fascinating dichotomy in mindset. Which raises the question: which is more likely to achieve desired results?

China is the world's largest market; in population equivalent to all the western economies. It is an engine that revved up to dominate world manufacturing and is now determined to rev up to dominate high tech. Yes, it is happily stealing from all and sundry to speed things up but the quantity and quality of their experts in various fields is good to world class; they'd get there sooner or later anyway, currently it looks like sooner. While the west, generally, cuts back investment in R&D preferring debt and dividends, me-ism over us-ism, ideology over reality. My Great Chinese Adventure keeps looking better and better.

Currently, operating within China requires some accommodation, commonly a Chinese agent or partner. And they regulate business and communication, including the web, differently; it's critical to follow any country's rules, differences trip one up everywhere, just more so currently in China.

Lastly Chinese webdev, webgorillas, and web usage have always been markedly different from the west. It's a different society. They, imo, do a great many online things better, several worse, and some similarly. Not simple, not easy, but with unbelievable potential.

And if Baidu is stepping out Alibaba and TenCent won't be far behind.
Although I expect them to concentrate on east/south Asia and Africa first. The battle of the webgorillas will be there rather than in each others' backyard. Popcorn time!

I see this tool as Baidu coming on out

Baidu has had a Japan presence for many years.

And that was all she wrote.

:)

Baidu has had a Japan presence for many years.

Yeah, it's been a little over 10 years now [webmasterworld.com...] since they opened operations in Japan.

However, the new legislation in China introduced in November is not opening them up at all. It's having the opposite effect. This is making it a lot more difficult to have a site in China. Even Amazon shuttered its Chinese AWS service and sold off the infrastructure. That's why a tool like this is necessary.

The Great firewall has always been a problem, and it's getting worse.