What user agent are you seeing?

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

714 - APPLE-ENGINEERING

17.241.75.129
17.241.219.120
17.241.219.217
17.241.227.10
17.241.219.187
17.241.227.118
17.241.219.63

Interesting, thank you sudo. Apple has the full 17.0.0.0/8 and it appears they are (hopefully) sticking to 17.241. for the crawler.

Yeah of course. I provided the ASN and specific addresses for the purposes of validating my claim. Page on identifying Applebot for anyone unfamiliar: [support.apple.com...]

Apple also have added a section "About Search Rankings" which lists the following:

Apple Search may take the following into account when ranking web search results:

- Aggregated user engagement with search results
- Relevancy and matching of search terms to webpage topics and content
- Number and quality of links from other pages on the web
- User location based signals (approximate data)
- Webpage design characteristics

Search results may use the above factors with no (pre-determined) importance of ranking. Users of Search are subject to the privacy policy in Siri Suggestions, Search & Privacy.

I see they also have a mobile bot that followed after a series of visits from the 17.246. range. The mobile UA was Mozilla/5.0 (iPhone; CPU iPhone OS 17_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) GSA/304.0.607401217 Mobile/21D61 Safari/604.1 and it came in on AT&T IPS from IP 107.72.178.99 and requested only images from the same page the crawler had visited.

not2easy, now that is interesting. I was grepping for applebot...I will go back and broaden my search. However, two thoughts enter my head:

1. Piggybacking off client devices?
2. Targeted image indexing based on user traffic?

Perhaps I'm thinking too deeply...though idk how else you explain that.

Spur says 107.72.178.99 is a call-back proxy IP, but it could also be a GC-nat for a cellular network.

"The mobile UA was Mozilla/5.0 (iPhone; CPU iPhone OS 17_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) GSA/304.0.607401217 Mobile/21D61 Safari/604.1"

I don't see anything in that UA that indicates it's a bot (apple or otherwise).

I think that's the point. Is it just a coincidence?

They were in immediate sequence, when the applebot UA quit, the same UA minus (Applebot/0.1; +http://www.apple.com/go/applebot) with (iPhone; CPU iPhone OS 17_3_1 like Mac OS X) inserted showed up from the AT&T IP. That mobile UA/IP requested only images from the page the applebot had visited. The applebot did not request images. Then the applebot from the same IP returned. The applebot requested only pages.

I see that the UA support from apple lists a mobile UA with the applebot in the UA. It just seemed unusual to have one in the middle of the applebot visit that matched everything except the applebot string. A peculiar coincidence.

Does the page that was requested rank 1 for any searches? Safari has a feature for pre-loading the top result for a search.

I don't think so, it is one of several dozens of travel articles on an old travel blog. The article was from at least 10 years ago. It is not my site, one that I work on and the owner doesn't care about rank. At all. It is his hobby site.

Strange. So what are you thinking?

No ideas. I was just checking the logs and spotted the applebot which I had not seen there before. I do think it was a coincidence, just weird that they were interspersed at the same page activity. I see other similar UAs viewing that same page on T-Mobile IPs. No ideas at all. Maybe there was some big deal on that day that I don't have a record of. I could ask the owner if he's aware of anything to spark topic interest on that day. I do not analyze all of those logs every day, but was asked to take a look so I did. He must have noticed something or got a visitor question, I don't know. It does not worry me, I've seen a lot of coincidences over the decades. That iPhone had a referer: "https://www.google.com/" so an image might have caught their eye, since that is all they requested. I don't use google so I have no idea where it ranks there either. Your guess may be better than mine, I have none.

Was there a referrer for any of these hits?

I don't think I've ever seen apple-bot as a referrer for any hit to my site.

What could have happened is that someone on Verizon with an iPhone did a search using apple's search interface (what-ever that is, maybe Siri? not google or bing) and your site came up on the short list and applebot grabbed or cached only the html files, created a user-agent by melding the iPhone's UA with applebot. iPhone user selects your site to visit from the search, they already have the html files so they directly grab only the image files, then the apple bot comes back and grabs more html files.

Here's a few things:

1) I see the applebot hit my site periodically, several times a week if not once a day (these days). I don't think I've ever seen it grab an image file (gif, jpg, etc).

2) apple could be offering inline / real-time malware scan of a website, scanning only html files but not image files. So browsing a site will look like it's coming from a combination of apple-bot and the user.

3) alternatively this could be apple iCloud private relay situation, but in that case none of the page requests would or should come from an apple IP.

Knowing if there was a referrer involved would help figure this out. Also - did the apple-bot request robots.txt during any of this?

The only referer in any of this was the non-applebot iPhone user. That iPhone had a referer: "https://www.google.com/" so an image might have caught their eye, since that is all they requested.

In about the 5th post in this thread, there is a link to Apple's page on identifying Applebot's IPs and UAs.

SamGuy isn't the behavior you described basically Safari's pre-loading the top search result feature that I mentioned several posts ago?

I'll check referrers on my side over the weekend.

Would not be surprised to see requests begin spiking again post anti-trust ruling

Anyone else seeing a lot more of Applebot?

:: business with archived logs ::

Nope, rock-steady over here. That is, the total requests from

^17\..+Applebot

so far this year work out to very close to 7/12 of the total requests from all of 2023.

I do see a fair number of redirected requests recently, which suggests they may be revisiting an old shopping list. (All search engines do this periodically. I’ve currently got G doing the same thing.)