I assume that most advertiser base their ads on the user more than the website.

Ads are contextual (the website) and interest-based (user's history). Clearly interest based ads are based on the users. To a large extent so are the contextual ads. Advertisers typically bid on keywords. You have a site targeting those keywords, so user interested in that specific keyword land your site but the advertiser is not explicitly targeting your site, theyr are targeting the user.

I would guess that despite the fact that site may have a very narrow and precise niche, users going to the site could still be attractive to advertisers that are advertising other products to that target market.

I think that your primary concern should be more focused on whether the site has sufficient interest from search to generate the volume of page view required to generate a decent revenue. If you only get 10 page views a day, it becomes completely irrelevant which ads appear on your pages.

Advertisers can target their ads to specific sites. They can see how the traffic converts and decide if they want to continue running their ad on your site. They can also exclude your site from showing their ads if the site sends poor quality traffic. It is about the traffic, not so much the site. If the traffic converts, then they will want to advertise with you.

Few sites get a very high proportion of placement targeting in AdSense. Ensure you have the content to attract the contextual and are bringing in a genuine audience so that you have the audience/behavioural element and you are doing most of what you can.
I see the following more through Ad Exchange than AdSense, but the principal still applies: A domain/placement will often attract more targeted spend once it has proven it can convert (it is discovered through a broad campaign, then the spend on that campaigned in slowly honed to where it brings results). Having content that is "closer to the sale" helps with that as the user is further along the funnel and more likely to convert. Using your example you might find that content like "where to stay on the Isle of Elephants" and "Isle of Elephants flights" helps drive converting traffic more than "How the Isle of Elephants got it's name".

Although placement targeting takes place isn't the point of the exercise for the advertiser normally to outsource the heavy lifting of finding sites to Adwords / Adsense?