If you're planning on experimenting with this yourself, one easy way is to use the RSS feed and aggregate them into each relevant page.

If the sites have RSS feeds its easy. There are a lot of feed aggregators that will do a reasonable job off the shelf: most have names with the word "planet" in them.

I built a news aggregator as part of a niche search engine a few years ago. A lot of sites did not have RSS feeds so we built a bot that was easily configurable for different sites using Scrapy and indexed the data with Solr. The biggest problem was getting blocked by things like Incapsula - many smaller site owners do not know how to configure things like that to allow a particular bot.

Thanks. Nobody has used public news API services for collecting news? Or it can’t be done this way?

Why couldn't it be done that way? An API is just another means of getting the same news. Buy the paper, crawl the news sites, read the RSS feeds, connect to an API, they're all ways to get the same data. Most likely, that (annoying) site uses an API and/or its own bot, but you'd have to ask them to be sure :-)

Depends on whether then API matches your need. Does the API crawl the sources you need? Can you rely on it being available? Does it updates fast enough?.....

The other thing about APIs is you're often hampered by limits, so its worth bearing that in mind.

Many of the best aggregators are quite sophisticated these days.

The other thing is if you crawl the website won’t the website block the bot?

Well, yes, and you'll probably find that the major services block most bots.

The RSS feed could be the solution.

And what if the website doesn’t have rss feed feature? How does one automate news collecting from these websites? To aggregate into my website.

Ask them? They may not be aware that others could share their news. Some sites may offer other options.

They may not block you depending on how much you crawl. If you are just getting headlines it may not be that much. If you are crawling the whole site you probably will get blocked. Follow directions in robots.txt, and robots.txt may also point to sitemaps (xml ones) as an alternative to RSS

There are lots of ways to get round bot blockers. There is a whole industry around doing this. The best way is if they know and and decide they want to allow your bot.

A lot of news sites have RRS feeds.

I experimented with a news aggregation site many years back, around 2004 (ish). It was really an RSS reader that I wrote specifically to deal with news sources. I had a cronjob that would download the URL, title and description every 20 mins (if the feed was modified).

Mack.