Forgive me for copying and pasting but:

Section 230 of the Communications Decency Act: Google, as an internet service provider, is generally protected from liability for content provided by third parties. This means that Google is typically not held responsible for the content that appears in its search results, even if it is incorrect or misleading.

Negligence and Duty of Care: To successfully sue for negligence, you would need to prove that Google owed you a duty of care, breached that duty, and that this breach directly caused your injury. Courts have generally found that search engines do not owe a specific duty of care to individual users regarding the accuracy of search results.

Suing Google: Protected under Section 230 and not typically found negligent for search result accuracy.
Suing the Phone Manufacturer: Hard to prove liability unless there is a clear defect or negligence in the phone itself.

Yes but wouldn't Section 230 change if they are writing the content? Technically, their own AI system crafts the answer, unlike traditional search results. I am not lawyer but I am pretty sure it would be considered content written by them, not just picked from search engines.

This means that Google is typically not held responsible for the content that appears in its search results, even if it is incorrect or misleading.

Sure, and the law is pretty much the same over here in the UK too.

However: the difference in this case is that Google themselves are providing answers, not websites.

Regarding owners of browsers and mobile phones: should they be defaulting to a 'search engine' that is known to give false and sometimes dangerous advice? Run by a company that is still 100% trusted by billions of people worldwide?

However: the difference in this case is that Google themselves are providing answers, not websites.

Which begs the question, that to disclose the sources via link citations and emphasise a disclaimer on accuracy would be the common sense thing to do. Then there will be copywrite issues. No doubt Google will muddy the legal arguments and authorities will be slow to act.

I spoke to a lawyer about this topic. He couldn't tell me anything about the USA, but if Google introduces SGE in Europe (EU), the company will legally become a publisher that can be held liable for the content. It would also mean a breach of copyright guidelines, for which there is also liability. Google will not be able to talk its way out of this. I think Google knows that too and that's why it won't be introduced here (fortunately)