Should you trust a chatbot’s stock tips?

Should you trust a chatbot’s stock tips?

AI stock recommendations are free, confident and widely followed. However, they underperformed the analysts they are meant to replace over a 20-month period.

A chatbot told investors to buy five stocks on most days over 20 months: NVIDIA, Microsoft, Amazon, Alphabet and Apple. It rarely mentioned what any of them cost, nor did it check the news for company updates. And the users who asked the chatbot paid closer attention to its picks than to a professional opinion.

That chatbot is available to anyone. Type “which stocks should I buy tomorrow?” into ChatGPT and an answer arrives in seconds. It costs nothing, reads like a broker’s note and comes with a list of sources attached. For the ordinary investor, who has never had a research analyst on call, this feels like the moment the playing field finally levels.

Professional investors have always had an edge. Sell-side analysts produce the recommendations, forecasts and price targets that fund managers rely on, and that research is paid for with brokerage commissions that households do not generate. Regulators have spent decades trying to narrow the gap, with free online access to company filings, bans on selective disclosure and machine-readable reports. The results have been mixed.

Generative AI promises something those reforms never could. It does not just make information available; it goes and finds it, reads it and turns it into a buy list on demand. A silicon analyst for everyone. My coauthors Yongxin Xu of Monash University and Yuhao Xuan of Sun Yat-sen University and I set out to test whether that promise holds.

What we did

On every US trading day from May 2024 to December 2025, we asked three generations of ChatGPT, through the same public website a household would use, for:

  • the twenty stocks it would buy the next day
  • reasoning behind each pick
  • sources it used.

We supplied no documents, no screening model and no expert framing. We wanted to measure the advice people actually receive, not what the technology can do when a finance professor builds the scaffolding around it.

Collecting the answers live, day after day, also sidesteps a problem that plagues much of the research on AI and markets. A model tested on past data may simply remember what happened. A model asked about tomorrow cannot.

We then benchmarked every AI recommendation against the sell-side analyst recommendations issued on the same day. The agents did their job. They returned a tradeable list of twenty stocks on the great majority of days they answered. The problem was what sat behind the list.

Three things the chatbot does not tell you

It often does not look anything up. ChatGPT attaches sources to its answers, and a user watching it browse would reasonably assume the picks rest on today’s news. By comparing the sources it cited with the web pages it actually retrieved, we found that about 1-in-3 recommendations rested entirely on the model’s training data. The answer to a question about tomorrow was drawn from a snapshot of the world more than one year old, and the agent did not say so. The share did not fall from GPT-4 to GPT-4o to GPT-5, despite more than two years of model development in between.

When it does look, what it finds is often old. Among recommendations that genuinely used retrieved pages, the newest source was more than 100 days old in a quarter to three fifths of cases, depending on the model. This was not for lack of news. The agents gravitated to the most heavily covered companies on the market, and same-day articles about those firms existed for roughly 7-in-10 recommendations.

It talks about the company, not the price. The reasoning was dominated by market leadership and growth prospects. Valuation appeared in only 7–17% of recommendations. By contrast, 80% of analyst buy recommendations came with a price target. The agent found good companies and treated them as good investments, which is precisely the mistake that behavioural finance has documented in human investors for decades. The result was the short, repetitive list this article opened with. The same five names appeared on most days in every generation, while the single most-recommended stock in the analyst sample appeared on fewer than one day in ten.

Table 1. AI stock recommendations versus sell-side analysts, May 2024 to December 2025

Source: Xu, Xuan and Zheng (2026). Analyst figures are from I/B/E/S buy recommendations over the same period.

 

GPT-4

GPT-4o

GPT-5

Analysts

Recommendations based on training data, not live search

30%

35%

34%

n/a

Newest cited source more than 100 days old

61%

31%

26%

Same-day news for 72% of buy calls

Reasoning that mentions valuation

7% to 17% across the three generations

80% carry a price target

How often the single most-recommended stock appears

81% of days

95% of days

95% of days

9% of days

The natural response from a sceptical user is to ask again. We did. We prompted each agent five more times to search further and revise, tried three differently worded prompts, and repeated the question every hour in separate sessions. The stocks changed. The defects did not. For two of the three models, reliance on training data was higher after five rounds of updating than before. Lists collected at different hours of the same day shared three quarters of their stocks, so an investor who reads consistency as a sign of reliability is drawing the wrong conclusion. Four of Google’s Gemini models, run through the same protocol, performed worse.

The part that should worry us

None of this would matter much if investors treated chatbot picks with appropriate caution. They do not. Using Google search volume to measure attention, we found that an AI recommendation drew a spike in investor interest more than 50% larger than an analyst recommendation for a comparable stock on the same day. The spike also appeared when the pick rested on year-old training data rather than on live search. Investors cannot see the difference, so they do not react to it.

And the picks did not earn their reputation. Compared with analyst-recommended firms chosen on the same day, AI-recommended firms were less likely to report a profit at their next earnings announcement, and their shares underperformed on the first trading day after the recommendation. Over the next ten trading days, a portfolio that followed the analysts earned higher abnormal returns than one that followed the chatbot.

Put the two findings together and the picture is uncomfortable. Retail investors are receiving a weaker signal than the professional benchmark and responding to it more strongly. Easy access to a lower-quality substitute for research does not close the gap between households and institutions. It may widen it.

What should change

For the platforms. The chatbot knows whether it searched, when its sources were published and whether it fell back on memory. The user does not. Every one of those facts is already recorded by the platform and could be shown next to the answer. A simple flag reading “no live sources were used for this response” would change how a reasonable person reads a stock pick. That is a product decision, and it is overdue.

For regulators. The current policy conversation focuses on disclosing that advice is AI-generated. Our evidence says that is not enough. Users already know they are talking to a machine. What they cannot see is whether the machine looked at anything current before it answered. Regulators examining how financial firms deploy AI should ask for disclosure of retrieval, not only disclosure of automation. The same standard applies to researchers and vendors who claim an AI agent outperforms because it searches the web and draws on the latest information: show that the search happened, and show which information it actually used and how recent it was.

For the industry. Investment firms are asking whether AI genuinely improves research or merely looks like it does. Our results suggest where the value lies. AI is a strong tool for gathering and summarising information, and a weak one for making the final call without a disciplined framework around it. Firms that separate the two, verify that retrieval occurred, date their sources and benchmark against the human process they are replacing will know what they have. Firms that back-test a chatbot on history it may have memorised will not.

For the rest of us. Use the chatbot the way you would use a bright research assistant. Ask it to explain a company, summarise an earnings call or find the bad news you might have missed. Do not ask it what to buy tomorrow, and do not mistake a confident answer, or a consistent one, for a current one.

Where this goes next

These findings come from a simple question asked the way a household would ask it. A more structured approach, designed with practitioners, could produce something far better, and that is the collaboration we are now pursuing.

Platforms willing to test disclosure designs, firms willing to share how they deploy AI in research, and policymakers thinking about the next round of guidance on AI in financial advice all have a role to play. I would welcome the conversation.

About the author

Dr Gaoping Zheng is a Senior Lecturer in the School of Economics, Finance and Marketing at RMIT University. Dr Zheng’s research examines how information environments shape financial markets, including the effects of social media, information access and artificial intelligence on investor behaviour and price efficiency.

Research referenced in this article

Xu, Y., Xuan, Y. and Zheng, G. (2026). Silicon Analyst: Do AI Agents Level the Playing Field? Evidence from Stock Recommendations. Working paper. SSRN 5651130

28 October 2026

Share

Return to Acumen
28 October 2026

Share

Engage

Related News

Detailed view of a line graph illustrating data trends with a prominent line representing variable measurements.

Should you trust a chatbot’s stock tips?

AI stock recommendations are free, confident and widely followed. However, they underperformed the analysts they are meant to replace over a 20-month period.

A person gently touching a wall featuring a bold, colorful striped pattern.

Are we still discovering new ways to improve the human experience?

Why does innovation matter? At the most fundamental level, because as humans we are constantly searching for ways to improve our experience of life. What exactly this means is difficult and perhaps impossible to define.

An elderly woman practices walking with a wheeled walker as a smiling nurse in blue scrubs offers support in a bright care room

How AI can support older adults without taking over care

AI can help older adults live more safely and independently. A useful system must also protect dignity, keep people accountable for consequential decisions and strengthen the relationships that make care human.

Various Australian dollar coins displayed on a banknote, illustrating the denominations and currency style.

Could Australia become a capital hub for global corporate finance? Insights from Alphabet’s $5.5 billion kangaroo bond issuance

Google’s parent company, Alphabet has struck a landmark AUD $5.5 billion bond issuance. A record-breaking deal in Australia’s debt market, this signals a fundamental shift in the geography of global corporate finance strategy.

Show more
aboriginal flag float-starttorres strait flag float-start

Acknowledgement of Country

RMIT University acknowledges the people of the Woi wurrung and Boon wurrung language groups of the eastern Kulin Nation on whose unceded lands we conduct the business of the University. RMIT University respectfully acknowledges their Ancestors and Elders, past and present. RMIT also acknowledges the Traditional Custodians and their Ancestors of the lands and waters across Australia where we conduct our business - Artwork 'Sentient' by Hollie Johnson, Gunaikurnai and Monero Ngarigo.

More information