YHK - yet another hacker news clone

AI chatbots give wrong answers to financial queries 'most of the time'

39 points 1vuio0pswjnm7 1 hour ago 12 comments

ehe78qhe 1 hour ago | parent

Article is just a vague summary of https://www.saturnos.com/report/artificial-authority

Anecdotally, current models seem to be decent at general personal finance principles - certainly better than the majority of personal finance education that people get exposed to unless they seek it out and read a variety of books and sources. But I wouldn't trust them with direct decision making with actual money due to the training lag time on current tax policy, etc.

NitpickLawyer 4 minutes ago | parent

> certainly better than the majority of personal finance education that people get exposed to unless they seek it out and read a variety of books and sources.

Also, models are now good enough that you can give them chapters from "authoritative" books, and they'll integrate that and come up with better answers even if their "vanilla" answers were average. And they'll tailor stuff to your particular situation. It's funny that the "agentic" stuff is only used in coding mostly, while it can and does work in other fields as well.

As always, you kinda need to check it (at least spot check) but all in all I'd agree it's better than the average stuff you used to find with a quick google search.

sixtyj 51 minutes ago | parent

I would prefer to use agent-assisted python scripts that chatbot.

k7peak 47 minutes ago | parent

Agreed, this works really well for me. Double check the math/python, execute many times without a LLM that can change o

demibabs 47 minutes ago | parent

I like how FT makes me accept cookies from their 46 “technology” (advertising) partners before showing me that the article is behind a paywall anyway.

jb1991 41 minutes ago | parent

You actually like that? I find it kind of annoying.

Wololooo 37 minutes ago | parent

No they do not like it, it is a figure of speech to underline how much they do not like it.

01100011 33 minutes ago | parent

Single shot or with reasoning enabled? My experience is that reasoning dramatically reduces hallucinations and improves output quality. I don't trust models without it.

simianwords 21 minutes ago | parent

These models do pretty well in benchmarks and real world so I'm highly suspicious of this article. Further more, in the original report, the examples of bad answers are from Haiku - at least 7 out of 10. Anyone who knows anything about LLMs know that haiku shouldn't be used for anything pretty much.

There's no reproducible set either. I'm not gonna trust this report.

in_absentia 12 minutes ago | parent

Now, compare this to a recent story that seemed to claim the opposite:

https://news.ycombinator.com/item?id=49139102

I don't have the time to review the underlying research and decide which one is more correct. My personal biases make me want to believe the current one. Your personal biases may be pulling you in the other direction. How do we make the conversation more intelligent than that?

74gee 5 minutes ago | parent

Well duh! If it's not using tools to look up the state of the market empirically it's not likely to be accurate financially.

lukeify 1 minute ago | parent

Given most financial advisors tend to vend out suboptimal advice and steer customers in favour of products they receive a kickback for, I'm happy to be accepting of an unbiased LLM that's trained on bogleheads.org.