Yet another Hacker News

Dayjob: Tax – evaluating AI agents on 50 professional tax assignments

67 points landonxi 1 hour ago 41 comments

holdingunsteady 1 hour ago | parent

My tax accountant mis-filed my taxes a couple years ago, which cost me about $20K (which I learned after I switched to another accountant later). Don't know if Claude / GPT would make the same mistake or not given SOTA today, but feels like there's an opportunity for models to be saving me thousands or tens of thousands of dollars a year.

ttthuy 1 hour ago | parent

Someone still needs to take liability for mistakes.

The real issue is that accountant most likely has a track record of those kinds of mistakes - but you have no info on other clients. So you weren’t as informed at the time you chose them vs the new one.

kjksf 1 hour ago | parent

In this case accountant DID NOT take liability for mistakes.

It's not the accountant that paid $20k for his mistake but the accountant's client.

holdingunsteady 1 hour ago | parent

Yeah, I looked him up just now actually, and he has a 3-star rating on Yelp... Which I think is pretty low for accountants?

I think the red flag should have been when he printed out all my tax returns and made me pick them up in person, and didn't even send a digital copy.

matkoniecz 1 hour ago | parent

> Someone still needs to take liability for mistakes.

I am not aware of setups available to me that would transfer liability onto accountant.

So in either case I am taking liability.

(also, I am pretty sure that if corporation loses 10 millions over accountant making bad decisions, they are not going to recover that funds either)

ffwang2 1 hour ago | parent

yeah i think liability ultimately lies on the individual. of course, you could sue your accountant, but that feels hard, maybe almost as hard as suing the model providers.

matkoniecz 1 hour ago | parent

At least in Poland it is not viable. Friend's business lost noticeable amount of money because accountant has not filed taxes and lied that they filed and lied that they checked it.

And there was no real path to recovering funds, except small part, after long court case.

At least they were able to fire them without severance pay.

skinfaxi 1 hour ago | parent

Yes, at least in the US the individual must sign their own return.

ffwang2 1 hour ago | parent

i'm a bit unclear who has the liability even when you use a CPA. accountants are humans that sometimes make mistakes.

raegis 59 minutes ago | parent

The IRS made it crystal clear to me long ago. The tax filer is responsible for everything on the tax return, regardless of the preparer. Of course, accountants can be liable in other ways, but the filer has to pay the fines if the accountant makes a mistake.

dghlsakjg 8 minutes ago | parent

From the IRS' point of view, the tax-payer is responsible for paying all of it. You can then turn around and sue to recover any of the fees/fines or penalties incurred as a result of an error made by the accountant, not the actual taxes though.

ElCapitanMarkla 39 minutes ago | parent

In NZ the liability is generally on you and not the accountant.

paxy 26 minutes ago | parent

People misunderstand the concept of liability. Yes your tax professional is liabile for fraud or gross misconduct, but if they make a mistake which results in unpaid taxes or fees you're still the one on the hook.

dghlsakjg 11 minutes ago | parent

Typically you are on the hook for paying the actual taxes and penalties, but if you acted in good faith on the negligent advice of a tax professional many jurisdictions allow you to recover the costs of penalties and interest accrued as a result of the negligent advice.

tchalla 9 minutes ago | parent

How much money does that take?

BeetleB 41 minutes ago | parent

My obligatory comment about my experience with tax accountants:

https://blog.nawaz.org/posts/2022/Nov/do-you-trust-your-tax-...

tristor 7 minutes ago | parent

I'm also 0 for 4 on tax preparers / tax accountants, so I concur with you its best to just do it yourself, which is what I now do. Only time I ever got audited (and failed) was when I had employed an accountant, since then I've never had a problem including passing audits. The tax code isn't /that/ complicated for most people's situation, and it's well worth understanding it.

itishappy 38 minutes ago | parent

Sounds like an opportunity which could end up costing or saving $10k and which may be difficult to evaluate.

ffwang2 1 hour ago | parent

i want to always have a fast way of reviewing my tax returns without having to hassle my accountant or hire another one.

aquafox 55 minutes ago | parent

In Germany, there are companies like Buhl data that let you do your tax return very easily and give a live estimate of your return based on the data you provided. From my experience over the past 5 years, its prediction error is less than 5%.

matkoniecz 1 hour ago | parent

Sharing all my financial info with LLM companies is not something that I like.

EDIT: good point that paying accountant does not ensure avoiding it

wonderful, you made a good point that make me dislike situation even more

cornholio 1 hour ago | parent

Your accountant sure will.

grey-area 20 minutes ago | parent

Most accountants will not in fact share client data with an LLM company.

vatsachak 1 hour ago | parent

Some inference providers have a Zero Data Retention policy

ChucklsTheBeard 1 hour ago | parent

Oh look, yet another reason to use local inference.

oklahomasports 57 minutes ago | parent

Worst case scenario what does OpenAI do with your tax return?

pawelduda 1 hour ago | parent

I'm surprised Sonnet 5.5 did noticeably better than Opus and Fable. Same pattern in their other benchmarks.

esafak 1 hour ago | parent

I like their benchmarks. They ask the right, not easy questions: https://surgehq.ai/blog/sudo-l7

Has anyone verified to see if they hold up?

andriy_koval 59 minutes ago | parent

What is human baseline? How it stacks against 10% LLM accuracy?..

waiwai933 57 minutes ago | parent

I have a non-trivial tax situation, as a US citizen living abroad. I prepare my own tax returns, never having found a cross-border tax preparer I trust, and have for the past few years, fed my draft return along with all the supporting paperwork into Claude. It has:

1. found errors, e.g. where I used the incorrect exchange rate (and managed to work out how I had made that error, which I was impressed by)

2. found items which would appear to be errors except for context it did not know

3. flagged areas where it disagreed with me as to the correct tax treatment, but accepted that the law is ambiguous (and in some cases I preferred its interpretation after additional research and in other cases I did not)

4. flagged areas where it disagreed with IRS guidance, and only acknowledged said guidance after I provided links to it

5. provided highly inaccurate estimated tax calculations, and not been able to fix its estimates even when challenged

So my takeaway is that it is a very useful tool for tax, but perhaps unsurprisingly, it is not good enough to dump all your paperwork into and expect a shiny return you don't need to check. (Possibly it is if you just have a W-2 and no other substantial income, but I can't speak for that scenario.)

ivanjermakov 47 minutes ago | parent

I also had a great experience being consulted by an LLM regarding registering a business as a foreigner. Except I'm still very sceptical since it's not my area of expertise and consequences for a mistake are very high.

redanddead 42 minutes ago | parent

you were consulted by an LLM?

sanex 31 minutes ago | parent

He said he's a foreigner, almost certainly a mistranslation.

ajhit406 38 minutes ago | parent

check out https://www.multiplierholdings.com/

it's the most ai-native tax experience for US citizens abroad

waiwai933 35 minutes ago | parent

Are you saying I should use this (or it looks like one of their portfolio companies) as my return preparer, or as a replacement for Claude my return-checker? If the former, I would want to know why I should trust it; if the latter, I would want to know why I should pay for their services given I am already playing Anthropic for a subscription plan anyway.

(It would also help to know if you had a financial interest in or prior experience with this company)

persedes 26 minutes ago | parent

https://claude.com/marketplace/connectors/aiwyn-tax

Try this, it uses a tax engine as the backbone and provides feedback for Claude if it goes off the rails. Albeit it doesn't support every tax scenario.

waiwai933 17 minutes ago | parent

I appreciate the recommendation, but as it happens I have already solved the estimated issue (mostly by just preparing a full return with estimated next year's figures, which while time-heavy is going to be more accurate). And for what it's worth, I suspect it wouldn't work for me - cross-border is extraordinarily complicated because of the tax treaties (indeed there is a dedicated IRS form to disclose the effect and rationale of any treaty based position), and given there is no obvious note on the documentation for it that there is cross-border support I doubt there is.

(And as I said to another commenter, I think it would be advisable in this thread for people to expressly acknowledge or disclaim any interest in or prior experience with any particular product)

bitbasher 56 minutes ago | parent

How much can someone trust a report about AI from a company who's domain ends in ".ai"

Not much, I assume.

newhotelowner 46 minutes ago | parent

Not related but I hope QuickBooks learn something and make their software better.

I downloaded 1700 of my transactions and use AI to build an app for it, so I can analyze the transactions and make sure that everything is properly categorized. It used to take me whole day because of the Quickbooks. It took me less than an hour to go through all the transactions.

persedes 25 minutes ago | parent

There's also: https://github.com/column-tax/tax-calc-bench