YHK - yet another hacker news clone

Document Review with Unsiloed and TypeSafe Jev

48 points ritzaco 4 days ago 7 comments

armcat 4 days ago | parent

They use the OCR + structured data extraction in this example, but I'm thinking you could extrapolate this to any scenario where you are extracting some data (from a table, markdown file, OCRed doc), and then use a decision model to perform a fast on-the-fly review/validation.

mmo93 4 days ago | parent

My masters thesis used a similar approach a couple of years ago, we finetuned a BERT model to categorize natural language questions as either ambigous or non-ambigous in context of the database schema and which queries where plausible candidates to the question. It worked really well, I suppose JEV is orders of magnitude more powerful. There's going to be many applications of good decision models.

rohansood15 4 days ago | parent

Jev is cool, but this example is really bad.

The example flags a '10 tablets' dose as a possible erroneous extraction, when the text says exactly that. So it is a false positive.

Furthermore, even if your goal was to catch logical errors rather than extraction errors, you are better off using programmatic checks on the extracted value and having a dedicated error handling path including additional full LLM calls, rather than decision model.

LewisDwyer 4 days ago | parent

The idea is that Unsiloed would check for values that are hard to read and Jev checks for semantic issues. No one is prescribing 10 tablets, 3 times a day. In this example it's Jev's job to check for obvious errors in units and so it flags this value. Unsiloed cannot flag it because, as you said, the text says exactly that. That's why the models work well together as they catch different issues.

rohansood15 4 days ago | parent

But you need to explicitly specify the prompt to check dosage. If you passed it as part of the original extraction model call, it would return a lower confidence to begin with. Or provide a different 'is_dosage_reasonable' value, which would eliminate the need for Jev.

And this is before one argues that dose <= 3 tablets is a better validation than the Jev prompt and costs nothing.

sixhobbits 4 days ago | parent

Well yes but then you'd have to manually code every set of expectations, the point of using a model like Jev is it can flag a "something is wrong here" entry without needing someone to define every thing that could go wrong

rohankhameshra 4 days ago | parent

I agree that Jev is pretty cool but example is pretty bad, there is lot more that can be done with it, particularly in unstructured data processing pipeline. One big problem I see is just figuring out if there is new training point is to be found in the data, because with stream of data coming in its waste of resource to process everything