Purus suspendisse a ornare non erat pellentesque arcu mi arcu eget tortor eu praesent curabitur porttitor ultrices sit sit amet purus urna enim eget. Habitant massa lectus tristique dictum lacus in bibendum. Velit ut viverra feugiat dui eu nisl sit massa viverra sed vitae nec sed. Nunc ornare consequat massa sagittis pellentesque tincidunt vel lacus integer risu.
Mauris posuere arcu lectus congue. Sed eget semper mollis felis ante. Congue risus vulputate nunc porttitor dignissim cursus viverra quis. Condimentum nisl ut sed diam lacus sed. Cursus hac massa amet cursus diam. Consequat sodales non nulla ac id bibendum eu justo condimentum. Arcu elementum non suscipit amet vitae. Consectetur penatibus diam enim eget arcu et ut a congue arcu.
Vitae vitae sollicitudin diam sed. Aliquam tellus libero a velit quam ut suscipit. Vitae adipiscing amet faucibus nec in ut. Tortor nulla aliquam commodo sit ultricies a nunc ultrices consectetur. Nibh magna arcu blandit quisque. In lorem sit turpis interdum facilisi.
Vitae vitae sollicitudin diam sed. Aliquam tellus libero a velit quam ut suscipit. Vitae adipiscing amet faucibus nec in ut. Tortor nulla aliquam commodo sit ultricies a nunc ultrices consectetur. Nibh magna arcu blandit quisque. In lorem sit turpis interdum facilisi.
“Nisi consectetur velit bibendum a convallis arcu morbi lectus aecenas ultrices massa vel ut ultricies lectus elit arcu non id mattis libero amet mattis congue ipsum nibh odio in lacinia non”
Nunc ut facilisi volutpat neque est diam id sem erat aliquam elementum dolor tortor commodo et massa dictumst egestas tempor duis eget odio eu egestas nec amet suscipit posuere fames ded tortor ac ut fermentum odio ut amet urna posuere ligula volutpat cursus enim libero libero pretium faucibus nunc arcu mauris sed scelerisque cursus felis arcu sed aenean pharetra vitae suspendisse ac.
TL;DR: We automate generating evaluation data in technical domains, primarily in the format of prompt/response pairs. Our API is in closed beta, but you can play with the stripped-down demo here!
A team we work with was using financial analysts to write prompt/answer pairs to improve their LLM agents. They'd write questions like “How does Dodd-Frank legislation affect X company action”, or “What filing would you need to make if an entity did Y?”. This process, however, took hours and was a drag for employees.
But what if we didn’t actually need humans to create labeled data?
Sometimes we’re not looking for net new knowledge. For example if we were trying to encode an HR policy into an AI model, all the knowledge we need is already written up. In this case, generating a dataset is not a problem of finding new knowledge, but transforming existing knowledge.
The challenge then becomes, “How do we transform unstructured knowledge into rows of data?”
Some of this approach has been adopted by teams like Microsoft with AdaptLLM, where they turn sentences into reading comprehension tasks, and then use the resulting tasks as training data for a general purpose LLM. We expand on these types of approaches by parsing unstructured text to form knowledge graphs.
At a high level, we:
For example, here’s a few snippets of banking regulation [1]
From these snippets, we could parse this fact:
JP Morgan must hold at least 4.5% of its assets in common stock and cash
When we add to that regulation knowledge of what banks are considered GSIBs, we can generate multiple scenarios that demand advanced financial knowledge:
There’s two important features of generating data this way.
And that’s it! Early customer teams have reported that data generated this way is as realistic and accurate as samples they could write themselves.
Currently this requires a bit of setup as every knowledge base is different, but we threw together a basic, stripped down MVP here that you can play with – check it out here!
Note that there are some serious limitations to this web demo. For example, we haven’t exposed the underlying config file – most of our customers see massive quality jumps after just a few config iterations, and this usually fixes the first few rounds of feedback. This felt too complicated to throw up as a toy demo, but I’m happy to revisit if there’s interest.
If you have any questions, find bugs, or have examples of mistakes we made, feel free to reach out directly at matt at talc dot ai!
[1] This is not actually how it’s phrased in Dodd Frank / Basel III but none of us want to read that today.
We're currently in a closed beta with early partners. To request access, drop your contact info so we can reach out!