Carragee Consulting

Are LLMs a gift or poison?

In English a gift is a present 🎁
In German Gift is Poison ⚰️

Giftig meaning poisonous is my favorite false cognate because it’s like a Trojan Horse. The word sneaks in looking like a present prepared to kill.

The overnight ubiquity of LLMs in our lives leaves us vulnerable to not just Trojan Horse attacks, but many other subtle forms of risk.

One of the methods of turning these new systems against us is to poison the training data, for instance by buying expired domains and replacing content referenced by URL in common crawl.

“More than you asked for” a new book by Rich Heimann & Clayton Pummill is an amazingly concise but comprehensive introduction into risks associated with AI and how we can start to mitigate them in system design.

Until I listed to their book I hadn’t realized how much the cyber security field is where we need to learn from to protect and manage AI systems. Every step in the process from data curation, to labeling, to prompting and usage requires careful design and testing due to the vulnerability it represents.

With the pace of innovation, it seems no two deploy systems have the same architecture and therefore the nuanced risks from choice of base model to manage of RAG repositories all present permutations of untested variables.

I highly recommend reading it both “More Than You Asked For” and Heimann’s longer book “Doing AI”.

This week I also watched two different presentations from the developers Ragas on developing custom metrics to evaluate RAG systems. Ragas is among a growing set of tools being built to both evaluate models globally with summary metrics like F1, but also combined with LangSmith enable you to explore nodes with low scores and hypothesize what can be done about it.

In LLM evaluation the term bias comes up a lot. Bias seems so squishy and relativistic, how can we mathematically measure something whose meaning is ambiguous?

I can’t help thinking that some states in the US have “science” text books that deny evolution and that we used to consider Pluto a planet.

Defining “Ground Truth” seems an unattainable Holy Grail which has implications both for legislation as well as anyone with positive intent to understand and manage the impacts of their own AI creations.

Due to the scale of the problem of observability and security, we’re now moving past the use of hand labeled data sets and using LLMs to generate both synthetic test scripts as well as synthetic answers.

What could go wrong?