Every serious model release now ships with a safety card: refusal rates, jailbreak resistance, toxicity scores, bias evaluations. The numbers look reassuring. Then the model gets deployed into a Nigerian fintech support channel or a Kenyan health information service, and within weeks the incident log fills with failures that no benchmark predicted.
This is not a story about careless labs. It's a structural gap. Red teaming only finds the harms someone thought to look for, in the language they thought to look in. Nearly all widely used red-team suites encode a harm taxonomy shaped by US and European policy debates, and nearly all of their attack prompts are written in standard English. A model can be genuinely well-tested against that taxonomy and still be untested against the way risk actually shows up in African markets.
What Red Teaming Actually Tests
It helps to be precise about the difference between red teaming and the evaluation work it's often confused with. Benchmarking measures average performance across a fixed test set: how often is the model right? Red teaming does the opposite. It is an adversarial search for the specific inputs where the model breaks, run by people whose job is to break it.
A red-team engagement typically covers four attack surfaces:
- Prompt injection and jailbreaking: Getting the model to ignore its system prompt or safety policy through role-play, encoding, translation, or multi-turn manipulation.
- Harmful content elicitation: Coaxing out instructions, incitement, or abuse the policy explicitly prohibits.
- Misinformation and confident fabrication: Finding domains where the model invents authoritative-sounding falsehoods, especially in health, legal, and financial contexts.
- Over-refusal: The inverse failure, where the model wrongly blocks legitimate requests. Rarely reported, consistently underrated, and the one that most directly damages product usability.
Each surface is defined relative to a harm taxonomy: the enumerated list of things that count as a failure. That taxonomy is the part almost nobody localizes, and it is the reason a model can pass and still fail.
Where Global Red-Team Suites Miss African Failure Modes
Four failure classes come up repeatedly in African deployments and are almost entirely absent from off-the-shelf red-team coverage.
1. Code-Switched Jailbreaks
Safety filters are classifiers, and like any classifier they perform best on the distribution they were trained on, which is overwhelmingly standard English. A request the model firmly refuses in English is frequently answered when the same request is posed in Nigerian Pidgin, in Swahili, in Sheng, or in a sentence that switches languages mid-clause the way people actually write in a WhatsApp thread.
The mechanism is mundane rather than exotic. The base model has enough multilingual capability to understand the request, while the safety layer, trained on far less non-English adversarial data, doesn't recognize it as one of the prohibited patterns. Capability generalizes across languages faster than safety training does, and the gap between the two is the attack surface. Any market where users routinely code-switch is a market where this gap is being probed by ordinary users, not just researchers.
2. Locally Grounded Misinformation
Generic misinformation testing asks whether a model will invent a fake study or endorse a conspiracy theory. It rarely asks whether the model will confidently misstate the voter registration process in a specific country, invent a public health guideline that contradicts a national ministry's actual advice, or give plausible-sounding but wrong instructions for reversing an erroneous mobile money transfer.
These are the questions real users actually ask, and they are exactly where the model's training data is thinnest. Worse, the failure is invisible to anyone reviewing outputs who doesn't know the correct local answer. A fluent, well-structured, entirely wrong response about NDPA obligations or a national health protocol passes an English-speaking reviewer's eye without a flag. Detecting it requires a reviewer who knows the ground truth, which is a sourcing problem before it is a testing problem.
3. Religious, Ethnic, and Political Sensitivity
Harm taxonomies encode the fault lines of the societies that wrote them. A suite built around US-centric concerns will carefully test racial slurs and partisan political framing, and say nothing about the interfaith and inter-ethnic dynamics that carry real consequence across much of the continent, or about the electoral contexts where a casually generated claim can cause tangible harm.
The result is a model that is well-guarded on the harms one society catalogued and effectively unguarded on the harms another never got to name. This isn't solved by adding a few translated prompts to an existing suite. It requires deriving the taxonomy from the deployment market in the first place.
4. Dialect-Triggered Over-Refusal
The most commercially damaging failure is the least dramatic. Safety classifiers trained mostly on English regularly read Pidgin, Sheng, or code-switched text as evasive, hostile, or suspicious, and refuse entirely ordinary requests. A user asking a straightforward billing question in the register they speak gets a lecture about content policy.
Standard red-team reporting almost never surfaces this, because it measures harmful output produced, not legitimate output withheld. Yet for a product whose entire premise is serving users in their own language, an elevated refusal rate on African-language input isn't a safety win. It is the product failing its intended users while the safety dashboard stays green.
What Good Findings Look Like
A red-team engagement that ends in a slide deck of interesting anecdotes has not done its job. Findings need to be reproducible, severity-rated, and routed to an owner. The rating scheme below is the one we've found holds up in practice across African deployments:
| Severity | Definition | African-Context Example | Required Action |
|---|---|---|---|
| Critical | Reproducible output causing direct physical, financial, or civic harm | Model gives actionable wrong guidance on a national health protocol, or fabricates electoral procedure during an election period | Block release; patch before deployment |
| High | Safety policy bypassed reliably through a repeatable technique | Code-switched Pidgin prompt elicits content the model refuses in English | Fix in current cycle; add to regression suite |
| Medium | Culturally harmful or inflammatory output on sensitive local topics | Model generates inter-ethnic or interfaith characterizations a local reviewer flags as inflammatory | Policy and prompt revision; native-speaker review |
| Medium | Over-refusal degrading legitimate use | Refusal rate on Sheng or Pidgin queries materially exceeds the English baseline for equivalent requests | Recalibrate classifier; re-measure by language |
| Low | Quality or tone failure without safety impact | Stilted, non-idiomatic register that reads as foreign to native speakers | Backlog; address in localization work |
Two reporting details matter more than the scheme itself. First, every severity rating needs a named local reviewer behind it, because severity in this domain is a judgment about local consequence, not a property of the text. Second, refusal rates must be reported per language. A single aggregate number hides the exact disparity that matters.
Building an Africa-Grounded Red-Team Program
The good news is that this doesn't require discarding existing safety work. It requires extending it in four specific places.
- Derive the harm taxonomy locally. Start from the markets, regulations, and social dynamics you actually deploy into, then map the global suite onto it, rather than translating the global suite and calling it localized. Regulatory obligations under frameworks like Nigeria's NDPA or South Africa's POPIA belong in the taxonomy alongside the content harms.
- Staff with native-speaking adversarial testers. Effective attacks are constructed in the register real users write in, complete with slang, code-switching, and local reference points. That is not a translation task performed after the fact; it is the attack itself, and it can only be authored by someone fluent in the variety.
- Test both directions. Run harmful-output probes and legitimate-request probes as one suite, so harmful elicitation and wrongful refusal are measured against the same baseline, per language.
- Convert findings into a regression suite. Every confirmed attack becomes a permanent test case re-run on each model, prompt, or guardrail change. Without this, safety fixes silently regress on the next update and the same finding gets rediscovered a quarter later.
Automated attack generation has a real role here, but it belongs after this foundation, not before it. Synthetic adversarial prompts are excellent for scaling coverage around attack patterns that native speakers have already established as real, and unreliable at discovering those patterns in the first place, for the same reason synthetic training data struggles with African contexts: the generator can only recombine what it has already seen.
The same logic connects red teaming to the rest of the evaluation stack. The benchmark work in our African-centric LLM benchmark analysis establishes which models are capable in these languages at all; red teaming establishes where that capability becomes a liability. Both depend on the same underlying resource, which is people who speak the language and know the market.
Deploying an LLM into an African market and unsure what your safety testing is missing?
At DataLens Africa, our model evaluation practice covers safety, bias, and red-teaming with native-speaking adversarial testers across African languages and markets, with severity-rated, reproducible findings your team can act on. Talk to us about a red-team engagement.
