Arabic quality is a service promise

A fluent reply can still produce the wrong record

A customer describes a claim in colloquial Arabic, names a product in English and attaches a formal document. The service must preserve the event date, identifier and any negation through to submission. A fluent reply can still produce the wrong claim. The investment decision is therefore which language route completes the promised task accurately, at an acceptable operating cost.

ArabicMMLU contains 14,575 multiple-choice questions across 40 tasks in Modern Standard Arabic, sourced from educational examinations across several regions. ¹ It provides a valuable knowledge-understanding test with a defined population. It does not establish that a model can resolve an informal complaint, recognize a spoken account number or explain an ambiguous policy. A benchmark should inform a shortlist and an evaluation design; it should not substitute for evidence that a customer can complete the intended task.

Define a testable Arabic service promise: who can complete which task, through which channels and language varieties, with what route to clarification or a person. This makes launch scope explicit and gives product teams a decision they can evaluate. A general claim to support Arabic leaves the consequential choices unresolved.

Test the language varieties the service promises

Arabic services operate across formal writing, local speech, mixed-language expressions and differing conventions for names and numbers. AraBench's dialectal Arabic-to-English translation suite includes four coarse, fifteen fine-grained and twenty-five city-level dialect categories across multiple genres. ² Those counts describe the benchmark's organization, not a definitive taxonomy of Arabic speakers. Their value for service design is to make variation visible instead of treating one formal test set as representative of every interaction.

The MADAR shared task likewise distinguishes travel-domain dialect identification from identification using Twitter users, with city- and country-level labels. ³ Recognizing a dialect is a narrower task than delivering an appropriate service in it. A company should not infer a user's identity, preferences or eligibility from a model's dialect guess. Where language selection matters, offer a clear choice or ask a practical clarification instead of making a consequential assumption.

ArabCulture-Dialogue adds a useful dimension to Arabic evaluation. The April 2026 preprint contains 3,471 dialogue cases in paired Modern Standard Arabic and dialect versions, covering 13 countries; its construction involved 26 annotators. The authors acknowledge incomplete country coverage, variation within national dialects and translation effects. ⁸ These are evaluation resources, not proof that a customer service works well in every market.

For a claims service, test the same intention through formal Arabic, the dialects in the launch market and mixed-language input. Keep cultural comprehension separate from the accuracy of the eventual claim record. A model can choose a plausible reply yet still reverse a date, omit an exception or route the case incorrectly. mayden proposes one release decision with separate evidence for comprehension, transaction accuracy and a usable escalation path.

Separate comprehension from transaction accuracy

LAraBench evaluated 33 tasks across 61 public datasets, including language and speech processing, in its 2024 study. ⁴ Its breadth helps identify which parts of the service require separate testing. Transcription, classification, question answering and text generation can have different failure patterns. A service that performs well on one may still fail where its operating journey joins them. Keep those components visible in the evaluation rather than collapsing them into one Arabic score.

For a voice-based support service, measure whether the transcript preserves the information required to act. A transcript can contain several harmless wording differences while preserving the correct request. It can also look fluent while changing a negation, a date or a product code. Review those fields against the source and intended action. The metric should reflect the cost of misunderstanding, with language specialists and service owners jointly defining what counts as a material error.

BALSAM's 2025 publication describes 78 tasks in fourteen categories and a blind evaluation platform, with separate development and test examples. ⁵ Its design addresses breadth and evaluation exposure. Enterprise teams can adopt the principle of holding back fresh cases without claiming that their private set is a representative benchmark. Preserve cases that were not used to tune prompts or choose a model, and refresh them as the service encounters new requests.

Choose the route using matched business cases

There are at least three practical routes through a bilingual service. The system can process Arabic directly, translate through an English intermediate representation, or combine language-specific components with a shared business workflow. Each route changes where meaning can be lost and where specialist review is required. Choose through a task-level comparison. The strongest general-purpose model may be preferable for one service; a specialized component may be better for another. Architecture should follow observed performance and operating constraints.

The original Jais and Jais-chat paper describes Arabic-centric foundation and instruction-tuned models trained with Arabic, English and code. ⁶ It demonstrates a concrete research approach to language-specific capability; its 2023 model comparisons are not current procurement rankings. The useful question is what a candidate model contributes to the service today, under the organization's data, deployment and review conditions. Brand association with a language or region cannot answer that question.

Run the alternatives on matched cases and retain the same acceptance criteria. Count missing qualifiers, altered names, unsupported conclusions, clarifications and specialist corrections. Include the full cost of translation, inference, review and recovery. Record whether any routing rule sends confidential material to a different processing environment. A route that improves language quality may still be unsuitable under a separate operational constraint. Make that trade-off explicit rather than hiding it inside a single technical score.

Preserve meaning through the interface

W3C's Arabic and Persian Layout Requirements describes right-to-left writing, bidirectional text and layout concerns. The October 2025 publication is a Group Draft Note, not an endorsed compliance standard. ⁷ It is a useful technical reference for a product problem that executives should understand: a correct underlying answer can become difficult to use when names, numbers, controls or document references are presented badly.

Test the complete journey on the devices and assistive technologies relevant to the audience. Include mixed-script account identifiers, dates, currency, error messages, attachments and downloaded records. Check whether copying a reference preserves its intended order and whether a customer can find the next action without changing language. These are proposed acceptance checks requiring direct usability testing. A mirrored navigation bar provides little evidence about the usability of a consequential form or decision notice.

Content ownership matters as much as rendering. Establish an approved vocabulary for products, policy terms and customer actions. Preserve distinctions that carry legal or operational meaning, while allowing natural explanations for users. A literal translation can be technically faithful yet unhelpful; an elegant paraphrase can accidentally remove a condition. Give the bilingual reviewer access to the original policy and the task context, with a clear process for resolving ambiguous terms rather than improvising them repeatedly.

Keep the claim’s facts intact from intake to submission

Consider a fictional insurer designing a claims-intake assistant. Customers may describe an incident in a local dialect, submit an English invoice and receive formal Arabic correspondence. The assistant can collect information and prepare a case summary; it cannot decide coverage. This scope creates a useful comparison between direct Arabic processing and a translation-based route while keeping the consequential decision with an authorized specialist. No performance or client outcome is assumed in this example.

Start with the English operational specification in Exhibit 02. The fictional customer says that no payment was received, gives an incident date of 12 September 2026 and supplies claim identifier CL-1042. An invoice contains a different billing date. The expected record must retain the negation, select the incident date from its labelled source and preserve the identifier exactly. It must not silently substitute the invoice date or infer coverage. These are synthetic facts used to define acceptance, not an Arabic-language test result.

A competent Arabic reviewer should then author and check matched utterances in the language varieties and channels the service intends to support. Keep the underlying facts and expected fields constant while varying the expression. Record who reviewed each case, which variety it represents and any unresolved interpretation. This report supplies neither those Arabic utterances nor a claim of native review. A date without a clear calendar or an identifier that cannot be read confidently should trigger clarification or human handover, not a guessed normalization.

Suppose one route requires fewer clarifications but produces more specialist corrections to mixed-language documents. That result would not automatically justify selecting either route. Examine whether the corrections concern decisive facts or cosmetic phrasing, whether they are concentrated in identifiable document types and whether routing can isolate those cases. The options include a document-specific path, a narrower launch scope or additional review. The test should reveal which operational choice the evidence supports.

Let customers inspect and correct the captured facts before submission in a language they can use. Preserve the original material alongside the approved summary under the applicable retention and access controls. When transferring to a person, retain context so the customer does not need to repeat the story. This is a proposed service design. Its commercial hypothesis is that accurate intake and effective handover may reduce avoidable rework; the organization must measure that outcome before claiming it.

Share business rules; specialise the language work

Language-specific work has a cost: duplicated operations, fragmented releases and specialist capacity. Arabic-first design need not require a separate foundation model or every dialect at launch. It requires an explicit supported experience and evidence from the people and tasks it is intended to serve.

Separate reusable business rules from language-dependent interpretation and communication. Keep eligibility, transaction limits and access rights consistent where policy is consistent; adapt vocabulary, clarification and interaction to the user. mayden proposes this separation as an option to test, not a universally superior architecture. Its value depends on scale, change frequency and the ability to operate the components.

Prioritize gaps by consequence and observed demand. A rare pronunciation issue affecting a critical identifier may deserve attention before a frequent stylistic preference. Maintain a supported-scope statement and an effective human route for cases outside it. Avoid presenting an untested dialect as covered merely because the model occasionally produces convincing examples. A credible limited promise is a stronger starting point for expansion than an expansive claim that users discover is unreliable.

Invest where the Arabic journey loses meaning

Review Arabic service quality through completion, correction, escalation and unresolved outcomes for the supported segments. Compare matched cases where possible and account for differences in case complexity. A raw performance gap does not establish a language effect; small samples should remain visibly uncertain. Use the review to find the specific journey that needs repair.

Give one product owner the authority and budget to maintain the complete experience. Language and domain specialists need access to source policies, terminology decisions and new failure cases. Reassess when the model, policy or channel changes. Launch approval is the beginning of the service commitment, not its final translation check.

Choose the next investment from the failure pattern. If decisive facts are lost in translation, change or narrow that route. If the record is correct but customers cannot submit it, fix the interface or handover. If specialists repeatedly correct policy terms, repair terminology ownership. mayden’s differentiation hypothesis is better completed work and lower avoidable effort; measure those outcomes before claiming a commercial advantage.