Localization A/B testing can produce a confident answer to the wrong question. If one locale receives stale treatment text, another falls back to English, and a third activates its variant after the measured event, the aggregate result does not isolate copy. It mixes translation state, configuration timing, and language-specific behavior. Fix that by treating the source hypothesis, translated branches, eligibility, assignment, activation, and analysis as one versioned contract. The payoff is a result you can act on per locale without mistaking delivery failures for user preference.

Diagnose what the experiment can accidentally measure

A normal copy test assumes that control and treatment differ in one intended way. Multilingual delivery adds several uncontrolled differences:

Translators may interpret the treatment more strongly or weakly than the source.
One branch may be approved while the other is missing or stale.
A broad fallback can silently show English to users assigned to another locale.
Configuration can activate after a screen impression or conversion event.
A user can change the app language after assignment.
An aggregate winner can be driven by one large locale while smaller locales move in the opposite direction.

Remote delivery alone does not solve these problems. Firebase lets a team define defaults, conditional values, and versioned templates through Remote Config parameters and conditions. Those controls determine which value reaches an app instance. They do not prove that two translated values preserve the same experimental contrast.

Developers are trying to use remote configuration as a runtime translation store. One Flutter developer wants to store localized JSON in Remote Config, while another wants translation changes to reach users without an app update. These are individual questions, not evidence that Firebase recommends Remote Config as a translation system. They expose a boundary the app must control: remote values can bypass the catalog, fallback rules, and review path that bundled strings normally provide.

Write the experiment claim before creating any parameter. A useful claim is specific enough to falsify:

Replacing the checkout button label from “Continue” to “Review order” will increase completed checkouts because users understand that payment is not submitted on the next screen.

The translated treatment must preserve that reassurance. A locale that changes it to “Place order” is not another rendering of the treatment. It is a different product promise and must not enter the same experiment.

Define one versioned experiment contract

Keep experiment state in a record that links product intent to localization and telemetry. The feature-flag service can store assignment values, but the contract should remain reviewable with the source copy and translations.

experiment_id: checkout-review-label-v3
hypothesis_revision: 3
assignment_unit: authenticated-user
activation_boundary: checkout-screen-open
primary_event: checkout-completed
control:
  message_id: checkout.continue
  source_revision: 8f21
  semantic_note: proceed without submitting payment
 treatment:
  message_id: checkout.review-order
  source_revision: a713
  semantic_note: open the final review before payment
eligible_locales:
  - en
  - de
  - ja
catalog_revision: mobile-copy-2026-09-14
fallback_policy: exclude-before-exposure
owner: checkout-growth

This schema is an implementation recommendation, not a Firebase requirement. It ties the hypothesis revision and semantic notes to message identities, locales, the catalog revision, the activation boundary, and the measured event. If any of those changes, start a new experiment revision rather than editing history in place.

Do not put a complete long-lived translation catalog into one generic JSON parameter. An Android developer asking whether Remote Config can update strings.xml at runtime describes why that shortcut is awkward: platform resources are not replaced by parsing remote JSON. An app can render remote strings through its own resolver, but then it owns validation, fallback, caching, formatting, accessibility context, and rollback.

Use remote configuration for the small branch selector or a versioned experiment overlay. Keep normal product copy in the localization system unless the experiment genuinely requires remote text delivery.

Prove semantic parity before enrollment

Use a semantic-parity matrix to prove every locale tests the same control and treatment meanings before enrollment.

Configuration setup pages focus on targeting and values, not whether translated branches preserve the source experiment. A locale is not eligible merely because both cells are non-empty. A localization lead must confirm that the same intended distinction survives in both branches.

For each locale, record:

The exact control and treatment strings.
The source revision each string translates.
The semantic contrast the pair must preserve.
Placeholder, plural, length, and accessibility checks.
Linguistic approval for both branches as a pair.
The app and catalog revisions used in verification.

Review branches side by side. Reviewing the treatment alone can miss that the control already uses equivalent wording in one locale, leaving no meaningful contrast. It can also miss a grammatical dependency, such as a noun gender or case that changes surrounding copy.

Do not demand literal similarity across languages. Semantic parity means that the intended product difference is comparable, not that sentence structure or word count matches. If a language cannot express the distinction naturally in the available space, exclude that locale or design a locale-specific experiment with its own hypothesis. Do not force weak copy into the global test to preserve sample size.

Treat any post-approval text change as a new artifact. Re-run pair review when source copy, terminology, variables, fallback rules, or the containing screen changes. The existing i18nAgent guide to safe over-the-air translation updates explains source hashes, validated activation, cached startup, and bundled fallback for remote translation artifacts. Apply the same version discipline to experiment branches, then add the assignment and analysis controls covered here.

Make locale eligibility explicit

Compute eligibility before exposure. The decision needs the effective app locale, supported experiment locales, approved branch versions, and minimum app version. A user is eligible only when both control and treatment can render correctly under the same catalog and screen contract.

eligible(user, experiment):
  locale = resolve_effective_app_locale(user)
  pair = approved_pair(experiment.revision, locale)

  if locale not in experiment.eligible_locales:
    return false
  if pair is missing or pair.catalog_revision != app.catalog_revision:
    return false
  if app.version < experiment.minimum_app_version:
    return false
  return validate_message_shape(pair.control, pair.treatment)

Do not enroll first and then rely on fallback. If a French treatment is missing and the resolver falls back to English, the user remains counted in the French segment while seeing a different language. That pollutes both intent and locale analysis. Render ordinary approved product copy outside the experiment instead, and record a language-neutral ineligibility reason for operations.

Separate locale from country. Firebase's Remote Config experiment guide supports targeting by languages and locales as well as countries and regions. Those dimensions answer different questions. French speakers in Canada and English speakers in Canada should not be grouped merely because they share a country. Likewise, a device language may differ from the language selected inside the app. If the platform cannot target the app's effective locale directly, pass a validated user property or perform an app-side eligibility check before exposure.

Preserve assignment and control activation timing

Keep variant assignment stable across language changes, but exclude a locale before exposure when either translated branch is unavailable.

Stable assignment preserves the experimental contrast. Derive the branch from a stable user or installation identifier plus the immutable experiment revision. Locale selects the translated rendering of that branch; it should not generate a fresh branch by itself. Otherwise a user switching from English to German can move from control to treatment, creating carryover and double-exposure problems.

If the new locale lacks an approved pair, stop experimental exposure and show canonical copy. Do not switch branches to find an available string. Record that the assignment still exists but the current locale is ineligible. If the user returns to an eligible locale, the same branch can resume under the same revision.

Activation must precede the event attributed to the variant. Firebase explicitly says the selected activation event must occur after the app activates fetched configuration values in its Remote Config A/B Testing documentation. Its loading-strategy guide also warns that fetching and activating during startup can cause visible UI changes.

Pick one boundary and test it:

Activate on the previous screen, then measure the target-screen impression.
Fetch in the background and activate on the next session.
Activate behind a blocking transition only when the delay is bounded and acceptable.

Never log exposure when the app receives an assignment. Log it when the approved localized branch is actually rendered. Include experiment ID, revision, branch, effective locale, translation-pair revision, app version, and activation revision. Avoid sending raw copy in analytics because it increases cardinality and can expose text that should remain in controlled artifacts.

Analyze each locale before trusting the aggregate

Start with an allocation check. For every eligible locale and app version, compare assigned users, activated users, rendered exposures, and primary events. A large drop between assignment and exposure often indicates loading, eligibility, or screen-routing differences rather than user response.

Then inspect the primary outcome by locale. The aggregate result is useful only after these counterchecks:

No locale has a sample-ratio mismatch between control and treatment.
Both branches used the expected translation-pair revision.
Exposure occurred before the primary event.
Missing-pair and fallback counts stayed at zero for enrolled users.
Crash, latency, cancellation, and support guardrails did not worsen.
Large locales did not hide a material reversal in another eligible locale.

Do not call every locale independently significant when samples are small. The practical decision may be to roll out globally, hold one locale, or continue collecting data. State the rule before reading results. For example, require a positive aggregate outcome, no locale breaching a defined harm guardrail, and manual review of any locale whose effect direction reverses.

Language performance can reflect culture, market, acquisition channel, price, device mix, or release timing. Segmentation reveals the pattern but does not automatically identify its cause. Follow up with a narrower test when a locale behaves differently instead of claiming the translation caused the difference.

Promote the winner and remove experiment debt

Promote the winning copy into the canonical localization catalog and retire the experiment parameter instead of creating permanent remote copy.

Firebase's experiment setup and rollout pages do not define how approved copy returns to a product's translation catalog. Without that handoff, the flag service becomes a second translation database. Once the decision is final, merge the approved source message and locale translations under normal keys, review any excluded locales, ship the catalog change, and verify bundled or validated OTA fallback.

Use this sequence:

Freeze the winning experiment revision and export its approved locale pairs.
Open a catalog change that links back to the experiment decision.
Complete normal placeholder, plural, terminology, screenshot, and linguistic review.
Ship the canonical copy while the experiment still resolves consistently.
Verify exposure no longer depends on the remote branch.
Remove the parameter, targeting rule, and obsolete analytics handling after the supported app-version window closes.

Firebase stores previous Remote Config template versions and supports rollback, according to its parameter documentation. Use that history for operational recovery, not as the permanent source of product language.

Handle failures without corrupting the test

Define failure behavior before launch:

Fetch fails: use the last activated valid assignment only if its experiment and translation revisions still match. Otherwise show canonical copy and log no exposure.
One translated branch fails validation: disable enrollment for that locale. Never compare a valid control with a malformed treatment.
Language changes mid-session: preserve branch assignment, resolve the same branch in the new locale, and stop exposure if the pair is unavailable.
Template changes during a session: activate at the declared boundary rather than replacing visible copy halfway through the measured task.
A translation must be corrected: create a new experiment revision or predeclared correction policy. Silent edits make earlier and later exposures incomparable.
A locale shows harm: stop that locale without rewriting historical assignment, then return users to canonical copy.

Rollback should restore a known translation-pair revision and activation policy together. Rolling back only the text while leaving new eligibility or telemetry rules active creates another mixed experiment.

Verify one locale pair end to end

Run a controlled test before opening traffic. Use one locale with a distinct script or longer copy so fallback and layout errors are visible.

Confirm that:

control and treatment preserve the documented semantic contrast
both branches pass message-structure and linguistic review
eligibility is false when either branch is missing or stale
the same user keeps the same branch after changing to another eligible language
an unsupported language receives canonical copy and no experiment exposure
configuration activates before the target impression and primary event
exposure records contain experiment, branch, locale, translation revision, and app version
dashboards can compare assignment, activation, exposure, and outcomes per locale
a locale-level guardrail can stop one language without changing other assignments
the winner can move into the catalog and the remote parameter can be retired

Create that semantic-parity matrix and run these checks on a test cohort before allocating production traffic. If any enrolled user can see fallback copy or log the outcome before variant activation, the experiment is not ready.

Store listings run their own experiments with the same risks; see how to localize storefront variants without corrupting App Store experiments.