Guides
  1. core-concepts

Experiences & Variations

How A/B tests work — the ExperienceManager and variation selection

Welcome back! In Context, we saw how the Context object acts as a personal guide for each visitor. When we wanted to know which headline variation 'user123' should see in our A/B test, we called the context's runExperience('headline-test') method. We learned that the Context object doesn't make this decision itself; it delegates the task to a specialist.

Let's meet that specialist: the ExperienceManager.

The Problem: Directing the A/B Test Show

Imagine you're running multiple A/B tests (which we call "experiences" in Convert):

  • A headline test on the homepage.
  • A button color test on the pricing page.
  • A special promotion shown only to users from Canada.

When a visitor like 'user123' navigates your site, how does the SDK ensure they get the correct variation for each relevant experiment? How does it know:

  1. Which experiences are even active?
  2. Does 'user123' meet the specific targeting rules for each experience (e.g., "only for users from Canada")?
  3. If they qualify, which variation (A or B) should they consistently see based on the traffic distribution (e.g., 50% see A, 50% see B)?

Managing all this decision logic for just the A/B tests requires a dedicated component.

What is ExperienceManager? The A/B Test Director

Think of the ExperienceManager as the Director specifically focused on running your A/B tests (experiences). While the ConvertSDK is the overall conductor, and the Context is the personal guide for a visitor, the ExperienceManager is the expert who knows all about the different A/B test "scenes" being filmed.

Its main jobs are:

  1. Knowing the Script: It can tell you about all the active experiences (using getList()) or give you details about a specific one (using getExperience(key) or getExperienceById(id)).
  2. Casting the Role: Its most crucial job is deciding which variation a specific visitor should see for a particular experience. This involves checking rules and splitting traffic fairly. This is done via the selectVariation() method (or selectVariationById() when working with IDs).
  3. Collaborating with Specialists: The ExperienceManager doesn't do everything alone! To make the final decision in selectVariation(), it relies heavily on other managers:
    • DataManager: To get the details of the experience and its rules.
    • RuleManager: To check if the visitor meets the targeting conditions (handled via DataManager).
    • BucketingManager: To assign the visitor to a variation based on traffic allocation (also handled via DataManager).

So, ExperienceManager directs the A/B test, but it uses the DataManager as its primary assistant, which in turn consults with the RuleManager and BucketingManager.

How it's Used (Mostly Indirectly)

As a developer using the SDK, you will rarely interact directly with the ExperienceManager. Why? Because the Context object handles it for you!

When you call the context's runExperience('headline-test') method, the Context object internally turns around and asks the ExperienceManager: "Hey, can you please figure out the variation for 'user123' in the 'headline-test' experience?"

Specifically, the Context calls the ExperienceManager's selectVariation() method, passing along the visitor's ID, the experience key, and any relevant visitor properties.

So, while ExperienceManager is doing the heavy lifting for choosing A/B test variations, you trigger its work through the convenient methods on the Context object.

Under the Hood: How selectVariation Works

Let's peek behind the curtain. What happens when the Context calls selectVariation('user123', 'headline-test', attributes)?

  1. Delegation to DataManager: The first thing the ExperienceManager does inside its selectVariation method is delegate the task to the DataManager. It calls the DataManager's getBucketing('user123', 'headline-test', attributes) method.
  2. DataManager Orchestrates: The DataManager takes over. It fetches the definition of the 'headline-test' experience (including variations, traffic split, and targeting rules).
  3. Rule Checking: The DataManager uses the RuleManager to check if 'user123' (with their attributes) matches the targeting rules defined for 'headline-test'. If not, the process stops here, and an error indicating the visitor didn't qualify is returned.
  4. Bucketing: If the rules pass, the DataManager uses the BucketingManager to determine which variation ('variation-A' or 'variation-B') 'user123' should see, based on their ID and the traffic distribution percentages set for the experiment. This ensures the visitor consistently sees the same variation.
  5. Result Returned: The DataManager returns the result (either the chosen variation object, a RuleError, a BucketingError, or null) back to the ExperienceManager.
  6. Logging and Pass-Through: The ExperienceManager logs diagnostic information about the result and then passes the result back to the Context object, which then returns it to your original runExperience call.

Here's a simplified diagram showing the flow:

sequenceDiagram
    participant Context as Visitor Context
    participant ExpManager as ExperienceManager
    participant DataManager as DataManager
    participant RuleManager as RuleManager
    participant BucketManager as BucketingManager

    Context->>ExpManager: selectVariation('user123', 'headline-test', attributes)
    activate ExpManager
    ExpManager->>DataManager: getBucketing('user123', 'headline-test', attributes)
    activate DataManager
    Note right of DataManager: Fetches experience data
    DataManager->>RuleManager: Check rules for 'user123' against 'headline-test' rules
    activate RuleManager
    RuleManager-->>DataManager: Rules Pass/Fail
    deactivate RuleManager

    alt Rules Pass
        DataManager->>BucketManager: Determine variation for 'user123' based on traffic split
        activate BucketManager
        BucketManager-->>DataManager: Return chosen variation (e.g., 'variation-B')
        deactivate BucketManager
        DataManager-->>ExpManager: Return variation object
    else Rules Fail
        DataManager-->>ExpManager: Return RuleError or null
    end
    deactivate DataManager
    Note right of ExpManager: Logs result diagnostics
    ExpManager-->>Context: Return variation object or error
    deactivate ExpManager

As you can see, ExperienceManager acts primarily as an entry point that quickly hands off the complex logic to the DataManager, which coordinates the rule checking and bucketing specialists.

Key Implementation Details

1. Construction

The ExperienceManager receives a reference to the DataManager (and optionally a LogManager) when it is created by the main SDK Core. It stores these references for later use.

2. The selectVariation Method

The ExperienceManager's selectVariation method calls the DataManager's getBucketing() with the exact same arguments it received (visitor ID, experience key, and attributes). It acts as a pass-through for this core function. After receiving the result, it logs diagnostic details including the specific reason for null results (experience not found vs. visitor not qualified), then returns the result.

The four possible outcomes of selectVariation are:

  • A successfully bucketed variation object
  • A RuleError (visitor did not match targeting rules)
  • A BucketingError (variation could not be decided)
  • Null (experience not found or other failure)

3. The selectVariations Method

This bulk method runs selectVariation for every experience and filters to only successful results. It iterates over all experiences from getList() and calls selectVariation for each one. Only successfully bucketed variation objects survive the filter -- errors and nulls are discarded.

4. Other Helper Methods

Methods like getExperience, getExperienceById, and getList also rely on the DataManager to look up and return the requested data about experiences. Both key-based and ID-based lookups are supported (getExperience / getExperienceById, getVariation / getVariationById), giving you flexibility in how you reference your experiences.

Conclusion

The ExperienceManager is the dedicated component within the Convert SDK responsible for managing A/B tests (experiences). While you typically interact with it indirectly via the Context object's runExperience method, it plays a vital role behind the scenes.

You've learned that:

  1. ExperienceManager acts as the "Director" for A/B tests.
  2. Its main job is to select the correct variation for a visitor using selectVariation (or selectVariationById for ID-based lookups).
  3. It achieves this by delegating heavily to the DataManager, which coordinates with the RuleManager and BucketingManager.
  4. Methods like getExperience, getExperienceById, and getList allow access to experience configuration data, again via the DataManager.
  5. The selectVariations method provides a convenient way to bucket a visitor across all experiences at once.

ExperienceManager handles the complexity of A/B testing variations. But what if you don't need multiple variations? What if you just want to turn a feature on or off for certain users, or gradually roll out a new feature? For that, we have another specialist.

Let's explore how the SDK handles feature flags next: FeatureManager!

Updated about 1 month ago