Examples & Case Studies

Learn from real-world examples of applying scientific methodology to business problems and data analysis challenges.

An online retailer wants to increase conversion rates by testing a new product page design.

Initial Question: “Does the new design improve conversions?”

Refined Scientific Question: “Does the new product page design increase conversion rates by at least 2% compared to the current design, when tested over a 4-week period with sufficient statistical power?”

Research Hypothesis: The new product page design will increase conversion rates due to:

• Improved visual hierarchy
• Better product information presentation
• Enhanced call-to-action buttons

Statistical Hypotheses:

• H₀: No difference in conversion rates between designs (difference ≤ 0%)
• H₁: New design increases conversion rates by ≥ 2%

Experimental Design: Randomized A/B test

• Control Group: Current product page design
• Treatment Group: New product page design
• Random Assignment: 50/50 split of traffic
• Duration: 4 weeks to account for weekly patterns

Power Analysis:

Current conversion rate: 3.5%
Minimum detectable effect: 2% (absolute increase to 5.5%)
Statistical power: 80%
Significance level: 5%
Required sample size: ~8,400 visitors per group

Controls:

• Time-based: Run both designs simultaneously
• Traffic source: Stratify by traffic source (paid, organic, direct)
• Product category: Ensure balanced distribution across categories
• Device type: Track mobile vs. desktop performance

Key Metrics:

• Primary: Conversion rate (purchases/visitors)
• Secondary: Add-to-cart rate, time on page, bounce rate
• Guardrail: Revenue per visitor (ensure no cannibalization)

Data Quality Checks:

• Verify random assignment is working
• Check for bot traffic and filter appropriately
• Monitor for technical issues affecting either design
• Validate tracking implementation

Statistical Test: Two-proportion z-test

Control Group (4 weeks):
- Visitors: 42,156
- Conversions: 1,475
- Conversion Rate: 3.50%

Treatment Group (4 weeks):
- Visitors: 42,203
- Conversions: 1,773
- Conversion Rate: 4.20%

Difference: +0.70 percentage points
Statistical Significance: p < 0.001
95% Confidence Interval: [0.45%, 0.95%]

Multiple Comparison Correction: Applied Bonferroni correction for secondary metrics

Statistical Significance: ✓ (p < 0.001) Practical Significance: Mixed

• Observed increase: 0.70% (below target of 2.0%)
• Revenue impact: $42,000 additional monthly revenue
• Implementation cost: $15,000

Effect Size: Small but meaningful given high traffic volume

Sensitivity Analysis:

• Excluded outlier days (site maintenance, major sales)
• Analyzed mobile vs. desktop separately
• Tested different time periods within the 4 weeks

Robustness Checks:

• Verified results hold across different product categories
• Confirmed no interaction effects with traffic sources
• Validated using different statistical methods (bootstrap, Bayesian)

Recommendation: Implement the new design Rationale:

• Statistically significant improvement
• Positive ROI ($27,000 net monthly benefit)
• No negative effects on guardrail metrics
• Room for further optimization

Limitations Acknowledged:

• Effect size smaller than originally targeted
• Long-term effects unknown
• May not generalize to major seasonal events

A SaaS company wants to predict which customers are likely to churn to enable proactive retention efforts.

Business Question: “Can we identify customers likely to churn in the next 30 days with sufficient accuracy to make retention efforts profitable?”

Analytical Question: “What combination of usage patterns, support interactions, and account characteristics best predicts customer churn with at least 75% precision and 60% recall?”

Domain-Based Hypotheses:

1. Usage Decline: Customers showing decreased product usage in past 30 days are more likely to churn
2. Support Issues: Customers with recent unresolved support tickets have higher churn risk
3. Feature Adoption: Customers using fewer core features are more likely to churn
4. Contract Timing: Customers approaching contract renewal dates have elevated churn risk

Study Type: Retrospective cohort study Observation Period: 12 months of historical data Outcome Window: 30-day churn prediction

Data Preparation:

Total Customers: 15,000
Churned Customers: 1,800 (12% churn rate)
Features: 47 behavioral and demographic variables
Time Windows: 7-day, 14-day, 30-day, and 90-day historical periods

Train/Validation/Test Split:

• Training: 60% (9,000 customers)
• Validation: 20% (3,000 customers)
• Test: 20% (3,000 customers)

Feature Categories:

• Usage Metrics: Logins, feature usage, session duration
• Engagement Signals: Email opens, in-app actions, feature adoption
• Support Interactions: Ticket volume, resolution time, satisfaction
• Account Characteristics: Plan type, company size, contract terms

Feature Selection Process:

1. Univariate Analysis: Statistical significance testing
2. Correlation Analysis: Remove highly correlated features (r > 0.8)
3. Recursive Feature Elimination: Systematic feature importance ranking
4. Domain Expertise: Include features known to be important

Models Tested:

• Logistic Regression (baseline)
• Random Forest
• Gradient Boosting (XGBoost)
• Neural Network

Cross-Validation Results:

Model                 Precision  Recall  F1-Score  AUC-ROC
Logistic Regression      0.72     0.58     0.64     0.83
Random Forest           0.78     0.65     0.71     0.87
XGBoost                 0.81     0.67     0.73     0.89
Neural Network          0.79     0.63     0.70     0.88

Best Model: XGBoost with hyperparameter tuning

Test Set Performance:

Precision: 0.79 (target: 0.75 ✓)
Recall: 0.64 (target: 0.60 ✓)
F1-Score: 0.71
AUC-ROC: 0.88

Feature Importance:

1. Days since last login (23%)
2. Support ticket count (last 30 days) (18%)
3. Feature usage decline (15%)
4. Contract renewal timing (12%)
5. Payment issues (8%)

Cost-Benefit Analysis:

Predicted Churners: 240 customers/month
Retention Campaign Cost: $50 per customer
Campaign Success Rate: 25%
Customers Saved: 60/month
Average Customer LTV: $2,400
Monthly ROI: (60 × $2,400) - (240 × $50) = $132,000

False Positive Analysis:

• False positives: 50 customers/month
• Cost of unnecessary outreach: $2,500
• Potential relationship damage: Minimal (gentle retention offers)

Deployment Strategy:

• Gradual Rollout: Start with 25% of predicted churners
• A/B Testing: Compare retention rates with/without predictions
• Human Review: Sales team reviews high-value accounts

Monitoring Plan:

• Model Performance: Weekly precision/recall monitoring
• Feature Drift: Monthly statistical tests for data changes
• Business Metrics: Monthly retention rate and revenue impact
• Model Refresh: Quarterly retraining with new data

Results After 6 Months:

• Churn rate reduced from 12% to 9.5%
• Retention campaign ROI: 380%
• Model performance maintained (precision: 0.77, recall: 0.62)

A multi-channel retailer needs to understand which marketing channels drive the most valuable customers to optimize budget allocation.

Strategic Question: “How should we allocate our $2M annual marketing budget across channels to maximize customer lifetime value?”

Analytical Question: “What is the true causal impact of each marketing channel on customer acquisition, controlling for customer quality and inter-channel effects?”

Attribution Hypotheses:

1. Last-Click Bias: Current last-click attribution undervalues upper-funnel channels
2. Interaction Effects: Certain channel combinations have synergistic effects
3. Customer Quality: Different channels attract customers with different lifetime values
4. Temporal Effects: Attribution windows significantly impact channel evaluation

Multi-Method Approach:

• Observational Analysis: Customer journey analysis with statistical modeling
• Incrementality Testing: Marketing mix modeling with external factors
• Causal Inference: Difference-in-differences for budget shifts

Data Requirements:

Time Period: 24 months
Customer Journeys: 450,000 complete journeys
Touchpoints: 2.8M marketing touchpoints
Channels: 8 major channels (paid search, social, display, email, etc.)
External Variables: Seasonality, competitors, economic indicators

Model 1: Multi-Touch Attribution (Data-Driven)

• Shapley Value: Game theory approach to credit allocation
• Markov Chains: Probability-based path attribution
• Time Decay: Weighted attribution based on recency

Model 2: Marketing Mix Modeling (MMM)

• Adstock Effects: Carryover effects of advertising
• Saturation Curves: Diminishing returns modeling
• Base vs. Incremental: Separate organic from paid effects

Model 3: Incrementality Testing

• Geo-based Tests: Randomly vary spend by geographic region
• Time-based Tests: Planned budget shifts with control periods
• Holdout Tests: Exclude portions of audience from campaigns

Multi-Touch Attribution Results:

Channel            Last-Click  Shapley  Time-Decay  MMM
Paid Search           35%       28%       31%      25%
Social Media          15%       22%       18%      20%
Display               8%        12%       10%      15%
Email                 12%       15%       14%      12%
Direct                20%       12%       16%      18%
Affiliate             10%       11%       11%      10%

Incrementality Testing Results:

Channel           Spend    Incremental   True ROAS   Recommended
                 (%)      Customers     (vs 4.2x)   Allocation
Paid Search       40%        +15%         3.8x        30%
Social Media      20%        +25%         5.1x        25%
Display           15%        +8%          2.9x        10%
Email             10%        +18%         6.2x        15%
Direct/SEO        10%        N/A          N/A         15%
Affiliate         5%         +12%         4.1x        5%

Cross-Validation:

• Holdout Period: Last 3 months held for validation
• Prediction Accuracy: MMM predicted actual revenue within 3%
• Attribution Consistency: Shapley and MMM showed similar patterns

External Validation:

• Industry Benchmarks: Results aligned with industry attribution studies
• Incrementality Validation: Test results confirmed MMM incrementality estimates
• Business Logic: Results passed domain expert review

Budget Reallocation Recommendations:

Channel          Current   Recommended   Change    Expected Impact
Paid Search        40%        30%        -25%     Reallocate to higher ROAS
Social Media       20%        25%        +25%     Significant underinvestment
Display            15%        10%        -33%     Lower incrementality
Email              10%        15%        +50%     Highest ROAS channel
Direct/SEO         10%        15%        +50%     Invest in organic growth
Affiliate           5%         5%         0%      Maintain current level

Projected Annual Impact:

• Revenue increase: $1.2M (+8%)
• Customer acquisition cost reduction: 12%
• Lifetime value improvement: 15%

Phased Implementation:

• Phase 1: 25% budget shift (3 months)
• Phase 2: 50% budget shift (3 months)
• Phase 3: Full implementation with ongoing optimization

Continuous Learning:

• Monthly MMM Updates: Refresh models with new data
• Quarterly Testing: New incrementality experiments
• Annual Deep Dive: Comprehensive attribution analysis

Challenges and Solutions:

• Privacy Changes: Adapted to iOS 14.5 and cookieless tracking
• Cross-Device: Improved identity resolution methodology
• New Channels: Extended framework to include emerging platforms

A mobile app company launched a new recommendation feature and needs to measure its impact on user engagement and retention.

Product Question: “Does the new recommendation feature improve user engagement?”

Scientific Question: “Does exposure to the new recommendation feature increase daily active usage by at least 10% and 7-day retention by at least 5%, measured over an 8-week period?”

Theoretical Framework: Social proof and personalization theory predict that relevant recommendations will:

1. Increase content discovery and consumption
2. Improve user satisfaction and engagement
3. Reduce churn through better content fit

Measurable Predictions:

• H₁: Users with recommendations will have 10%+ higher daily session time
• H₂: Recommendation users will have 5%+ higher 7-day retention
• H₃: Users will interact with recommended content at >15% rate

Design Type: Stratified randomized controlled trial Population: New users (to avoid learning effects) Assignment:

• Control: 40% (no recommendations)
• Treatment A: 30% (basic recommendations)
• Treatment B: 30% (advanced ML recommendations)

Stratification Variables:

• Device type (iOS/Android)
• Geographic region
• App version
• User acquisition source

Feature Implementation:

# Pseudo-code for randomization
def assign_user_to_group(user_id, user_attributes):
    # Stratified randomization
    strata = get_strata(user_attributes)
    random_seed = hash(user_id + strata + experiment_salt)

    if random_seed % 100 < 40:
        return "control"
    elif random_seed % 100 < 70:
        return "treatment_a"
    else:
        return "treatment_b"

Data Collection:

• User Events: App opens, session duration, content views
• Recommendation Events: Impressions, clicks, engagement
• Retention: Daily/weekly active status
• Quality Metrics: App store ratings, crashes

Sample Size and Power:

8-week experiment period
Control group: 45,000 users
Treatment A: 35,000 users
Treatment B: 35,000 users
Power: 90% to detect 10% relative increase

Primary Analysis Results:

Metric                  Control   Treatment A   Treatment B
Daily Session Time      8.5 min   9.1 min      9.8 min
(95% CI)               (8.3-8.7)  (8.9-9.3)    (9.6-10.0)
Relative Change           -        +7.1%        +15.3%
P-value                   -        0.023        <0.001

7-Day Retention         72.3%     74.1%        76.8%
(95% CI)               (71.8-72.8) (73.6-74.6) (76.3-77.3)
Relative Change           -        +2.5%        +6.2%
P-value                   -        0.089        <0.001

Recommendation CTR        -        12.3%        18.7%
(95% CI)                  -       (11.9-12.7)  (18.2-19.2)

Secondary Analysis:

• Segmentation: Power users showed stronger response
• Time Trends: Effect strengthened over time (learning curve)
• Content Categories: Recommendations worked better for entertainment vs. news

Instrumental Variable Analysis:

• Used random assignment as instrument for recommendation exposure
• Confirmed causal interpretation of observational correlations

Difference-in-Differences:

• Compared user behavior before/after feature launch
• Validated experimental results with observational data

Robustness Checks:

• Outlier Analysis: Results robust to removing extreme users
• Covariate Balance: Verified randomization worked properly
• Missing Data: Multiple imputation confirmed results

Decision Criteria:

Feature Success Criteria:
✓ Statistical significance: p < 0.05
✓ Effect size: >10% engagement increase (Treatment B: 15.3%)
✓ Retention improvement: >5% (Treatment B: 6.2%)
✓ No negative impacts on app performance
✓ Positive user feedback (4.2/5.0 rating)

Cost-Benefit Analysis:

Implementation Cost: $200,000 (development + infrastructure)
Ongoing Costs: $50,000/year (ML infrastructure)

Benefits (Annual):
- Improved retention: +$1.2M revenue
- Increased engagement: +$800K ad revenue
- Reduced churn: +$400K saved acquisition costs
Net Benefit: +$2.15M/year
ROI: 430%

Rollout Plan:

• Week 1: Deploy advanced recommendations (Treatment B) to 10% of users
• Week 2-3: Scale to 50% monitoring for issues
• Week 4: Full rollout with ongoing optimization

Success Metrics Dashboard:

• Real-time monitoring of key engagement metrics
• Weekly cohort retention analysis
• Monthly business impact assessment
• Quarterly model performance review

Long-term Learning:

• A/B Testing Platform: Integrated learnings into experimentation framework
• Personalization Strategy: Informed broader personalization roadmap
• Data Science Team: Developed reusable causal inference methodology

1. Clear Question Definition: Specific, measurable, business-relevant questions
2. Hypothesis-Driven: Theory-based predictions tested with data
3. Rigorous Design: Appropriate methodology for causal questions
4. Quality Control: Data validation and assumption testing
5. Multiple Validation: Cross-validation and robustness checks
6. Business Integration: Results translated to actionable recommendations

Pitfall: Testing too many variables simultaneously Solution: Focus on primary hypothesis with pre-planned secondary analyses

Pitfall: changing analysis after seeing results Solution: Pre-register analysis plan and follow systematically

Pitfall: Ignoring practical significance for statistical significance Solution: Always consider business impact and implementation costs

Pitfall: Overgeneralizing from single studies Solution: Replicate findings across different contexts and time periods