added CUPED and outliers info #248
Conversation
chris-absmartly
requested review from
bmsilva,
calthejuggler,
marcio-absmartly and
mario-silva
as code owners
✅ Deploy Preview for absmartly-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
WalkthroughThis pull request adds CUPED guidance and clarifications across metrics documentation: it updates docs/web-console-docs/goals-and-metrics/metrics/create.mdx to include CUPED lookback cautions near Time filters and Retention, inserts inline info blocks clarifying that Quantile and Standard Deviation outlier capping uses the full experiment population, and duplicates CUPED guidance in multiple locations. It also introduces a new file, docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx, documenting CUPED purpose, adjustment formula, applicability, examples, metric compatibility, statistical validity, benefits, and integration notes. Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Fix all issues with AI agents
In `@docs/web-console-docs/goals-and-metrics/metrics/create.mdx`:
- Around line 117-119: Fix the typo and punctuation in the outlier info blocks:
change "Ouliers capping is done using the full experiment population so all
variants have the same capping limits." to "Outliers capping is done using the
full experiment population, so all variants have the same capping limits."
(apply the same correction to the other identical block).
- Around line 343-345: The two sentences about CUPED read choppy; replace them
with a single smoother sentence that references the variance-reduction-cuped
link and uses correct verb agreement and phrasing — e.g., instruct the reader to
enable CUPED (when applicable) by checking the checkbox and choosing a lookback
period that matches user behavior — updating the lines that mention "CUPED" and
the link "variance-reduction-cuped" to this single polished sentence.
In
`@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`:
- Around line 109-110: Fix the grammar in the sentence "When creating a new
metric or a new version of an existing metrics you can enabled CUPED." by
changing it to "When creating a new metric or a new version of an existing
metric, you can enable CUPED." — update the text in the metrics/CUPED paragraph
so "metrics" becomes singular ("metric"), add the missing comma after the
clause, and change "enabled" to the correct verb form "enable."
- Around line 29-46: Tighten wording and range formatting in the "When CUPED is
Most Effective" section: reword the θ line to "θ (theta): optimal coefficient
(typically the correlation between pre- and post-metrics)" and ensure the prose
sentences are concise; normalize list ranges to use consistent en-dash ranges
(e.g., "2–4 weeks", "1–4 weeks") and remove the extra comma/verbosity in the
pre-period selection line so it reads like "In ABsmartly you can choose 1–4
weeks; the default is 2 weeks." Also make minor grammar tweaks for parallelism
in bullets (e.g., "Revenue metrics typically correlate 0.5–0.7", "Engagement
metrics 0.4–0.6", "Conversion metrics may be lower but still useful").
| :::info | ||
| Ouliers capping is done using the full experiment population so all variants have the same capping limits. | ||
| ::: |
There was a problem hiding this comment.
Fix typo and add comma in outlier info blocks.
Suggested edits
-:::info
-Ouliers capping is done using the full experiment population so all variants have the same capping limits.
-:::
+:::info
+Outlier capping is done using the full experiment population, so all variants have the same capping limits.
+:::
-:::info
-Ouliers capping is done using the full experiment population so all variants have the same capping limits.
-:::
+:::info
+Outlier capping is done using the full experiment population, so all variants have the same capping limits.
+:::Also applies to: 149-151
🧰 Tools
🪛 LanguageTool
[uncategorized] ~118-~118: Use a comma before ‘so’ if it connects two independent clauses (unless they are closely connected and short).
Context: ...one using the full experiment population so all variants have the same capping limi...
(COMMA_COMPOUND_SENTENCE_2)
🤖 Prompt for AI Agents
In `@docs/web-console-docs/goals-and-metrics/metrics/create.mdx` around lines 117
- 119, Fix the typo and punctuation in the outlier info blocks: change "Ouliers
capping is done using the full experiment population so all variants have the
same capping limits." to "Outliers capping is done using the full experiment
population, so all variants have the same capping limits." (apply the same
correction to the other identical block).
| If applicable you can enable [CUPED](variance-reduction-cuped) for this metric to help reduce variance. | ||
| Enable [CUPED](variance-reduction-cuped) for this metric by checking the checkbox and choose a lookback period which matches the user behaviour. | ||
|
|
There was a problem hiding this comment.
Polish the variance-reduction sentence for flow.
Suggested edit
-If applicable you can enable [CUPED](variance-reduction-cuped) for this metric to help reduce variance.
-Enable [CUPED](variance-reduction-cuped) for this metric by checking the checkbox and choose a lookback period which matches the user behaviour.
+If applicable, you can enable [CUPED](variance-reduction-cuped) for this metric to help reduce variance.
+Enable [CUPED](variance-reduction-cuped) for this metric by checking the checkbox and choosing a lookback period that matches user behaviour.🧰 Tools
🪛 LanguageTool
[uncategorized] ~343-~343: Possible missing comma found.
Context: ...r reissued. ## Variance reduction If applicable you can enable [CUPED](variance-reducti...
(AI_HYDRA_LEO_MISSING_COMMA)
🤖 Prompt for AI Agents
In `@docs/web-console-docs/goals-and-metrics/metrics/create.mdx` around lines 343
- 345, The two sentences about CUPED read choppy; replace them with a single
smoother sentence that references the variance-reduction-cuped link and uses
correct verb agreement and phrasing — e.g., instruct the reader to enable CUPED
(when applicable) by checking the checkbox and choosing a lookback period that
matches user behavior — updating the lines that mention "CUPED" and the link
"variance-reduction-cuped" to this single polished sentence.
| - **θ (theta)**: An optimal coefficient chosen to maximize variance reduction (typically the correlation between pre and post metrics) | ||
|
|
||
| The adjusted values maintain the same average (mean) as the raw values but have reduced variance, making treatment effects easier to detect. | ||
|
|
||
| ## When CUPED is Most Effective | ||
|
|
||
| CUPED provides the greatest benefit when: | ||
|
|
||
| 1. **High correlation between pre and post metrics** (correlation ≥ 0.3) | ||
| - Revenue metrics typically show correlation of 0.5-0.7 | ||
| - Engagement metrics often show correlation of 0.4-0.6 | ||
| - Conversion metrics may show lower but still useful correlation | ||
|
|
||
| 2. **Sufficient pre-experiment data is available** | ||
| - Minimum: 7-14 days of historical data | ||
| - Recommended: 2-4 weeks for stable baseline estimates | ||
| - The pre-period should reflect normal user behavior | ||
| - In ABsmartly you can choose between, 1, 2, 3 or 4 weeks with 2 weeks being the default |
There was a problem hiding this comment.
Tighten wording and range formatting in the effectiveness section.
Minor grammar and punctuation tweaks will read more cleanly and consistently.
Suggested edits
-- **θ (theta)**: An optimal coefficient chosen to maximize variance reduction (typically the correlation between pre and post metrics)
+- **θ (theta)**: An optimal coefficient chosen to maximize variance reduction (typically the correlation between pre- and post-experiment metrics)
-1. **High correlation between pre and post metrics** (correlation ≥ 0.3)
+1. **High correlation between pre- and post-experiment metrics** (correlation ≥ 0.3)
-- Minimum: 7-14 days of historical data
-- Recommended: 2-4 weeks for stable baseline estimates
+- Minimum: 7–14 days of historical data
+- Recommended: 2–4 weeks for stable baseline estimates
-- In ABsmartly you can choose between, 1, 2, 3 or 4 weeks with 2 weeks being the default
+- In ABsmartly, you can choose between 1, 2, 3, or 4 weeks, with 2 weeks as the default🧰 Tools
🪛 LanguageTool
[grammar] ~29-~29: It appears that hyphens are missing.
Context: ...tion (typically the correlation between pre and post metrics) The adjusted values maintain the same...
(PRE_AND_POST_NN)
[grammar] ~37-~37: It appears that hyphens are missing.
Context: ...it when: 1. High correlation between pre and post metrics (correlation ≥ 0.3) - Revenue metr...
(PRE_AND_POST_NN)
[typographical] ~43-~43: If specifying a range, consider using an en dash instead of a hyphen.
Context: ...iment data is available** - Minimum: 7-14 days of historical data - Recommende...
(HYPHEN_TO_EN)
[typographical] ~44-~44: If specifying a range, consider using an en dash instead of a hyphen.
Context: ...ys of historical data - Recommended: 2-4 weeks for stable baseline estimates ...
(HYPHEN_TO_EN)
🤖 Prompt for AI Agents
In `@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`
around lines 29 - 46, Tighten wording and range formatting in the "When CUPED is
Most Effective" section: reword the θ line to "θ (theta): optimal coefficient
(typically the correlation between pre- and post-metrics)" and ensure the prose
sentences are concise; normalize list ranges to use consistent en-dash ranges
(e.g., "2–4 weeks", "1–4 weeks") and remove the extra comma/verbosity in the
pre-period selection line so it reads like "In ABsmartly you can choose 1–4
weeks; the default is 2 weeks." Also make minor grammar tweaks for parallelism
in bullets (e.g., "Revenue metrics typically correlate 0.5–0.7", "Engagement
metrics 0.4–0.6", "Conversion metrics may be lower but still useful").
| When creating a new metric or a new version of an existing metrics you can enabled CUPED. | ||
| When CUPED is enabled for your metrics in ABsmartly: |
There was a problem hiding this comment.
Fix grammar in the CUPED enablement sentence.
Suggested edit
-When creating a new metric or a new version of an existing metrics you can enabled CUPED.
+When creating a new metric or a new version of an existing metric, you can enable CUPED. 📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| When creating a new metric or a new version of an existing metrics you can enabled CUPED. | |
| When CUPED is enabled for your metrics in ABsmartly: | |
| When creating a new metric or a new version of an existing metric, you can enable CUPED. | |
| When CUPED is enabled for your metrics in ABsmartly: |
🧰 Tools
🪛 LanguageTool
[uncategorized] ~109-~109: Possible missing comma found.
Context: ... metric or a new version of an existing metrics you can enabled CUPED. When CUPED is e...
(AI_HYDRA_LEO_MISSING_COMMA)
[grammar] ~109-~109: The modal verb ‘can’ requires the verb’s base form.
Context: ... version of an existing metrics you can enabled CUPED. When CUPED is enabled for your ...
(MD_BASEFORM)
🤖 Prompt for AI Agents
In `@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`
around lines 109 - 110, Fix the grammar in the sentence "When creating a new
metric or a new version of an existing metrics you can enabled CUPED." by
changing it to "When creating a new metric or a new version of an existing
metric, you can enable CUPED." — update the text in the metrics/CUPED paragraph
so "metrics" becomes singular ("metric"), add the missing comma after the
clause, and change "enabled" to the correct verb form "enable."
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In
`@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`:
- Around line 90-93: Update the phrasing in the bullet that currently reads
"Metrics where pre/post correlation is very low" to hyphenate the term
consistently as "pre-/post-experiment correlation" (match the earlier "pre- and
post-experiment" phrasing). Locate the bullet in the CUPED list and replace
"pre/post correlation" with "pre-/post-experiment correlation" to improve
readability and consistency.
- Around line 26-30: Update the θ description to state that θ is not the
correlation but the optimal coefficient estimated from covariance and variance
(commonly θ̂ = Cov(pre, raw)/Var(pre)), used to adjust the raw metric in CUPED;
replace the phrase “typically the correlation between pre and post metrics” with
a short sentence like “θ is estimated as the covariance of pre‑ and
post‑experiment metrics divided by the variance of the pre‑experiment metric (θ̂
= Cov(pre, raw)/Var(pre))” and keep the terms “Raw Metric” and “Pre‑experiment
Metric” as-is.
| Where: | ||
| - **Raw Metric**: The user's observed value during the experiment | ||
| - **Pre-experiment Metric**: The same metric measured before the experiment | ||
| - **θ (theta)**: An optimal coefficient chosen to maximize variance reduction (typically the correlation between pre and post metrics) | ||
|
|
There was a problem hiding this comment.
Clarify θ so it is not described as the correlation itself.
In CUPED, θ is estimated from covariance/variance; describing it as “the correlation” is misleading. Consider wording that reflects the actual estimator.
✏️ Proposed edit
-- **θ (theta)**: An optimal coefficient chosen to maximize variance reduction (typically the correlation between pre and post metrics)
+- **θ (theta)**: An optimal coefficient estimated from pre-/post-experiment data (often Cov(pre, post) / Var(pre))🧰 Tools
🪛 LanguageTool
[grammar] ~29-~29: It appears that hyphens are missing.
Context: ...tion (typically the correlation between pre and post metrics) The adjusted values maintain the same...
(PRE_AND_POST_NN)
🤖 Prompt for AI Agents
In `@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`
around lines 26 - 30, Update the θ description to state that θ is not the
correlation but the optimal coefficient estimated from covariance and variance
(commonly θ̂ = Cov(pre, raw)/Var(pre)), used to adjust the raw metric in CUPED;
replace the phrase “typically the correlation between pre and post metrics” with
a short sentence like “θ is estimated as the covariance of pre‑ and
post‑experiment metrics divided by the variance of the pre‑experiment metric (θ̂
= Cov(pre, raw)/Var(pre))” and keep the terms “Raw Metric” and “Pre‑experiment
Metric” as-is.
| CUPED is less effective for: | ||
| - Metrics without meaningful pre-experiment analogs | ||
| - Completely novel user behaviors introduced by the treatment | ||
| - Metrics where pre/post correlation is very low |
There was a problem hiding this comment.
Hyphenate pre-/post-experiment correlation.
This reads smoother and matches the earlier “pre- and post-experiment” phrasing.
✏️ Suggested edit
-- Metrics where pre/post correlation is very low
+- Metrics where pre- and post-experiment correlation is very low📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| CUPED is less effective for: | |
| - Metrics without meaningful pre-experiment analogs | |
| - Completely novel user behaviors introduced by the treatment | |
| - Metrics where pre/post correlation is very low | |
| CUPED is less effective for: | |
| - Metrics without meaningful pre-experiment analogs | |
| - Completely novel user behaviors introduced by the treatment | |
| - Metrics where pre- and post-experiment correlation is very low |
🧰 Tools
🪛 LanguageTool
[grammar] ~93-~93: It appears that hyphens are missing.
Context: ...oduced by the treatment - Metrics where pre/post correlation is very low ### Statistical Validity ...
(PRE_AND_POST_NN)
🤖 Prompt for AI Agents
In `@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`
around lines 90 - 93, Update the phrasing in the bullet that currently reads
"Metrics where pre/post correlation is very low" to hyphenate the term
consistently as "pre-/post-experiment correlation" (match the earlier "pre- and
post-experiment" phrasing). Locate the bullet in the CUPED list and replace
"pre/post correlation" with "pre-/post-experiment correlation" to improve
readability and consistency.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In
`@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`:
- Around line 102-106: Update the four-item benefits list in
metrics/variance-reduction-cuped.mdx to avoid unsupported absolute claims: for
the "Faster decisions" item (the line starting with "1. **Faster decisions**")
remove or qualify "Reduce time to statistical significance by 30-50% on average"
unless you add a citation; for "No downside" (the line starting with "4. **No
downside**") change to a qualified statement such as "generally conservative" or
"rarely harmful" or add a source that supports the absolute claim; ensure any
retained numeric claim is accompanied by a citation or replaced with softer
language like "can reduce" or "may reduce" to avoid presenting unsourced
definitive benefits.
🧹 Nitpick comments (1)
docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx (1)
69-83: Make the sample-size figures explicitly illustrative.
These numbers read as prescriptive; clarifying they are illustrative avoids overpromising.✏️ Suggested edit
-- Requires 100,000 users to reach significance +- Illustrative only: might require ~100,000 users to reach significance -- Can detect the same effect with ~65,000 users -- Or detect a smaller 2% improvement that would have been undetectable before +- Illustrative only: could detect the same effect with ~65,000 users +- Or detect a smaller ~2% improvement that would have been undetectable before
| 1. **Faster decisions**: Reduce time to statistical significance by 30-50% on average | ||
| 2. **Cost efficiency**: Achieve the same statistical power with fewer users | ||
| 3. **Detect smaller effects**: Find wins that would otherwise remain hidden in the noise | ||
| 4. **No downside**: CUPED is conservative, when it doesn't help, it doesn't hurt | ||
|
|
There was a problem hiding this comment.
Soften absolute/quantified benefits unless sourced.
“30–50% on average” and “No downside” read as definitive. If no source is cited, consider qualifying or removing the numbers.
✏️ Suggested edit
-1. **Faster decisions**: Reduce time to statistical significance by 30-50% on average
+1. **Faster decisions**: Can reduce time to statistical significance, especially when pre-/post correlation is high
-4. **No downside**: CUPED is conservative, when it doesn't help, it doesn't hurt
+4. **Typically no downside**: CUPED is conservative; when correlation is weak, it usually offers little benefit but remains unbiased🤖 Prompt for AI Agents
In `@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`
around lines 102 - 106, Update the four-item benefits list in
metrics/variance-reduction-cuped.mdx to avoid unsupported absolute claims: for
the "Faster decisions" item (the line starting with "1. **Faster decisions**")
remove or qualify "Reduce time to statistical significance by 30-50% on average"
unless you add a citation; for "No downside" (the line starting with "4. **No
downside**") change to a qualified statement such as "generally conservative" or
"rarely harmful" or add a source that supports the absolute claim; ensure any
retained numeric claim is accompanied by a citation or replaced with softer
language like "can reduce" or "may reduce" to avoid presenting unsourced
definitive benefits.
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In
`@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`:
- Line 112: Fix the typo in the sentence "Pre-experiment data already collectd
is automatically used" by changing "collectd" to "collected" so it reads
"Pre-experiment data already collected is automatically used"; locate this exact
sentence in the metrics/variance-reduction-cuped.mdx content (look for the
phrase "Pre-experiment data already collectd") and update the spelling.
- Line 117: Typo in the metric docs: change the word "corrolation" to
"correlation" in the sentence "When corrolation is < 0.1 then ABsmartly does not
use the CUPED data" so the sentence reads "When correlation is < 0.1 then
ABsmartly does not use the CUPED data"; update the string in the
metrics/variance-reduction-cuped.mdx content where that sentence appears (search
for "When corrolation is") to correct the spelling.
- Around line 119-122: The "CUPED at booking.com" link currently points to an
endpoint that redirects to Medium's login; update the link target in the
"Further Reading" list item (the line containing the "CUPED at booking.com"
text/URL) to a direct, publicly accessible URL (e.g., the article's canonical
Medium URL or an archived copy) and verify the new URL returns a 200 without
redirecting to an auth page.
🧹 Nitpick comments (1)
docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx (1)
69-69: Consider qualifying unsourced quantified claims.The claim "30-40% faster or with 30-40% fewer users" is presented as a definitive result. Unless this is sourced or based on ABsmartly's own data, consider softening to "may detect effects significantly faster" or citing the source.
Suggested edit
-**Result:** You might detect the effect 30-40% faster or with 30-40% fewer users. +**Result:** You may detect the effect significantly faster or with substantially fewer users, depending on pre-/post-experiment correlation.
| When creating a new metric or a new version of an existing metrics you can enabled CUPED. | ||
| When CUPED is enabled for your metrics in ABsmartly: | ||
|
|
||
| - Pre-experiment data already collectd is automatically used |
There was a problem hiding this comment.
Fix typo: "collectd" → "collected".
Suggested edit
-- Pre-experiment data already collectd is automatically used
+- Pre-experiment data already collected is automatically used 📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - Pre-experiment data already collectd is automatically used | |
| - Pre-experiment data already collected is automatically used |
🤖 Prompt for AI Agents
In `@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`
at line 112, Fix the typo in the sentence "Pre-experiment data already collectd
is automatically used" by changing "collectd" to "collected" so it reads
"Pre-experiment data already collected is automatically used"; locate this exact
sentence in the metrics/variance-reduction-cuped.mdx content (look for the
phrase "Pre-experiment data already collectd") and update the spelling.
| - Adjusted metrics are computed alongside raw metrics | ||
| - Statistical significance calculations use the variance-reduced estimates | ||
| - CUPED runs automatically in the background without requiring changes to your experiment setup or tracking implementation. | ||
| - When corrolation is < 0.1 then ABsmartly does not use the CUPED data |
There was a problem hiding this comment.
Fix typo: "corrolation" → "correlation".
Suggested edit
-- When corrolation is < 0.1 then ABsmartly does not use the CUPED data
+- When correlation is < 0.1, ABsmartly does not use the CUPED data📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - When corrolation is < 0.1 then ABsmartly does not use the CUPED data | |
| - When correlation is < 0.1, ABsmartly does not use the CUPED data |
🤖 Prompt for AI Agents
In `@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`
at line 117, Typo in the metric docs: change the word "corrolation" to
"correlation" in the sentence "When corrolation is < 0.1 then ABsmartly does not
use the CUPED data" so the sentence reads "When correlation is < 0.1 then
ABsmartly does not use the CUPED data"; update the string in the
metrics/variance-reduction-cuped.mdx content where that sentence appears (search
for "When corrolation is") to correct the spelling.
| ## Further Reading | ||
|
|
||
| - Original CUPED paper: [Deng et al., 2013 - "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data"](https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf) | ||
| - CUPED at booking.com: [Simon Jackson, 2018, "How Booking.com increases the power of online experiments with CUPED"](https://booking.ai/how-booking-com-increases-the-power-of-online-experiments-with-cuped-995d186fff1d) No newline at end of file |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify external documentation links are accessible
echo "Checking CUPED paper URL..."
curl -s -i "https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf" 2>&1 | head -20
echo ""
echo "Checking Booking.com article URL..."
curl -s -i "https://booking.ai/how-booking-com-increases-the-power-of-online-experiments-with-cuped-995d186fff1d" 2>&1 | head -20Repository: absmartly/docs
Length of output: 2922
Fix the Booking.com article link - it redirects to Medium's login page.
The CUPED paper link is publicly accessible. However, the Booking.com article link returns a 307 redirect to Medium's authentication page, making it inaccessible. Update the link to the direct Medium article URL or verify the correct endpoint.
🤖 Prompt for AI Agents
In `@docs/web-console-docs/goals-and-metrics/metrics/variance-reduction-cuped.mdx`
around lines 119 - 122, The "CUPED at booking.com" link currently points to an
endpoint that redirects to Medium's login; update the link target in the
"Further Reading" list item (the line containing the "CUPED at booking.com"
text/URL) to a direct, publicly accessible URL (e.g., the article's canonical
Medium URL or an archived copy) and verify the new URL returns a 200 without
redirecting to an auth page.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@docs/web-console-docs/goals-and-metrics/metrics/create.mdx`:
- Around line 93-96: Edit the caution block text so it reads grammatically
correct: add a comma after "For example," and change "measure conversions
achieved 2 weeks after exposure" to "measures conversions achieved 2 weeks after
exposure" (i.e., ensure subject-verb agreement and correct punctuation in the
sentence inside the caution block).
| :::caution | ||
| If your metric uses [CUPED](variance-reduction-cuped), the lookback window must incorporate the time filter. | ||
| For example if your metric measure `conversions achieved 2 weeks after exposure` then the CUPED lookback window must be at least 2 weeks. | ||
| ::: |
There was a problem hiding this comment.
Fix grammar and punctuation in the caution block.
Line 95 has two issues:
- Missing comma after "For example"
- "measure" should be "measures" (subject-verb agreement)
📝 Suggested fix
:::caution
If your metric uses [CUPED](variance-reduction-cuped), the lookback window must incorporate the time filter.
-For example if your metric measure `conversions achieved 2 weeks after exposure` then the CUPED lookback window must be at least 2 weeks.
+For example, if your metric measures `conversions achieved 2 weeks after exposure`, then the CUPED lookback window must be at least 2 weeks.
:::🧰 Tools
🪛 LanguageTool
[typographical] ~95-~95: After the expression ‘for example’ a comma is usually used.
Context: ...w must incorporate the time filter. For example if your metric measure `conversions ach...
(COMMA_FOR_EXAMPLE)
🤖 Prompt for AI Agents
In `@docs/web-console-docs/goals-and-metrics/metrics/create.mdx` around lines 93 -
96, Edit the caution block text so it reads grammatically correct: add a comma
after "For example," and change "measure conversions achieved 2 weeks after
exposure" to "measures conversions achieved 2 weeks after exposure" (i.e.,
ensure subject-verb agreement and correct punctuation in the sentence inside the
caution block).
New documentation for January's release
Summary by CodeRabbit
✏️ Tip: You can customise this high-level summary in your review settings.