alihan/growthbook
1
0
Fork 0
mirror of https://github.com/growthbook/growthbook.git synced 2021-08-07 14:23:53 +03:00
Code Issues Projects Releases Wiki Activity

Update docs.

Browse Source 
-  Update js docs to v0.11.0
-  Update react docs to v0.4.0
-  Add new 'build-your-own' client library docs page
-  Update in-app instructions for js/react to match latest versions
-  Fix url regex escaping bug in inline-script code
This commit is contained in:
Jeremy Dorn
2021-06-11 13:11:13 -05:00
parent abeda1f517
commit 150626e417
9 changed files with 488 additions and 362 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
CONTRIBUTING.md
View File

@@ -2,6 +2,8 @@
Interested in making Growth Book better? So are we! This guide should help get you setup with a local development environment so you can make changes, create PRs, and get your code merged.
If you just want to contribute a client library in a new language and not make changes to the app itself, you can skip the instructions here and view https://docs.growthbook.io/lib/build-your-own instead.
## Requirements
- MacOS or Linux (Windows may work too, but we haven't tested it)

4
packages/docs/pages/_app.tsx
View File

@@ -64,6 +64,10 @@ const navLinks = [
href: "/lib/ruby",
name: "Ruby",
},
{
href: "/lib/build-your-own",
name: "Build Your Own",
},
],
},
{

3
packages/docs/pages/api-docs.mdx
View File

@@ -50,5 +50,4 @@ We offer official client libraries that work with these data structures in a few
* [React](/lib/react)
* [PHP](/lib/php)
* [Ruby](/lib/ruby)
* Go - *coming soon*
* Python - *coming soon*
* [Build your own](/lib/build-your-own)

281
packages/docs/pages/lib/build-your-own.mdx Normal file
View File

@@ -0,0 +1,281 @@
# Build Your Own Client Library
This guide is meant for library authors looking to build a Growth Book client library in a currently unsupported language.
Growth Book client libraries are very simple and do not interact with the filesystem or network. Because of this, they can often be kept to **under 500 lines of code**.
All libraries should follow this specification as closely as the language permits to maintain consistency.
## Data structures
These are the 3 main data structures used in the client libraries.
`Context + Experiment = Result`
### Context
Defines the experimental context (attributes used for variation assignment and targeting).
At a minimum, the context should support the following optional properties:
- **enabled** (`boolean`) - Switch to globally disable all experiments. Default true.
- **user** (`Map`) - Map of user attributes that are used to assign variations
- **groups** (`Map`) - A map of which groups the user belongs to (key is the group name, value is boolean)
- **url** (`string`) - The URL of the current page
- **overrides** (`Map`) - Override properties of specific experiments (used for Remote Config)
- **forcedVariations** (`Map`) - Force specific experiments to always assign a specific variation (used for QA)
- **qaMode** (`boolean`) - If true, random assignment is disabled and only explicitly forced variations are used.
- **trackingCallback** (`function`) - A function that takes `experiment` and `result` as arguments.
An example of a `user`:
```json
{
"id": "123",
"anonId": "abcdef",
"company": "growthbook"
}
```
An example of `trackingCallback` in javascript:
```js
function track(experiment, result) {
analytics.track("Experiment Viewed", {
experimentId: experiment.key,
variationId: result.variationId
});
}
```
### Experiment
Defines a single experiment:
- **key** (`string`) - The globally unique tracking key for the experiment
- **variations** (`any[]`) - The different variations to choose between
- **weights** (`number[]`) - How to weight traffic between variations. Must add to 1.
- **status** (`string`) - "running" is the default and always active. "draft" is only active during QA and development. "stopped" is only active when forcing a winning variation to 100% of users.
- **coverage** (`number`) - What percent of users should be included in the experiment (between 0 and 1, inclusive)
- **url** (`RegExp`) - Users can only be included in this experiment if the current URL matches this regex
- **include** (`() => boolean`) - A callback that returns true if the user should be part of the experiment and false if they should not be
- **groups** (`string[]`) - Limits the experiment to specific user groups
- **force** (`number`) - All users included in the experiment will be forced into the specific variation index
- **hashAttribute** (`string`) - What user attribute should be used to assign variations (defaults to `id`)
The only required properties are `key` and `variations`. Everything else is optional.
### Result
The result of running an Experiment given a specific Context
- **inExperiment** (`boolean`) - Whether or not the user is part of the experiment
- **variationId** (`string`) - The array index of the assigned variation
- **value** (`any`) - The array value of the assigned variation
- **hashAttribute** (`string`) - The user attribute used to assign a variation
- **hashValue** (`string)` - The value of that attribute
The `variationId` and `value` should always be set, even when `inExperiment` is false.
## Running an Experiment
The main export of the libraries is a simple `GrowthBook` wrapper class with a `run` method that returns a `Result` object.
```js
growthbook = new GrowthBook(context);
result = growthbook.run(experiment);
```
There are a bunch of ordered steps to run an experiment:
1. If `experiment.variations` has fewer than 2 variations, return immediately (not in experiment, variationId `0`)
2. If `context.enabled` is false, return immediately (not in experiment, variationId `0`)
3. If `context.overrides[experiment.key]` is set, merge override properties into the experiment
4. If `context.url` contains a querystring `{experiment.key}=[0-9]+`, return immediately (not in experiment, variationId from querystring)
5. If `context.forcedVariations[experiment.key]` is defined, return immediately (not in experiment, forced variation)
6. If `experiment.status` is "draft", return immediately (not in experiment, variationId `0`)
7. Get the user hash attribute and value (`context.user[experiment.hashAttribute || "id"]`) and if empty, return immediately (not in experiment, variationId `0`)
8. If `experiment.include` is set, call the function and if "false" is returned or it throws, return immediately (not in experiment, variationId `0`)
9. If `experiment.groups` is set and none of them are true in `context.groups`, return immediately (not in experiment, variationId `0`)
10. If `experiment.url` is set, evaluate as a regex against `context.url` and if it doesn't match or throws, return immediately (not in experiment, variationId `0`)
11. If `experiment.force` is set, return immediately (not in experiment, variationId `experiment.force`)
12. If `experiment.status` is "stopped", return immediately (not in experiment, variationId `0`)
13. If `context.qaMode` is true, return immediately (not in experiment, variationId `0`)
14. Compute a hash using the [Fowler–Noll–Vo](https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function) algorithm (specifically fnv32-1a)
```js
n = (fnv32_1a(id + experiment.key) % 1000) / 1000
```
15. Apply coverage to weights:
```js
// Default weights to an even split
weights = experiment.weights
if(!weights) {
weights = Array(experiment.variations.length)
.fill(1/experiment.variations.length)
}
// Multiple each weight by the coverage (or 1 if not set)
weights = weights.map(w => w*(experiment.coverage || 1));
```
16. Loop through `weights` until you reach the hash value
```js
cumulative = 0
assigned = -1
for (i=0; i<weights.length; i++) {
cumulative += weights[i]
if (n < cumulative) {
assigned = i
break
}
}
```
17. If not assigned a variation (`assigned === -1`), return immediately (not in experiment, variationId `0`)
18. Fire `context.trackingCallback` if set
19. Return (**in experiment**, assigned variation)
## Remote Config
`context.overrides` allows overriding a subset of experiment properties without changing code. For example, changing an experiment's status from `running` to `stopped`.
The overrides are typically stored in a database or cache so there should be an easy way to pass them in as a JSON-encoded object. The object will look something like this:
```json
{
"my-experiment-key": {
"status": "stopped"
},
"my-other-experiment": {
"coverage": 0.5,
"groups": "beta-testers"
}
}
```
The full list of supported override properties is:
- weights
- status
- force
- coverage
- groups
- url
Note that the above list specifically does not include `variations`. The only way to change the variations is to change the code.
This restriction makes testing and maintaining code much much easier.
## Developer Experience
Having a good developer experience is super important.
### Basic Usage
It should be very simple to run a basic A/B test:
```js
result = growthbook.run({
key: "my-experiment",
variations: ["A", "B"]
})
print(result.value) // "A" or "B"
```
And it should feel natural to scale up to more complex use cases:
```js
// 50% of beta testers, 80/20 split between variations
result = growthbook.run({
key: "complex-experiment",
variations: [
{color: "blue", size: "small"},
{color: "green", size: "large"}
],
weights: [0.8, 0.2],
coverage: 0.5,
groups: ["beta-testers"]
})
print(result.value.color, result.value.size) // "blue,small" OR "green,large"
```
### Type Hinting
Most languages have some sort of strong typing support, whether in the language itself or via annotations. This helps to reduce errors and is highly encouraged for client libraries.
If possible, use generics to type the return value. If `experiment.variations` is type `T[]`, then `result.value` should be type `T`.
If your type system supports specifying a minimum array length, it's best to type `experiment.variations` as requiring at least 2 elements.
The `experiment.status` field should be typed as a string union or enum if possible. The only valid values are `draft`, `running`, and `stopped`.
### URL Regexes
If your language has support for a native regex type, you should use that instead of strings for `experiment.url`.
However, in all languages, `context.overrides` needs to remain serializeable to JSON, so strings must be used there. When importing overrides from JSON, you would convert the strings to actual regex objects.
Since the regex deals with URLs, make sure you are escaping `/` if needed. The string value `"^/post/[0-9]+"` should work as expected and not throw an error.
### Handling Errors
The general rule is to be strict in development and lenient in production.
You can throw exceptions in development, but someone's production app should never crash because of a call to `growthbook.run`.
For the below edge cases in production, just act as if the problematic property didn't exist and ignore errors:
- `experiment.weights` is a different length from `experiment.variations`
- `experiment.weights` adds up to something other than 1
- `experiment.coverage` is greater than 1
- `context.trackingCallback` throws an error
- URL querystring specifies an invalid variation index
For the below edge cases in production, the experiment should be disabled (everyone gets assigned variation `0`):
- `experiment.url` is an invalid regex
- `experiment.coverage` is less than 0
- `experiment.force` specifies an invalid variation index
- `context.forcedVariations` specifies an invalid variation index
- `experiment.include` throws an error
- `experiment.status` is set to an unknown value
- `experiment.hashAttribute` is an empty string
### Subscriptions
Sometimes it's useful to be able to "subscribe" to a GrowthBook instance and be alerted every time `growthbook.run` is called. This is different from the tracking callback since it also fires when a user is *not* included in an experiment.
```js
growthbook.subscribe(function(experiment, result) {
// do something
})
```
It's best to only re-fire the callbacks for an experiment if the result has changed. That means either the `inExperiment` flag has changed or the `variationId` has changed.
If it makes sense for your language, this function should return an "unsubscriber". A simple callback that removes the subscription.
```js
unsubscriber = growthbook.subscribe(...)
unsubscriber()
```
In addition to subscriptions you may also want to expose a `growthbook.getAllResults` method that returns a map of the latest results indexed by experiment key.
### Memory Management
Subscriptions and tracking calls require storing references to many objects and functions. If it makes sense for your language, libraries should provide a `growthbook.destroy` method to remove all of these references and release their memory.
## Tests
We strive to have 100% test coverage for all of our client libraries. Since they all use the same data structures, the test suites are pretty transferrable between languages.
1. Copy/paste the test suite from an existing client library
2. Do global search/replace to fix common syntax differences (e.g. replacing `->` in php with `.` in javascript)
3. Manually fix the rest of the syntax errors and coding patterns
4. Run the tests!
5. Go back to step 3 a bunch of times :)
## Getting Help
Join our [Slack community](https://join.slack.com/t/growthbookusers/shared_invite/zt-oiq9s1qd-dHHvw4xjpnoRV1QQrq6vUg) if you need help or want to chat. We're also happy to hop on a call and do some pair programming.
## Attribution
Open a [https://github.com/growthbook/growthbook/issues](GitHub issue) with a link to your project and we'll make sure we add it to our docs and give you proper credit for your hard work.

8
packages/docs/pages/lib/index.mdx
View File

@@ -1,14 +1,12 @@
# Client Libraries
We offer official client libraries in a few popular languages with more coming soon:
We offer official client libraries in a few popular languages:
* [Javascript/Typescript](/lib/js)
* [React](/lib/react)
* [PHP](/lib/php)
* [Ruby](/lib/ruby)
* Go - *coming soon*
* Python - *coming soon*
It's not required to use these libraries. You can do variation assignment yourself or use another library like PlanOut.
There is also a guide if you want to [build your own](/lib/build-your-own).
The only requirement is that you track in your datasource when users are put into an experiment and which variation they received.
It's not required to use any of these libraries. The only requirement is that you track in your datasource when users are put into an experiment and which variation they received.

232
packages/docs/pages/lib/js.mdx
View File

@@ -14,7 +14,7 @@ or use directly in your HTML without installing first:
```html
<script type="module">
import GrowthBookClient from 'https://unpkg.com/@growthbook/growthbook/dist/growthbook.esm.js';
import GrowthBook from 'https://unpkg.com/@growthbook/growthbook/dist/growthbook.esm.js';
//...
</script>
```
@@ -22,86 +22,71 @@ import GrowthBookClient from 'https://unpkg.com/@growthbook/growthbook/dist/grow
## Quick Usage
```ts
import GrowthBookClient from '@growthbook/growthbook';
import GrowthBook from '@growthbook/growthbook';
// Create a client and setup tracking
const client = new GrowthBookClient({
onExperimentViewed: ({experimentId, variationId}) => {
// Use whatever event tracking system you have in place
analytics.track("Experiment Viewed", {experimentId, variationId});
// Define the experimental context
const growthbook = new GrowthBook({
// The attributes used to assign variations
user: { id: "123" },
// Called when a user is put into an experiment
trackingCallback: (experiment, result) => {
analytics.track("Experiment Viewed", {
experimentId: experiment.key,
variationId: result.variationId
})
}
});
})
// Define the user that you want to run an experiment on
const user = client.user({id: "12345"});
// Run an experiment
const {value} = growthbook.run({
key: "my-experiment",
variations: ["A", "B"]
})
// Put the user in an experiment
const {value} = user.experiment({
key: "my-experiment",
variations: ["A", "B"]
});
console.log(value); // "A" or "B"
console.log(value) // "A" or "B"
```
## Client Configuration
## GrowthBook class
The GrowthBookClient constructor takes an optional `options` argument.
The GrowthBook constructor takes a `Context` object. Below are all of the possible Context properties:
Below are all of the available options:
- **enabled** - Default true. Set to false to completely disable all experiments.
- **debug** - Default false. If set to true, console.log info about why experiments are run and why specific variations are chosen. Only works when NODE_ENV is not set to production.
- **onExperimentViewed** - Callback when the user views an experiment.
- **url** - The URL for the current request (defaults to `window.location.href` when in a browser)
- **enableQueryStringOverride** - Default true. If true, enables forcing variations via the URL. Very useful for QA. https://example.com/?my-experiment=1
### SPA support
With a Single Page App (SPA), you need to update the client on navigation in order to target tests based on URL:
```ts
client.config.url = newUrl;
```
Doing this with Next.js for example, will look like this:
```tsx
export default function MyApp({ Component, pageProps }) {
const router = useRouter()
useEffect(() => {
const onChange = (newUrl) => client.config.url = newUrl;
router.events.on('routeChangeComplete', onChange);
return () => router.events.off('routeChangeComplete', onChange);
}, [])
return <Component {...pageProps} />
}
```
- **enabled** (`boolean`) - Switch to globally disable all experiments. Default true.
- **user** (`{}`) - Map of user attributes that are used to assign variations
- **groups** (`{}`) - A map of which groups the user belongs to (key is the group name, value is boolean)
- **url** (`string`) - The URL of the current page (defaults to `window.location.href` when in a browser environment)
- **overrides** (`{}`) - Override properties of specific experiments (used for Remote Config)
- **forcedVariations** (`{}`) - Force specific experiments to always assign a specific variation (used for QA)
- **qaMode** (`boolean`) - If true, random assignment is disabled and only explicitly forced variations are used.
- **trackingCallback** (`function`) - A function that takes `experiment` and `result` as arguments.
## Experiments
As shown above, the simplest experiment you can define has 2 fields: `key` and `variations`.
There are a lot more configuration options you can specify. Here is the full list of options:
Below are all of the possible properties you can set for an Experiment:
- **key** (`string`) - The globally unique tracking key for the experiment
- **variations** (`any[]`) - The different variations to choose between
- **weights** (`number[]`) - How to weight traffic between variations. Must add to 1.
- **status** (`string`) - "running" is the default and always active. "draft" is only active during QA and development. "stopped" is only active when forcing a winning variation to 100% of users.
- **coverage** (`number`) - What percent of users should be included in the experiment (between 0 and 1, inclusive)
- **url** (`string`) - Users can only be included in this experiment if the current URL matches this regex
- **url** (`RegExp`) - Users can only be included in this experiment if the current URL matches this regex
- **include** (`() => boolean`) - A callback that returns true if the user should be part of the experiment and false if they should not be
- **groups** (`string[]`) - Limits the experiment to specific user groups
- **force** (`number`) - All users included in the experiment will be forced into the specific variation index
- **randomizationUnit** - What user attribute you want to use to assign variations (defaults to `id`)
- **hashAttribute** (`string`) - What user attribute should be used to assign variations (defaults to "id")
### Running Experiments
Run experiments by calling `user.experiment()` which returns an object with a few useful properties:
## Running Experiments
Run experiments by calling `growthbook.run(experiment)` which returns an object with a few useful properties:
```ts
const {inExperiment, variationId, value} = user.experiment({
const {
inExperiment,
variationId,
value,
hashAttribute,
hashValue
} = growthbook.run({
key: "my-experiment",
variations: ["A", "B"]
});
@@ -114,15 +99,21 @@ console.log(variationId); // 0 or 1
// The value of the assigned variation
console.log(value); // "A" or "B"
// The user attribute used to assign a variation
console.log(hashAttribute); // "id"
// The value of that attribute
console.log(hashValue); // e.g. "123"
```
The `inExperiment` flag can be false if the experiment defines any sort of targeting rules which the user does not pass. In this case, the user is always assigned variation index `0`.
The `inExperiment` flag is only set to true if the user was randomly assigned a variation. If the user failed any targeting rules or was forced into a specific variation, this flag will be false.
### Example Experiments
3-way experiment with uneven variation weights:
```ts
user.experiment({
growthbook.run({
key: "3-way-uneven",
variations: ["A","B","C"],
weights: [0.5, 0.25, 0.25]
@@ -132,11 +123,15 @@ user.experiment({
Slow rollout (10% of users who opted into "beta" features):
```ts
// User is in the "qa" and "beta" groups
const user = client.user({id: "123"}, {
qa: isQATester(),
beta: betaFeaturesEnabled()
});
user.experiment({
const growthbook = new GrowthBook({
user: {id: "123"},
groups: {
qa: isQATester(),
beta: betaFeaturesEnabled()
}
})
growthbook.run({
key: "slow-rollout",
variations: ["A", "B"],
coverage: 0.1,
@@ -146,29 +141,35 @@ user.experiment({
Complex variations and custom targeting
```ts
const {value} = user.experiment({
const {value} = growthbook.run({
key: "complex-variations",
variations: [
{color: "blue", size: "large"},
{color: "green", size: "small"}
],
url: /^\/post\/[0-9]+$/i
include: () => isPremium || creditsRemaining > 50
});
console.log(value.color, value.size); // blue,large OR green,small
```
Assign variations based on something other than user id
```ts
const user = client.user({
id: "123",
companyId: "abc"
});
user.experiment({
const growthbook = new GrowthBook({
user: {
id: "123",
company: "growthbook"
}
})
growthbook.run({
key: "by-company-id",
variations: ["A", "B"],
randomizationUnit: "companyId"
hashAttribute: "company"
})
// Users in the same company will now always get the same variation
// Users in the same company will always get the same variation
```
### Overriding Experiment Configuration
@@ -177,13 +178,17 @@ It's common practice to adjust experiment settings after a test is live. For ex
For example, to roll out a winning variation to 100% of users:
```ts
client.overrides.set("experiment-key", {
status: 'stopped',
force: 1
});
const growthbook = new GrowthBook({
user: {id: "123"},
overrides: {
"experiment-key": {
status: "stopped",
force: 1
}
}
})
// Later in code
const {value} = user.experiment({
const {value} = growthbook.run({
key: "experiment-key",
variations: ["A", "B"]
});
@@ -197,26 +202,32 @@ The full list of experiment properties you can override is:
* weights
* coverage
* groups
* url
This data structure can be easily seralized and stored in a database or returned from an API. There is a small helper function if you have all of your overrides in a single JSON object:
```ts
const JSONFromDatabase = {
"experiment-key-1": {
"weights": [0.1, 0.9]
},
"experiment-key-2": {
"groups": ["everyone"],
"coverage": 1
}
};
client.importOverrides(JSONFromDatabase)
```
* url (can use string instead of regex if serializing in a database)
If you use the Growth Book App (https://github.com/growthbook/growthbook) to manage experiments, there's a built-in API endpoint you can hit that returns overrides in this exact format. It's a great way to make sure your experiments are always up-to-date.
## Typescript
This module exposes Typescript types if needed.
This is especially useful if experiments are defined as a variable before being passed into `growthbook.run`. Unions and tuples are used heavily and Typescript has trouble inferring those properly.
```ts
import type {
Context,
Experiment,
Result,
ExperimentOverride
} from "@growthbook/growthbook"
// The "number" part refers to the variation type
const exp: Experiment<number> = {
key: "my-test",
variations: [0, 1],
status: "stoped" // Type error! (should be "stopped")
}
```
## Event Tracking and Analyzing Results
This library only handles assigning variations to users. The 2 other parts required for an A/B testing platform are Tracking and Analysis.
@@ -229,43 +240,36 @@ For A/B tests, you just need to track one additional event - when someone views
```ts
// Specify a tracking callback when instantiating the client
const client = new GrowthBookClient({
onExperimentViewed: ({experimentId, variationId}) => {
// ...
}
});
const growthbook = new GrowthBook({
user: {id: "123"},
trackingCallback: (experiment, result) => {
// ...
}
})
```
The object passed to your callback has the following properties:
- experimentId (the key of the experiment)
- variationId (the array index of the assigned variation)
- value (the value of the assigned variation)
- experiment (the full experiment object)
- user (the full user object)
- randomizationUnit (which user attribute was used to assign a variation)
Below are examples for a few popular event tracking tools:
#### Google Analytics
```ts
ga('send', 'event', 'experiment', experimentId, variationId, {
ga('send', 'event', 'experiment', experiment.key, result.variationId, {
// Custom dimension for easier analysis
'dimension1': `${experimentId}::${variationId}`
'dimension1': `${experiment.key}::${result.variationId}`
});
```
#### Segment
```ts
analytics.track("Experiment Viewed", {
experimentId,
variationId
experimentId: experiment.key,
variationId: result.variationId
});
```
#### Mixpanel
```ts
mixpanel.track("$experiment_started", {
'Experiment name': experimentId,
'Variant name': variationId
'Experiment name': experiment.key,
'Variant name': result.variationId
});
```

275
packages/docs/pages/lib/react.mdx
View File

@@ -15,23 +15,28 @@ or
### Step 1: Configure your app
```tsx
import {
GrowthBookClient, GrowthBookProvider
GrowthBook, GrowthBookProvider
} from '@growthbook/growthbook-react'
// Create a client instance and setup tracking
const client = new GrowthBookClient({
onExperimentViewed: ({experimentId, variationId}) => {
// Create a GrowthBook instance
const growthbook = new GrowthBook({
// The attributes you want to use to assign variations
user: {
id: "123"
},
// Called every time the user is put into an experiment
trackingCallback: (experiment, result) => {
// Mixpanel, Segment, GA, or custom tracking
mixpanel.track("Experiment Viewed", {experimentId, variationId})
mixpanel.track("Experiment Viewed", {
experiment: experiment.key,
variation: result.variationId
})
}
});
// Specify the user id you want to experiment on
const user = client.user({ id: mixpanel.get_distinct_id() })
export default function App() {
return (
<GrowthBookProvider user={user}>
<GrowthBookProvider growthbook={growthbook}>
<OtherComponent/>
</GrowthBookProvider>
)
@@ -40,6 +45,8 @@ export default function App() {
### Step 2: Run an experiment
#### Hooks (recommended)
```tsx
import { useExperiment } from '@growthbook/growthbook-react'
@@ -53,20 +60,32 @@ export default function OtherComponent() {
}
```
Use class components? We support that too! [See Example](#react-class-components)
#### Class Components
**Note:** This library uses hooks internally, so still requires React 16.8 or above.
```tsx
import { withRunExperiment } from '@growthbook/growthbook-react';
class MyComponent extends Component {
render() {
// The `runExperiment` prop is identical to the `useExperiment` hook
const {value} = this.props.runExperiment({
key: "headline-test",
variations: ["Hello World", "Hola Mundo"]
});
return <h1>{value}</h1>
}
}
// Wrap your component in `withRunExperiment`
export default withRunExperiment(MyComponent);
```
### Step 3: Analyze results
Query your raw data, calculate significance, decide on a winner, and document your findings.
Typically, this is done with one of the following:
* Online A/B testing calculators
* Built-in A/B test analysis in Mixpanel/Amplitude
* Python or R libraries and a Jupyter Notebook
These are all pretty tedious to set up and maintain, which is why created the [Growth Book App](https://www.growthbook.io)
that automates all the messy and annoying bits and lets you focus on building your product.
The easiest way to accomplish this is with the Growth Book App (https://github.com/growthbook/growthbook), but it's not required. You can use an online A/B test calculator or a Jupyter notebook if you prefer.
## Dev Mode
@@ -76,222 +95,8 @@ Dev Mode adds a variation switcher UI that floats on the bottom left of pages.
[View Live Demo](https://growthbook.github.io/growthbook-react/)
![Dev Mode Variation Switcher](/images/variation-switcher.png)
![Dev Mode Variation Switcher](variation-switcher.png)
## Experiments
## Configuration and Usage
The simplest experiment you can define has just 2 fields: `key` and `variations`.
There are a lot more configuration options you can specify. Here is the full typescript definition:
- **key** (`string`) - The globally unique tracking key for the experiment
- **variations** (`any[]`) - The different variations to choose between
- **weights** (`number[]`) - How to weight traffic between variations. Must add to 1.
- **status** (`string`) - "running" is the default and always active. "draft" is only active during QA and development. "stopped" is only active when forcing a winning variation to 100% of users.
- **coverage** (`number`) - What percent of users should be included in the experiment (between 0 and 1, inclusive)
- **url** (`string`) - Users can only be included in this experiment if the current URL matches this regex
- **include** (`() => boolean`) - A callback that returns true if the user should be part of the experiment and false if they should not be
- **groups** (`string[]`) - Limits the experiment to specific user groups
- **force** (`number`) - All users included in the experiment will be forced into the specific variation index
- **randomizationUnit** - What user attribute you want to use to assign variations (defaults to `id`)
## Running Experiments
The useExperiment hook returns an object with a few useful properties
```ts
const {inExperiment, variationId, value} = useExperiment({
key: "my-experiment",
variations: ["A", "B"]
});
// If user is part of the experiment
console.log(inExperiment); // true or false
// The index of the assigned variation
console.log(variationId); // 0 or 1
// The value of the assigned variation
console.log(value); // "A" or "B"
```
The `inExperiment` flag can be false if the experiment defines any sort of targeting rules which the user does not pass. In this case, the user is always assigned index `0`.
### Example Experiments
3-way experiment with uneven variation weights:
```ts
useExperiment({
key: "3-way-uneven",
variations: ["A","B","C"],
weights: [0.5, 0.25, 0.25]
})
```
Slow rollout (10% of users who opted into "beta" features):
```ts
// User is in the "qa" and "beta" groups
const user = client.user({id: "123"}, {
qa: isQATester(),
beta: betaFeaturesEnabled()
});
// Later in a component
useExperiment({
key: "slow-rollout",
variations: ["A", "B"],
coverage: 0.1,
groups: ["beta"]
})
```
Complex variations and custom targeting
```ts
const {value} = useExperiment({
key: "complex-variations",
variations: [
{color: "blue", size: "large"},
{color: "green", size: "small"}
],
include: () => isPremium || creditsRemaining > 50
});
console.log(value.color, value.size); // blue,large OR green,small
```
Assign variations based on something other than user id
```ts
const user = client.user({
id: "123",
companyId: "abc"
});
// Later in a component
useExperiment({
key: "by-company-id",
variations: ["A", "B"],
randomizationUnit: "companyId"
})
// Users in the same company will now always get the same variation
```
### Overriding Experiment Configuration
It's common practice to adjust experiment settings after a test is live. For example, slowly ramping up traffic, stopping a test automatically if guardrail metrics go down, or rolling out a winning variation.
For example, to roll out a winning variation to 100% of users:
```ts
client.overrides.set("experiment-key", {
status: 'stopped',
force: 1
});
// Later in a component
const {value} = useExperiment({
key: "experiment-key",
variations: ["A", "B"]
});
console.log(value); // Always "B"
```
The full list of experiment properties you can override is:
* status
* force
* weights
* coverage
* groups
* url
This data structure can be easily seralized and stored in a database or returned from an API. There is a small helper function if you have all of your overrides in a single JSON object:
```ts
const JSONFromDatabase = {
"experiment-key-1": {
"weights": [0.1, 0.9]
},
"experiment-key-2": {
"groups": ["everyone"],
"coverage": 1
}
};
client.importOverrides(JSONFromDatabase)
```
If you use the Growth Book App (https://github.com/growthbook/growthbook) to manage experiments, there's a built-in API endpoint you can hit that returns overrides in this exact format. It's a great way to make sure your experiments are always up-to-date.
## Event Tracking and Analyzing Results
This library only handles assigning variations to users. The 2 other parts required for an A/B testing platform are Tracking and Analysis.
It's likely you already have some event tracking on your site with the metrics you want to optimize (Google Analytics, Segment, Mixpanel, etc.).
For A/B tests, you just need to track one additional event - when someone views a variation.
```ts
// Specify a tracking callback when instantiating the client
const client = new GrowthBookClient({
onExperimentViewed: ({experimentId, variationId}) => {
// ...
}
});
```
The object passed to your callback has the following properties:
- experimentId (the key of the experiment)
- variationId (the array index of the assigned variation)
- value (the value of the assigned variation)
- experiment (the full experiment object)
- user (the full user object)
- randomizationUnit (which user attribute was used to assign a variation)
Below are examples for a few popular event tracking tools:
### Google Analytics
```ts
ga('send', 'event', 'experiment', experimentId, variationId, {
// Custom dimension for easier analysis
'dimension1': `${experimentId}::${variationId}`
});
```
### Segment
```ts
analytics.track("Experiment Viewed", {
experimentId,
variationId
});
```
### Mixpanel
```ts
mixpanel.track("$experiment_started", {
'Experiment name': experimentId,
'Variant name': variationId
});
```
## React Class Components
If you aren't using functional components, we offer a `withRunExperiment` Higher Order Component instead.
**Note:** This library uses hooks internally, so still requires React 16.8 or above.
```tsx
import {withRunExperiment} from '@growthbook/growthbook-react';
class MyComponent extends Component {
render() {
// The `runExperiment` prop is identical to the `useExperiment` hook
const {value} = this.props.runExperiment({
key: "headline-test",
variations: ["Hello World", "Hola Mundo"]
});
return <h1>{value}</h1>
}
}
// Wrap your component in `withRunExperiment`
export default withRunExperiment(MyComponent);
```
This package is a small React wrapper around the [javascript client library](/lib/js). Look at those docs for more info on how to configure your GrowthBook instance and define Experiments.

33
packages/front-end/components/Experiment/InstructionsModal.tsx
View File

@@ -9,6 +9,7 @@ import useForm from "../../hooks/useForm";
import { useEffect } from "react";
import {
generateJavascriptSnippet,
getUrlRegex,
TrackingType,
} from "../../services/codegen";
import TextareaAutosize from "react-textarea-autosize";
@@ -17,7 +18,8 @@ import Code from "../Code";
type Experiment = {
key: string;
variations: string[];
// eslint-disable-next-line
variations: any[];
weights?: number[];
status?: string;
coverage?: number;
@@ -45,6 +47,25 @@ function indentLines(code: string, indent: number = 2) {
const spaces = " ".repeat(indent);
return code.split("\n").join("\n" + spaces);
}
function withHashAttribute(expDef: Experiment) {
const { anon, ...otherProps } = expDef;
if (anon) {
return {
...otherProps,
hashAttribute: "anonId",
};
} else {
return {
...otherProps,
};
}
}
function withRealRegex(stringified: string): string {
return stringified.replace(/("url"\s*:\s*)"([^"]+)"/, (match, key, value) => {
return key + getUrlRegex(value);
});
}
const InstructionsModal: FC<{
experiment: ExperimentInterfaceStringDates;
@@ -166,8 +187,8 @@ const InstructionsModal: FC<{
</p>
<Code
language="javascript"
code={`const { value } = user.experiment(${stringify(
expDef
code={`const { value } = growthbook.run(${withRealRegex(
stringify(withHashAttribute(expDef))
)})\n\nconsole.log(value${
!variationParam
? ""
@@ -191,7 +212,7 @@ const InstructionsModal: FC<{
</p>
<Code
language="tsx"
code={`function MyComponent() {\n const { value } = user.experiment(${indentLines(
code={`function MyComponent() {\n const { value } = useExperiment(${indentLines(
stringify(expDef)
)})\n\n return <div>{value${
!variationParam
@@ -288,6 +309,10 @@ const InstructionsModal: FC<{
maxRows={6}
{...inputProps.funcs[i]}
/>
<small className="form-text text-muted">
Will be executed if user is assigned variation:{" "}
<code>{v.name}</code>
</small>
</div>
))}
</form>

12
packages/front-end/services/codegen.ts
View File

@@ -4,10 +4,18 @@ export type TrackingType = "mixpanel" | "ga" | "segment" | "custom";
const fnvHash = `(n)=>{let o=2166136261;const t=n.length;for(let e=0;e<t;e++)o^=n.charCodeAt(e),o+=(o<<1)+(o<<4)+(o<<7)+(o<<8)+(o<<24);return o>>>0}`;
export function getUrlRegex(url: string): string {
return `/${url
// JSON strigify adds extra escaping for backslashes
.replace(/\\\\/g, "\\")
// Need to do this replace twice to catch 2 slahes in a row (e.g. `http://`)
.replace(/([^\\])\//g, "$1\\/")
.replace(/([^\\])\//g, "$1\\/")}/i`;
}
function getUrlCheck(exp: ExperimentInterfaceStringDates): string {
if (!exp.targetURLRegex) return "";
const escaped = exp.targetURLRegex.replace(/([^\\])\//g, "$1\\/");
return `if(!location.href.match(/${escaped}/i)){return}`;
return `if(!location.href.match(${getUrlRegex(exp.targetURLRegex)})){return}`;
}
function getUserIdCode(t: TrackingType): string {