A/B Test a New Menu Launch



A/B Test allows business to make decisions with confidence when data is limited. This is a good guide on designing and running an AB Test. This project teaches us how to use Alteryx to analyse A/B Test results.


For this project, we will be looking at a coffee chain (round roasters) who wants to introduce gourmet sandwiches to the menu, along with limited wine offerings. Management team believes that a TV advertising campaign is crucial to driving people into these stores. Due to the TV campaigns requiring high marketing budget, management team have decided to test the changes in two cities (Denver and Chicago).


The test ran for a period of 12 weeks (2016-April-29 to 2016-July-21) where five stores in each of the test markets offered the updated menu along with television advertising. The comparative period is the test period, but for last year (2015-April-29 to 2015-July-21). If there is at least an increase in 18% profit, then management will proceed with the changes.


Provided are these three data files.

  1. Transaction data for all stores from 2015 January 21 to 2016 August 18. (this file is fairly large (200mb), with over 4 million rows, you can download this here if you want.)
  2. A listing of all Round Roasters stores
  3. A listing of the 10 stores (5 in each market) that were used as test markets.

As always in a Udacity project, we go through the project in 3 steps:


Step 1: Plan Your Analysis

What is the performance metric you’ll use to evaluate the result of your test?

As mentioned above, we want to have at least 18% increase in profit when comparing the test results to the comparative period. Increasing profit will be our performance metric.


What is the test period?

The test period if 12 weeks between 2016 April 29 to 2016 July 21.


At what level (day, week, month etc) should the data be aggregated?

The data should be aggregated on a weekly level. (We’ll use Alteryx to aggregate this later).


Step 2: Clean Up Your Data

To proceed to the next section, you will need to clean your data and aggregate it to a weekly level. You can see above the workflow I used on Alteryx to get the store_list.yxdb & weekly_store_traffic .yxdb output (which we will use to match control stores to the treatment store list). You can ignore the store_sales_analysis.yxdb, we won’t use this for the projects, but instead weekly_store_traffic. You can see the weekly_store_traffic data here and store_list data here.


Step 3: Match Treatment and Control Units

Apart from trend and seasonality, what control variables should be considered? Note: Only consider variables in the RoundRoastersStore file.

If we look at the round roaster store file, there is really only two other control variables we could consider, and that is Sq_ft and Avg Monthly Sales.



What is the correlation between your each potential control variable and your performance metric?

Our performance metric is profit (gross margin). In the previous step when we cleaned our data, we found the gross margin of each store in the weekly_store_traffic file. We can use this data to find the correlation of Avg Monthly sales and Sq_ft on gross margin. This can be done in excel using the correlation formula. We find that Sq_ft has a negative correlation of 0.086 and Average monthly sales has a positive correlation of 0.85. So in this case, we will just use the Average monthly sales (and Trend & Seasonality) as a control variable to match our treatment and control stores.


Above is the workflow in Alteryx that will help us match the control store to the treatment stores(provided). The settings used in the AB/Trend tool on Alteryx is simple enough: 12 is the number of periods used to calculate trend & our start date is 29th of April 2016. Below is a summary result of what control stores is matched to each treatment store.


Treatment StoreControl Store 1Control Store 2

Step 4: Analysis and Writeup

Let’s now use the control_treatment_pair.yxdb we got from the last step and the weekly_store_traffic file to conduct our AB Analysis. This can be easily done on Alteryx using the AB Analysis tool. You can find the control_treatment file here.

We have a filter [Region] = Central to allow us to get the lifts for each region (AB Analysis tool connected to True gives us the central region lift, whilst False give us the west region lift) and the third AB Analysis tool gives us both regions combined lift. Remember, we will use gross margin as our performance measurement.


Should the company roll out the updated menu to all stores?

Yes, the company should roll out the updated menu to all stores. The lift from the new menu for Central Region is 43.5% with a statistical significance of 99.6, and the lift from the new menu for West Region is 37.9% with a statistical significance of 99.5%, both exceeded our 18.5 criteria.


Here are the results of the central store test:

And the west:

Overall, the new menu lift is 40.7% (average of both West and Central lift).

You can learn more on A/B Testing for Business Analyst at Udacity for free here.

Feel free to message me if you need tech help, have work opportunities, or if you want restaurant recommendation in Sydney! Contact Icon