A/B Testing Automation - Custom Python Module

NOTE***: Certain information has been redacted/changed to protect proprietary information

While responsible for analyzing and visualizing A/B tests on the website, I noticed the process required many repetitive tasks from the analyst. Tasks like connecting to the AWS data lake, writing a query, and changing scripts for new tests and/or metrics began taking up time that could be better spent elsewhere.

The goal was to create a Python module, which could be imported by the analyst, to abstract away much of these tasks while still allowing for customization when needed.

In a Python script, the analyst is asked to provide relevant (non-sensitive) information such as their AWS directory, test number, and the date the test started. They are also asked to provide the expected (or desired) conversion rate difference between the control & variant, as well as the metrics they would like to test. With this info, they can call run_query() and analyze() and voila! An A/B test is executed.

Want to dig into the redacted code yourself? Head on over to my Github repo!