How to: Write experiment docs

Product managers run tons of experiments every week. A good way to be proactive and intentional about your experiments (and signal that to leadership) is by documenting your experiments and results.

In my four years as a product manager, I’ve learned that while critical thinking and proactiveness are essential, effectively communicating those skills to others is even more critical. At my company, I’m known for writing clear, informative documentation. In this article, I’ll show you how to document your experiments effectively so you can achieve the same.

Why document your experiments?

As product managers, we’re constantly testing hypotheses. Will this copy improve conversion? Will removing a step in onboarding reduce friction? Will adjusting a third-party tool’s configuration boost operational efficiency?

If you haphazardly make changes to answer these questions, you might get results, but you won’t know why those results occurred—or how to replicate success in the future. Worse, if you leave your company, your replacement or teammates will struggle to understand what’s been tried and what hasn’t without a historical record.

By documenting your thought process, you not only clarify your thinking but also become more intentional about your hypotheses, goals, and resource usage. Clear documentation shows respect for your team by transparently sharing successes and failures. It also signals your efforts to leadership, increasing the likelihood that your work will be recognized and understood.

The general structure

Here’s a proven structure for documenting experiments. I recommend sharing this iteratively with your team. At the first stage, share with the team after completing sections 1-6 so they can understand what you’re testing. After completing the experiment, share the document again, with sections 7-8 filled out, showcasing your results and proposing next steps.

1. Background/Key Problem Statement

Explain the background and context of the problem. Highlight its importance, what prompted you to address it, and how it ties into the company’s overarching goals.

2. Goal

Define your experiment’s objective within the broader company goals. For example, if the company’s aim is to increase revenue by 10%, your sub-goal might be increasing user conversions by 5% during onboarding.

3. Key Definitions

Clarify any jargon or terminology your readers need to understand, including bespoke metrics or user journey terms specific to your company.

4. Driving Hypothesis

State your hypothesis: Why do you think a particular change will help achieve your goal? This is often the most challenging part of the process.

5. Success Metrics

List the metrics you’ll use to measure success. Be specific about tools, data sources, and how the metrics align with your objectives. For example:

• Metric: Conversion rates during onboarding

• Tool: GA4 funnel analysis

• Goal: Increase onboarding conversion by 5%

6. Experimental Design

Detail the changes you’ll make and any dependencies. Will you run an A/B test? What’s the split? Do you need marketing approval? Be thorough in outlining the timeline, process, and resources required. Identify factors you can’t control and how they might impact results.

7. Results and Analysis

Once the experiment concludes, document the outcomes. Include graphs or data visualizations and analyze whether the results validate your hypothesis.

8. Next Steps

Reflect on the experiment and determine what you’ll need to do next to clarify new questions or continue achieving your goal.

Example

This is an example of a real experimental doc that I wrote for a family business that I’m advising. As you can see below, it doesn’t have to be super fancy or complicated. It just has to be clear and easy for everyone to understand.

Status

Actively in progress

Background

Exam Bytes Academy (EBA) seeks to increase revenue by attracting more parents to enroll their children in tutoring lessons, aiming for £5,000 in revenue by March 2025. Founded in 2020, EBA initially grew through strong word-of-mouth referrals. However, signups have declined as the pool of parents familiar with EBA's successes has diminished, and new parents seeking resources for the 11+ exam remain unaware of EBA's offerings.

Goal

Sign up at least 10 students for the Fundamentals Group Lesson Course running February–March.

Definitions

• Group Lessons Course: A one-hour session for multiple students, held five times weekly, with each session covering a different subject. Each student generates £60 in weekly revenue.

• Parents: EBA's primary customers who make purchasing decisions based on EBA's perceived value.

• Students: The users receiving the tutoring services.

Hypotheses

• Parents are unaware of EBA:

  • Reasoning: EBA's reliance on word-of-mouth marketing has limited its visibility. With no social media or online presence, potential customers lack awareness of its services.

• Parents don’t know whether they can trust EBA courses

  • Rationale: Competitors like Atom Learning and Bond 11+ have strong signals of trust on their website. Atom Learning displays a trustpilot rating of 4.3/5 and publishes weekly blogs to establish its dominance as a trusted source of knowledge for parents seeking 11+ resources. Bond 11+ advertises that it has been around for more than 30 years and is owned by Oxford Publishing, thus utilizing its parent company to convey trust and legacy. EBA must build trust to compete against legacy competitors.

Success Metrics

1. Unique Blog Views (30 days):

   • Method: Use Substack's writer dashboard to track unique views.

   • Goal: Exceed 50 unique views within 30 days.

2. Google Search Ranking:

   1. Method: Check the top 10 Google search results for “Exam Bytes Academy.”

   2. Goal: Ensure EBA articles occupy at least 5 of the top 10 results.

3. Signups for February–March Group Lessons:

   • Method: Monitor bank deposits to track parent signups.

   • Goal: Achieve at least 10 signups.

Experimental Design

Initiative: Publish blog posts on the EBA Substack every 2–3 days.

• Objective: Answer common 11+ exam questions to improve EBA’s SEO ranking and establish thought leadership.

• Hypotheses Addressed: Both

Execution Requirements:

• Articles: 500–800 words each, with cover photos.

• Distribution: Cross-post articles to LinkedIn and Facebook with captions.

• Frequency: Publish 3 articles weekly.

• Owner: Ranjani

• Duration: 1 month

Considerations:

• We must remove internal views from unique visitor counts

• In the future, we will need to evaluate content performance to improve engagement and conversion rates

Results

Unique Blog Views (30 days): The blog initiative showed promising progress in improving Exam Bytes Academy's online presence. Over the past 30 days, the Substack blog achieved 64 unique views (excluding internal views), exceeding the initial goal of 50. While this indicates growing awareness, the views did not lead to any subscriptions, suggesting the need to refine content to better engage readers or reach a more targeted audience.

Figure 1: Views, users, and subscription information from the last 30 days

Google Search Ranking: Google search rankings also improved significantly. EBA now occupies 6 of the top 10 results for searches of "Exam Bytes Academy," exceeding the goal of securing 5 top spots. These results include links to the blog and other EBA-related pages, highlighting the academy’s growing online footprint.

Figure 2: “exam bytes academy” incognito search on Google

Signups for February–March Group Lessons: Enrollment in the February–March group lessons has been slow, with just one parent signing up. This enrollment came through word-of-mouth rather than blogging efforts, suggesting that while the blog may be building brand awareness, it is not yet driving conversions. This indicates a need to explore additional marketing strategies to attract and convert potential customers more effectively.

Next steps

Moving forward, the blog will remain a key focus to continue improving search engine rankings and solidify EBA's reputation as a trusted authority in 11+ exam preparation. Simultaneously, EBA will broaden its marketing efforts to include platforms like Instagram and TikTok, which have the potential to reach a larger, more engaged audience.

Additionally, a deeper analysis of the blog content will help identify which topics resonate most with readers. By focusing on high-performing content, EBA can tailor its messaging to address parents' needs more effectively and improve conversion rates. These combined efforts aim to both expand EBA’s reach and enhance its ability to turn awareness into signups.

Practice makes perfect

It might take a while to get used to documenting your thinking. But I promise, once you start doing it consistently, you’ll never want to go back. You’ll gain recognition for your work, provide your team with clear guidance, and sharpen your critical thinking as a product manager.

Have questions about this framework? Want to know more about how I approach other frameworks? Let me know!