Running Growth Experiments — CXL Review

Understanding the concepts concerning growth marketing is easy.

However, knowing which growth experiments to test first can be extremely tricky.

Oftentimes, most people would look online for a list of different conversion optimization tactics to run. But unfortunately, these listicles don’t tell you which experiments you should prioritize first.

In this Review, I’m going to go through what I learned in the second week at CXL Institute’s Growth Marketing MiniDegree. I’ll go through their Running Growth Experiments Chapter to show you how to conduct conversion research to gain sustainable results over time.

Let’s go!

1. How to Optimize Optimization

Before we jump into the details of conversion rate optimization, here are a few general rules to keep in mind.

1. Test (or make) more effective changes: Make sure you don’t blindly copy market leaders or your competitors. Test effective changes that are relevant to your site.
2. Reduce the duration (and cost) of optimization: Conversion optimization is an ongoing process and systematic way for finding opportunities for growth. The faster and cheaper your experiments are, the more you can run.
3. Improve the Speed of Optimization: To optimize quickly, you’ll need to implement a framework so you have all the problems and ways you can test conversion laid out ready for you to implement. They should also be in the order of prioritization so you can quickly get going.

CRO (Conversion Research and Testing mistakes to avoid while optimizing):

1. Precious time wasted on stupid tests
2. You think you know what will work
3. You copy other people’s test
4. Your sample size is too low
5. You run tests on pages with too little data
6. Your tests don’t run long enough
7. You don’t test full weeks at a time
8. Test data is not set to third party analytics
9. You give up after your first hypothesis fails
10. You’re not aware of validity threats:

• History Effect
• Instrumentation effect
• Selection Effect

11. You ignore small gains

12. You’re not running tests at all times

For brand new businesses, look into customer development first. Unless you have product-to-market fit, traffic, sales, and data you should forget about CRO and work on understanding your customers.

To get started, I recommend you look at:

• Startup Owners Manual: Steve Blank
• Lean Startup: Eric Ries
• Lean Analytics: Alistair Croll

2. The Research XL Process

80% of CRO is Research. Once you’ve conducted extensive research, then you spend 20% of the time implementing A/B tests and experiments. So before we dive into A/B Testing, we’re going to look at all the ways you can conduct research so you know where and how the website is leaking money!

1. Heuristic Analysis

Being data-driven is important, but there’s no substitute for a human-led evaluation of a website. That’s why, to begin, we’ll use experience-based techniques for problem-solving, learning, and discovery. While it’s a qualitative form of research, we’ll have to back everything up with data.

Evaluating Websites

There are a number of different frameworks you can use. The 7 Levels of Conversion by Web Arts, Invesp Conversion Framework, or even use a conversion formula developed by marketing experiment.

For example, Let’s try the Marketing Experiments Formula.

While you don’t need to understand too much depth, the formula shows you the probability of conversion and how it’s dependent on:

The visitor’s motivation + the clarity of the value proposition + (incentives to take action now — friction) — anxiety.

Another Framework you can use is the LIFT Framework.

How to Get Started

1. Perform Walkthroughs of the site with all the top browsers and each device category (desktop, tablet, mobile). Pay attention to site structure, go through the checkout, and form filling process. Make sure you familiarize yourself with the site and its structure. Identify any cross-browser and cross-device issues.
2. Assess each page for clarity

3. Evaluate how relevant the page is for visitors

4. Assess incentives to take action

5. Evaluate all sources of friction on the key pages

6. Assess any distractions

While you’re going through the site, write down everything you notice. Here, map out different areas of interest.

Then go through the website again with Google Analytics, survey data, and other sources and see if you can find data to confirm or disprove your findings. If you can’t find the data, make sure you note it down as something you’ll find out.

Buying Stages

The buying cycle has many stages, however, for the purpose of CRO, we can use 3 stages.

1. Awareness: When a lead first becomes aware of your product. Or could also refer to the point where a lead first becomes aware of a need that they want to fulfill.
2. Consideration: When a lead starts evaluating solutions to their need
3. Purchase: When a lead to ready to spend money and become a customer

The best way to bring people back to your site and develop a relationship is to capture their email address. Once you have it, put them through a lead nurturing process so that you add value and nurture the leads over time.

2. Technical Analysis

While many people consider technical analysis and technical issues, it has a major impact on conversion rates. Many conversions are lost due to poor cross-browser and cross-device compatibility issues.

Here are some questions you want to answer:

1. Does the site work with every major browser?
2. Does the site work with every device?
3. What’s the user experience like with every device?

How to Get Started:

1. Create a custom report to see conversions per browser.

2. Look at Conversion rates by Browser Version.

If you notice any drops in conversion concerning different browsers and devices, make sure you explore that situation further:

The great thing about technical analysis is that all the findings are super practical. They’re also easy wins as well. But be prepared to spend 1–2 days on Technical analysis. Alternatively, you can outsource this to technical people or other companies.

3. Digital Analytics

Here, we’ll look at digital analytics to identify problems in the flow stack. You’ll need to look at how people are flowing through your website, going from page-to-page and seeing where they drop out.

Usability Evaluation

Usability is the black horse of boosting conversion. If your website is difficult to use or hard to understand, it results in poor conversions. That’s why making websites easy to use are key for higher conversions.

• Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?
• Efficiency: Once users have learned the design, how quickly can they perform tasks?
• Memorability: When users return to the design after a period of not using it, how easily can they reestablish proficiency
• Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors
• Satisfaction: How pleasant is it to use the design?

To test usability, you can find every issue you find and write it on a spreadsheet. Then rank the items by the ease of implementation from 1 (a no-brainer, easy to implement), 2 (easy, but can be done within 1–2 hours), 3 (lots of development and/or designer hours needed).

1. Identify drop-off points
2. Correlate behaviors with outcomes
3. Fix measurements and verify data

4. Qualitative Research

Once you’ve improved the flow stack, it’s time to undergo surveys and user-testing.

Surveys

Here are a few tips when it comes to constructing your own survey:

1. Directly address the desired issue
2. Don’t make the survey too long (10–15 questions)
3. Make sure your questions are not asked in a leading way
4. Aim for large sample size 100–200
5. Implement an intuitive scale (the highest number should be the highest value)
6. Don’t mix questions of behavior and attitude (Group them together.)
7. Be aware of Cognitive biases in survey design
8. Be aware of Cognitive biases on the user end
9. Use the voice of the customer

One way to conduct these surveys is via email. Ideally, you want to survey people who still freshly remember their purchase and the friction they experienced in the buying process. Filter out repeat buyers or people who purchased a long time ago.

What to ask?

Analyzing Survey Results

1. Be clear about the goals and what you’re looking for

2. Conduct an initial review of all the info — gain an initial sense of the data

3. Code the data: Organize the information into a manageable form (reducing the data) — Develop some codes or categories (while keeping the raw data)

For example:

4. Interpret the data

5. Write a summary report of the findings

Here, write down the key learning to keep them at hand when you need to formulate hypotheses. I also recommend paying attention to their language — how do the people phrase the problem?

If you’re a visual person, I recommend generating word clouds from the responses. You can use tools like Wordle, TagCrowd, or Worditout to generate word clouds.

Web and Exit Surveys

It’s always great to ask questions from people who have just purchased something from your website. To capture this data, you can leverage web surveys. Web and exit surveys are pop-up boxes that appear to the visitor based on certain rules (e.g. time spent on site, number of pages visited, activity).

While your visitors may not like this, not all of them will see it, it’s only temporary, it’s well worth it and there’s no other good way to get the same data (understand why they’re not completing the purchase).

When to ask

But before you decide to pop the questions, you need to qualify the visitor and ask the right question at the right time.

Questions to Ask

How to get more people to respond

Here, you’ll need to experiment around.

User Testing

User testing is about insight into the user’s mind and usability (but not the same as usability evaluation). Usability is how easy user interfaces are to use. User testing helps you understand what stops people from doing stuff.

Here, you identify bottlenecks:

• Is it that they can’t find key stuff?
• Is something distracting them?
• Is the copy clear enough?

With user testing, you’ll find out whether the existing solution has shortcomings and what they are. It’s a great approach if the process makes sense or if the functionality of a page contains a lot of friction or anxiety.

Unlike A/B Testing, user testing involves people observing recruited testers complete a given set of tasks on the website. It can take place anywhere and your site doesn’t need to be online yet. User testing also requires only 10 people or less.

For sample size, the minimum number of people to conduct user testing with is 5 and the maximum 15. You can use tools like usertesting.com and TryMyUI if you need to recruit testers online. Ideally, these participants are seeing your site for the first time during the test.

Running the Tests

1. Over the shoulder testing: Here you can see what they’re doing and see how they feel about using the site (relevance, trust, orientation, security, convenience)
2. Unmoderated remote testing: Here you’re running the test remotely with online tools. Here, it’s unmoderated and you don’t have a physical facilitator.
3. Moderated remote testing: Here you can moderate testing from a distance. While you can clarify tasks if the user doesn’t get it, you’ll need to arrange a mutually convenient time.

Questions I tend to ask:

We recommend starting with a mindset as well.

Things you should be aware of:

1. Only ask questions you can learn from
2. Testers are not buyers
3. Sample sizes (and its limitations)
4. Don’t jump to conclusions
5. Record your testing sessions
6. Be open-minded

Customer Support, Live Chat

If you need more qualitative information, we recommend you talk to customer support staff. Here, see what people ask for the most, what their objections and concerns are. You can also use a live chat tool to gain further insights on frictions and objections as well.

5. Mouse Tracking

Mouse tracking provides valuable insights into viewing and information processing patterns.

Here, you can identify:

• Where people click and where they don’t (click maps)

• How far they scroll on any given page (scroll maps)

You can also undergo user session video replays — seeing how actual users interact with your site. They fill in the missing links for insight and help you understand how actual users use your forms, check out, and the issues they face. Tools you can use include Hotjar, Clicktale, Inspectlet, CrazyEgg, Mouseflow, Ghostrec.

6. Google Analytics Health Check

A health check is one of the first things you should do with Google Analytics after gaining access to the account. It’s vital if you want your analytics of CRO to work for you. Health checks are serious. Running with bad data renders all of your work useless.

Here, before you begin working on a new Google Analytics setup, make sure that everything that needs to be measured is being measured (goals, funnels, e-commerce, and event tracking setup), and the data is not corrupted.

While you can outsource this to an analytics expert who is familiar with the client’s goals, you as an optimizer need to be able to check most of these things yourself. You need to know how to find out what’s working and what’s missing.

What’s involved in a health check?

Output and Benefits

Checklist

Funnels and Goal Flows

Funnels are vital because you need to measure drop-offs per step. It’s the easiest way to see where the website is leaking money. It’s the optimizer’s gold mine. You should check funnel visualization to see the major problem steps, then move to goal Flows for detailed analysis.

Goal Flows
Goal flows are similar but not the same. With Goal flows you can see actual user flows towards goals (no backfilling). You can also segment the flow by traffic sources, landing pages, events, technical data like browsers, and so on.

You can also use tools like Heap Analytics, Mixpanel, Kissmetrics, and DashRocks. These tools are all supplemental and not a replacement to digital analytics tools.

Key Audience Insights

If you sell to big markets, it can be interesting to know where they are from (and can help you better understand your buyers.

Some regions might send a ton of traffic that doesn’t convert. Check your stats with an advanced segment and exclude those countries to get a better picture from those who actually buy.

Behavior: Visitor Type

New Vs Returning — Returning visitors convert typically 2x better. Sometimes you can see major differences which can be useful. Very low conversions for new visitors and reasonably high for returning visitors can indicate poor clarity or usability.

Demographics Reports

To get the demographics and interest data into your GA account, you will need to perform the following steps:

If you use Google Tag Manager, you should select “Add Display Advertiser Support” in your Google Analytics tag template.

Age, Gender, and Interests

You can break down your metrics using these parameters:

• Age
• Gender
• Affinity Categories (technophiles, music lovers, gamers)
• Other categories

Insights are in the segments.

You can try:

1. Male and Female
2. Age Groups
3. Affinity Categories

Check A/B Test results across demographics for insights. For example, if you’re testing imagery/ creatives, you might find out that the best creative for 18–24 is couples, 25–34 is babies, and 35–44 is families.

Site Search

Internal site search is important for conversion. To make sure your site search is configured, take a look at the Behavior → Site Search → Overview report and if it shows nothing, it’s most likely unconfigured.

How to Set it Up

Compare the Conversion Rate for Visits (With and Without site search)

Usually, the conversion rate for visits with site search is usually TWICE AS HIGH as visits without. Make sure you subtract bounced visits from the comparison.

Where does the search happen

The Site Search → Pages report tells you what pages people are on when they use search. Most of it happens on the home or blog page.

Content Reports

Content reports help to identify pages that work better than others.

Top Content:

Here, you can find high-traffic, low-performance pages (high bounce, exit rate).

In this example, you should look at the pages ranking 7,8,10, and 13. Filter out the pages with less than 1000 page views and increase the observed time if the sample size is too small. Sort the table with bounce rate or exit rate. Then identify the pages with the worst bounce rates that still get quite a bit of traffic.

Landing Pages
Use the comparison button to look at bounce rates for top landing pages compared to the site’s average rate. Top landing pages with bounce rates higher than the site’s average should go on the list of problem pages to investigate. The goal is to identify high traffic, low-performance landing pages.

Navigation Summary

Use the navigation summary to see what people are doing on a specific page. Which links get clicked on and which links don’t. It’s a great way to evaluate which links on any given page are needed and which ones are not.

You can also look at in-page analytics, to view behavioral metrics for a specific page.

Screen Resolutions, Browsers

You can also use Google Analytics to see which browsers are being used and which ones perform worse than others. Here, run the “conversions per browser” report, and segment by device category.

You can see here that chrome converts less than IR, and somewhat less than Firefox. Opera only has 3 transactions so it’s not a statistically valid sample size.

It’s also worth checking out funnel steps per browser/device.

Mobile Devices

Look at the distribution between desktop, tablet, and mobile.

Here, you can see that smartphones converted just 0.76% compared to 5.74% desktop. Mobile UX is probably pretty bad — tons of money leaking here!

Tech: OS and Screen Resolution

Here, have a look at whether people are on Mac or Windows. What’s the most common screen resolutions that determine where the fold is and how much of your site they’re seeing. Is there a big conversion rate difference between resolutions?

7. Copy Testing

Copy is the single most important element when it comes to conversion rates. Copy is two times more influential than design. That’s why you need data on your copy to assess and see whether it’s good — whether these words and terms land with your audience.

With copy testing, you gain data on which parts are good and which parts fall flat.

The key things to assess in copy are:

• Clarity and Information Completeness
• Value (How much do they care)

You can easily have people within your target audience test your copy using copytesting.com.

8. After Research

Once you finish gathering your data, writing down your observations and findings, you can then list out your problems to see where your biggest leaks are. Here you can use the Ring Model to visualize this.

The Ring model helps you identify where your flow is stuck — where traffic is not flowing down to the next level. To construct it, go to your Google Analytics report. Go to Behavior, Site Content, and select all pages.

1. Map out the traffic flow per layer of the site (see where the flow is stuck)
2. Verify whether the goal funnel has been configured properly.

If it helps, you can visualize it like a funnel.

List of Custom Reports

If you can’t create any custom reports of advanced segments yourself, you can’t call yourself an optimizer. You need to be able to run any report that’s needed.

Here are a few common reports to run in addition to standard reports.

• Conversion rate per device
• Conversion rate per browser
• Site search usage
• Conversion rate per time of day
• Conversion rate per day of the week
• Conversions per traffic source
• Conversions per keyword
• Top-performing landing pages
• Social media performance
• PPC Analysis
• Conversions per location (country, city)
• New Vs Returning Users
• 404 Pages

Conversion Research Reports

Conversion Research is mainly for you to know what’s going on. While there’s no universal best structure for a report, you can start with:

• Individual research outcome (surveys, Qualaroo, Google Analytics) and then propose changes page by page (combining heuristics with data)
• Make sure it’s presented in a live presentation format (in person or webinar)

Conclusion

Overall, I found the second Chapter on Growth Marketing concerning Conversion Research Optimization extremely in-depth.

I was impressed at how much-condensed knowledge was given in each video, paragraph, and section. While the content was a bit overwhelming at first, I decided to push myself through all the challenging concepts and move forward.

This was difficult at first but highly rewarding. I walked away with new knowledge, new skills, and a completely new mindset and approach towards CRO (Conversion rate optimization).

The three key points I’ve taken away from this week are:

1. CRO is 80% research and 20% experimentation. You need to do thorough quantitative and qualitative research before you begin testing.
2. Make sure you have a solid framework before rushing off to research and test. Having a framework helps you get shit done faster so that you can research and test quickly and affordably.
3. There’s no point in gathering data that has no use. Make sure you only gather data that can help you improve your research and can add value. Data alone is passive. Know what you’re looking for and how that data can validate/invalidate your research.

If this is a lot to take in, be prepared for Week 3. Week 3 is when we take our research and learn how to leverage it to implement effective A/B Tests. We’ll also be looking at the key statistics fundamental for A/B Tests.