How to challenge your own analysis so others won’t | by Torsten Walbaum | Jul, 2024

Master the art of sanity checks to level up the quality of your work

Have you ever created an analysis only for it to be torn apart by your manager? Or have you ever gotten a question during a presentation that made you think “Why didn’t I check this beforehand?”

Sometimes it can feel like managers and executives have an uncanny ability for finding the one weak spot in your work. How did they identify the issue so quickly, especially if they are seeing your work for the first time?

What seems like a superpower can be learned by anyone, and this post will show you how.

By routinely applying “sanity checks” to your work, you can proactively identify the weak spots and ensure the result makes sense before you share it with a broader audience.

I am going to go over:

• What sanity checks are and why they matter
• How sanity checks are different from how most people check their work
• How to do a sanity check
• How to use sanity checks to increase your credibility
• How to use AI to sanity check your work for you

We’ve got a lot to cover, so let’s jump in.

Imagine you’re building a detailed model from scratch, carefully choosing each assumption and combining them to get to your final output (e.g. a forecast, company valuation etc.).

Each assumption seemed reasonable and you checked the math twice, so the output should be solid. Right? Right??

My experience over the last decade has been that more often than not, we miss the forest for the trees when building models or doing analyses. We layer so many assumptions on top of each other that the final result can quickly change from reasonable to ridiculous.

This is where sanity checks come into play: Sanity checks help us determine whether the result of our analysis falls has a good chance of being correct.

We’re all wrong from time to time. That’s fine; reality doesn’t always play out the way we expect. But you should try to be right most of the time.

Let’s dive into how to do that.

When checking our work, most of the time we go through it step-by-step to check for errors. Are the cells linked correctly? Did I pull the formulas all the way down? Do all the joins in my SQL work correctly?

This mechanical “Quality Control” approach of checking all inputs can help us find issues, but it doesn’t ensure that the output makes intuitive business sense.

Sanity checking, on the other hand, is about taking a step back and validating the output from a different angle. If you come to the same conclusion both ways, you can be much more confident in your work.

There are three broad categories of sanity checks: Bottom-up vs. top-down, benchmarking and intuition. I will go through each of them in detail and show how you can apply them at work.

Bottom-Up vs. Top-Down

Our analyses are typically either top-down or bottoms-up. But what does that mean?

Let’s look at a (simplified) example. Let’s say you work in a B2B SaaS company and want to figure out how many customers you can acquire for the new product you’re launching.

• In the top-down approach, we are trying to understand what share of the market we can win. So we’d start by looking at the total number of businesses in the US, exclude industries that we are not targeting and company sizes we can’t support, assume what % of companies is looking to switch software providers and finally assume what share of those companies we can win (vs. our competitors)
• In the bottom-up approach, we are trying to understand how many companies we can acquire based on the channels we have available. So we’d look at prior launches to figure out what lead volume we can get from LinkedIn, analyze keywords to determine expected SEM volume, project Email leads based on the number of companies we can target and expected conversion rates etc.

Both approaches can give us a directional idea of what we can expect, but they each have a crucial weakness.

The top-down approach does not consider how we are going to get these customers, while the bottom-up approach ignores the size of our target market.

As a result, the best way to sanity check your work is to combine top-down with bottom-up analysis.

Benchmarking

The best way to make sure a plan or projection is reasonable is to compare it against benchmarks. For example, if you’re forecasting the performance of a new market, it helps to compare it against prior launches of similar countries.

If your analysis massively deviates from the benchmark, you need to be able to explain why.

A few common things you should check for any model, forecast or projection:

• Magnitude: How do the final outputs compare to benchmarks? E.g. are you projecting that France will be a larger market for the company than the UK?
• Growth assumptions: What trend are you forecasting over time? E.g. is the new product projected to grow more quickly compared to past launches?
• Seasonality: Does your projection show the same repeating patterns as the benchmarks? E.g. if all other markets show a slowdown during the December holiday period, why does your projection for the new country not show this?

This doesn’t mean you always have to model everything in line with benchmarks; but you always need to be able to explain why something is deviating.

Benchmarking Example 1: New Market launch

Scenario: You plan to enter the UK market and forecast user growth

✅ Sanity checks: You compare the forecast against the two most recent launches, Germany and France. Your forecast is more aggressive than the last two launches, and doesn’t show any seasonality.

❓ Questions you need to be able to answer:

• What gives you confidence that that’s possible? Is the UK structurally different (e.g. the market is larger)? Is our product a better fit for the UK market? Are we using a different go-to-market strategy?
• Why is there no seasonality in France? Are holiday periods different? Do B2B buyers have different seasonal buying patterns?

👉 If you are not able to make a strong case for why the new market is different from past launches, you are better off keeping the forecast similar.

Benchmarking Example 2: Marketing Plan

Scenario: You are forecasting Marketing spend and performance metrics by channel. The plan is to double the Marketing budget year-over-year.

✅ Sanity check: You compare the projected marketing efficiency against past trends. Your forecast projects that efficiency (Cost per Lead) will improve as we increase Marketing spend, but past data shows the opposite trend.

❓ Questions you need to be able to answer:

• Why do you expect better efficiency?
• What specific improvements are we deploying in each Marketing channel that will drive this?
• Are we doing anything that would improve overall Marketing performance, e.g. investing in our Brand?

👉 If you don’t have a concrete plan to improve Marketing efficiency, you should assume that the historical relationship between spend and efficiency holds.

Comparing against intuition

Many times, you can use common sense to sanity check your analysis. Your intuition is not always going to be right, but it often highlights potential issues that need further validation.

A few examples:

• You build a Discounted Cash Flow financial model and the terminal growth rate is much higher than that of the Gross Domestic Product (GDP); this means you are implicitly assuming that the company will outperform the broader economy forever. Is that realistic?
• You are building an Account Scoring model and it scores companies as “good fit” that Sales believes are a waste of time. This doesn’t mean Sales is right (you build the model to surface new insights, after all), but you should take their experience into account since counterintuitive outputs often highlight model weaknesses

Sanity checks are not just there to prevent others from poking holes in your work. They also give you a tool to increase your credibility.

Instead of doing them behind the scenes and then sharing the improved work, share the sanity checks as well. By showing how you validated the output of your analysis, you will build trust with your audience. If you don’t share the sanity checks you did, the audience has to assume they have to check your work on the spot.

You can do this visually on a slide by showing how both top-down and bottom-up approaches get to the same result, or comparing your data against benchmarks.

But you can also do it verbally:

✅ “We are planning to grow to 50 Email leads in the UK by October; that is based on similar conversion assumptions as in Canada, and corresponds to 3% monthly penetration of our Total Addressable Market for Email

Sanity checking can be pretty time consuming; after all, you have to approach the same problem from multiple angles. Luckily, AI tools can save you a lot of time.

This is not a replacement for your sanity checking skills; ChatGPT needs your guidance to do a good job, so you still need to know how to perform a robust sanity check. The AI’s job is simply to do the heavy lifting for you, and to bring up a few points you might have missed.

Here is a step-by-step guide on how to do this with ChatGPT; all screenshots are from actual conversations I had where I asked ChatGPT to sanity check my forecast for a new market launch.

Disclaimer: Always check your employer’s policies on using AI tools like ChatGPT before uploading any proprietary data.

Step 1: Upload your work to ChatGPT

The first step is to upload the work that you want ChatGPT to sanity check. ChatGPT can handle a variety of file types, including PDFs, Excel, CSV files and more.

You can also integrate directly with several tools; e.g. in this example, I linked my Google Sheet that contained my forecast:

Even if your actual model lives outside a spreadsheet (e.g. in Python), I recommend dumping the outputs in Google Sheets for the sanity check; after all, you want ChatGPT to validate your outputs, not the mechanics of your model.

For this example, I gave ChatGPT this simple Go-To-Market Forecast for a new country launch (you can make a copy and try your own sanity check with it).

I went through four sanity checks; here are the logs:

1. First attempt (Grade: Intern)
2. Second attempt (Grade: Intern)
3. Third attempt (Grade: First-year analyst)
4. Fourth attempt (Grade: Over-confident first-year analyst)

• Attempt #1 and attempt #2 were okay, but I felt like I had to provide quite a lot of guidance and didn’t always get exactly what I wanted.
• The third attempt was pretty good, but both ChatGPT and I forgot to dig into the Marketing channel mix (and when I remembered a few days later, ChatGPT was unable to continue where we left off).
• Attempt #4 was promising, but when re-forecasting to include seasonality, it adjusted the wrong months at first.

You might have to try a few times until you get a really good performance. And remember:

Don’t blindly use anything that AI produces; it can (and will) make mistakes, and you’ll be on the hook for the end result. AI can give helpful input and save you time, but it’s not a replacement for critical thinking.

Step 2: Write a prompt asking ChatGPT to sanity check

After ingesting your file, you need to write a prompt asking ChatGPT to sanity check your work.

Here’s what I used to sanity check the Go-To-Market Plan I linked above:

I’ve found that giving a little bit of context on the dataset is helpful (although not absolutely necessary). Also, don’t forget to say “please” and “thank you” in case the AI ever gets sentient; this will keep you off the naughty list.

You could also give ChatGPT this article, or another summary of how to do sanity checks, so you don’t have to include too many instructions in your prompt.

Step 3: Make sure ChatGPT ingested the data correctly

After ingesting the file, ChatGPT will often typically give you a brief summary of what it sees and what it believes the data represents, as well as the steps it will take for the analysis:

ChatGPT will also often restate some of the data in the chat, which is helpful for making sure it pulled the numbers correctly:

Note: Initially, ChatGPT had some problems ingesting my spreadsheet. Here’s how you can troubleshoot:

• If ChatGPT throws an error or obviously didn’t ingest the data correctly, you can ask it to reprocess by clicking on the ♻️ icon
• If it repeatedly has issues with your file, the you might have to clean it up. For example, I found that I could significantly reduce errors if I removed non-essential rows and columns (e.g. section headers, comments etc.). If the file only contains the relevant tables, it’s easier for ChatGPT to convert them to data frames in Python. Giving descriptive column and row headers also helps ChatGPT make sense of the data

Step 4: Work through the sanity checks with ChatGPT

Next, ChatGPT will start providing some initial observations, like this:

ChatGPT correctly identified that the forecast is conservative and catches up with the UK eventually after a slow start. It would have been better if it had pulled in some stats for validation such as the number of small businesses in each country, but in my experience you need to ask it explicitly to do that.

Sometimes it will also proactively visualize key trends; other times, you have to prompt it for that.

Here’s how the conversation continued:

It’s nice that ChatGPT was able to pull the launch dates from the separate tab, and plot the performance for each country indexed by launch date.

We then got into seasonality; as you can see, I needed to provide the initial nudge and it took some back-and-forth, but ChatGPT did the work of identifying the correct patterns:

In several of my attempts, ChatGPT also gave a brief, but on-point summary of the Marketing mix, like this:

I’d say this is overall okay.

It correctly highlights Organic and Email as key areas that deserve a closer look. Unfortunately, it also draws some odd conclusions; it doesn’t make a lot of sense that Referrals in France would be as high as in the US given it’s a new market without an established customer base.

Step 5: [Optional] Ask ChatGPT to re-do your work

If you used ChatGPT to sanity check a forecast, it sometimes offers to re-forecast based on your discussion.

In my case, I asked it to incorporate seasonality:

It sucks if you put a lot of effort into an analysis only for it to be torn apart by others. By sanity checking your work before you share it you can massively reduce the chance of that happening.

It’s also a great way to build trust with more senior stakeholders and show that you are thinking like an executive.

For more hands-on analytics advice, consider following me here on Medium, on LinkedIn or on Substack.