All pages in this guide
Start here
- Getting out ready: start here
- Is industry right for you?
- Roles and who hires economists
- The five-year timeline
Build your proof
Be memorable
Communicate like industry
Apply
Interviews
- How industry interviews work
- Behavioral interviews
- Technical interviews
- Coding and SQL
- Your research and interview day
Offers and first job
About
Other path
During the PhD · Interviews
Technical interviews: causal inference, experiments and statistics
This is where economists should shine and often do not, because they answer like a seminar instead of like a colleague solving a business problem.
Technical interviews for economist and data scientist roles test whether you can use your training to answer business questions. Economists usually know the methods well. Where they lose points is the approach: they jump straight to an estimator, answer in seminar language, or solve a slightly different problem from the one the business has. This page gives you a framework for causal and experimental questions, then the questions themselves, grouped by topic, with what the interviewer is looking for in each.
The two ground rules and the five-step framework below are adapted from Evan’s guide to technical interviews for economists on The Most Dismal Scientist. The topic coverage in the question banks draws in part on Scarlet Chen’s Tech Industry Jobs for Econ PhDs, Part 2. The questions, explanations and examples are my own.
Two ground rules
Stay within what you know. If a question takes you beyond your expertise, say so plainly: “I haven’t used that method in practice, but here is how I would think about it.” Interviewers almost always respect that and move on. Bluffing is the fastest way to fail.
Lead the conversation. Ask clarifying questions, state your assumptions and offer the interviewer choices: “We could run an experiment or use the historical rollout. Which is more realistic here?” Leading keeps you away from unproductive tangents and shows exactly the behavior they want in a colleague.
A framework for causal questions
Many technical interviews are some version of “did X cause Y, and by how much?” Use these five steps every time. The method you pick matters much less than getting the first three steps right.
1. Confirm what the business wants to know, and why
Ask what decision depends on the answer. “Does the new checkout button increase sales?” might really mean “should we roll it out to every country?” That changes what you measure and how precise you need to be. If it becomes clear this is a forecasting or prediction question rather than a causal one, say so; the approach is different.
2. Agree on the treatment and the outcome
Define exactly what the treatment is and which metric you will measure: click-through rate, units sold, revenue or profit, and over what time window. These become the key variables in your analysis. Getting this wrong is a serious mark against you, because a correct method on the wrong metric is still a wrong answer.
3. Describe the ideal experiment
State it explicitly as a thought experiment: “If we could randomly assign the button to half of users, comparing the two groups would give us the effect on purchases. Is that the effect you want?” This clarifies your own thinking, shows you understand selection and omitted-variable problems, and lets the interviewer correct you early. Do not assume an experiment is possible; many businesses find them costly, slow or risky for customer trust.
4. Check feasibility, then find variation
Ask whether an experiment is feasible. If yes, design it (sample, randomization unit, duration, power). If not, look for natural variation that approximates the experiment, and propose it as a question to get buy-in before you go further:
- Difference-in-differences when the change rolled out to some groups or places at different times. Versatile; relies on parallel trends.
- Regression discontinuity when eligibility depends on a threshold in a score, a date, a distance or a geographic boundary.
- Instrumental variables when something shifts treatment without affecting the outcome directly, such as a randomized email encouraging people to try a feature.
- Synthetic control when one market or region was treated and you need a comparison built from several untreated ones.
- Matching or regression adjustment when you believe you observe everything that drives selection. State that this is a strong assumption.
5. Work out the details
Power and sample size, controls that add precision, pre-trends and placebo checks, spillovers between groups, novelty effects, and how you will explain the identifying assumptions to the business. Prefer simple methods executed well. A careful linear regression with a clean comparison often does most of the work in practice; mention more advanced methods only if you know them well.
The framework in action
Example question
“We launched free shipping for orders over $35 in Texas last quarter. Did it work?”
A strong candidate starts by asking what “work” means: higher revenue, higher profit after shipping costs, more orders or higher order values? Suppose the answer is profit per customer over three months. The candidate then states the ideal experiment, random assignment of the threshold across customers, and notes it was not done. Since the change applied to one state at one date, they propose difference-in-differences, comparing Texas to similar states before and after the launch, or a synthetic control built from a weighted mix of states. They mention checking that Texas and the comparison states moved together before the launch, watching for customers bunching just above $35, and accounting for anything else that changed in Texas that quarter. They finish with how they would present it: “Free shipping raised profit per customer by about X, with a range of Y to Z; the main risk to that estimate is…”
Question bank: causal inference
T1We gave a discount to customers who were about to cancel, and they stayed longer than customers who did not get it. Did the discount work?
Looking forRecognizing selection: the people who received the discount were chosen differently. You need a comparison group that was similar before the discount, ideally from a randomized holdout, otherwise a design that uses how the discount was assigned.
T2A new feature was rolled out city by city over six months. How would you estimate its effect?
Looking forStaggered difference-in-differences, a check of pre-trends, awareness that simple two-way fixed effects can be biased when effects change over time and that newer estimators address this, and attention to why cities were chosen in that order.
T3Customers with a credit score above 700 get a lower interest rate. How would you estimate the effect of the rate on default?
Looking forRegression discontinuity at 700, comparing borrowers just above and below, plus checks for manipulation of scores near the cutoff and for other things that change at 700.
T4We cannot randomize price. How would you estimate price elasticity for our product?
Looking forThe endogeneity of price (prices respond to demand), and sources of variation: cost shocks, promotions set in advance, differences across regions, past price tests. Mention instruments and their assumptions, and the difference between short-run and long-run elasticity.
T5How would you measure the effect of a TV advertising campaign on sales?
Looking forGeographic variation (markets with and without the campaign), matched or synthetic comparison markets, the timing of effects, carry-over after the campaign ends, and the difficulty of separating the ad from seasonal demand.
T6What is the difference between correlation and causation? Give an example from business.
Looking forA clear, short explanation with a realistic example, such as heavy app users seeing more ads and also buying more, without the ads necessarily causing the purchases. Then how you would get at causation.
T7What are the assumptions behind difference-in-differences, and how would you check them?
Looking forParallel trends in the absence of treatment, no anticipation, stable group composition and no spillovers. Checks: pre-trend plots and event studies, placebo tests on fake dates or untreated groups.
T8What makes a good instrument? What goes wrong with a weak one?
Looking forRelevance and the exclusion restriction, explained in plain words. Weak instruments give biased, imprecise estimates; check the first stage. Note that IV estimates the effect for people whose treatment the instrument changes.
T9Our loyalty-program members spend twice as much as non-members. Should we push everyone into the program?
Looking forSelection again: big spenders join loyalty programs. Propose an encouragement design, such as randomly inviting some customers, and estimate the effect for those who join because of the invitation.
T10How would you explain your causal estimate and its uncertainty to an executive?
Looking forPlain language and a range: “The program increased spending by about 6%, most likely between 3% and 9%. The main thing that could make this wrong is…” No p-values or jargon.
Question bank: experiments and A/B testing
T11Walk me through how you would design an A/B test for a new recommendation algorithm.
Looking forHypothesis, primary metric and guardrail metrics, randomization unit, sample size from a power calculation, duration covering full weekly cycles, a plan to check randomization balance, and a decision rule set in advance.
T12How do you choose the unit of randomization: user, session, day or city?
Looking forRandomize at the level where treatment is experienced and where spillovers are contained. Users for most product changes; cities or markets for pricing, delivery or two-sided marketplaces where treated and control users interact. Note the loss of power with larger units.
T13How would you decide the sample size and the duration of a test?
Looking forA power calculation using the baseline rate and its variance, the minimum effect worth detecting, the significance level and the power (often 80%). Run full weeks to cover weekly patterns; watch for novelty effects.
T14The test shows a 2% lift in clicks but no change in purchases. What do you recommend?
Looking forGo back to the business goal. Clicks are a proxy; if purchases do not move, the feature may just shift behavior. Check whether the test was powered to detect a purchase effect and look at segments, then recommend based on the metric that matters.
T15The result is not statistically significant. What does that mean, and what do you do?
Looking forNot significant does not mean no effect. Report the confidence interval: if it rules out effects large enough to matter, you can conclude the change is not worth it; if it is wide, the test was underpowered. Do not extend the test or slice the data until something is significant.
T16A product manager checked the dashboard every day and stopped the test on the day it became significant. What is the problem?
Looking forRepeated peeking inflates the false-positive rate. Fix it with a fixed sample size decided in advance, or a sequential testing method designed for repeated looks.
T17We tested 20 versions of a headline and one was significant at 5%. Should we ship it?
Looking forMultiple testing: with 20 tests you expect about one false positive by chance. Adjust for multiple comparisons or confirm the winner with a fresh test.
T18How would you test a change in a two-sided marketplace, such as a ride-sharing or delivery platform?
Looking forInterference: treated riders take drivers from control riders, biasing a user-level test. Use market-level or time-based randomization (switchback designs) and discuss the tradeoffs.
T19The treatment group has 10% more users than expected. What do you do?
Looking forA sample ratio mismatch suggests a bug in assignment or logging. Investigate before trusting any result; the experiment may be invalid.
T20Why might the effect measured in a two-week test differ from the long-run effect?
Looking forNovelty or learning effects, seasonality, users adapting, competitor responses. Mitigations: longer tests, long-term holdout groups, looking at how the effect changes over the test.
Question bank: metrics and product sense
T21How would you measure the success of a new feature that lets users save items for later?
Looking forTie metrics to the goal: adoption, conversion of saved items, and the effect on overall purchases, not just saves. Add guardrails such as time to purchase and cart abandonment. Say how you would separate the feature’s effect from the type of users who adopt it.
T22Daily active users dropped 8% yesterday. How would you investigate?
Looking forA structured approach: check whether the data is right (logging, pipeline), then segment by platform, region, app version and traffic source, look for external events and recent launches, and narrow to the cause before guessing.
T23What is wrong with using revenue as the only success metric for a promotion?
Looking forDiscounts can raise revenue while cutting profit, pull purchases forward from later weeks, or attract customers who do not return. Use profit, measure over a longer window and check for pull-forward.
T24Our average order value went up, but total revenue went down. How is that possible?
Looking forA change in composition: fewer, larger orders, for example because small-order customers left. Explain mix effects and how to decompose the change.
T25Propose a metric for the health of a marketplace and explain its weaknesses.
Looking forA reasonable choice, such as successful transactions per active buyer or the share of requests fulfilled, with honest weaknesses and a second metric to cover them.
Question bank: statistics and probability
These often appear inside other questions. Make sure you can answer each one in plain language in under a minute.
T26Explain a p-value to a product manager.
Looking for“If the change truly had no effect, this is how often we would see a difference this large just by chance.” Avoid saying it is the probability the effect is real.
T27What are type I and type II errors? Which is worse?
Looking forFalse positive versus false negative, with a business example of each, and the answer “it depends on the cost of each mistake,” with an example.
T28What is statistical power, and what increases it?
Looking forThe probability of detecting an effect that really exists. Larger samples, larger true effects, lower variance (for example through controls or pre-period data) and a higher significance threshold increase it.
T29How would you interpret a 95% confidence interval of 1% to 5% for a lift?
Looking forThe procedure that produced this interval captures the true effect 95% of the time; plausible values are between 1% and 5%; the effect is positive and could be small or meaningful, so the decision depends on what lift is worth the cost.
T30When would you use the median instead of the mean?
Looking forWith skewed data or outliers, such as revenue per user or delivery times. Note that the business may still care about the mean (total revenue), so you may need both.
T31What is the central limit theorem, and why does it matter for A/B testing?
Looking forThe average of many independent draws is approximately normal regardless of the underlying distribution, which justifies normal-based tests on averages even for skewed metrics, given enough data.
T32What is Simpson’s paradox? Give an example.
Looking forA trend that appears in every group but reverses when the groups are combined, because the groups have different sizes. Example: a treatment that wins on both mobile and desktop but loses overall because it was shown mostly to mobile users, who convert less.
T33You roll two fair dice. What is the probability the sum is 7? At least one six?
Looking for6/36 = 1/6 for a sum of 7. For at least one six, 1 − (5/6)² = 11/36. Show the reasoning, not just the number.
T34A test for a rare fraud pattern is 99% accurate, and 1 in 1,000 transactions is fraudulent. If a transaction is flagged, how likely is it to be fraud?
Looking forBayes’ rule. In 100,000 transactions, about 99 of the 100 frauds are flagged and about 999 of the 99,900 legitimate ones are flagged, so roughly 99 of 1,098 flags, about 9%, are fraud. The base rate matters.
T35How would you estimate the uncertainty of a statistic that has no simple formula, such as a ratio of medians?
Looking forThe bootstrap: resample the data with replacement many times, recompute the statistic each time, and use the spread. Resample at the right level (for example by user if users have many sessions).
Question bank: econometrics, forecasting and machine learning
T36What assumptions does ordinary least squares need to give an unbiased causal estimate?
Looking forMost importantly, that the error is unrelated to the regressors (no omitted variables, no reverse causality, no measurement error in the regressors). Distinguish this from the assumptions needed for correct standard errors.
T37When and why would you cluster standard errors?
Looking forWhen observations within groups are correlated, such as users within cities or repeated observations of the same user, and especially when treatment is assigned at the group level. Cluster at the level of assignment.
T38What is the difference between a model built for prediction and a model built for causal inference?
Looking forPrediction cares about accuracy on new data and can use any correlated feature; causal inference cares about the coefficient on one variable being unbiased. A great predictor can give misleading “effects.”
T39How would you forecast weekly demand for each of our stores for the next quarter?
Looking forClarify the horizon, the level and how the forecast will be used (for example to avoid stock-outs, which means you may need a high percentile, not the average). Start with simple baselines and seasonal models, add drivers such as promotions and holidays, validate on held-out past periods, and report uncertainty.
T40What is overfitting, and how do you prevent it?
Looking forA model that fits noise in the training data and predicts poorly on new data. Prevent it with train and test splits, cross-validation (time-based for time series), regularization such as lasso or ridge, and simpler models.
T41Explain the difference between lasso and ridge regression.
Looking forBoth shrink coefficients to reduce overfitting. Lasso can set some coefficients exactly to zero, so it also selects variables; ridge shrinks all coefficients without removing any and handles correlated predictors more smoothly.
T42How would you evaluate a model that predicts which customers will cancel?
Looking forUse a held-out set, look at precision and recall rather than accuracy when cancellations are rare, an ROC curve and AUC, and above all the business value: what it costs to contact a customer versus to lose one.
T43When would you choose a tree-based model, such as gradient boosting, over a linear model?
Looking forWhen relationships are non-linear or involve interactions and the goal is prediction. Tradeoffs: less transparency, more tuning and a greater risk of overfitting on small data.
Case and product questions
Case questions describe a real problem the company faces and ask how you would approach it. There is no single right answer. The interviewer is watching how you structure the problem, which questions you ask, whether you choose sensible metrics and methods, and whether you end with a recommendation.
A reliable structure: restate the problem and ask what decision depends on it; agree on the goal and the metric; lay out two or three possible approaches with their tradeoffs; pick one and go deeper; name the main risks; finish with what you would recommend and what you would need to know to be confident.
Practice with questions like these:
- Our food-delivery app is considering a subscription with free delivery. How would you decide whether to launch it and at what price?
- Bookings at our hotels fell in one region. Is it our pricing, a competitor or something else?
- We want to reduce late deliveries. Where would you start?
- Should we show fewer ads to new users in their first week?
- How would you decide where to open our next ten stores?
To prepare for a specific company, use its product as a customer, read its engineering and data science blog, and watch talks its economists and data scientists have given. Case questions are usually drawn from the company’s real problems, and you will be able to ask much better questions.
Credits
Framework and ground rules adapted from Evan, The Most Dismal Scientist, “Technical Interview Framework.” Topic coverage informed by Scarlet Chen, Tech Industry Jobs for Econ PhDs, Part 2 (2020).