There are many factors which contribute to a house model performing well. Ideally, each should be considered when evaluating a model or exploring your modeling options. I’ve prepared a list of the factors I’ve found most influential in the effectiveness of predictive customer models for existing buyers. They are ranked from the most important to those least important, based upon my experience.
Successful models are primarily determined by their data inputs. A well-defined set of customer summaries derived from behavioral transactions (purchases, promotions, email activity, web visits, etc.) goes a long way towards producing good models.
Experience matters. An expert modeler will automatically incorporate best practices.
Defining what you wish to predict is also important. While once this was relatively easy, now it is almost as complex as defining the predictor set. Revenue allocation must be considered, as well as whether you are predicting profit contribution or its components (response, average order size, return, payment, margin, etc.).
Business knowledge is valuable. The more the modeler understands the data, the marketing events, and the general business, the better the model. While science is certainly a key factor in modeling, it can also be an art, so knowing the environment is extremely useful.
Financial forecasts of the event utilizing the model dictate the contact depth and circulation volume. Accurate forecasts yield correct depths – knowing whether to increase or decrease volume from past experience. Ideally the forecasts are independent of past models and cover the entire customer universe. Inaccurate forecasts can make a good model look bad.
Choosing a past marketing event that is representative of future events yields a better model. It also helps for the sample to be less screened by previous models so it can accurately score the full customer universe.
Identifying which unique customer segments need a unique model is a good practice that generates value. Creating unique models based upon purchase recency, purchase frequency, channel preference, or predictor data coverage will often produce big gains over trying to model all customers with one model.
Execute the model correctly. If the model is not scored and customers not selected properly, then the model’s performance results are tainted, regardless of the model’s relevancy. It is often too easy to blame the model when something else went wrong with execution.
While frequently asked, determining which marketing events need a unique model is surprisingly less important than many of the other factors. Events don't typically produce different types of buyers as much as we may think. After you've established a good model, there can be worthwhile opportunities to expand the number of unique models to capture seasonal or product variations.
Although it is tempting to apply a model beyond what it was intended, this can lead to sub-par results. Avoid applying a model to scenarios like; the customer universe to which the model is applied changed or expanded (for example, deeper recency); or the applying the model to a different event (different season, different product mix, etc.).
It is not uncommon that the measurement of the model’s effectiveness is somehow flawed. When comparing models, all other factors must be controlled to ensure that any variance is due solely to the different models.
House models, generally, do not get much value beyond that of historical behavior data. Cooperative data can be useful in certain models. Demographic, geographic, and lifestyle data also can have value for the right model. It is good practice to consider all data sources, but do not expect the non-behavioral ones to provide significant lift.
Surprisingly, the statistical technique used in modeling has little impact. While modelers may find gains with new methods, modeling has generally matured to the point that most modelers utilize similar methods.
A relatively new factor on my list concerns how the customer scores are deployed. Specifically, are the scores collapsed into score segments or left naturally at the customer level? Raw scores preserve the full power of the models; whereas any score aggregation restricts model effectiveness. If segmentation is required, the more segments the better.
While not strictly influencing model effectiveness, model costs are always a consideration. There is abundant evidence that most new models will pay back within a couple usages. Technology advancements of recent years (cloud computing, web-based modeling software, etc.) make development, maintenance, and implementation less costly and affordable to any-sized marketing environment. Most marketers now deploy multiple models which are refreshed regularly as they seek marketing effectiveness and efficiency benefits.
While these won't answer every question as you create your modeling plans, it does remove some of the myths and mystery surrounding prioritization. Also, keep in mind that this isn't an exhaustive list, or one that ranks identically for each situation. Either way, the good news is your next modeling plan will likely be your best one yet.