Saturday, December 23, 2017

Movie Reception and Follow-Up Success: A Correlational Analysis

There has been a great deal of conversation among the superhero fan community about the reasons behind Justice League's underperformance at the box office. It is currently on its way to $650+ Million worldwide, which will place it as the 12th biggest movie of the year -- nothing to scoff at, but everyone agrees this is a disappointment.

What we don't agree about is the root cause(s) of the disappointment. There are two main contentions that I want to address in this post.

Contention A: Justice League is underperforming because Warner Brothers executives inserted themselves into the creative process, enacting reshoots (and a new score) that changed the tone of the film and altered characters in ways that betrayed the loyal fans of prior DCEU movies. Thus the fanbase is disappointed with the film (i.e., not giving it as much business) and the rushed final production resulted in an inferior film, which explains the relatively low box office totals. People in this camp are quick to point out that prior DCEU movies, even those that were critically maligned, earned more money at the box office than Justice League.

Contention B: Justice League is underperforming because the mixed reception of Man of Steel and the generally negative reaction to Batman v Superman have finally caught up to WB. A substantial portion of the general audience stayed home from Justice League because they either watched BvS once and didn't care for it or they never saw BvS and just accepted the pervasive notion that it was an overblown, poor-quality movie. (If you read this blog or listen to my podcast, then you know I wholeheartedly disagree with this assessment. I think BvS is a brilliant and well-executed film, but I've heard countless times at work, in stores, on TV, and even in political discourse people stating as an accepted fact that BvS is a bad film.) What about the higher performance of past DCEU films? Well, these were benefiting from the good will of prior films, particularly The Dark Knight trilogy (Batman and Joker boosted BvS and Suicide Squad both) and also Man of Steel to an extent (MoS was better received across the board than BvS, although not as universally praised as TDK).

Of course, the above are simplifications and many complex factors are at play, but the descriptions will suffice for the purposes here. What I decided to do for this post was actually test these two contentions statistically. This is just a rough-and-dirty correlational analysis, but I think it is revealing. And one of the contentions certainly does come out ahead.

What I decided to do was collect 22 film pairs (22 originals and 22 follow-ups, though the two sets are not mutually exclusive) in the modern superhero-movie era and investigate which of the following factors was most strongly correlated with the follow-up movie's box office performance:
  • Predictive Variables
    • Original film total box office
    • Original film Cinemascore
    • Original film user rating
    • Follow-up film Cinemascore
    • Follow-up film user rating
  • Outcome Variables
    • Follow-up film opening weekend
    • Follow-up film total box office
For consistent comparisons, all box office numbers are in the U.S. User ratings were taken from Metacritic. I didn't include critic scores because I'm tired of critics and I think they have too much power as it is, so I don't want to give them further attention. Full disclosure: I came into this analysis suspecting that Contention B has more merit than Contention A, primarily because Justice League's opening days were already substantially lower than expected, before any of its perceived flaws or its own word-of-mouth could take off. Moreover, millions of people seeing BvS and a year-and-a-half of nonstop online coverage and BvS-word-of-mouth seems like it would be more potent than a Justice League marketing campaign. But even though Contention B was my hypothesis going in, I chose the list of 22 film pairs a priori and did not add or remove any films to attempt to shift the results. I just ran the correlations and am reporting them now.

Note: I am using BvS, rather than Wonder Woman, as the lead-in for Justice League because the continuity of Superman, Batman, Zack Snyder, and the crew all indicate a close link between those films. The coverage of Wonder Woman over the summer also made it seem like its own entry in the superhero film landscape, and the announcement of a separate Wonder Woman sequel made it clear, and I think it was clear in the mind of the general audience, that fans of Wonder Woman should stay tuned for that direct sequel, while Justice League is a continuation of BvS.

Correlational Results

First of all, a follow-up's total box office is positively correlated with the original's box office. This makes sense, of course, and is why studios love producing sequels to successful films. The correlation coefficient in this case is 0.485, so that gives us somewhat of a benchmark for comparison. (The outlier visible at the top, by the way, is The Dark Knight, which performed much higher than expected given the modest box office of Batman Begins. Though, as a preview of results below, Batman Begins was highly regarded by audiences.)

Correlation: 0.485

Now, if Contention A is true, then a follow-up film's own Cinemascore and user ratings should be highly correlated with its box office performance. That would explain at least a substantial portion of Justice League's underperformance because it has only a B+ Cinemscore and a 6.9 user rating -- not bad but not especially good either.

However, looking across the 22 follow-up films in my sample, a follow-up's Cinemascore is weakly correlated with its final box office (0.263) and almost not related to its opening weekend at all (0.161). The user ratings tell a similar story, though they are slightly more correlated. Follow-up user ratings to final box office have a correlation of 0.415 and user ratings to opening weekend have a correlation of 0.373.

Correlation: 0.161

So overall we can say that a follow-up's opening weekend performance has very little to do with its own audience reception or word-of-mouth. This makes sense because, as I said before, it has just been released and so hasn't had any time to build up its word of mouth. Yet even the final box office total of a follow-up film is only moderately correlated to its Cinemascore and user rating.

On the other hand, if Contention B is true, then we would expect a follow-up's film to be more strongly correlated with the original's Cinemascore and user ratings. In other words, if people were left with a positive feeling about a movie, then they're going to be more likely to head to the theater to see the follow up. And indeed, that's exactly what the data show. For these 22 superhero film follow-ups, the original film's Cinemascore and user rating was more strongly correlated with their success. For original film Cinemascore, the correlation to follow-up opening weekend was 0.531 and to total box office was 0.524. For original film user ratings, the correlation to follow-up opening weekend is 0.528 and to total box office was 0.483. Each of these correlations is higher than its counterpart from the follow-up itself; that is, by all measures, the original is more predictive of follow-up performance than the follow-up is for itself.

Correlation: 0.528


As I wrote earlier, this is a quick-and-dirty analysis so obviously it doesn't prove anything, but the evidence here clearly supports Contention B over Contention A. Justice League seems to be suffering more from BvS's B Cinemascore (tied for the lowest in my entire set) and BvS's 7.0 user rating (bottom third) more than from its own merits or faults. Now, these correlations between the original film and the follow-up performance still leave a lot of variability to be explained by other factors, so this is certainly not the entire story, and one could still argue that if Justice League had been a better film itself, it could've overcome the drag of BvS. But the data indicates that that probably wouldn't have happened opening weekend anyway, because a follow-up film's own merits don't really seem to kick in until the second weekend and beyond.

But when told that BvS dampened Justice League's box office performance, some people might say, "That doesn't make sense, because BvS made quite a bit more money than Justice League is making." To this I would say notice that the correlation between original reception and follow-up success is actually stronger than the correlation between original box office and follow-up success. In other words, even if tons of people saw the original movie, if they weren't that thrilled with it the follow-up can underperform. This happened with TMNT: Out of the Shadows and Transformers: The Last Knight, and it is happening with Justice League too.

To close, I just want to make clear that although I like Justice League quite well, this post is not meant to be taken as an endorsement of WB's interference. I personally would've much preferred Zack Snyder and Chris Terrio's original vision for the film. But I also think -- and the data supports -- that WB was facing an uphill climb at the box office regardless of whether they instituted reshoots or not, because of the divisive/derided films leading into Justice League. Because that challenge was going to be there regardless, WB probably would've been wise to save the money they spent on the reshoots because the changes failed to turn it into a universally acclaimed film. But superhero production budgets are a topic for another post.


  1. Interesting, and the stats would be interesting to see refined.

    Anecdotally, I was surprised how negative word of mouth was. I had many students I encouraged to go when they said generally they go to superhero films but had heard that this wasn't worth it.

    Most negative comments I heard revolved around jumpy story (that's the changes at least in part and cutting the length) and the "horrible CGI." Don't know what stage that is from. (For me, you have Ciaran Hinds, show his face.) Regardless, that word of mouth would seem to stem from part critic, part disappointing experience.

    Of course, not liking the changes doesn't mean they would have preferred what it was before. Instead of jumpy, get criticisms that it drags, etc.

  2. Another good article. So in the end we do agree that the unfair critical reception of BvS is what dented the success of JL.

    However, while I have seen people universally agree that the hate BvS received is what affected the BO, what I think is dividing people the most is whether BvS deserved the hate or not. Some blame the people that worked in BvS for this year's underperformance, while the people on the other side of the spectrum (myself included) blame the close-minded and arrogant critics.

  3. I like the math but I think what you're trying to get at with it is distinguishing between opening reception (as influenced by prior films) versus subsequent reception (as influenced by the film itself, WOM, etc). There is justification in the figures you pick for the model but problems as well as we've discussed elsewhere.

    That said, thinking about it, I feel like there's already a ton of math already addressing what you're trying to get at... and I think it's embodied in tracking. Professional tracking is by no means infallible but it's definitely highly multi-dimensional analysis attempting to predict pre-release box office. I have to believe that any credible attempt at tracking for JL would have factored in the general reception to BvS as part of the analysis. I have to believe that tracking is based on data, polls, etc. and not just blind guessing, otherwise no one would pay them for their analysis.

    So that suggests that tracking firms collected data and still came away with generally optimistic (if soft-ish) views on OW and, therefore, whatever effect BvS may have had on JL (etc). Unless the tracking can be totally discredited and discounted, it seems that the outlook for JL WAS healthy and THEN turned.

    I don't agree with the narrative that the turning is entirely because of what JL was as a product, but I don't think it's completely off the hook just because reception to BvS was mixed either. If it was, the tracking should have been lower throughout.

    I could be putting too much faith in tracking... but then that means dedicated professional maths are failing to capture / predict the audience anyways, so we're not going to solve / absolve / vindicate / prove / exonerate narratives with maths regardless. We need people to be open to alternative explanations that don't agree with their preconceptions just in principle and this is a good attempt at trying to crack some of them.

    1. Good points. I agree that tracking numbers must use prior films in the series (hopefully they use reception and not just box office), and the mixed-to-negative reception of BvS is probably a big reason why JL's opening tracking numbers were much lower than BvS and Suicide Squad. But it's also still possible that the tracking estimates were just wrong and didn't accurately account for BvS carry-forward.

      Of course predictions are always imprecise and hindsight leads to bias, but my frustration, as you acknowledge, was people who prefer BvS to Justice League and who refuse to see that BvS may have harmed Justice League. Instead, they just want to drag JL through the mud. (And personally I love BvS, as you know, I just am trying to get a clearer sense of what has actually occurred with JL's reception and box office performance.)