Is VAR a problem for MLS?

An in-depth analysis of the issues with video review in soccer

By John K. Fuller@JohnKFuller May 25, 2018, 10:05am EDT

Ten months ago I wrote an article explaining how video review would be implemented in MLS. My dek (that’s journalese for subheading) for that article was “It’s going to be better than you think.” Well, we’ve had nearly a full season’s worth of games with live video review (from here on out I will refer to it as VAR, even though that’s not technically correct. After all, everyone, even FIFA itself, is doing so). There have also been a few articles here at Dirty South Soccer both for and against it, most recently Josh Bagriansky’s post this week. It’s about time to review my prediction.

Before I dive into the details, I have to state that I remain overall a proponent of VAR. You need to keep that in mind, because as you read through this screed you will probably conclude the opposite. However, VAR remains in a testing phase and the final version may well be significantly different from what we are seeing now. IFAB approved VAR for permanent use at its Annual General Meeting in March 2018, so it’s here to stay whatever that final form may be.

To date, Atlanta United has been both a beneficiary and a victim of VAR, and that has been amplified over the past two home games. This is not to suggest that Atlanta has had an experience better or worse than any other MLS team. It probably feels that way to Atlanta fans, of course, but every other fan base likely feels the same way. That is, except Portland Timbers fans, who complain about everything anyway, and Orlando City fans as, well, you know. However, because Atlanta fans are obviously more familiar with their own team’s games, these two games give us a very convenient basis for analysis.

I’m going to put 6 plays under the microscope. One from the game against Sporting Kansas City and the rest from the game against the New York Red Bulls. So go grab a beer, this is going to take a while. Maybe a soft toy to throw at your computer screen too, just in case.

Play #1: Josef Martinez’ called-off goal against SKC

I already analyzed this one to death here. The analysis did not address the applicability of VAR to the play though. What I focused on was the intricacies of the offside law itself and how its numerous variables could be interpreted in differing ways as they applied to the play. My conclusion was this was a case of referee’s judgment.

As far as Mark Geiger deciding to take a second look is concerned, I think he was right to do so. There was after all a very clear possibility that Josef was offside on the goal. He needed to make sure the original decision was correct. Obviously, he concluded that it wasn’t.

There are two problems here. First, Benny Feilhaber of Los Angeles FC (and previously of Sporting KC, it is worth noting) pointed out that in pre-season information sessions with MLS players it had been categorically stated that this type of play is not offside. If true (and we will have a similar and more serious issue later), then this is a broader problem about the overall approach to refereeing in MLS. Inconsistencies between instruction and application should not exist. Moreover, inconsistency in general is one of the problem areas VAR is intended to address.

Second, if this is indeed a judgment call, is it in fact reversible by VAR? Here we get into the “clear and obvious error” standard that is supposedly applicable to VAR decision-making. It strikes me that a judgment call is by definition not a clear error (the “obvious” part that everyone is familiar with is not in fact in the official protocols and is pretty much redundant). Most offside calls are pretty simple. This one was not, thanks to the law itself. About the only clear thing on the play was that Jimmy Medranda touched the ball. Was it intentional? Was it a save? Very hard to say. Was it like the example Feilhaber referred to? Who knows. Surely, then, the conclusion should have been that there was insufficient basis to reverse the call on the field. That is, the call was not clearly wrong.

Play #2: Josef Martinez’ called-off goal against NYRB

That Mark Geiger was the center referee for SKC and the VAR for NYRB is, I would say, mere coincidence. Conspiracy theorists abound in sports fandom, of course, and they will undoubtedly be shaking their heads at that statement. But consider this: Geiger only made one of those two decisions against Josef. On this one he only alerted center ref Chris Penso to the need for a review. If Geiger had told Penso there was a clear foul on the play and he had accepted that conclusion on its face (as he has the right to do under VAR rules), then yes, Geiger would have been responsible for both decisions. But Penso decided to review the play himself and make the call. Tata Martino directed his frustration at Geiger, but he was in fact wrong to do so.

There are several issues to be considered in this decision:

Did Josef make contact with Tim Parker?
Was that contact “careless” (or worse), that is, did it rise to the level of a foul?
Was the contact clear and obvious (yes, that again)?

As far as the first question is concerned, the flip side to it is whether Parker made contact with Josef. Let’s review the play at full speed (and that is important from the VAR perspective too, in fact):

Although Parker probably has the better opportunity to play the ball, he is fact the challenger for it, because he is intercepting a pass from Julian Gressel to Josef. Essentially, he cuts Josef off. That’s quite legal, but he comes from Josef’s blind side and leaves him very little time to react to his presence. Parker may not have initiated the contact, but he gave Josef little opportunity to adjust his run to avoid him.

That being said, I think Josef did attempt to avoid contact. As soon as he realizes Parker is cutting in front of him he makes a little jump to clear over his legs. And if he was attempting to avoid contact, that means he was taking care and the final contact was inadvertent. And if he was taking care, then obviously that inadvertent contact was not careless. And therefore not a foul.

Also, the contact was minimal and very difficult to see. Andrew Wiebe and Bobby Warshaw of mlssoccer.com reviewed this play as part of their weekly Instant Replay feature (it’s the first play reviewed). In order to even ascertain that contact was made, they had to enlarge the shot and slow it down, even freeze-framing it at one point. To my mind, the slowed video shows Josef’s attempt to skip over Parker’s legs even more clearly than the full-speed video. However, as far as VAR is concerned, that’s out of bounds. Unless I have misread the protocols, slow-motion video in video review is only permitted for “point of contact” fouls. All other reviews must be performed using full-speed video only. Now this was a point of contact foul, so slo-mo is okay, but freeze frame and blow-ups are not allowed. On this play, that should make it very difficult to see enough to change the decision on the field.

I’m confident that Penso only looked at the permitted video feeds. He only spent about 10 seconds reviewing it, so anything more detailed would be unlikely anyway. Regardless, it is hard to identify what he saw in that brief examination that made it “clear and obvious” that a foul had occurred.

Ultimately, this is another judgment call: was there enough contact and was that contact of a careless enough nature to merit calling a foul? Regardless of your interpretation of Josef’s movements (I know Josh Bagriansky disagrees with me on this). I submit that the answer is no on both counts. Moreover, if this is a judgment call, what makes the original decision any less valid than the final decision?

A brief aside: in Instant Replay Andrew Wiebe mentions that “incidental contact” does not appear anywhere in the Laws of the Game. While this is true, it does not mean that the concept is not applicable in interpreting the laws. Not all contact is a foul. Even when exercising a reasonable degree of care, a player cannot always avoid contact. Zidane headbutting Materazzi is a foul; two players clashing heads when going for a header is not.

The lack of clarity in this decision leads us to the next play…

Play #3: Bradley Wright-Phillips’ first goal

All goals are subject to VAR, but in this case no review was deemed necessary by the officiating crew. But here’s the rub: for a goal, all plays leading up to the goal in the “attacking phase of play” are to be considered. If at any point in that phase the attacking team committed an infraction not called on the field, the goal is called off.

Which brings us to this:

This was how NYRB gained the possession that resulted in the score, and therefore it is part of the reviewable phase. As with the play above, we have minimal contact between an Atlanta attacker (Miguel Almiron) and an NYRB defender (Kemar Lawrence). But is the degree of contact any less severe than the contact on Josef’s goal? Both are ticky-tack, I grant, but I fail to see any meaningful difference of degree. If you are going to call it once, you have to call it every time. It’s called consistency.

Note that Lawrence getting the ball first is irrelevant. It is a common misconception that getting the ball first excuses any contact that would otherwise be a foul. That is not so. The Laws are completely silent on that point. There are limited circumstances in which an attempt to play the ball makes a difference as to whether a foul merits a yellow card or a red, but that’s as far as it goes.

So the question in this case is why did the officials not consider this play worthy of review given what they had done earlier in the game?

Play #4: Jeff Larentowicz’ rescinded red card

Atlanta United have now had two red cards rescinded by VAR this season. Both rescissions were clearly correct. The first one was to Chris McCann in the game against LAFC for a tackle that was definitely hard but probably not quite enough to merit a red. The ejection was reversed and McCann given a yellow. That was a correctly applied use of VAR.

Calling off Jeff’s red card was definitely also correct, and he too ended up with a yellow. But that last is where the problem lies. I don’t think the play merited a card at all, and if anyone was at fault here it was NYRB’s Sean Davis. Tata Martino certainly thought so. So did Taylor Twellman, who in his weekly ESPN+ show MLS Rewind said that Davis went in studs up (that show also includes a lengthy and revealing interview with Howard Webb, the head of VAR operations). That’s not so clear to me and if anything it looks like he may have chop-blocked Jeff, causing him to go flying. Or, looked at yet another way, the two realized they were going to collide and Davis intentionally went low to allow Jeff to go high. However you view the play, there doesn’t seem to be much contact. Also, the video on this one is not very helpful in deciding if there was any contact:

What exactly it was that Penso thought he saw in issuing the red is anyone’s guess. But these things do happen. Human perception is not perfect. But then he is faced with a problem. With the red evidently off the table after review, what does he do? Completely wiping the foul and restarting with a dropped ball would be the correct decision in my opinion. That would certainly have pleased the home crowd. But what of the 22 guys on the field? Would he have lost some credibility and authority with them? Possibly. This seems like a face-saving move to me. And that is another potential problem with VAR. It’s tough to admit to being a complete idiot in front of 40,000+ people and who knows how many more watching on TV. That takes some cojones.

Play #5: Greg Garza’s red card

I’m going out on a limb here. Atlanta has grounds to appeal this red.

Let’s look at the video to see why:

OK, I pulled a switcheroo there. That was indeed Greg, but it wasn’t NYRB’s Sean Davis (who was somehow involved in another red card play. Cue the conspiracy theories…) on the receiving end, it was the Montreal Impact’s Alejandro Silva. Let’s look at the play in this game:

What’s the difference? Greg got a yellow card for the first one, but a red for the second. I defy you to see any material difference between the two incidents. Yes, Greg goes in hard, but the contact with the opposing player in both cases results from his foot rolling over the ball. I can certainly accept an argument that Greg used excessive force and was deserving of a red in both cases.

But that’s the problem. Both cases. Worse, since the first one was judged worthy only of a yellow, every player in the league was given notice that such a challenge will not result in an ejection, not least among whom is the player who committed the original challenge. Moreover, the first incident was either not reviewed by MLS’ Disciplinary Committee or it was reviewed and judged correctly sanctioned, thereby sending the same message to the league.

As far as I am aware, neither incident was subjected to VAR. At one point after issuing the red card, Penso is seen putting his hand to his ear (which is the official signal that a center ref is talking to the VAR, and that has to be the dumbest refereeing signal ever), but it looks to me like he is merely adjusting his headset. However, all potential and actual red cards are subject to review and that means that in the second case at least the VAR looked at (“checked”) it and agreed with the call. But that still doesn’t get the referee crew past the consistency issue.

Play #6: Leandro Gonzalez-Pirez’ yellow card

I honestly have no idea how LGP got away with a yellow on this play. Although I have in the past discussed how complex the denial of a goal-scoring opportunity rule is, this seems like a fairly easy instance of that rule. Florian Valot was clear to goal on the play with only Brad Guzan to beat. Greg Garza was in the general vicinity, but I don’t think he was anywhere near close enough to make a play. In the MLS Rewind interview Howard Webb opined that the refs thought he was close enough, but at professional playing speeds that’s hard to fathom.

Since this was a potential red card, VAR was available to judge the decision. For whatever reason, Penso and Geiger chose not to take that route. It is rather mystifying why they did that, because this strikes me as the precisely the kind of situation in which VAR could be most helpful.

Alright, that was a lot to take in, and I didn’t even address the Miles Robinson foul that gave up the penalty kick (it was very soft, but probably correct). Since you’re probably pretty upset by now (all over again – sorry), go take a few deep breaths, chant a few mantras, scream “I’m mad as hell and I’m not going to take this any more!” out the window, or whatever it is you do to calm down. Then go get another beer and settle in for the second half. We’re nowhere near done yet.

So we’ve reviewed all these plays, but what do they tell us about VAR? Well, a few things, I would say.

Take #1: We need some definition

In short, it is not clear and obvious what clear and obvious means. This came up several times in these plays. Channeling my inner Inigo Montoya, I think it means whatever the center ref and VAR think it means. Here’s what I managed to uncover on the subject searching through all the resources I could muster:

***crickets***

That’s right. Nothing. Nada. Nix. Not through MLS, not through PRO (we’ll get on them in a while), not through FIFA, not through IFAB, not through US Soccer. Or anywhere else that might be expected to know. But this question comes up time and again with respect to VAR. It’s a key point, because we are told repeatedly that the primary objective of VAR is reduce the number of clear errors. Therefore it is incumbent on the implementers of VAR to define for the benefit of the paying customers just exactly what that is supposed to mean.

This is the soccer equivalent of the NFL’s “incontrovertible visual evidence” standard for instant replay. The NFL has had instant replay review in effect one way or another since 1986, so it is much better understood than this new system just on that basis, but most football fans are well aware of what that means. Not so in soccer.

For sure, when VAR is ultimately incorporated into the Laws IFAB will have to develop a standard definition, even if there is some flexibility in it (as there are in so many of the Laws’ definitions. Since updates to the Laws are issued at the end of the European season, that doesn’t seem likely to happen until next year at the earliest. But that doesn’t mean they shouldn’t already be operating under some working definition.

This is an important issue not just in its own right, but also because it contributes to the second issue:

Take #2: There is much inconsistency in the use of VAR

Consistency in the adjudication of any matter is neither foolish nor the hobgoblin of little minds. Indeed, it is a necessity. However, looking at the various plays discussed above (and taking numerous other games into consideration as well), it appears to the outside observer that consistency is lacking in VAR.

It is lacking first in that it is far from evident that it is being properly used in all applicable instances. It didn’t take fans long to start making the TV screen signal when questionable calls were made. Granted, fans don’t always have a strong understanding of the Laws, but there have been more than enough instances where VAR was not invoked when it clearly could have been. Worse, there is little to no communication as to why it was not invoked. In the long term, this is probably not necessary, but since we are still in the experimental phase, I think it behooves the powers-that-be to be far more open about how VAR is being used.

Earlier this year, IFAB held a press briefing on the progress of VAR to date. This relied heavily on the findings of a study performed by the Belgian university KU Leuven. There were two key findings:

The study revealed that…the accuracy of reviewable decisions increased by 5.8 per cent to 98.8 per cent. The average playing time lost due to VAR was only 55 seconds.

This highlights two further weaknesses I see in VAR’s level of consistency. To say the least, those findings seem dubious to me.

A near 6% improvement in the accuracy of calls would be sufficiently material to justify the permanent implementation of VAR. That would be true regardless of the base level of accuracy. Almost 99% final accuracy stretches credulity to the breaking point though, and without knowing the methodology of the study (it is not publicly available) it is impossible to know how that conclusion was reached. Not stated in that press briefing was how many correct calls were incorrectly reversed by VAR, nor how many incorrect calls were upheld by VAR, nor how many were changed but remained wrong after the change. Based on the experience in MLS, I suspect those numbers may be unacceptably high.

I am also skeptical that VAR stoppage time averages less than 55 seconds. That might be true if games where VAR was available but never needed are included in the average, but otherwise that’s low. Apart from anything else, depending on the positioning of the VAR monitor on the field, that could suggest referees are capable of sprint speeds to put Usain Bolt to shame.

On the two called-off goals, Geiger and Penso spent very little time actually reviewing the plays, and also happened to be conveniently close to the monitors, such that very little time was eaten up. Those would support that finding. But that brings up the opposite problem: are referees spending enough time reviewing plays? In both those cases there were intricacies and difficulties in the video feeds that suggest a little more care may have been advisable.

But that’s a problem too. Soccer does not lend itself to extended stoppages. It’s not like the refs can order a 2-minute TV timeout while they review a play. It is also why VAR can only be invoked if there is a normal stoppage in play. But there is a trade-off between keeping the game flowing and getting the call right. That’s is a compromise that referees are going to have to arrive at, and I don’t think they are anywhere near it yet.

Take #3: The Laws of the Game do not lend themselves easily to VAR

This could be the biggest argument against VAR. Let’s consider the NFL again. Here is the list of situations that are eligible for review under the current NFL rules (per Wikipedia):

Scoring plays
Pass complete/incomplete/intercepted
Runner/receiver out of bounds
Recovery of a loose ball in or out of bounds
Touching of a forward pass, either by an ineligible receiver or a defensive player
Quarterback pass or fumble
Illegal forward pass
Forward or backward pass
Runner ruled not down by contact
Forward progress in regard to a first down
Touching of a kick
Other plays involving placement of the football
Whether a legal number of players is on the field at the time of the snap

Every one of these is a matter of concrete fact. It either happened or it didn’t. None of them are personal fouls (such as pass interference, holding, facemasks). Those can be judgment calls. The NFL recognizes that fact and therefore excludes them.

The VAR system in soccer applies to 4 areas: goals, penalty kicks, red cards and mistaken identity. Of those, only mistaken identity is entirely a matter of concrete fact, and is by far the least common occurrence of VAR (I do not know of a single case of that in MLS to date). With regard to goals, the only concrete issue – whether the ball crossed the line – is covered by goal-line technology and is not a matter for VAR.

Everything else is, to varying extents, judgment calls. That is unavoidable though, because it is in the nature of the Laws. By comparison with most other sports, soccer rules are very short and very simple. Even more, VAR only addresses matters in Law 12 (Fouls and Misconduct) and indirectly in Law 11 (Offside). That’s a whopping 12 pages of text.

The Laws are intentionally written to allow referees to exercise their discretion, much like corporate policy manuals that rely heavily on “guidelines” as opposed to actual rules, thereby granting employees latitude to exercise their professional expertise.

This is not just a problem for VAR, it’s refereeing problem in general. There are going to be disagreements, but VAR is only likely to intensify those disagreements. Moreover, fans tend to see plays in relatively black-and-white terms. VAR is just giving them another refereeing matter to get mad about.

It looks like VAR is here to stay, though, so we are going to have to get used to this.

Take #4: VAR is only as good as the referees operating it

VAR is a tool. Well, to be accurate, it’s a person (since the abbreviation stands for Video Assistant Referee) so take that statement as you will.

Tools are entirely dependent on the skill of their operators. A tool with a poor operator will invariably produce a poor result. Thus VAR can only be as good as the referees who use it.

Referees of course are and always have been the targets of fan indignation. This is especially the case in MLS, where referee quality has long been a topic of fierce debate. If the quality of the referee pool in MLS is weak, then the results of VAR as used in MLS are going to be weak.

Without pointing any fingers at individual referees, we have then to evaluate the quality of the overall referee program. Noting first that refereeing North America in general (i.e., CONCACAF) is notoriously poor, do we have the best refereeing available to us in the US and Canada?

I can already hear the screams of “No!” as I write this. And if that is true, then we have to take a close look at the Professional Referees Organization. PRO, in existence since 2012, supplies all referees for MLS, the USL, the NWSL and the US Open Cup. As of right now it has the following pool of referees:

23 referees
47 assistant referees
7 fourth officials
17 video assistant referees

The referees in each pool don’t work solely in that pool; they can be assigned to work in any position down the scale. Hence Mark Geiger, who is listed as a referee, worked as VAR for the NYRB game.

This is the case because even though only MLS uses VARs currently, and there can never be more than 11 MLS games on any given day, the pool of 17 VARs has to be supplemented. Why? Because the vast majority of the referees in the pool are part-time. I have been unable to uncover the precise number of full-time refs, but I believe it to be between 15 and 20. Out of a total pool of 94. Which given the number of games they have to cover over three leagues isn’t much to begin with (they also used to cover the NASL).

Now that may not seem like a lot, and in fact it isn’t. However, consider that until the 2017 season the NFL had precisely zero full-time referees (they now have 24). Full-time referees in a sport that doesn’t have an extended schedule are relatively uncommon in the US. Nevertheless, part-time referees cannot make the same time commitment to officiating that full-time referees can and this will have an adverse effect on the overall quality of refereeing. It can get worse: some of you may recall the incident last year in which referee Ted Unkel’s business address (which happens to be near Orlando) was doxxed by the Orlando SG Ruckus after they were especially aggrieved by his calls in a game.

The pay scale is also a problem. PRO referees are paid on a scale based on position and experience, but the highest-paid referees get just $800 per game. They also receive a salary, maxing out at around $75,000 a year in total. That’s about half what referees in the English Premier League make. Referees in the upcoming World Cup will receive a $70,000 retainer and $3,000 per game. NFL referees average about $200,000. NBA referees (who admittedly work far more games) earn a minimum of $150,000 and as much as $550,000. MLB umpires (also with way more games) start at $120,000 and can earn as much as $350,000.

Well, at least PRO refs don’t have to deal with the infield fly rule. That’s probably worth about $100,00 by itself. But the point is apparent: you can’t attract the talent if you aren’t willing to pay for it. Further, you increase the risk of match-fixing if you pinch the pennies.

Take #5: The implementation of VAR as currently used in MLS may not be optimal

According to the media briefing materials on VAR issued by IFAB after the AGM earlier this year, each VAR is supposed to have two assistants: an AVAR (assistant video assistant referee, and yeah, that’s awkward) and an RO (replay operator). The AVAR must be a qualified match official. As far as I am aware, that has not been the case in MLS. This may be the recommendation for future competitions, though.

Another aspect of the recommended implementation is this:

The VAR (with the AVAR and RO(s)) work in a specially equipped Video Operation Room (VOR) into which footage is transmitted. The VOR may be located in/close to the stadium or in a match centre.

By “match centre” what they mean is a permanent video control studio at some offsite location. This is what will be used at the World Cup in fact, and I think it makes a lot of sense.

In Russia, all VARs will work out of a specially constructed studio in Moscow, obviously communicating remotely with game officials at each stadium. This strikes as a very good idea. First, it means that the exact same technology will be available for every single game. Second, that technology will be much more easily maintained. Third, it means that the VARs will have a greater degree of independence. Fourth, they wouldn’t have to travel. Fifth, they would be directly answerable to management. They mess up, Howard Webb could be in an office right next door and on them in a flash.

VAR implementation at the World Cup is going to be totally over the top, by the way. Each VAR will have 3 AVARs, they’ll have access to a boat-load of broadcast cameras and there will be special offside cameras at each stadium for their exclusive use (one AVAR will be assigned solely to offside calls, and the monitor he will use will employ technology similar to the first down line seen in football broadcasts).

TL;DR: OK. Full time. My final point: VAR is still a work in progress. It’s by no means perfect, nor will it ever be. Doubtless it will change over time. In the meantime, we can rant about it to our hearts’ content. I’m done, so now it’s your turn. But please, if you’re going to throw bottles, throw them at Josh Bagriansky, not me.