The Infinite Zenith

Where insights on anime, games, academia and life dare to converge

Caveats and shortcomings of the quantitative anime review

Imagine if the Newton was defined to be 7.23 pounds-feet per second square rather than the 1.00 kilogram-meter per second square, or if such some individuals insisted on referring to a kilometre as 4.97 furlongs (roughly 3281 feet, rounded for sake of readability). With such a diverse, convoluted system of measurement, hardly any progress would be made in engineering and research, as the participants would spend a majority of their time converting between units. Fortunately, this is not the case: most professionals rely on SI units, or in the United States and Great Britian, the Imperial units. Converting between the common units between the two systems is reasonably straight forward, and those working entirely in the SI system will find it trivially easy to interconvert (1000 metres to a kilometre is easier than 5,280 feet to a stature mile) between units.

  • For those in a rush, this entire article is summarised as “anime reviews that involve a numerical score generally are inconsistent and do not necessarily reflect on the anime’s merits”.

Contrasting the use of standardised units in sciences and engineering, no such universal agreement exists as to how to rate anime effectively, with reviewers online adopting seemingly arbituary systems for numerically rating shows. This is analogous to party A defining a mile to be 1000 feet, and party B declaring 1000 feet to be a kilometer. Granted, media is to be subjectively rated, and this subjectivity must factor into account the individual’s interests and preferences. This alone cannot justify for all the inconsistencies within the system: in fact, inconsistency appears to affect anime reviews more significantly than reviews on movies and TV Shows. When one considers that reviewers of local media are oftentimes professionals who possess some standards for evaluating works, it is clear that reviews of the aforementioned media should be, and are, very fair and consistent. For instance, in the local papers, movies are given a star rating and the writer of the article justifies what makes the movie worth watching, as well as outlining the weak points. There is no lateral criticism or praise of a work, but simply, an explaination of its content, as well as standing points in said work. This short written section makes clear the explaination behind the star rating, allowing readers to decide for themselves whether they wish to see a particular film. Thus, a five star movie will consistently be worthwhile, while two star movies will be consistently less substantial.

  • The best reviews out there are those written by those with an open mind. Vehement complaints, commonly referred to as ‘rants’, have very little weight and should not be regarded as a valid justification for not watching something.

Whereas professionals have standards, anime reviewers do not; some reviewers use a star system, some use a percentage system and yet others use a letter grade system similar to the University. In spite of inconsistent scoring, it appears that at least one trend has persisted throughout the community: the tendency to break a review down into component parts dealing with characters plot, animation, music and so forth. The only movie I ever review in this manner is the Gundam 00 movie, and the decision to do so was based on several factors, primarily to ensure that the review was clean. Otherwise, I review content in a radically different manner than most reviewers. I enschew the procedural approach because that methodology requires consistency to be effective. For instance, my standards of what forms a good plot and characters must be rigidly defined such that I may provide as fair an evaluation as possible. This approach paradoxically renders the review ineffectual, given that one’s expectations going into a work is usually influenced by the nature of said work itself. I would not walk into the K-On! Movie with the same expectations as I would Five Centimeters per Second: if my standards were “all films must have a coherent plot”, then both movies should be abysmal, although in reality, they are both fine movies. Reviewers tend to be unaware of this and thus, judge harshly that which they cannot follow. This is not dissimilar to complaining about a difficult course on the assumption that the course was going to be straightforward- in both approaches, the individual’s displeasure appears to come from having the incorrect expectations about what was to come.

  • People who watch anime for serious stories are barking up the wrong tree, so to speak. Most anime is light entertainment in the same vein as Futurama. Those who demand life-changing plots would do better to go visit their local library and pick up a copy of Crime and Punishment or Hamlet.

I have encountered several anime review sites that seem to follow no fixed standard for reviewing a work. Anime News Network is an unexpected example: contrary to common belief that they are a superbly reliable site for reviews, the consistency of their grading schema vary greatly. While some reviews demonstrate consistency (for instance, the Five Centimetres per Second film gets an overall score of A-, and all of its aspects are A-), other reviews assign scores in an arbituary fashion. I have supplied a standarised definition of letter grades, and how the movie’s actual scores seem to be misrepresented.

Example of a Standardised Letter Grade System

Letter Grade Value Description
A+ 4.00 Revolutionary
A 4.00 Excellent: superior execution and presentation of material
A- 3.70
B+ 3.30
B 3.00 Good: above average execution and presentation of material
B- 2.70
C+ 2.40
C 2.00 Satisfactory: average execution and presentation of material
C- 1.70 Anime with components below this are not typically reviewed
D+ 1.30
D 1.00 Anime with this in any component are not considered for watching
F 0.00 Fail: unsatisfactory execution and presentation of material

Now that we have a (hastily) designed but standardised system, let’s apply these values to two movies found at Anime News Network. The overall score is objectively calculated by the sum of the values represented by the letter grades over the total number of components, and the story component has been (for the purpose of discussion) given twice the weight of the other components (not that this changes anything: a bit of fundamental math reveals that the final letter grade actually does not change even if the weight of the story is the same as everything else). The K-On! Movie scores higher than it would under a consistent system, and the Gundam 00 Movie scores lower than it would under the same system.

K-On!

Component Letter Grade Value
Story B+ 3.30
Animation A 4.00
Art A- 3.70
Music B+ 3.30
ANN Overall A- 3.70
Actual Overall B+ 3.52

Gundam 00 Movie

Component Letter Grade Value
Story D 1.00
Animation A- 3.70
Art B+ 3.30
Music B 3.00
ANN Overall C 2.00
Actual Overall B- 2.80

At a different site, The Nihon Review, the staff there have a diverse set of reviews and a different standard for reviewing. While at least one of the staff have disagreed with me on semantics, the score they give appears to be inversely proportional to the age of the show and directly proportional to how much plot is present. Thus, older anime generally score well there, alongside with series that were designed to have a strong story from the start, while newer series are shafted by various complaints about how “inappropriate moralizing robbed [the anime] of its excellence” or how a show is “mediocrity at its quintessence”. Moreover, it appears that the numerical values exist solely for the purpose of enhancing the credibility of the review: there is no basis for the scoring system aside from the one I’ve proposed to be their methodology. Earlier, I stated that if I followed a plot-only scoring paradigm, I would have likely rated several good movies poorly, as I would doubtlessly fail to comprehend the purpose of seemingly story-less shows. Thus, if I were to depend on solely sites like Nihon Review, rather than something more reliable (say, my own judgement), I would have passed on several excellent series. Then again, I am unbound by the same constraints as those that enslave those at The Nihon Review in that I freely choose what I wish to watch, whereas their reviews appear honour-bound to watching everything. The time spent on watching shows that one hated would probably be better used for pursuit of goals in the real world.

  • One K-On! Movie rants out there was an incoherent stream about how the K-On! Movie was lacking, how the London Citizens were incorrectly depicted (and how he alone understands the mannerisms and culture of London citizens), as well as how the music sucks, how the story sucks ad infinitum. For purposes of discussion, rants are always lacking in value, as the individual does not think things through before putting pen to paper (well, fingers to keyboard).

At this point, it is prudent to discuss good paradigms for reviewing anime. There is exactly one site I read consistently for following a good paradigm: Random Curiosity is perhaps one of the best anime review sites out there, and has in fact inspired my own methodologies. Their reviews are useful because they inform a viewer about what they can expect from the show, and what audience the show appears to be optimised for. Each individual episode discussion considers highlights and weaker elements, as well as providing insightful comments on various aspects in the aforementioned episode. When a series concludes, the reviewer provides their final impressions, how they felt at the end of the show, what they felt were the best and worst moments, and finally, which audiences they would recommend the show to. No quantitative score is provided because it isn’t necessary: the reviewer either enjoyed the show or didn’t. When a waiter or waitress asks whether or not I’m enjoying something at a restaurant, I respond yes or no: “it’s good, but…” is certainly an uncommon and superfluous answer. A similar case holds in anime: either one enjoyed it or didn’t. I follow this paradigm exclusively in my reviews, choosing to discuss the points on only series that I recommend. If I am forced to provide a numerical score, I base it on the percentile system: a score of seven would imply that this show is better (e.g. more amusing, enjoyable, well-written) than seven of ten series I’ve seen. Such a system is consistent and applicable, free of the constraints that otherwise force reviewers to provide assessments that do not match their numerical ratings.

  • Not all acts of malevolence go unpunished: a quick inspection of the individual’s Twitter reveals that he received an infraction on April 24, 2012 for another rant. According to his Twitter feed, he was forced to watch the movie on a lost wager, and pretty much posted complaints every 15 minutes as he watched said movie. I conclude that he either lacks self-awareness, or his ‘friends’ tied him to a chair and forced him to watch the movie. Judging from what I’ve heard so far, it’s probably the former.

Readers who have reached this point deserve a medal. The take-home message of all this is that one should always regard anime reviews (including mine) as pure opinion. What determines whether or not something is worth watching is not dependent on what some supposedly well-respected reviewer says, but rather, one’s own interests.

4 responses to “Caveats and shortcomings of the quantitative anime review

  1. vimitsu May 31, 2014 at 19:35

    I know you wrote this post a long time ago, but I agree with everything here. My experience seeing the chaotic “reviews” on MyAnimeList led to me to the same conclusion recently.

    Like

    • infinitezenith June 1, 2014 at 17:12

      I found myself in disagreement with some of the negativity surrounding K-On! when I wrote this, especially regarding the poor reviews found at Nihon Review. The reviews written there did not follow any standards and were inconsistent across the board. Usually, the best anime reviews are those that tell the author’s story without any pretense at objectiveness, which a quantitative approach aims to achieve: what did the author find enjoyable or unpleasant? Are there things that make said anime suited/unsuited for certain viewers?

      Like

Please provide feedback!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: