• Log In | Sign Up

  • News
  • Reviews
  • Games Database
  • Game Discovery
  • Search
  • New Releases
  • Forums

Adventure Gamers - Forums

Welcome to Adventure Gamers. Please Sign In or Join Now to post.

You are here: HomeForum Home → Other → Site & Forum Feedback → Thread


   

AG’s Reviews Rating (DEBATE!)

Avatar

Total Posts: 7620

Joined 2011-10-21

PM

* Pardon the double post, this one is with considering the last five posts… Tongue

kelmer - 09 July 2012 12:35 PM

I still believe it would be healthier if it was the whole team commenting on these mistakes/incoherencies between score and review, but I guess that’s just my personal taste.

You can’t have the entire team do it. These things need to pass through a single Lead Editor, otherwise you’d get too many opinions, leading to too much of a compromise in the texts. Editing also means you have to make sure the reviews aren’t too generic…

Besides, reading how every reviewer describes Jackal’s job, I can only state that it sounds like he’s doing a proper Lead Editor job. It’s what’s expected, and it’s what apparently happens. It’s also why this site rocks… Smile

     

Now playing: Blade Runner (post-CPT) | The Witcher: Enhance Edition (on hold) | Danganronpa: Trigger Happy Havoc (on hold) | Professor Layton and the Azran Legacy (3DS)
Recently finished: Whispers of a Machine (CPT) - 4/5 | Beneath a Steel Sky (CPT) - 3/5 | 3 in Three - 3.5/5 | Puzzle Gallery: At the Carnival - 2.5/5 | The Fool’s Errand (replay) - 3.5/5 | The Dig (replay) - 4.5/5 | Return of the Obra Dinn (CPT) - 4/5 | Beavis and Butt-Head in Virtual Stupidity - 3.5/5 | League of Light: The Game (CCPT) - 3/5 | realMyst: Masterpiece Edition - 2.5/5 | Contradiction - 3/5 | Tex Murphy: Mean Streets - 2/5 | The Last Express - 3.5/5 | South Park: The Fractured But Whole - 4/5 | Indiana Jones and the Fate of Atlantis (replay, CPT) - 5/5

Avatar

Total Posts: 1178

Joined 2012-02-17

PM

We actually tried the group edit idea at one point.  Most people didn’t seem to like having their work (potentially) picked apart by twenty people instead of one.  And many didn’t have time to contribute to the process, so the idea died a quick and quiet death. Which isn’t surprising, really.  It’s hard enough coordinating a group of volunteers without complicating matters further.  And as TimovieMan says, the final authority still has to rest somewhere, no matter how many people are involved.

TimovieMan - 09 July 2012 01:14 PM

Editing also means you have to make sure the reviews aren’t too generic…

This is actually the much bigger challenge for me.  I’d never impose my own opinion over a reviewer’s about a game, but it is tempting to over-edit content at times.  It’s a constant balancing act between saying something the best way possible and preserving each writer’s own “voice”.  The goal is to finish with the best work THEY can do, not the best work I can rewrite.  Tongue

     

Editor-in-Chief, some obscure little site called Adventure Gamers

Avatar

Total Posts: 3933

Joined 2011-03-14

PM

let me first say that i wholeheartedly disagree with the opinion that the ratings doesn’t matter, or that you cant compare two games by their ratings. That is the whole purpose of the ratings, to offer a quick and standardized way to compare games to others.

lets face the facts and not have an utopian view on this, not everybody is going to read the whole review. Especially if a game has gotten a low score, many are just going to conclude that it is a poor game, and search for a game with a higher score, before bothering to read a two page review, and some might never read the full reviews.

Jackal - 08 July 2012 12:27 PM
Iznogood - 08 July 2012 05:32 AM

(I also want it to reflect the objective qualities and flaws, but that should be self evident)

...

There are some reviews that has gotten a much higher rating, than not only the pro/con box can justify, but also compared to how positive/negative the whole review was.
Perhaps some reviewers are simply more generous when it comes to handing out stars?
Regardless of the cause, it is something that needs to be addressed.

You’ve actually answered your own question here.  It’s precisely because of the need to “reflect the objective qualities and flaws” that even a positive review can have many criticisms. Even though flaws may not bother a reviewer personally, or really impact their experience with the game particularly negatively, their mere existence means we need to mention them (they could certainly bother other people more).

Obviously not all “cons” are weighted equally (and this goes to Advie’s point about Yesterday).  Some are deal-breakers, some really hamper one’s enjoyment of a game, and some are just minor annoyances that are easily overlooked by all the good stuff.  (Overlooked while playing, but not while reviewing. Wink) If a game nails its story and puzzles but has lots of little polish issues, that’s why what could have been a classic is marked all the way down to 4 or 3.5.  It’s still a pretty darn good game, but it’s our duty to point out all the warts that prevent it from being so.

I don’t think you are always correct in this.

lets take the Yesterday as an example, the review was very positive and it got a high score of 4 stars, so far so good, but there are also some objective flaws with the game, mainly the interface and the length, and i fell that these objective flaws simply didn’t reflect in the the score at all.
(I am also a bit puzzled by the fact, that the lenght of the game wasn’t mention in the con box at all, but only in the actual review, and yes that might explain something regarding this specific review)

There might be examples where the objective pros/cons have gotten too much influence of the score, but my general expirence is that it has too low influence, and that this is what is causing the inconsistencies in the ratings.

I think that much of this could be solved, simply if you all agreed on a standard, of how much of the score should reflect the reviewers personal like or dislike of the game, and how much it should reflect the objective cons/pros of the game.

And let me just clarify, i’m am only talking about the scores here, not the actual reviews.

Also i still think that there are games, that have gotten a higher rating then the reviews can justify, especially if you look at this over time, and compare some older reviews with never ones, then there is an overall tendency towards some inflation in the rating system.

     

You have to play the game, to find out why you are playing the game! - eXistenZ

Avatar

Total Posts: 7750

Joined 2012-01-02

PM

Woow, i am surpised that this thread was taken seriously by Us(members),the Reviewers and Mr.Jack as well.

i am sitting in an expensive Cafe with WiFi service near the Red Square as i read as fast as i could all the posts i had missed in the last Couple of days here, so much interesting posts that i want to Quote All and comment over, but i will have to hold this/myself until i am back Home as just thinking about the differences of the prices for anything between Moscow and Cairo can simply make me lose it and my wallet as well.Shifty Eyed

@Mr.Jack.
my date of joining AG is almost less than 7 monthes but i had been lurking around as a guest since 2005 (not as much since i became a member but i have been following AG for Quite a while), so… i just Happen to have A Notice and A Question,

The notice: is that always before 2010 i found AG’s Ratings tend to be VERY Careful/Unforgiving/Stingy/Picky ....etc (and all the terms in the English vocabulary that would explain my point).

But that kinda had Changed since then (2010) and i always thought it was/is intended in a way to praise the Genre by praising it’s Games…. which is acceptable , (i.e I can not imagine Mourinho mentioning/talking about Benzema’s Flaws 24/7 as he is an absolute disgrace for the Real’s and for football in General , he eventually must push him and encourage him by praising/enlarges his efforts/pros and forget about his Cons/Mistakes//flaws as least most of the time,..........

SO this introduction is to just ask you a straight Question, which i am sure you will answer it honestly as usual .

Does this has to do anything with promoting AdventureShop Releases and Games?
i surly dont know what is the connection between you both (AG and AS) but i just had to connect and state that with at least a reason or an excuse , not to forget that my Notice (i mentioned above) has been dimishing little by little and the ratings became so generous with AdventureShop and its Games/Releases Appearing and taking a front and a big Advertising(reklama)space at the AdventureGamers.

P.S:Mr.Jack i see this Site is no obscure or little toward the Genre and it’s Fans , it is a leading one, as it does the Players very well and even helps the Developers to maintain/take a strong feedback with their spectators…. you are doing a Great Hell of A job at the biggest/reliable and most important site for all the Adventurers (players) around whole world. Smile

     
Avatar

Total Posts: 1178

Joined 2012-02-17

PM

Iznogood - 10 July 2012 05:10 PM

let me first say that i wholeheartedly disagree with the opinion that the ratings doesn’t matter, or that you cant compare two games by their ratings. That is the whole purpose of the ratings, to offer a quick and standardized way to compare games to others.

It’s a quick and standardized way of telling if a game has succeeded in its own right.  You can certainly compare two similar games by score; doing so with drastically different games really doesn’t tell you anything comparatively.  If you blindly decide to choose between Yesterday and Dark Fall (simply as an example from earlier in this thread) based on nothing but score, you’re hardly getting an accurate read on the “better” game for you personally (let alone the two different reviewers covering them). 

Iznogood - 08 July 2012 05:32 AM

lets take the Yesterday as an example, the review was very positive and it got a high score of 4 stars, so far so good, but there are also some objective flaws with the game, mainly the interface and the length, and i fell that these objective flaws simply didn’t reflect in the the score at all.

At all?  It got 4 stars, not 5.  Wink  Sure there were flaws, which is why they were mentioned, but clearly the reviewer didn’t feel they were very detrimental to the overall experience. Obviously if the story, characters, dialogue, or puzzles were bad, the score would have been marked down more measurably. 

(I am also a bit puzzled by the fact, that the lenght of the game wasn’t mention in the con box at all, but only in the actual review, and yes that might explain something regarding this specific review)

Because she didn’t think it was particularly short, simple as that.  Then again, she clocked her game at 8 hours, which is double what others are claiming for theirs.  8 hours isn’t a long game, but nor is it an obscenely short one. 

There might be examples where the objective pros/cons have gotten too much influence of the score, but my general expirence is that it has too low influence, and that this is what is causing the inconsistencies in the ratings.

Put the word “objective” completely out of your mind.  When it comes to reviews, it is an absolute myth.  Almost everything is subjective.  Even play time, as above.  Even technical stability, which is often dependent on a player’s own hardware.  Sure, the odd blatant dead end or design glitch might be universal, but everything besides that is subject to our personal slant.  Everything. 

I think that much of this could be solved, simply if you all agreed on a standard, of how much of the score should reflect the reviewers personal like or dislike of the game, and how much it should reflect the objective cons/pros of the game.

The problem with that idea is that there’s simply no way to quantify personal slant.  If you tried, that would be the most subjective number of all.  You want to turn scoring into a science, and it’s not, it’s an art (so to speak). 

Also i still think that there are games, that have gotten a higher rating then the reviews can justify, especially if you look at this over time, and compare some older reviews with never ones, then there is an overall tendency towards some inflation in the rating system.

If you were comparing old and new reviews of the same game, that might mean something.  Otherwise, this is a non-starter.  Maybe there are more good games now than there have been in years past.  Maybe you’re just ignoring all the games that got low scores because it doesn’t fit the theory.  Sweeping generalizations like this are rarely based on fact.  (Where’s Lucien to crunch numbers when you need him? Tongue)  I’m not accusing you of deliberately skewing the truth, just saying that perception is sometimes quite different than reality.

Advie - 10 July 2012 06:15 PM

Does this has to do anything with promoting AdventureShop Releases and Games?
i surly dont know what is the connection between you both (AG and AS) but i just had to connect and state that with at least a reason or an excuse , not to forget that my Notice (i mentioned above) has been dimishing little by little and became so generous since (then) AdventureShop and its Games/Releases Appearing and taking a front and a big Advertising(reklama)space at AdventureGamers.

No, not one iota. 

Haven’t you read all the people saying we just underrated The Dark Eye and Deponia?  Wink

Affiliate links (and we use them for Big Fish, GOG, Amazon, and App Store as well) all work the same way.  For every sale made through AG links, we get a tiny fraction back as commission.  A finder’s fee, of sorts. 

We advertise Adventure Shop prominently because we were its original media partner, and we were pleased to support a fledgling venture that was ahead of the curve in a genre that usually has to be dragged kicking and screaming into any new change.  And now, a mere couple years later, it’s hard to remember when downloads weren’t a standard option.  We like to think we were an important cog in that transition.

We knew that the affiliation risked some accusations of corruption, as if the pittance earned per game would be enough to motivate us to artificially inflate scores.  But really there would be no WORSE idea than directly linking to games we’d deliberately overrated.  That would be the quickest and easiest way of advertising our own untrustworthiness, short of painting a big “we’re lying!” sign on our backs.  It would be a horribly short-sighted strategy that would result in long-term disaster.

Here’s our site policies regarding this matter.  We’ve written longer articles over the years, though I’m not sure they all made the transition to the new site.  The point is, we’ve been open and up front about these things all along.

     

Editor-in-Chief, some obscure little site called Adventure Gamers

Total Posts: 31

Joined 2005-07-14

PM

I’m finding this discussion fascinating. I have found that working with Jack as he edits my reviews has been one of the most rewarding writing experiences I’ve had. My review and the score that goes along with it absolutely reflect only my review and my experience of the game. Trying to encapsulate your 8-hour experience playing a game into one score is no easy task. I could write pages and pages about my thoughts on the game’s interface, puzzle frequency, puzzle difficulty, writing, story, music, animations, art direction, voice acting, etc. To boil all that down to a succinct, cohesive review that covers all of the bases without putting myself, let alone someone else reading the review, to sleep is difficult. Boiling those remaining thoughts down to a single score? That’s the hardest part of the review. In the process of distilling those thoughts, Jack’s edits let me know if I’ve left out portions of the review that will make the final score I’ve arrived at seem inconsistent. And, if I’m struggling to articulate a thought, Jack always manages to zero in on what I was trying to say, which is what a skilled editor does.

The other part of this discussion I find interesting is the thought of relying on a score to make a decision about a game. I love reading book and movie reviews and would never rely on a score alone to sway my purchase decision. While I do tend to gravitate toward reviewers that I know have tastes similar to mine, I also read the ones that are fantastic writers and manage to reveal truths about the work that I wouldn’t get anywhere else, making me seek the reviews out even after I’ve read the book or seen the movie.

     
Avatar

Total Posts: 7750

Joined 2012-01-02

PM

Jackal - 10 July 2012 09:01 PM

Haven’t you read all the people saying we just underrated The Dark Eye and Deponia?

Of course i read! ,and i am one of those people if not the 1st to say it through My (user’s) review and rating to the Dark Eye.

i just was speaking generally of my feeling toward an observation i made/gotten through the years of following the site,that concerns the main Issue of debating over the Review’s Rating here not trying to go out of the subject. and claiming such stupid accusation .

Advie - 10 July 2012 09:01 PM

  i just had to connect and state that with at least a reason or an excuse

that might prove what i am saying now.

i mean was trying to figure out a reason or an excuse for the ratings which became more generous lately or just more than before,

but i ought to be sorry for my last post Pan  , if it was understood as an accusation more than just a Debate by any chance.

 

and i gotta say ,i was just thinking of the rating of Myst at the time !......and what/how would it have rated if it was released these days (literary!) , lets say/just imagine after Resonance , would it still carry 4/5 stars , after resonance’s 4.5/5 ! .

P.S: i know that Time changes and StarWars (i.e) the old Trilogy is not the same as it seemed to the spectators like more than 30 years ago Grin

 

     
Avatar

Total Posts: 4011

Joined 2011-04-01

PM

Advie - 11 July 2012 12:11 AM

and gotta say ,i was thinking of the rating of Myst at the time !......and what/how would it have taken/rated if it was released these days (literary!) , lets say/just imagine after Resonance , would it still carry 4/5 stars , after resonance’s 4.5/5 ! .

I think Myst would (or should) score worse today. The innovation factor in Myst should be worth something, which means it would get lower than a 4/5 if released today. So worse than JULIA, A New Beginning and So Blonde. That can’t be right - it’s a better game than that even without creating a whole genre singlehandedly. But I probably value creativity more than most people, which is why I’d disagree with, for example, Portal 2 scoring above Portal.

But while creativity is good, I don’t think games should be punished for being made in an old style either. I don’t think (or hope) AGers has such a bias. If you look at a site like Gamespot you find a game like Anacapri getting a ‘terrible’ score for being a ‘relic from the 90s’ while retro-style Gemini Rue scores ‘great’. AG’s review had no mention of either game being outdated, which is good because they’re not.

     
Avatar

Total Posts: 1310

Joined 2005-12-06

PM

I remember choking a bit realising that Art of Murder: FBI Confidential and Gray Matter got the same score, 3 and a half stars. For AoM that is extremely generous and while I’m not demanding 5 stars to Gray Matter, 3 and a half seems rather low - especially when compared to AoM. Though the whole AoM is anomaly, players mostly agree that the series keeps getting better while the AdventureGamers score keeps getting worse.

So in a few cases I raise my brows but mostly I don’t really find anything to complain about. When I disagree I can leave my own review or a comment.

     

Currently playing: Stasis
Recently finished: Detective Grimoire, Portal, Tormentum: Dark Sorrow, Emerald City Confidential (CPT)
My game reviews and other stuff: Lux Atarnia

Avatar

Total Posts: 5866

Joined 2007-07-22

PM

Also, AG utilizes a five-star system, which often leads to somewhat questionable or even paradoxical situations if we’re to compare games by scores. There’re no many options for a score - 1, 1½, 4½ or 5 stars are rare, which leave us with majority games getting something between 2 and 4 stars.

But I love scores, and I hate when I read a game review without it. And I’m not saying scores from 0-100 are significantly better - although, let’s say if one specific reviewer is followed by readers, 0-100 system does bring more information looking at the score - maybe the second episode or a sequel scored 85% compared to 81% of the first episode, which clearly says the reviewer thinks the second episode is slightly better. In 5-star system, stretch from 3½ to 4 stars is bigger.

And funny how Mobygames “translates” AG scores into percents - 3 stars is 60%, which are not the same IMO.

     

Recently finished: Four Last Things 4/5, Edna & Harvey: The Breakout 5/5, Chains of Satinav 3,95/5, A Vampyre Story 88, Sam Peters 3/5, Broken Sword 1 4,5/5, Broken Sword 2 4,3/5, Broken Sword 3 85, Broken Sword 5 81, Gray Matter 4/5\nCurrently playing: Broken Sword 4, Keepsake (Let\‘s Play), Callahan\‘s Crosstime Saloon (post-Community Playthrough)\nLooking forward to: A Playwright’s Tale

Avatar

Total Posts: 1310

Joined 2005-12-06

PM

I think I would prefer 10 star system or 5 stars with quarters but this is decent enough.

diego - 11 July 2012 07:47 AM

And funny how Mobygames “translates” AG scores into percents - 3 stars is 60%, which are not the same IMO.

One would think that half a star = 10% but I think in reality no game gets for example 11% score even though we have some one star or half a star games. So yeah, they are not directly comparable, percentages seem to usually be something in between 1-10 and 4-10 scoring ranges.

     

Currently playing: Stasis
Recently finished: Detective Grimoire, Portal, Tormentum: Dark Sorrow, Emerald City Confidential (CPT)
My game reviews and other stuff: Lux Atarnia

Avatar

Total Posts: 40

Joined 2009-04-25

PM

Iznogood - 10 July 2012 05:10 PM

I think that much of this could be solved, simply if you all agreed on a standard, of how much of the score should reflect the reviewers personal like or dislike of the game, and how much it should reflect the objective cons/pros of the game.

I think this would be almost impossible.

You couldn’t just count up how many things are in each pro/con column, because not all pros and cons carry equal weight. It would be absurd for “horrible, game-breaking bugs made it impossible to finish” and “some of the voice acting was bad” to factor the same way into the numerical score, for example.

Translating words and opinions into numbers is never going to be an exact science. When we recognize (as we must) that different aspects of the game have different importance, exactly how important each thing is will obviously vary from reviewer to reviewer.

I think you just have to take any review with a grain of salt. A review is one opinion, period. I try to read more than one review of any game I’m considering buying, to try to get a more balanced picture.

     
Avatar

Total Posts: 1178

Joined 2012-02-17

PM

I used to be a staunch supporter of percentage scores, which made for quite an adjustment when I first joined AG.  But I came to appreciate the freedom from not being so specific.  I mean, really, how does one decide whether a game gets 77 or 78%?  It’s folly.

What people often forget is that star scores represent a range.  Just because two games get the exact same score does not mean they’re both absolutely equal.  Of course there are better 3.5 games and worse ones.  But they fall in the same general quality range, so they get the same score.  I admit (and I know many AG writers have felt the same at one point or another) that it would sometimes be nice to have “quarter” stars.  But if we did that, we’d probably end up wanting eighths after that.  Grin You need to draw the line somewhere. 

There isn’t a direct math correlation between our star scores and percentages.  Our scoring system means what it means.  That said, I don’t think a straight numerical translation is that far off.  The bigger problem with doing that is that percentages don’t mean the same thing to different people.  To some sites, 70% is a very solid, respectable game.  (That would be true here of 3.5 games.)  To others 70% is mediocre at best.  Just having a number doesn’t make it any more self-explanatory.  You’d still need to gauge it against each site’s own standards. 

I think scores are useful as a quick snapshot of a game’s quality.  But even that isn’t science, since every site’s numbers mean something different.  And when people start imposing their own interpretations, that’s where understanding really goes off the rails.

     

Editor-in-Chief, some obscure little site called Adventure Gamers

Avatar

Total Posts: 105

Joined 2003-11-15

PM

Jackal - 11 July 2012 11:56 AM

The bigger problem with doing that is that percentages don’t mean the same thing to different people.

That’s one of the big problems with scores. The other is that, as people have witnessed about in this thread, some people are mad enough to only look at the score at make an opinion from that. Unless it’s an extreme score that either screams “avoid!” or “must play!”, it’s IMHO a strange attitude to take.

No matter how careful AG are about the scores, my recommendation remains to mostly ignore the numbers, and read the text.

     

You can play my game Frasse and the Peas of Kejick for free! (AG review here.)

Avatar

Total Posts: 2573

Joined 2005-08-12

PM

I like the star ratings a lot more than percentages, but people really need to stop thinking of them as scores that can be used to rank games on some sort of universal scale that cannot exist. I think the only way to think about them is as recommendations:

* 2 star and below means the game should be avoided.
* 2.5 and 3 stars means the reviewer thinks the game does not really succeed at doing what it’s trying to do, but if you’re really passionate about this style of games and you lower your expectations, you might enjoy it.
* 3.5 stars means the game tries to do something and acquits itself competently; if it’s your style of games and you have the time and money, you should check it out.
* 4 stars means the game is really, really good at what it does; if it’s your style of games, you really don’t want to miss it.
* 4.5 and 5 stars means the game is exceptional; even if you’d think it’s not your style of games, you should give it a chance anyway, because it’s Just That Good.

And I think that’s enough. Sure, sometimes you might need to make up your mind between two 3.5-star games, but I don’t think knowing one is 68% and the other 71% would help much. At that point, it boils down to personal preference and deciding for yourself which game appeals to you more based on style, gameplay, etc. (i.e. RTFR!)

     

Currently playing:
Recently finished: Sam & Max: The Devil’s Playhouse (R) (4/5) | Detective Di (3.5/5) | Ace Attorney: Trials and Tribulations (R) (4.5/5) | Heaven’s Vault (3/5) | Ace Attorney: Justice For All (R) (3.5/5) | Phoenix Wright: Ace Attorney (R) (4.5/5) | Sam & Max Beyond Time and Space (R) (4.5/5) | The Infectious Madness of Doctor Dekker (abandoned) (2/5) | Assassin’s Creed Odyssey (3.5/5) | The Sexy Brutale (R) (4/5)

You are here: HomeForum Home → Other → Site & Forum Feedback → Thread

Welcome to the Adventure Gamers forums!