Mini Normal Stats Update

mhsmith0 · Post Post #31 (isolation #0) » Thu Jul 07, 2016 7:32 pm

In post 3, Ether wrote:But with that said, I do agree that mods/reviewers are often too quick to nerf things, and everyone forgets that towns are
really really stupid
.

Yeah that point. Most normal setups I think towns should crush, and I think the Matrix6 newbie setup should be ridiculously town-sided (except for the tracker/BP one). Towns just have an incredible tendency to be dumb and/or lazy.

In post 22, Antihero wrote:
In post 20, callforjudgement wrote:For example, excessive swing is less fun than an imbalance, often: if you make a setup where the scum can be effectively defeated by D2 and it actually happens, the scumteam will be very upset.
let 'em be upset

it's not the game designer's duty to protect scum from getting steamrolled when they're playing badly

Also, basically any game with a vig or SK can result in wolves getting destroyed.

That said, a mod who's designed a game where the wolves can get destroyed on D2 (for instance, cop/doc vs RB) has only himself to blame if the RB sucks and gets lynched D1 (/rants against cops).

PS if 11v2 mountainous is "disastrously scum-sided" then lol towns. 11v2 mountainous should be balanced AT WORST. I'm sorry but it's outright pathetic for towns to get pwned in setups where wolves need FIVE mislynches to just TWO correct lynches to win.

mhsmith0 · Post Post #38 (isolation #1) » Fri Jul 08, 2016 1:13 am

wrt balance and some of these formats being "scum sided" (even though they shouldn't be), one of my favorite quotes (from scum chat in my first game as mafia):

Never worry. The town just hasn't figured out how they are going to lose yet. It always happens.

(town won that one though

)

mhsmith0 · Post Post #59 (isolation #2) » Fri Jul 08, 2016 11:32 pm

What are some examples of town sided normal games that have happened? My gut was that mini normal 1775 was town sided, and I remember complaints on mini normal 1782 (though I disagreed there). Any that stand out from those who know a lot on those things?

mhsmith0 · Post Post #61 (isolation #3) » Sat Jul 09, 2016 3:26 am

In post 53, Antihero wrote:
In post 52, zMuffinMan wrote:dont think i ever bothered calculating past 4 scum in a vanilla because break-even on 4 scum was something like 80+ town?
sounds like a good time

someone run that

/pre-in

mhsmith0 · Post Post #63 (isolation #4) » Sat Jul 09, 2016 4:30 am

Iirc your neighborhoods were no day talk right? That makes it super hard to do anything useful w those IMO.

mhsmith0 · Post Post #67 (isolation #5) » Sat Jul 09, 2016 4:53 am

In post 66, Ircher wrote:Yep, that's exactly what hapened. Part of the problem was scum didn't have too many good fakeclaims when ran up. Swing was a big part of it though.

Luck ran both ways though. Wolves shot prs n1 AND n3 while killing players they wanted dead vs pr hunting; that kind of luck is abnormal IMO. I still get chuckles from the "I bet they just role copped smith" bit after my death. But I think the result spoke to town playing better than wolves, when one mafia death snowballed through the rest of the game.

mhsmith0 · Post Post #80 (isolation #6) » Sun Jul 10, 2016 4:54 am

In post 77, zoraster wrote:neighborhoods aren't power neutral when a neighbor is a power role.

Especially when both are. Otoh no day talk severely nerfs the power of the hood.

mhsmith0 · Post Post #90 (isolation #7) » Mon Jul 11, 2016 2:59 am

I agree with hoopla there, 2 town PR's is super low even if they're strong PR's. Moreover, the mafia tracker could just claim town tracker and give honest results without any real negative consequence (barring a "why are you still alive" bit after a couple days).

mhsmith0 · Post Post #92 (isolation #8) » Mon Jul 11, 2016 3:24 am

10v3 with just 2 cops, no daytalk or other mafia powers is crazy swingy to be sure, and super dependent on the cops not getting NK'd or overlapping peeks or outing each other or the like. But that's a setup where you could potentially see the game basically locked for the town by like D3 through no fault of the wolves other than failing to PR hunt and getting unlucky w night peeks. That kind of potential is why that (might) be town sided, although I think it's really just swingy, with a result that's likely to be an easy win by one side or the other depending mainly on night actions.

That said, I've played 10v3 open setup with one cop with a N0 villa peek (random townie) on MU, and it seems reasonably balanced, but that's mainly because it's an open setup, and the N0 peek structure means every single NK is high info (since the wolves are always hunting the cop). Of course, there's nothing more fun than faking a guilty on someone who turns out to be guilty and then getting NK'd for your trouble

.

But in a closed setup, that wouldn't be anything close to balanced, because fake claims can be made reasonably effectively with little room to prove them false, and the rest of the town won't be engaged in "seer cover" if they aren't aware of the need to do so ahead of time.

As far as JK goes, JK is very OP in small games or in late game. Early game in a 13p, it's much less so. I'd say cop is stronger than JK in a 13p, though JK is a more entertaining and fun role to have in a game.

PS As stated before, towns are dumb and lazy. Like, never underestimate this. In my (admittedly limited) experience, I've seen WAY more mediocre to outright terrible towns than I have legitimately good towns.

mhsmith0 · Post Post #96 (isolation #9) » Mon Jul 11, 2016 8:53 am

In post 95, RadiantCowbells wrote:Like, in an open, town would slaughter that shit.

In an open, I'd agree. The only way it's any kind of game is if the wolves kill a cop N1. Otherwise it'd be completely ridiculous. Every player would be forced to make a "cop peek" every day, and every single NK would be chock full of info. If the wolves kill cops N1 AND N2 (or maybe N1 and N3) they can win, otherwise they get slaughtered.

That said, opens are big edges for towns, especially full opens (as opposed to semi-opens like C++, JK++ etc). And this is a great example of why.

mhsmith0 · Post Post #101 (isolation #10) » Tue Oct 18, 2016 7:40 am

One setup I think was scum sided:
http://forum.mafiascum.net/viewtopic.php?f=2&t=68243

Town:
2-shot vig
Tracker
2-shot bodyguard
3-shot gunsmith
Ascetic

Scum:
Goon
Doctor
Vanilla cop

IMO the scum power structurally acted against every single edge town had, actively nerfing vig AND gunsmith AND tracker (2 potential false innocents), and giving scum ability to narrow down who was or wasn't PR, and giving doc/vanilla cop believable safe claims if needed (since they were true). Only real town edge was no day talk.

mhsmith0 · Post Post #103 (isolation #11) » Tue Oct 18, 2016 11:47 am

Let's say doc was vigged n2 after a 2nd vt lynch, with bugs on vt/doc, and nk on say tracker/bg. Is scum really screwed in that case? Gunsmith has a chance for a guilty, but the rest of town power is basically spent. D1/n1 doc death is brutal, but by n2 I'm thinking it's pretty survivable.

mhsmith0 · Post Post #105 (isolation #12) » Tue Oct 18, 2016 12:45 pm

Well mainly it was as an example of what I felt was a scum sided setup, as that idea came up earlier and iirc no one could give good examples.

mhsmith0 · Post Post #107 (isolation #13) » Tue Oct 18, 2016 1:49 pm

Lolol me

mhsmith0 · Post Post #109 (isolation #14) » Sun Nov 13, 2016 8:36 am

Is there a database of the mini normal setups with roles, results etc, like there is for newbie games? There's something I'm curious about but would rather not do manual work of inputting into excel if avoidable.

mhsmith0 · Post Post #111 (isolation #15) » Sun Nov 13, 2016 11:59 am

Yeah was hoping that parsing had already been done. If not I may do it when I'm in the mood, but figured I'd rather find out if it'd a,ready happened.

mhsmith0 · Post Post #112 (isolation #16) » Sun Nov 13, 2016 2:30 pm

OK I've done a bit of analysis on the mini normal games from 1401 up, ONLY looking at 10v3 games (and excluding a number of ones where the mods didn't bother to update the OP, and didn't bother to post full role sets in the game-ending post or anywhere else that was easy to find).

Overall (now through 1832):
105 games, 36 town wins, 69 scum wins, 34% town win rate (win rates will be town win rates from here on in)

Games with Mafia Traitor
Town is 4-4, 50% win rate

Results by VT count:
3 VT: 0-2
4 VT: 0-3
5 VT: 1-9 10%
6 VT 13-25 34%
7 VT 22-27, 45%
8 VT 0-3

Results by Goon count:
0 goons: 3-4, 43%
1 goon: 14-27, 36%
2 goons: 15-27, 36%
3 goons: 4-11, 29%

The basic conclusion here seems to be that (ignoring odd games with just 1-2 town PRs) there's a pretty consistent over-estimation on value of sheer number of PRs, both for town AND for mafia.

Out of curiosity, I dug into some of the results in a bit more detail:

Spoiler: 5 VT games

Without REALLY digging into the details, one notable tendency here is that there are a bunch of town roles that don't REALLY do much of anything useful (neighbor most notably, also bodyguard), and that having them around doesn't actually seem to help town in any meaningful way.

Spoiler: 0 goon games

Interestingly, in games where ALL mafia have roles of some sort (even if just neighbor or encryptor), town usually seems to have enough actual power to reasonably counter it.

Spoiler: 3 goon games

*1705 is incorrectly marked on http://forum.mafiascum.net/viewtopic.php?f=53&t=29549, it was nea, mason, mason, not nea and all VT's

There's a real variety of power at work here, but in general, even though it seems at first glance like town ought to have reasonably decent power, it's consistently not enough and often not even close to enough.

(counting full roles without modifiers only)
Games with:

Town Cop 9-13, 41%
Town Doctor 10-15, 42%
Town Jailkeeper 7-14, 34%
Town Bodyguard 3-11, 21% (this suggests this role in particular tends to be drastically overvalued)
Town Tracker 7-13, 35%
Town Roleblocker 4-3, 57%
Mafia Roleblocker 7-7, 50% (shockingly, it seems like roleblocker may actually be considered so strong that there's a relative over-compensation here)
2 masons 4-9, 31% (a theoretically strong role that in practice has not been strong)
2 or 3 town neighbors 1-5, 17% (even less useful than bodyguard)

Spoiler: mafia roleblocker games

There's a pretty wide variety of power here, but overall a .500 record suggests that, while some games have probably too much town power (doctor, cop, rolecop, JK vs RB/RC/goon) and others too little (doctor, cop, and miller or 1-shot vig vs RB/goon/goon), in general mafia roleblockers are actually part of setups that are overall pretty balanced.

mhsmith0 · Post Post #113 (isolation #17) » Sun Nov 13, 2016 3:16 pm

Other stats:

Town Odd night whatever: 9-6, 60%
Town Even night whatever: 6-6, 50%
Town Two-shot whatever: 6-12, 33%
Town One-shot whatever: 8-18, 31%
Town Two-shot whatever: 6-12, 33%
Mafia One-shot whatever: 6-6, 50%
Mafia Two-shot whatever: 3-4, 43%

more stuff tbd

mhsmith0 · Post Post #114 (isolation #18) » Sun Nov 13, 2016 3:32 pm

TLDR:

1) Non-vanilla roles seem to be generally over-valued, especially roles that don't actually do much of anything useful (town neighbor, bodyguard, 1-shot whatever).

2) 3 goon setups in particular seem to tend to attract nowhere near enough town power needed to make games tossups. One decent power (tracker, JK, watcher, doctor, vig - obviously some of these are better than others) and two mediocre powers (bodyguard, hider, vengeful, 1-shot anything) is simply not enough. Ditto one good power and just ONE mediocre power (like just a night 1 doc in 1701). Even against an all-goon team, you probably need TWO good powers out of 3-4 overall power roles in order to achieve balance.

3) If the goal is to have a balanced win rate in 10v3 mini normals, then towns need to keep getting boosted even beyond the current norms, especially when it comes to what level of power to apply against all goon teams or how heavily to value five town PR's solely for the sake of HAVING five town PR's.

mhsmith0 · Post Post #116 (isolation #19) » Sun Nov 13, 2016 4:07 pm

I'd definitely agree that the % of vets vs newer players has an impact on mini normal win rates. That said, I think it's interesting to see what the stats are showing in more detail, most notably where there seem to be more consistent imbalances. And it seems like there's a relatively more common imbalance wrt "roles that do something, even if the value of them is very low" being much less valuable than they seem to be given credit for. Whether this is due to the inherent design, or whether those are the kinds of roles that newer players can especially struggle with, I'm less sure.

It's also, FWIW, possible that the randomness inherent in mafia PRs tends to be more pro town in that if a newbie rolls mafia roleblocker, that's much tougher on the scum team than if, say, it's all goons and bussing can be done with much less damage done.

mhsmith0 · Post Post #118 (isolation #20) » Sun Nov 13, 2016 4:36 pm

In post 117, callforjudgement wrote:Bodyguard is a little weaker than 1-shot Cop, and (in most setups) stronger than Doctor. (If it stops a kill, you effectively confirmed yourself via "swapping places with" the best nightkill target, but the nightkill target doesn't become any less confirmed in the process). It's possible that I'm overestimating how good scum are at choosing nightkill targets, though, and thus overestimating Bodyguard as a result.

This is completely incorrect. In zero setups that I have ever seen is bodyguard stronger than doc. Moreover, doc has the potential to outright stop a kill, which can push a game from evens to odds and add a mislynch that town can afford which is a HUGE advantage.

Bodyguard is like the epitome of a trash role that is over-valued in terms of game design, and town's horrific win rate in games with bodyguards largely confirms this. Bodyguard MIGHT be ok in a game with a full cop or a vig or a jailkeeper (really strong town roles that can continue to be useful as the game goes on), but otherwise is very close to worthless. Successfully BG'ing an IC isn't zero value, but is close to it. Successfully Doc'ing an IC is MUCH higher value for obvious reasons (it stops a night kill, someone on town might see you do it, and if it takes it evens to odds, then you just gave town a HUGE advantage).

Town roles with especially horrid win rates include:
bodyguard (3-11, 21%)
IC (2-5, 29%)
1x neighbor (2-4, 33% - neighborizer may be mixed up in this fwiw)
2 or 3x neighbor (1-5, 17%)

and a few more are pushing it:
2x masons (4-9, 31%)
tracker (7-13, 37%)
jailkeeper (7-14, 33%)

mhsmith0 · Post Post #119 (isolation #21) » Sun Nov 13, 2016 4:59 pm

In post 117, callforjudgement wrote:
I suspect your sample size is too small to get really meaningful information about how much roles are being overvalued/undervalued (e.g. 1472 was won by town despite being clearly scumsided; 1590 is a very similar setup that's almost as scumsided (you can do Follow the Tracker in it to get a slight edge) but also was won by town). (There's also the issue of role interactions, e.g. one good reason to put Neighbours in a setup is to make any Rolecops in the setup more powerful by reducing their target pool, as the Neighbours tend to be exposed fairly quickly and a Rolecop rarely has reason to scan them.)

I'd agree that sample size is limited. However, it's enough to draw some useful conclusions. P-value of the 1-goon, and 2-goon sets are each around 5% independently, and 3-goon data is more like 10% (presuming a base null set probability of 50% win rate), which for that sample size is remarkable.

Similarly, the 5 VT and 6 VT data sets are both well under 10% p-value, which is again remarkable. It's pretty clear that, while individual setups have varying success rates, as a whole, high #s of town PR's (the crappy ones anyway) are consistently overvalued if the goal is 50% win rate. That's a pretty meaningful conclusion when it comes to game design/review IMO.

Neighbors in particular seem to be put into games as a "well that's a town PR that adds to the list" sort of thing, even though scum neighbors are clearly a thing, and there's no particular indication that towns are especially good at sorting out people inside a neighborhood based on the data results.

The results for 7 VT games are very interesting, mostly because 7 VT = 3 town power roles is the easiest point on the scale to balance. (It wouldn't surprise me if a major part of the reason more complex setups tend to be scumsided is that the harder a setup is to balance, the lower the bar for allowing it to run seems to be, because eventually nobody can be sure what effect a change would have.)

Yeah I think it's good that more straightforward setups are by and large being reasonably balanced (7 VT vs 3 goons is 4-6, 7 vs 2 is 9-13, 6v3 is 0-2, 7v1 is 8-8, 6v2 is 5-10, 5v3 is 0-0). Probably means that more focus needs to be put into the more complex ones, since that's where the biggest outlier scum win rates seem to be.

Finally, the perception of balance affects balance. If players believe a setup is unbalanced, that makes it more scumsided (because it causes them to draw incorrect assumptions from the lylo massclaim). This could well be why we seem to be stuck in a balance rut at 30-40%; it could be that this effect is forcing 50% games down to 40% no matter what we do. (I note that most new mods in the Mini Normal queue submit obviously scumsided setups and have to be persuaded to make them more townsided; it's likely that the site as a whole has similar opinions on balance to the new moderators.)

Anyway, I have a suggestion: we decide upon an intentionally "townsided" balance standard (along the lines of a 1-shot Cop or actiated Innocent Child more townsided than a typical setup), and allow moderators to opt into the more townsided standard in reviews, then advertise it during signups (the previous standard would also be available). After we've had a few games we can compare to see if it's a) making win rates more balanced, b) more fun. (It seems plausible that the two goals will be in conflict; I can imagine that making games unusually townsided might increase the chance of a town blowout, which tend to be unfun. Or to view things through a simpler lens, games that go through 3:2 tend to be more fun than games that go through 4:1.)

That might make some sense. One thing I'd like to find is a list of setups that are relatively balanced over a long term in mini normals, and then at least have a reasonable baseline to look at other setups off of that.

mhsmith0 · Post Post #121 (isolation #22) » Mon Nov 14, 2016 10:38 am

A few responses:

1) That particular incident is a pretty good indication of what I'm talking about, yes. Consider the counter-factual where I'd actually doc saved that night. Now you have 6v1 (instead of 5v1) and town has bought itself an extra mislynch. I'd have made myself very nearly clear from the save and claim, which means that, despite the suspicion the slot was under, it would have become extremely difficult ot mislynch me, from the night action alone.
EVEN IF I had still been mislynched (which again, is pretty dubious given the fact that scum would virtually never pull a no kill gambit in that situation), town would STILL be better off than it was given my BG use. Why? Because you have another day of voting data that can be used to analyze. Scum-hunting primarily comes from catching people making in-thread manipulations, and mislynching a declared doctor with a near-proven night kill save is the sort of thing that needs to be worked hard on to achieve, which then makes it likelier that someone will through that effort become more obviously town or mafia.

2) Using your example: IC, BG/Doc, VT, Goon (presumably going into night), you are correct in that specific instance that it's basically even value, since with a doc you have a no kill loop that only really resolves itself when the doc gets shot the next night. Your example of a scum who fake claims some other kill stopping role, though, gets in trouble here because now you have essentially cleared the VT claim (obviously not 100% since it's POSSIBLE there are two actual kill-stopping roles), and now it's head to head (doc vs fake claiming goon), with two town clears slightly able to better sort the claims since they can work together to solve the game. So either there's a no lynch loop that eventually ends with the goon shooting the doc (or maybe a draw depending on rule set), or you're able to narrow down the field a bit. Either way it's at least a bit more helpful than just having the BG save the IC at night.

3) Perhaps even more importantly, despite the clarity of the real world instance in which doc was massively and obviously more useful than BG, the stats bear it out. Town bodyguard is just about the single lowest win rate town PR out there in mini normals (2-shot vig is worse, and miller/JOAT are around same level, but that's really it), with a pretty reasonable sample size of 14 10v3 games. Town bodyguard is actively useful in limited examples where you can save a town PR with remaining shots (such as cop tracker or vig), but in general it's barely helpful, and in just about zero cases is it ever MORE helpful than doc.

4) "Someone on town might see you do it" is a POSITIVE, at least before LYLO. Someone sees you visit someone, and there is no kill. If pressed on the matter (which usually is inappropriate, fwiw, as most non-killing visitors are town PRs), you claim doctor, you are not counter-claimed, and you are now confirmed doctor (or maybe people might tinfoil you as some kind of visiting scum PR like role cop, but at least you're confirmed to have not taken a shot).

mhsmith0 · Post Post #122 (isolation #23) » Mon Nov 14, 2016 10:40 am

PS here are some more detailed stats about various town/scum roles that pop up in games, and the resulting town W/L rates (also p-values!). Note that these don't exactly correspond to what I'd shown earlier, since I've made a few spreadsheet corrections (naming conventions haven't always been consistent, so there are manual corrections going on)

Spoiler: town PR win stats