Vote Count Analysis: How Do You Do it?

[joqiza]

Perhaps not quite VCA, but about a year ago I ran an analysis on a particular voting pattern (unvoting without revoting). I didn't end up drawing any definitive conclusions, but I published the dataset and a discussion around it can be found here: https://www.mafiauniverse.com/forums/th ... tistics%29

[joqiza]

Sure, I'd be happy to help advise if you're looking to run any kind of analysis. Honestly, I think the data collection and sanitization is the tougher part. For the analysis I linked, I picked only mountainous games to keep it simple. Once you start introducing different types of mechanical information, I think a lot more thought than I was willing to do needs to go into how you are going to standardize data across different types of games.

My general approach in the analysis I ran was pretty straightforward. For the "sonny tell" (unvoting without revoting >1 hr) I defined the experiment as "exhibits the tell at least once on a given day." So for example in Game 1 on Day 1, 3 town players exhibited the tell of 12 overall, so I record this as 3 town successes and 9 town failures. For the wolves on Game 1 Day 1, 1 wolf player exhibited the tell of 3 overall, so I recorded 1 wolf success and 2 wolf failures. Etc. for all game days for all games in the dataset.

I found that the simple tell without any criteria was not obviously indicative in either way. A player named soah proposed a list of criteria which, when applied, led to greater separation between the estimated wolf and town parameters. At that point we were potentially data snooping, though.

It was an interesting exercise and I'd be happy to talk to anyone more about it if they have data they want to test. That said, I suspect this is all probably more of interest to people who are into math/stats than really useful to anyone seeking some kind of competitive advantage in games.

[joqiza]

My own thoughts have gone down similar avenues in the past and I wish you luck. I'll check out your scraper when I get home. I'd built something to grab MU vote histories in the past but that's more structured I think. An MS dataset would be really interesting.

A word of caution--there's a rule on this site prohibiting players from creating machine learning networks to classify posts. I don't think that's what you're proposing, but the mention of ChatGPT may make some folks a bit nervous.

[joqiza]

Oh, this scraper is pretty cool. Nice work. Do you already have a script which constructs a votecount dataset from the json? I could give a shot at that if you'd like.

I imagine one approach might be to create a dataset of votes with variables for the following:
- Day Phase
- Postnum
- Time
- Voter
- Voter Alignment
- Voted
- Voted Alignment
- Vote Type (vote or unvote)
- Vote Count
- Votes Needed for Elimination
- # Town Alive in Day Phase
- # Wolves Alive in Day Phase
- # Town on Wagon
- # Wolves on Wagon
- Alignment of Prior Vote
- Count of Prior Vote
- Alignment of 2nd Highest Wagon
- Vote Count of 2nd Highest Wagon
- # Town on 2nd Highest Wagon
- # Wolves on 2nd Highest Wagon
... (etc for 3rd, 4th ...)
- # People Not Voting

[joqiza]

If you really want to develop a theory, you'll probably need to find some way to capture things like "oh, this guy just got red-checked so everyone moved on to him." Not sure how to do that in a programmatic way. I just stuck with mountainous games in the past which made things easy but limited my data.

[joqiza]

Don't worry about messy code lol, I'm sure anything I've written is worse. I'm an econ guy not CS, all my programs are an abomination unto god. Anyway I might give a shot tonight or tomorrow at writing something, if I don't get it done by then I might not get it done for awhile tbh but I'll let you know. FWIW I am not aware of any publically available large dataset of votes in mafia games, and I think just creating something like that would be a huge accomplishment.

[joqiza]

Someeeee thinking around analysis plan...

Reading through your analysis plan, I'm not sure I quite understand your approach, but I do like the idea of shuffling alignments around and re-running as a test. Yeah I am not immediately sure what that is telling you though. I would need to think about it.

I suppose if you had found that say, scum were much more likely to land on, idk, position #3 or something, and then you find that it is very unlikely to observe such a result when alignments are assigned at random, maybe that suggests it is a significant result? Hmm. Yeah got to think about it.

My guess is you found everything with scum position on EoD votecounts closer to NAI though. Yeah if that's the case when you start shuffling you'll probably find that it's pretty easy to generate something similarly NAI.

It is worth noting that NAI is itself kind of an interesting result and you probably want a methodology that can distinguish between things that are NAI and things for which the data is insufficient to categorize.

[joqiza]

are you stats savvy? this is maybe the trickiest part of all of this for me. the stats courses ive taken have almost been theoretical to a fault (and also i didnt do well)

relatively ig. not a phd or anything but i studied math/econ in undergrad and used to work in market risk. I can give my thoughts but don't take me as an authority

[joqiza]

It is worth noting that NAI is itself kind of an interesting result and you probably want a methodology that can distinguish between things that are NAI and things for which the data is insufficient to categorize.

So it occurs to me that the way to test this kind of NAI result under your shuffling approach might be to, rather than assigning the alignments completely at random, instead assign the alignments

as if it were

towny or scummy, and then rerun the analysis, and then do that 5000 times or whatever and then see how likely it is to observe the NAI result.

[joqiza]

It is worth noting that NAI is itself kind of an interesting result and you probably want a methodology that can distinguish between things that are NAI and things for which the data is insufficient to categorize.

So it occurs to me that the way to test this kind of NAI result under your shuffling approach might be to, rather than assigning the alignments completely at random, instead assign the alignments

as if it were

towny or scummy, and then rerun the analysis, and then do that 5000 times or whatever and then see how likely it is to observe the NAI result.

I should say that I'm not sure that the shuffling is actually different from a typical approach with a beta distribution, though. It sort of seems like the same idea to me.

[joqiza]

- Correspondingly, it's probably better to frame analyses as examining how players navigate "forks in the road" throughout a Day instead of the Day's final state. We can measure players' propensities to join, leave, or stay on a wagon as a function of its size (or other wagons' size) instead of just looking at their position in a final snapshot. In this way, analyses use more data and provide more direct insight into players' decision-making and Days' overall trajectories.

Thinking bout this... might be worth extracting a "Time Before Phase Deadline" variable if possible.

[joqiza]

It might be true that scum are less likely to bus players they perceive as valuable/ endgame pieces. We might be able to assign a "value" to scum players based on info like: how many votes they received throughout the game, how close they've been to elimination, for how long, etc. Then test whether scum players are less likely to vote in a way that would endanger other high value scum players. Not saying it's necessarily true but it might be testable.

[joqiza]

cool idea!
it looks like the term for this is equivalence testing
maybe there's also a source directly describing how to do this in a nonparametric way like i think im doing to try to detect differences. i should probably already be tracking this stuff somewhere
but my guess is that it'll say something like what you described? -- that i can use the CI from an analysis applied this "equivalence" distribution and interpret complete overlap with the CI from applying the analysis to my actual data as unambiguous evidence that a behavior is NAI

Yeah you should be able to calculate a CI by iterating over the different towny/scummy ratios essentially.

I guess the thing is, when you're shuffling alignments you're still basically doing a binomial experiment. Cuz it's the same rate each time. In the original example you had where you're assigning at random, it's just a 50/50 rate basically.

So I don't really see why the end result would be different from using a beta-binomial to model the uncertainty about the ratios. Except your approach is monte carlo so is going to be super computation heavy. I could totally be thinking about this wrong so definitely still worth doing as verification at the least, but I feel like it's the same approach at the end of the day.

Tbh I'm not sure exactly what nonparametric means in this context. At my old job the term mainly referred to the use of a historical distribution of an asset's log-returns versus a normal distribution in things such as VaR calculations. I'm trying to figure out what the parallel here is but I'm not quite sure.