Can we talk about mathematical analysis of rules?

[Neurosis]

Hi guys.

Often a complaint I see on here about various game systems (Pathfinder, various editions of D&D, etcetera) is that the designers "obviously" failed to use "basic statistical and/or mathematical analysis" which is, we all agree, bad.

As an aspiring game designer (I have designed about a dozen full length RPGs, but considering that only about four of them are really current and finished and none of them are raking in the big bucks, and considering that the internet likes humility, I'll go with 'aspiring') I am really curious about this topic.

What exactly ARE the mathematical and statistical analysis techniques that we're talking about? What is the procedure for going about each of them? How much academic knowledge of statistics is required? What kind of computational and software resources are needed? How much playtest data, and therefore how many playtesters?

If there are multiple techniques with varying levels of complexity, it is the *basic* ones that I want to talk about. The mathematical analysis that is one step above eyeballing. Especially the stuff that can be done to the rules themselves without requiring a lot of playtesting. I'd also like the discussion of this issue to be one that a layman wouldn't have too much trouble understanding.

-Thankee Sai,
-Schwarzkopf

[echoVanguard]

If you've passed an undergraduate-level statistics course, you're qualified. In general, you should understand and be able to apply the following statistical concepts:

1. Mean, median, and mode (and know the differences)
2. Standard Deviations and why they matter
3. Core principles of probability (such as how often p and not p will occur in set X, and how "at least one of" differs from "one of")
4. Probability Density and why anyone should care about it

Generally speaking, however, the problem with mathematical analysis is not whether it is done properly, but how and where it is done. Getting a statistics equation to spit out a correct value is simple, but getting it to spit out a meaningful value is non-trivial - and getting it a meaningful value for a specific question is challenging indeed.

As far as the rest of your question, accuracy approaches 1 as X increases, where X is the total volume of your playtest data. If you know enough math to completely understand the previous sentence, you will have a very clear idea of why people constantly complain about poor analysis in TTRPG products.

echo

[souran]

Honestly,

For the complexity of any game you would actually be able to get humans to play (no mandatory computer assistance), you need little more than a high school level knowledge of probability and some very elementary statistics.

The place where most games fail mathematically is that they don't really consider what happens when you inflate the games inputs beyond early or low level play.

the most obvious example of htis is what happens to the D&D skill system. At low levels the system mostly works, a players attribute modifiers differentiate them from the other players and the npcs but the DCs and total modifiers are not so great as to make a task impossible or trivial for most characters.

However, a character who puts any real effort (or honestly even a half assed effort) into being good at a skill will get so far ahead of the "expected challenge" level of the game that they cannot be challenged. Or if they are challenged then no other character would have even the slightest chance of success.


These are not unsolvable problems. In truth, they can be solved mathematically fairly trivially, implimenting a solution in the game that doesn't leave players board can be a little harder.

However, as long as you don't have a lot of contingent rolls or other obtuse mechanics (keep the game simple) you shouldn't need any more training than simple probability and really basic statistics (mean/median/mode/variance/margin of error)

[fbmf]

Standard Deviations and why they matter

I know what SD is, but I have no idea why it matters.

Game On,
fbmf

[Grek]

For games that use dicepools, you'll also want to throw in a working knowledge of the binomial distribution, so that you can work out the probability of rolling at least X hits with a dicepool of Y and what happens to that probability if you get more dice.

E: Standard deviations only really apply to systems where you roll multiple dice and add them up. Using standard deviations allows you to predict the liklihood of rolling X or better on your unmodified dice roll, which is an important thing to know.

[Josh_Kablack]

Assuming that you are analyzing the combat minigame of an RPG, here are some basic questions you should be able to figure answers for

For a fight with easy/normal/hard difficulty:

• About how many rounds will it take to resolve? (which breaks down further into characters' chances of hitting each round and expected damage through soak and immunities and the amount and speed of healing and recovery available)
• Roughly what are the PC's chances of winning/losing/stalemating?
• Do these chances match up with the intent of the system?

Assuming that you are looking at simple non-combat skill use, here are some basic questions you should be able to figure answers for

• What is the chance of botch/failure/success/critical success for a typical character who is unskilled/dabbles/skilled/highly skilled/the most skilled possible at an easy/moderate/difficult/nigh impossible task?
• What happens to those chances when modifiers increase or decrease the skill roll? What about when modifiers increase or decrease the difficulties? You'd think this would be obvious, but especially in dice-pool systems, you get oddities like 1e Exalted's system where as your dots increase, the chance of a regular success goes down.
• Do those results reflect the intent of the system?
• Are there problems where characters fall off the random number generator on tasks that all PCs should to be able to participate in? This happens more in single-die systems.

[Lago PARANOIA]

The other thing to keep in mind is that even if you know the mathematical processes of the game it's also important for other people to be able to go through your work as well. Players, of course, but also designers.

The d20 is a popular system not only because it's the base of 3E D&D and the SRD is very flexible, but because the outputs for a given input are predictable. If someone with high-school level math cannot give you a rough idea of what a common RNG operand (reroll, modifier, automatic hits, exploding dice, etc.) would do to the probability you need to think really long and hard about how you construct your system.

This also means that you need to get a strong handle on the numbers in your system. Your game should generally or better yet completely avoid the following:

Negative integers that are not subtraction operators. If you have a negative integer output or as a result of multiplication or devision, start from scratch.
Fractions. If one happens as a result of division, you have to immediately round it. I recommend rounding it down always to keep it consistent and to reduce the math.
Triple-Digit Addition or Subtraction
Multiplication or Division of anything other than 2 or 10, unless it's already been worked out ahead of time (like a chart)
Multiplication or Division in the middle of a sequence. Multiplication of something that's done before inputing it into the sequence is fine (like power attack), but not if it's nested in the function (like sneak attack on critical hits).
Rolls that require more than five operators.
Operations that require the specific tracking of a die unless the die is immediately rerolled. Exploding dice are fine, Dragon Age's Dragon Die crap is not. You want to limit this anyway not because of mathematical concerns but because of time concerns.

Well, obviously you should also avoid stuff like exponential functions and trigonometric functions, but that goes without saying.

Secondly, it's neither required nor desired for a game to become infinitely extensible. It makes the math harder for something you don't expect to use. You should have an expected top range (this can be beyond what you expect PCs to ever achieve), an expected bottom range. Meaning that it's vitally important for you to not only playtest low levels but also really high levels. If both of these ends work then it's probable (though not guaranteed, but I've never seen a game that works at both high and low but not the middle) that your middle game would work, too. Testing the low levels and middle levels is pretty much doomed to failure.

[echoVanguard]

Standard Deviations and why they matter

I know what SD is, but I have no idea why it matters.

Game On,
fbmf

It's very useful as a rough estimate of percentile membership for a group of results. For example, given a sample of 100 character attack bonuses, you can compute the mean and standard distribution. This will tell you, at a glance, the ranges of "acceptable" attack bonuses within one standard distribution of the mean - that is to say, those which comprise the majority of the data, and by how much they tend to vary. Exceptionally important when you're trying to determine the difference between "an acceptable challenge" and "a very difficult challenge" with mathematical precision.

You can get a very good overview of its purpose at http://en.wikipedia.org/wiki/Standard_deviation.

echo

[Neurosis]

If you've passed an undergraduate-level statistics course, you're qualified.

Full Disclosure:

I in fact outright failed an Undergraduate Level Statistics class. It was the only class in all of college I actually failed failed. I blame the 8AM time slot and my complete inability to cope with it (never again, I told myself, and I stuck to it) more than the actual subject matter, however. I attended less than 30% of the classes, so failure wasn't exactly a shock.

I guess, let's focus on: procedural discussion of these techniques as they apply to game rules, mechanics, and options specifically. Like, actual examples would be awesome. Especially the stuff that can be done to the game rules themselves without using playtest data.'

For games that use dicepools, you'll also want to throw in a working knowledge of the binomial distribution, so that you can work out the probability of rolling at least X hits with a dicepool of Y and what happens to that probability if you get more dice.

This is relevant to my interests. Any convenient charts or tables you can plug right in to this discussion?

Posted: Tue Jul 26, 2011 3:32 pm Post subject:
Assuming that you are analyzing the combat minigame of an RPG, here are some basic questions you should be able to figure answers for

For a fight with easy/normal/hard difficulty:

About how many rounds will it take to resolve? (which breaks down further into characters' chances of hitting each round and expected damage through soak and immunities and the amount and speed of healing and recovery available)
Roughly what are the PC's chances of winning/losing/stalemating?
Do these chances match up with the intent of the system?

I know about this stuff, but this has always seemed less like math or stat to me and more like a general understanding of balance/eyeballing. Especially with the use of abstract terms like "About", "roughly", and "match up".

Are there specific, named techniques, applied to games, which are industry standard? Anything of that sort. What does the output of such an analysis as this look like?

[DragonChild]

Binomial distribution: http://en.wikipedia.org/wiki/Binomial_distribution

I'm not going to go into detail here, but if you're confused ask.

One thing I didn't see mentioned: Learn to use a spreadsheet, like Excel. Seriously, this is really important because it lets you make visual representations of data.

Excel let me make these graphs relatively easy:

http://s457.photobucket.com/albums/qq29 ... graph1.png
http://s457.photobucket.com/albums/qq29 ... graph2.png
http://s457.photobucket.com/albums/qq29 ... graph3.png
http://s457.photobucket.com/albums/qq29 ... graph4.png

Even if you're not good at stats, it's very easy to understand these. In my opinion, shit like standard deviation, etc, isn't anywhere near as important as just making a good visual representation of the data so you can learn from it. Chances are you don't know what a standard deviation of 1.58 even means, but can understand two colored lines.

[Bobikus]

I need to get more used to working with Excel, since the graphing on it is pretty helpful.

I've been trying to get a better understanding of statistics and analysis on this regard, especially when attempting to make suggestions and discussing mechanics on some homebrew additions to games, and amateur made systems. It's irritating looking at the system and realizing there wasn't any consistent logic behind much of the mechanics. (PTA being an example, although the PTA community seems to reject any mathematics imbalance, ie: having Trainer vs Pokemon combat not wanting any expected/average correlation between Trainer and Pokemon Levels/stats).

[Josh_Kablack]

I know about this stuff, but this has always seemed less like math or stat to me and more like a general understanding of balance/eyeballing. Especially with the use of abstract terms like "About", "roughly", and "match up".

It's a bit of both.

Sometimes you can use logical reasoning, exhaustive listing and/or N-chose-K to compute exact odds of one or more given outcomes occurring.

Other times there will be so many variables involved that the best you can do is write a short computer program to iterate through the probability space for most likely inputs. Sometimes you can't even do that and have to resort to monte carlo methods even within a computer program.

And even when you do get the numbers exactly right, there's still an art to it. There's not a single right answer for "how many rounds should combat take", but there is a right answer for "if PC's hit 50% of the time and it takes 4 hits to KO an enemy, how many rounds will the average combat take?" And there is a right answer for "if my game is going to be run for groups of 4 players in 4-hour blocks at Gencon, and each participant needs 3 minutes per round to handle their character will adventures be able to include more than one combat in such a system?

[Lokathor]

For games that use dicepools, you'll also want to throw in a working knowledge of the binomial distribution, so that you can work out the probability of rolling at least X hits with a dicepool of Y and what happens to that probability if you get more dice.

http://topps.diku.dk/torbenm/troll.msp
this might help, if you read the manual to learn how to turn your dice mechanic into a "troll" expression you can model even the absurdly complex dice mechanics. Then the program can do the probability for you.

If you don't want to read the manual, the website already has examples of how to do stat rolls (d20), dice pools (wod), and some other roll types.

[Draco_Argentum]

Are there specific, named techniques, applied to games, which are industry standard? Anything of that sort. What does the output of such an analysis as this look like?

The industry standard technique is known as SBRM, She'll Be Right Mate. Its an incredibly powerful yet simple to use. The methodology is:

1) Perform a cursory examination of the difficulty of the system.

2) Decide its too hard.

3) Assume that the system produces whatever result you want.

[Neurosis]

Binomial distribution: http://en.wikipedia.org/wiki/Binomial_distribution

I'm not going to go into detail here, but if you're confused ask.

Less confused and more SAN loss inducing brain melt from the density and complexity of mathematical formula vomited all over that page. Representative of all wikipedia math/science pages; totally incomprehensible to anyone not already an expert in the field.

One thing I didn't see mentioned: Learn to use a spreadsheet, like Excel. Seriously, this is really important because it lets you make visual representations of data.

Excel let me make these graphs relatively easy:

http://s457.photobucket.com/albums/qq29 ... graph1.png
http://s457.photobucket.com/albums/qq29 ... graph2.png
http://s457.photobucket.com/albums/qq29 ... graph3.png
http://s457.photobucket.com/albums/qq29 ... graph4.png

These graphs, on the other hand, are awesomely easy to understand and immediately useful. Care to post the excel spreadsheet(s) used to create them, for education purposes?

For games that use dicepools, you'll also want to throw in a working knowledge of the binomial distribution, so that you can work out the probability of rolling at least X hits with a dicepool of Y and what happens to that probability if you get more dice.

http://topps.diku.dk/torbenm/troll.msp
this might help, if you read the manual to learn how to turn your dice mechanic into a "troll" expression you can model even the absurdly complex dice mechanics. Then the program can do the probability for you.

If you don't want to read the manual, the website already has examples of how to do stat rolls (d20), dice pools (wod), and some other roll types.

This seems a touch finnicky and difficult to use, but also like an invaluable tool for any game designer.

Are there specific, named techniques, applied to games, which are industry standard? Anything of that sort. What does the output of such an analysis as this look like?

The industry standard technique is known as SBRM, She'll Be Right Mate. Its an incredibly powerful yet simple to use. The methodology is:

1) Perform a cursory examination of the difficulty of the system.

2) Decide its too hard.

3) Assume that the system produces whatever result you want.

But that would be wrong.

[sabs]

Earthdawn had that problem where there were break points where your chances of succeeding went significantly down, for raising your dicepool by 1.

[DragonChild]

Less confused and more SAN loss inducing brain melt from the density and complexity of mathematical formula vomited all over that page. Representative of all wikipedia math/science pages; totally incomprehensible to anyone not already an expert in the field. Sad

Ok, so, breaking it down as easy as possible. If you're doing a dice pool system, the bionmial distribution is how you figure out the average number of hits and whatnot. The formula is:

f= (n!)(p^k)(1-p)^(n-k) / k!(n-k!)

It's that image block right at the start, it's all you care about.

f is the actual probability of getting the number of hits you want.
n is the number of dice you're rolling.
k is how many "hits" you want.
p is the odds of getting a hit on each die (as in, for After Sundown and the graphs I posted, p=1/3)

These graphs, on the other hand, are awesomely easy to understand and immediately useful. Care to post the excel spreadsheet(s) used to create them, for education purposes?

Unfortunately I lost the files due to copy reformatting, the graphs survived due to being on photobucket.

[vermithrx]

f is the actual probability of getting the number of hits you want.

Precisely the number of hits you want, or at least the number of hits you want?

[DragonChild]

Good catch. Precisely the number of hits you want. Sorry that was unclear. So you'd have to do that out to f=n, and then add up the probabilities.

[Chamomile]

I'm curious as to how adding an extra die to your pool could possibly make you more likely to fail. Although, I guess if it's one of those systems where rolling a one removes successes? Those always seemed like a bad idea on a gut level to me, but I never thought about it enough to realize why. Is there another reason, or have I stumbled across the truth?

[Grek]

That's pretty much it.

If you don't have a calculator on hand to sum up all of the probabilities for you, you can usually approximate it pretty well with a normal distribution with a mean of n/3 and a standard deviation of (2/9n)^0.5

2 standard deviations below the mean = 95% chance of success.
2^0.5 standard deviations below the mean = 75% chance of success.
1 standard deviations below the mean = 66% chance of success.
0 standard deviations from the mean = 50% chance of success.
1 standard deviations above the mean = 33% chance of success.
2^0.5 standard deviations above the mean = 25% chance of success.
2 standard deviations above the mean = 5% chance of success.

[wotmaniac]

I'm curious as to how adding an extra die to your pool could possibly make you more likely to fail. Although, I guess if it's one of those systems where rolling a one removes successes? Those always seemed like a bad idea on a gut level to me, but I never thought about it enough to realize why. Is there another reason, or have I stumbled across the truth?

not sure if this counts, but Scion's system is: 7+=success; 10=2 successes; 1's subtract from successes.
I guess that if 10's didn't count double that extra dice might actually increase your chance of failure (don't know -- haven't done all the math) .... as it is, it's just a wash.

[Neurosis]

I'm curious as to how adding an extra die to your pool could possibly make you more likely to fail. Although, I guess if it's one of those systems where rolling a one removes successes? Those always seemed like a bad idea on a gut level to me, but I never thought about it enough to realize why. Is there another reason, or have I stumbled across the truth?

IIRC it's a dice step system, where for instance, when you increase a dice step, you might go from 3d4 to 2d8 or vice versa or something of the like. (Those aren't meant to be exact numbers.)

I'm curious too.

[wotmaniac]

I'm curious as to how adding an extra die to your pool could possibly make you more likely to fail. Although, I guess if it's one of those systems where rolling a one removes successes? Those always seemed like a bad idea on a gut level to me, but I never thought about it enough to realize why. Is there another reason, or have I stumbled across the truth?

IIRC it's a dice step system, where for instance, when you increase a dice step, you might go from 3d4 to 2d8 or vice versa or something of the like. (Those aren't meant to be exact numbers.)

I'm curious too.

you mean like with Savage Worlds?
yeah -- I just got schooled with that one recently: basically (within that system), if your target # = die size, you're actually better off using the next smaller die (because of exploding dice).
Example:
In Savage Worlds, your skill die is between a d4 and a d12 (depending on how much you invest in the skill), and your base target # is always 4.
well, that's a 25% chance of success on a d4; a 50% chance on a d6; etc. Intuition would tell you that the larger your skill die, the better chance of success you should have.

However, Savage Worlds also encourages you to do all sorts of crazy cinematic shit, which imposes penalties. So let's say that I'm doing something that imposes a -2 penalty -- that means that my target # is now 6. On a d6, that's a 16.67% chance of success (duh); but on a d4, it's a 18.75% chance (25% to hit a 4 -- which explodes -- and a 75% chance to hit the subsequent 2 ...... .25*.75=.1875).
So, a d8 hits a target # of 10 better than a d10; etc., etc.

[Neurosis]

That's interesting, but (to me) the difference between 16.67% and 18.75% is almost negligible. I mean, that's a really, really small difference.