Angband.oook.cz
Angband.oook.cz
AboutVariantsLadderForumCompetitionComicScreenshotsFunniesLinks

Go Back   Angband Forums > Angband > Vanilla

Reply
 
Thread Tools Display Modes
Old October 28, 2018, 22:15   #41
fph
Knight
 
Join Date: Apr 2009
Location: Berlin / Italy
Posts: 631
fph is on a distinguished road
Quote:
Originally Posted by Nick View Post
This is a common complaint, usually refuted by an appeal to advanced mathematical statistics
As an expert in advanced mathematics , the binomial test for 26 failures out of 335 at 4% failure rate gives a one-sided p-value of 0.00114. This means that you can expect to observe 26 or more failures about 0.11% of the times.

Usually, in a single, non-repeated experiment, a result is considered statistically significant if the p-value is 0.05 or below. So yes, that seems a statistically unusually high failure rate, and worth investigating. There is usually a reporting bias, in that only 'unusual' results get reported to the devs, so the best thing would be collecting completely new data to test it.
__________________
Dive fast, die young, leave a high-CHA corpse.
--
You read a scroll labeled 'lol gtfo' of Teleport Level.

Last edited by fph; October 28, 2018 at 22:21.
fph is offline   Reply With Quote
Old October 28, 2018, 22:16   #42
Pete Mack
Prophet
 
Join Date: Apr 2007
Location: Seattle, WA
Posts: 4,971
Donated: $40
Pete Mack is on a distinguished road
BTW: for the rest one turn keymap, I suggest 's', and make it rest 5 or 10 turns (or until disturbed.) I used to use the old search command for this all the time: wait until a nearby monster moves, or appears around a corner .
Pete Mack is offline   Reply With Quote
Old October 28, 2018, 22:36   #43
fph
Knight
 
Join Date: Apr 2009
Location: Berlin / Italy
Posts: 631
fph is on a distinguished road
Quote:
Originally Posted by fph View Post
There is usually a reporting bias, in that only 'unusual' results get reported to the devs, so the best thing would be collecting completely new data to test it.
Ran some tests: created a human mage, advanced him with debug commands to lvl21 and enough int to get 4% failure rate, and cast detect monsters 335 times. Got 14 failures. p-value of 0.47. Verdict: the RNG seems to work OK. Complaint "refuted by an appeal to advanced mathematical statistics".
__________________
Dive fast, die young, leave a high-CHA corpse.
--
You read a scroll labeled 'lol gtfo' of Teleport Level.
fph is offline   Reply With Quote
Old October 28, 2018, 22:51   #44
Pete Mack
Prophet
 
Join Date: Apr 2007
Location: Seattle, WA
Posts: 4,971
Donated: $40
Pete Mack is on a distinguished road
For low probability events, a poisson distribution is generally appropriate. It always has a fat tail (because you can't have fewer than zero events.) Yes, it will eventually converge to normal (as any distribution will.) But even at expectation value E=14, while the curve looks close to normal, the tail is significantly fatter at 1.5*E. The estimate of 0.11% is too low by a significant amount.
Pete Mack is offline   Reply With Quote
Old October 28, 2018, 23:57   #45
fph
Knight
 
Join Date: Apr 2009
Location: Berlin / Italy
Posts: 631
fph is on a distinguished road
Quote:
Originally Posted by Pete Mack View Post
For low probability events, a poisson distribution is generally appropriate. It always has a fat tail (because you can't have fewer than zero events.) Yes, it will eventually converge to normal (as any distribution will.) But even at expectation value E=14, while the curve looks close to normal, the tail is significantly fatter at 1.5*E. The estimate of 0.11% is too low by a significant amount.
I don't follow you. These are independent trials, so their exact distribution is binomial/Bernoulli. I'm not approximating anything with a normal. 0.11% is what is returned by
Code:
scipy.stats.binom_test(26,335,0.04, alternative='greater')
__________________
Dive fast, die young, leave a high-CHA corpse.
--
You read a scroll labeled 'lol gtfo' of Teleport Level.
fph is offline   Reply With Quote
Old October 29, 2018, 03:58   #46
Pete Mack
Prophet
 
Join Date: Apr 2007
Location: Seattle, WA
Posts: 4,971
Donated: $40
Pete Mack is on a distinguished road
Right. My bad.
Pete Mack is offline   Reply With Quote
Old October 29, 2018, 09:17   #47
PowerWyrm
Veteran
 
PowerWyrm's Avatar
 
Join Date: Apr 2008
Posts: 2,406
PowerWyrm is on a distinguished road
Note: I've already failed a spell with 1% failure rate five times in a row
__________________
PWMAngband variant maintainer - check http://powerwyrm.monsite-orange.fr (or http://www.mangband.org/forum/viewforum.php?f=9) to learn more about this new variant!
PowerWyrm is offline   Reply With Quote
Old October 29, 2018, 12:37   #48
Sky
Veteran
 
Join Date: Oct 2016
Location: somewhere on the south coast
Age: 45
Posts: 1,159
Sky is on a distinguished road
Quote:
Originally Posted by fph View Post
Ran some tests: created a human mage, advanced him with debug commands to lvl21 and enough int to get 4% failure rate, and cast detect monsters 335 times. Got 14 failures. p-value of 0.47. Verdict: the RNG seems to work OK. Complaint "refuted by an appeal to advanced mathematical statistics".
All you've proven is that under controlled test circumstances the rng doesnt fail.
I hope it's evident why that isnt enough.

Everyone has reported unusual rng results. At some point you gotta accept that it's not an epidemic of bias and something in the code is doing something it shouldnt. And god knows what it is. What is the rng based on anyway, system clock? That has to go through the OS which could unwittingly manipulate the results.

Empyrical observation does have its merits, y'know.

Oh btw congrats PowerWyrm on a 1:10,000,000,000 chance.
__________________
"i can take this dracolich"
Sky is offline   Reply With Quote
Old October 29, 2018, 19:52   #49
fph
Knight
 
Join Date: Apr 2009
Location: Berlin / Italy
Posts: 631
fph is on a distinguished road
Quote:
Originally Posted by Sky View Post
All you've proven is that under controlled test circumstances the rng doesnt fail.
I hope it's evident why that isnt enough.

Everyone has reported unusual rng results. At some point you gotta accept that it's not an epidemic of bias and something in the code is doing something it shouldnt. And god knows what it is. What is the rng based on anyway, system clock? That has to go through the OS which could unwittingly manipulate the results.

Empyrical observation does have its merits, y'know.

Oh btw congrats PowerWyrm on a 1:10,000,000,000 chance.
As far as I understand (I didn't write the code) the RNG does one call to time() in Rand_init() when the game is started, then it's all state-based. If I read the branches correctly it uses this RNG.
__________________
Dive fast, die young, leave a high-CHA corpse.
--
You read a scroll labeled 'lol gtfo' of Teleport Level.
fph is offline   Reply With Quote
Old October 29, 2018, 20:42   #50
Nick
Vanilla maintainer
 
Nick's Avatar
 
Join Date: Apr 2007
Location: Canberra, Australia
Age: 53
Posts: 7,188
Donated: $60
Nick is on a distinguished road
Quote:
Originally Posted by Sky View Post
Everyone has reported unusual rng results. At some point you gotta accept that it's not an epidemic of bias and something in the code is doing something it shouldnt.
Actually, no. Humans are built to recognise patterns; if there is no actual pattern, we will still find one. The correct thing to do is to have a mathematically sound source of randomness, check the implementation from time to time, and otherwise accept that people will continue to tell you that the RNG is broken
__________________
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
Nick is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Variant list caruso Oook! 13 July 5, 2013 19:14
list experience value in Monster List view fbas Vanilla 1 November 24, 2010 20:51
object list fyonn Vanilla 17 June 23, 2010 17:04
Hellband 0.8.8 Wish list konijn_ Variants 33 February 10, 2010 17:22
Variant list pav Oook! 19 September 19, 2008 00:07


All times are GMT +1. The time now is 05:27.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.