Angband Forums list of bugs and wishes
 Register FAQ Members List Calendar Search Today's Posts Mark Forums Read

October 28, 2018, 21:15   #41
fph
Knight

Join Date: Apr 2009
Location: Berlin / Italy
Posts: 741
Quote:
 Originally Posted by Nick This is a common complaint, usually refuted by an appeal to advanced mathematical statistics
As an expert in advanced mathematics , the binomial test for 26 failures out of 335 at 4% failure rate gives a one-sided p-value of 0.00114. This means that you can expect to observe 26 or more failures about 0.11% of the times.

Usually, in a single, non-repeated experiment, a result is considered statistically significant if the p-value is 0.05 or below. So yes, that seems a statistically unusually high failure rate, and worth investigating. There is usually a reporting bias, in that only 'unusual' results get reported to the devs, so the best thing would be collecting completely new data to test it.
__________________
Dive fast, die young, leave a high-CHA corpse.
--
You read a scroll labeled 'lol gtfo' of Teleport Level.

Last edited by fph; October 28, 2018 at 21:21.

 October 28, 2018, 21:16 #42 Pete Mack Prophet   Join Date: Apr 2007 Location: Seattle, WA Posts: 5,217 Donated: \$40 BTW: for the rest one turn keymap, I suggest 's', and make it rest 5 or 10 turns (or until disturbed.) I used to use the old search command for this all the time: wait until a nearby monster moves, or appears around a corner .
October 28, 2018, 21:36   #43
fph
Knight

Join Date: Apr 2009
Location: Berlin / Italy
Posts: 741
Quote:
 Originally Posted by fph There is usually a reporting bias, in that only 'unusual' results get reported to the devs, so the best thing would be collecting completely new data to test it.
Ran some tests: created a human mage, advanced him with debug commands to lvl21 and enough int to get 4% failure rate, and cast detect monsters 335 times. Got 14 failures. p-value of 0.47. Verdict: the RNG seems to work OK. Complaint "refuted by an appeal to advanced mathematical statistics".
__________________
Dive fast, die young, leave a high-CHA corpse.
--
You read a scroll labeled 'lol gtfo' of Teleport Level.

 October 28, 2018, 21:51 #44 Pete Mack Prophet   Join Date: Apr 2007 Location: Seattle, WA Posts: 5,217 Donated: \$40 For low probability events, a poisson distribution is generally appropriate. It always has a fat tail (because you can't have fewer than zero events.) Yes, it will eventually converge to normal (as any distribution will.) But even at expectation value E=14, while the curve looks close to normal, the tail is significantly fatter at 1.5*E. The estimate of 0.11% is too low by a significant amount.
October 28, 2018, 22:57   #45
fph
Knight

Join Date: Apr 2009
Location: Berlin / Italy
Posts: 741
Quote:
 Originally Posted by Pete Mack For low probability events, a poisson distribution is generally appropriate. It always has a fat tail (because you can't have fewer than zero events.) Yes, it will eventually converge to normal (as any distribution will.) But even at expectation value E=14, while the curve looks close to normal, the tail is significantly fatter at 1.5*E. The estimate of 0.11% is too low by a significant amount.
I don't follow you. These are independent trials, so their exact distribution is binomial/Bernoulli. I'm not approximating anything with a normal. 0.11% is what is returned by
Code:
`scipy.stats.binom_test(26,335,0.04, alternative='greater')`
__________________
Dive fast, die young, leave a high-CHA corpse.
--
You read a scroll labeled 'lol gtfo' of Teleport Level.

 October 29, 2018, 02:58 #46 Pete Mack Prophet   Join Date: Apr 2007 Location: Seattle, WA Posts: 5,217 Donated: \$40 Right. My bad.
 October 29, 2018, 08:17 #47 PowerWyrm Prophet     Join Date: Apr 2008 Posts: 2,586 Note: I've already failed a spell with 1% failure rate five times in a row __________________ PWMAngband variant maintainer - check http://powerwyrm.monsite-orange.fr (or http://www.mangband.org/forum/viewforum.php?f=9) to learn more about this new variant!
October 29, 2018, 11:37   #48
Sky
Veteran

Join Date: Oct 2016
Location: somewhere on the south coast
Age: 46
Posts: 1,249
Quote:
 Originally Posted by fph Ran some tests: created a human mage, advanced him with debug commands to lvl21 and enough int to get 4% failure rate, and cast detect monsters 335 times. Got 14 failures. p-value of 0.47. Verdict: the RNG seems to work OK. Complaint "refuted by an appeal to advanced mathematical statistics".
All you've proven is that under controlled test circumstances the rng doesnt fail.
I hope it's evident why that isnt enough.

Everyone has reported unusual rng results. At some point you gotta accept that it's not an epidemic of bias and something in the code is doing something it shouldnt. And god knows what it is. What is the rng based on anyway, system clock? That has to go through the OS which could unwittingly manipulate the results.

Empyrical observation does have its merits, y'know.

Oh btw congrats PowerWyrm on a 1:10,000,000,000 chance.
__________________
"i can take this dracolich"

October 29, 2018, 18:52   #49
fph
Knight

Join Date: Apr 2009
Location: Berlin / Italy
Posts: 741
Quote:
 Originally Posted by Sky All you've proven is that under controlled test circumstances the rng doesnt fail. I hope it's evident why that isnt enough. Everyone has reported unusual rng results. At some point you gotta accept that it's not an epidemic of bias and something in the code is doing something it shouldnt. And god knows what it is. What is the rng based on anyway, system clock? That has to go through the OS which could unwittingly manipulate the results. Empyrical observation does have its merits, y'know. Oh btw congrats PowerWyrm on a 1:10,000,000,000 chance.
As far as I understand (I didn't write the code) the RNG does one call to time() in Rand_init() when the game is started, then it's all state-based. If I read the branches correctly it uses this RNG.
__________________
Dive fast, die young, leave a high-CHA corpse.
--
You read a scroll labeled 'lol gtfo' of Teleport Level.

October 29, 2018, 19:42   #50
Nick
Vanilla maintainer

Join Date: Apr 2007
Location: Canberra, Australia
Age: 53
Posts: 7,588
Donated: \$60
Quote:
 Originally Posted by Sky Everyone has reported unusual rng results. At some point you gotta accept that it's not an epidemic of bias and something in the code is doing something it shouldnt.
Actually, no. Humans are built to recognise patterns; if there is no actual pattern, we will still find one. The correct thing to do is to have a mathematically sound source of randomness, check the implementation from time to time, and otherwise accept that people will continue to tell you that the RNG is broken
__________________
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.

 Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 Thread Tools Display Modes Linear Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home Angband     AAR     Vanilla     Development     ToME     Sil     Variants     Competition The real world     Idle chatter     Oook! Obsolete     v4

 Similar Threads Thread Thread Starter Forum Replies Last Post caruso Oook! 13 July 5, 2013 18:14 fbas Vanilla 1 November 24, 2010 19:51 fyonn Vanilla 17 June 23, 2010 16:04 konijn_ Variants 33 February 10, 2010 16:22 pav Oook! 19 September 18, 2008 23:07

All times are GMT +1. The time now is 06:28.