SDI stats

[Viluin]

Darkfall's nuclear stats for the past month:

Thwarted: 170
Direct hits: 147

Every single one of our opponents had an SDI, and so did we. The expected amount of thwarted nukes is 221, yet it was only 170.

Umbrella's nuclear stats for the past month:

Thwarted: 568
Direct hits: 644

All but 3 Umbrella's fighting nuclear nations have SDIs, and their NS is so high it's safe to assume practically all of their opponents did as well. Expected amount of thwarted nukes: 900+!


With sample sizes as large as these.. is there something admin isn't telling us? There is just no way the SDI really is 60/40. Pretty much every alliance has a very large number of direct hits compared to thwarted nukes (in fact, Darkfall is the only one I could find with less direct hits than nukes thwarted), but they often include a portion of nations that do not have SDIs, so it's hard to use them as an example.

Edited February 4, 2011 by Viluin

[Executive Minister]

Almost every single nuke, thrown or recieved amongst my wars were hit on first try (1 Offense, 3 Defense).

[Viluin]

Let's take a look at MK.

Around 3/4 of their alliance has a MP / SDI. Let's assume only 3/4 of their direct hits involved an SDI, which is an extremely conservative figure and highly unlikely considering their top heavy NS. Plus I have looked over MK's current opponents and their expired wars in their lower NS range and the majority of them had an SDI. Not to mention the very nature of the SDI causes more nuclear attacks to be performed, heavily weighing the number of nuclear attacks to SDI-equipped nations.

Stats:

Thwarted: 1247
Direct Hits: 1511
Expected number of thwarted nukes: 1511 * 0.75 * 1.5 = 1699

I am too lazy to do the math for alliances that aren't easy, but so far it appears the SDI doesn't do what it's supposed to do. Something is wrong. Since it's kind of hard to wrongly code a 60/40 RNG, it seems something else might be influencing the outcomes here, something that may be biased against certain nations. I'm not talking about admin bias because the numbers are incorrect for both sides of the war, but maybe a bug in the code somewhere else that affects nations with certain characteristics, or nations that attack at a certain time. Either that or there is something secret about the SDI we don't know. What does everyone else think?

Edited February 4, 2011 by Viluin

[Mompson]

Good stats. Really makes you wonder...

[Viluin]

It should be easy for admin to run a query that filters out all nuclear attacks where no SDI is involved, leaving only the relevant stats. I would be very interested in the result.

http://www.cybernations.net/stats_news.asp

Total nuclear attacks: 41,114
Thwarted: 17,220
Direct hits: 23,894

If it really is 60/40, that means roughly 17220 / 1.5 = 11480 of the direct hits were vs an SDI. So a little over 50% of all direct hits were against non-SDI equipped nations? That seems like a very high percentage to me.

Edited February 4, 2011 by Viluin

[der Rote Baron]

I think the SDI effectiveness depends largely on the amount of tech involved. Last week I was at war and had no problem nuking nations at my level or under, but two consecutive times - against a nation with 2,000 more tech than I do - it took [b]FIVE[/b] tries (ie 4 attempts thwarted each time).

[thedestro]

Have you considered that not every nuke launched has had an SDI to block it? Nvm

Edited February 4, 2011 by thedestro

[Viluin]

[quote name='der Rote Baron' timestamp='1296840936' post='2619371']
I think the SDI effectiveness depends largely on the amount of tech involved. Last week I was at war and had no problem nuking nations at my level or under, but two consecutive times - against a nation with 2,000 more tech than I do - it took [b]FIVE[/b] tries (ie 4 attempts thwarted each time).
[/quote]

This is one possible explanation, because Umbrella is very tech-heavy. But in the case of Darkfall, both incoming and outgoing thwarted attacks are quite a bit lower than they should be. Of course Darkfall's sample size is smaller so the theory is still plausible.

EDIT: Actually, it doesn't make sense, because it would work both ways. A tech heavy alliance would thwart a lot of incoming nukes while at the same time dishing out a lot of direct hits. So the end result should still be the same ~60/40 when you look at the total number of nuclear attacks.

Edited February 4, 2011 by Viluin

[der Rote Baron]

[quote name='Viluin' timestamp='1296842758' post='2619392']
This is one possible explanation, because Umbrella is very tech-heavy. But in the case of Darkfall, both incoming and outgoing thwarted attacks are quite a bit lower than they should be. Of course Darkfall's sample size is smaller so the theory is still plausible.

EDIT: Actually, it doesn't make sense, because it would work both ways. A tech heavy alliance would thwart a lot of incoming nukes while at the same time dishing out a lot of direct hits. So the end result should still be the same ~60/40 when you look at the total number of nuclear attacks.
[/quote]
You have to take the match ups into account, is my point. You might have several nations at war with any one particular target, but you always want the nation with the most tech to be the one doing the nuking if possible. That is going to dramatically increase the odds of the nuke avoiding the SDI and causing more damage. If these guys are coordinating their attacks they are likely taking advantage of the match ups on both sides, and that might be what's represented in the numbers you are quoting.

Edited February 4, 2011 by der Rote Baron

[Stelios]

It actually has looked like to me the closer your infra was to your tech that it blocked more, but at the least more tech is blocking more. Even as my nation with the one with more tech.

[Viluin]

[quote name='der Rote Baron' timestamp='1296844733' post='2619411']
You have to take the match ups into account, is my point. You might have several nations at war with any one particular target, but you always want the nation with the most tech to be the one doing the nuking if possible. That is going to dramatically increase the odds of the nuke avoiding the SDI and causing more damage. If these guys are coordinating their attacks they are likely taking advantage of the match ups on both sides, and that might be what's represented in the numbers you are quoting.
[/quote]

Tech isn't supposed to affect the SDI, though. I think only admin really knows the answer. Personally I've never really noticed anything with regards to tech, people seem to thwart my nukes all the same. My 10-day streaks of not thwarting any enemy nukes were all against opponents with way less tech than me.

Edited February 4, 2011 by Viluin

[Ironfist]

Personal stats for me were 4 nukes taken, and 2 thwarted. I had an SDI the whole time. 3 direct hits, and one instance in which the SDI actually did something (blocked 2 in a row); this was my last nuke taken, too.

Needless to say, I was checking daily to make sure that I still had an SDI and hadn't deleted by accident >_>

[Bakunin's Dream]

so I saw this and randomly did a chi-square test just for !@#$% and giggles. Darkfall's chi-squared was 5.36 which is significant at p=.02 with 1 degree of freedom and Umbrella's was 23.80 which is significant at p<.00001, also with 1 df. If you take both of them together you get a chi-squared of 29.14 which has a p-value on the order of 10^-8 so yeah something's up (for non-math geeks the p-value is the probability of the result occurring by chance alone). I'm ready to reject the null hypothesis that this is good, randomly sampled data of nations that all had SDIs giving a hit probability of .4 but that doesn't tell us what's actually going on. here are some possibilities:

-not all nations sampled actually had SDIs
-sampling error, i.e. hits were more likely to be recorded as data points than misses for some reason related to the methodology
-selection bias, i.e. alliances being analyzed were selected in the first place for having unusual hit rates
-i'm tired and i didn't do the chi-square test right
-the probability of a nuke hitting given the defender having an SDI is not actually equal to 0.4

I'm leaning towards some combination of the first three or maybe just #4. Or some other reason I haven't thought of. But I kind of doubt that there would be something off with the SDI hit rate since it seems like it would be really hard to screw up the code for it.

Edited February 5, 2011 by Bakunin's Dream

[Viluin]

[quote name='Bakunin's Dream' timestamp='1296887359' post='2620141']
so I saw this and randomly did a chi-square test just for !@#$% and giggles. Darkfall's chi-squared was 5.36 which is significant at p=.02 with 1 degree of freedom and Umbrella's was 23.80 which is significant at p<.00001, also with 1 df. If you take both of them together you get a chi-squared of 29.14 which has a p-value on the order of 10^-8 so yeah something's up (for non-math geeks the p-value is the probability of the result occurring by chance alone). I'm ready to reject the null hypothesis that this is good, randomly sampled data of nations that all had SDIs giving a hit probability of .4 but that doesn't tell us what's actually going on. here are some possibilities:

-not all nations sampled actually had SDIs
-sampling error, i.e. hits were more likely to be recorded as data points than misses for some reason related to the methodology
-selection bias, i.e. alliances being analyzed were selected in the first place for having unusual hit rates
-i'm tired and i didn't do the chi-square test right
-the probability of a nuke hitting given the defender having an SDI is not actually equal to 0.4

I'm leaning towards some combination of the first three or maybe just #4. Or some other reason I haven't thought of. But I kind of doubt that there would be something off with the SDI hit rate since it seems like it would be really hard to screw up the code for it.
[/quote]

#1 does not apply to Darkfall. It does apply to Umbrella, but we're talking about a very small, pretty much insignificant percentage. Not something that would dramatically inflate the number of hits. #3 is sort of correct, but only because it takes very large amounts of effort to correctly retrieve the stats of other alliances, because they generally have more low-NS nukers. At first glance it seems to me like every alliance has too many direct hits compared to thwarted nukes.

As for #4.. I've seen worse screw-ups in the game, lol. I remember one time when everyone's environment was completely devastated due to a typo in the code. Also, people are still getting nuked twice a day or spied on 3 times a day so I'm a bit skeptical of the game's integrity by default.

Edited February 5, 2011 by Viluin

[Viluin]

All right, I looked through all of MK's nuclear reports and filtered out the non-SDI ones. Yes, srsly. What I did was, I skimmed through the reports (sorted by defending nation) and looked for nations that had never thwarted a single nuke. Then I checked them out to see if they had an SDI. If they did not have an SDI, I counted all of the direct hits on their nation, and added them to the total number of non-SDI hits. I'm pretty sure I got them all, even the nations that were only involved in a single nuclear attack. It took me about 20 minutes, but here's the result:

241 out of 1568 direct hits involved no SDI.

So 1568 - 241 = 1327 direct hits did involve an SDI.

Expected number of thwarted nukes: 1327 * 1.5 = 1991.

Actual number of thwarted nukes: 1294.

Once again the numbers are [b]way[/b] off. This just can't be right. Maybe I'll do another alliance tomorrow, but at this point I'm absolutely convinced there is something fishy going on with the SDI. People have been complaining about it for years, I'm surprised it has taken this long for someone to look into it.

Edited February 5, 2011 by Viluin

[Viluin]

Nueva Vida:

Total number of nuclear attacks: 1102

# of direct hits: 638
# of direct hits not involving an SDI: 150
# of direct hits involving an SDI: 488
[b]# of thwarted nukes (expected): 736
# of thwarted nukes (real): 464![/b]

Do I have to continue? It's like admin implemented a 50/50 RNG instead of 60/40.

Edited February 5, 2011 by Viluin

[Quinoa Rex]

Like Bakunin alluded to, your data is statistically invalid unless you have data on all the nations that were receiving the nukes. There are a surprising number of very large nations that don't have an SDI, and you can't just throw that away as not significant unless you actually have the numbers.

Your project is nice but doesn't tell anyone anything in terms of statistics.

[Viluin]

[quote name='Quinoa Rex' timestamp='1296928271' post='2620630']
Like Bakunin alluded to, your data is statistically invalid unless you have data on all the nations that were receiving the nukes. There are a surprising number of very large nations that don't have an SDI, and you can't just throw that away as not significant unless you actually have the numbers.

Your project is nice but doesn't tell anyone anything in terms of statistics.
[/quote]

Darkfall's stats do not include any non-SDI nations. Umbrella has a near-100% SDI rate, as do the enemies they were fighting. As for more statistics, I would like to refer you to my last two posts about MK and NV, which are completely accurate. I can do more alliances if you want, but I doubt the results will be any different.

Edited February 5, 2011 by Viluin

[Quinoa Rex]

[quote name='Viluin' timestamp='1296928371' post='2620633']
Darkfall's stats do not include any non-SDI nations. Umbrella has a near-100% SDI rate, as do the enemies they were fighting. As for more statistics, I would like to refer you to my last two posts about MK and NV, which are completely accurate.
[/quote]
It's not valid to do it by aggregate percentages and total hits; you need to do it with each nation's individual percentage to get your data points.

[Viluin]

[quote name='Quinoa Rex' timestamp='1296928560' post='2620637']
It's not valid to do it by aggregate percentages and total hits; you need to do it with each nation's individual percentage to get your data points.
[/quote]

That doesn't matter at all. If we look at 1000 random nuclear attacks involving an SDI, we can expect roughly 400 to be direct hits and roughly 600 to be thwarted. There is no need to look at each nation's individual percentage. In this case, the distribution seems to be around 50/50 instead of 60/40.

Edited February 5, 2011 by Viluin

[Quinoa Rex]

[quote name='Viluin' timestamp='1296928697' post='2620640']
That doesn't matter at all. If we look at 1000 random nuclear attacks involving an SDI, we can assume roughly 400 will be direct hits and roughly 600 will be thwarted. There is no need to look at each nation's individual percentage. In this case, the distribution seems to be around 50/50 instead of 60/40.
[/quote]
No, it does matter, because the percentage is calculated with each individual nuclear attack. You could flip a penny 1000 times and I can almost guarantee it's never going to come out at exactly 50-50.
Also, for Umbrella, "near-100%" does not mean "100%" and given that Umbrella is of relatively small size compared to their strength, that has the potential to skew the results a lot. MK also doesn't have 100% SDI by any means.

I'm not saying you're wrong, I'm saying that your data collection and calculation methods are not useful.

[Viluin]

[quote name='Quinoa Rex' timestamp='1296929271' post='2620663']
No, it does matter, because the percentage is calculated with each individual nuclear attack. You could flip a penny 1000 times and I can almost guarantee it's never going to come out at exactly 50-50.
Also, for Umbrella, "near-100%" does not mean "100%" and given that Umbrella is of relatively small size compared to their strength, that has the potential to skew the results a lot. MK also doesn't have 100% SDI by any means.

I'm not saying you're wrong, I'm saying that your data collection and calculation methods are not useful.
[/quote]

You could flip a penny 1000 times and expect a result that is near 500-500. If it comes out as 400-600 then either a small miracle occurred or something was slightly off during the tests.

Did you check out the stats? I only included nuclear attacks that involved an SDI in MK's and NV's stats. The sample was not diluted by non-SDI attacks, so it is in fact useful.

In the case of Umbrella, their number of non-SDI nations and opponents is so small that it will not somehow correct the results and create a ~60-40 distribution, these nations couldn't have been involved in more than 50 nuclear attacks, if that. If you want, I can filter them out properly and produce a "useful" result, but it shouldn't be necessary as it is pretty obvious their stats will never come close to 60-40. Umbrella might be relatively small, but we're talking about a sample size of over 1000 nuclear attacks. Nearly 3000 for MK and 1000+ for NV as well, and each of these large samples shows ~50-50. I bet other alliances will show 50-50 as well if I go through their nuclear reports.

Edited February 5, 2011 by Viluin

[Quinoa Rex]

[quote name='Viluin' timestamp='1296929730' post='2620680']
You could flip a penny 1000 times and expect a result that is near 500-500. If it comes out as 400-600 then either a small miracle occurred or something was slightly off during the tests.

Did you check out the stats? I only included nuclear attacks that involved an SDI in MK's and NV's stats. The sample was not diluted by non-SDI attacks, so it is in fact useful.

In the case of Umbrella, their number of non-SDI nations and opponents is so small that it will not somehow correct the results and create a ~60-40 distribution, these nations couldn't have been involved in more than 50 nuclear attacks, if that. If you want, I can filter them out properly and produce a "useful" result, but it shouldn't be necessary as it is pretty obvious their stats will never come close to 60-40. Umbrella might be relatively small, but we're talking about a sample size of over 1000 nuclear attacks. Nearly 3000 for MK and 1000+ for NV as well, and each of these large samples shows ~50-50. I bet other alliances will show 50-50 as well if I go through their nuclear reports.
[/quote]
I did look at the stats, and they might prove something if your methods were statistically valid, which they aren't. In order to have a properly random sample, you would have to pull data regardless of alliances from nations that hold an SDI.

[Viluin]

[quote name='Quinoa Rex' timestamp='1296930160' post='2620691']
I did look at the stats, and they might prove something if your methods were statistically valid, which they aren't. In order to have a properly random sample, you would have to pull data regardless of alliances from nations that hold an SDI.
[/quote]

Alliance affiliation has absolutely no effect on the SDI, so once again this doesn't matter. My methods are statistically valid because every nuclear attack is equal, so all we need to look at is the nuclear attacks. These are thousands of nuclear attacks all involving an SDI, and unless not every SDI is equal (flipping the doomswitch on certain alliances? yeah right) then there is nothing wrong with the sample. The only thing that would make this sample non-random (for statistical purposes) is if I purposefully ignored some thwarted nukes and included an inflated amount of direct hits in the sample.

Not to mention an alliance's nuclear reports also include nuclear attacks that have been performed on their enemies, so we've been looking at the SDIs of both sides.

Edited February 5, 2011 by Viluin

[Viluin]

Alpha Omega:

Total number of nuclear attacks: 564

# of direct hits: 295
# of direct hits not involving an SDI: 57
# of direct hits involving an SDI: 238
[b]# of thwarted nukes (expected): 357
# of thwarted nukes (real): 269[/b]


Global Order of Darkness:

Total number of nuclear attacks: 1170

# of direct hits: 663
# of direct hits not involving an SDI: 196
# of direct hits involving an SDI: 467
[b]# of thwarted nukes (expected): 701
# of thwarted nukes (real): 507[/b]

I have yet to find an alliance that hasn't thwarted significantly less than 60%. They all seem to be in the 45-55% range. Maybe I'll do a big one like NpO later.

EDIT:

Proper Umbrella stats:

Total number of nuclear attacks: 1170

# of direct hits: 676
# of direct hits not involving an SDI: 52
# of direct hits involving an SDI: 624
[b]# of thwarted nukes (expected): 936
# of thwarted nukes (real): 598[/b]

NOTE: While searching through thousands of nuclear reports of all these alliances, I encountered a handful of nations that had never thwarted a nuke, but were deleted, so I couldn't check their wonders. I've always included those nations in the non-SDI category, to be on the conservative side. So, if anything, I've recorded too many (1%) non-SDI hits in all these samples.

Edited February 5, 2011 by Viluin