interpreting t-test results

[slynton]

Here's a question about interpreting t-test results for some microprobe data. Simply looking for statistical guidance.

I've got three populations of microprobe data (olivine, plagioclase, and pyroxene). For each of these populations (>100 to >300 samples each), I'm trying to determine whether I can distinguish between two subpopulations (from two different stratigraphic units) on the basis of their compositions. For the olivine subpopulations, there's a relatively large difference (in terms of their Fo values) between t-stat and t-critical (14.2 versus 2.3). For the plagioclase subpopulations, there's a significant but smaller difference (in terms of their An values) between t-stat and t-critical (6.8 versus 2.6). For the pyroxene subpopulations, there's no significant difference between the two subpopulations' means in terms of the two subpopulations' Wu values (because t-critical exceeds t-stat); and in terms of their En and Fs values, t-stat just barely exceeds t-critical (by 3.0 versus 2.0 or by 4.3 versus 3.3).

My question is whether it's statistically meaningful or appropriate to point out the different amounts of these differences between t-stat and t-critical values in describing this dataset. For example, is it statistically meaningful to say that the two olivine subpopulations are more distinguishable than the two plagioclase populations? Or would it be statistically meaningful to say that, for the two pyroxene subpopulations, the narrow difference between t-stat and t-critical indicates they're only barely distinguishable in their mean En and Fs values? Or would that sort of characterization of the data be an improper way to characterize t-test results? Any thoughts on this?

[Probeman]

I'm not an expert in t-test statistics, but if someone asked me if two compositions were statistically different I would just apply the analytical sensitivity t-test which is calculated by clicking the Calculate Detection Limits and Sensitivity option in the Calculation Option dialog from the Analyze! window in Probe for EPMA. One can also export these values from a number of output/export formats in PFE.

This issue was discussed here a few years ago:

https://smf.probesoftware.com/index.php?topic=1307.0

The analytical sensitivity output to the log window looks like this in PFE:

Analytical Sensitivity (t-test) in Elemental Weight Percent (Average of Sample):

ELEM:       Si      Al      Ca       K      Ti      Cr      Fe      Mn      Na      Mg       S      Ni
  60ci    .333    .000    .044    .000    .000    .000    .052    .002    .000    .299    .355    .000
  80ci    .520    .001    .069    .000    .000    .000    .081    .004    .000    .466    .554    .000
  90ci    .687    .001    .091    .000    .000    .000    .107    .005    .000    .615    .731    .000
  95ci    .844    .001    .112    .000    .000    .000    .132    .006    .000    .757    .899    .000
  99ci   1.201    .001    .159    .000    .000    .000    .187    .008    .000   1.076   1.279    .000

The analytical sensitivity t-test calculation is documented in the PFE User's Reference manual/help file:



So, if the average of one population is different from another population's average by an amount that exceeds the t-test analytical sensitivity, then I would say the populations are distinct, for that t-test statistics confidence, for that element at least.  Each element will be different of course.

But I'm no expert at these things... and certainly no geologist!

One other thing I might do is to plot up some variation plots in Probe for EPMA as described here:

https://smf.probesoftware.com/index.php?topic=1170.0

[Probeman]

Here's a question about interpreting t-test results for some microprobe data. Simply looking for statistical guidance.

I've got three populations of microprobe data (olivine, plagioclase, and pyroxene). For each of these populations (>100 to >300 samples each), I'm trying to determine whether I can distinguish between two subpopulations (from two different stratigraphic units) on the basis of their compositions. For the olivine subpopulations, there's a relatively large difference (in terms of their Fo values) between t-stat and t-critical (14.2 versus 2.3). For the plagioclase subpopulations, there's a significant but smaller difference (in terms of their An values) between t-stat and t-critical (6.8 versus 2.6). For the pyroxene subpopulations, there's no significant difference between the two subpopulations' means in terms of the two subpopulations' Wu values (because t-critical exceeds t-stat); and in terms of their En and Fs values, t-stat just barely exceeds t-critical (by 3.0 versus 2.0 or by 4.3 versus 3.3).

My question is whether it's statistically meaningful or appropriate to point out the different amounts of these differences between t-stat and t-critical values in describing this dataset. For example, is it statistically meaningful to say that the two olivine subpopulations are more distinguishable than the two plagioclase populations? Or would it be statistically meaningful to say that, for the two pyroxene subpopulations, the narrow difference between t-stat and t-critical indicates they're only barely distinguishable in their mean En and Fs values? Or would that sort of characterization of the data be an improper way to characterize t-test results? Any thoughts on this?

OK, just for kicks I used the above questions from Steve and asked Grok. Here is how it responded:

Your question is a great one, and it dives into the nuances of interpreting t-test results in a statistically meaningful way. Let's break this down step-by-step to clarify what's appropriate and meaningful when describing differences between t-statistics (t-stat) and critical values (t-critical) for your microprobe data.

Understanding the t-test and its components

The t-test compares the means of two groups to determine if they are statistically distinguishable, with the t-stat reflecting the magnitude of the difference between the means relative to the variability in the data, and the t-critical serving as the threshold for significance (typically at a chosen alpha level, e.g., 0.05). The difference between t-stat and t-critical isn't a standard metric in itself, but it can inform your interpretation if handled carefully.

Is it meaningful to compare the "amount of difference" between t-stat and t-critical across populations?

Yes, but with caveats. The magnitude by which t-stat exceeds t-critical can qualitatively suggest how "strong" the evidence is against the null hypothesis (i.e., no difference between means). A larger excess (e.g., 14.2 vs. 2.3 for olivine) indicates a more pronounced difference between the subpopulation means relative to the noise in the data, compared to a smaller excess (e.g., 6.8 vs. 2.6 for plagioclase). However, this isn't a formal statistical measure like a p-value or effect size—it's more of an intuitive guide. Here's how you might think about it for your specific cases:

    Olivine (t-stat = 14.2, t-critical = 2.3)
    The large gap suggests a highly significant difference in Fo values between the two stratigraphic units. This isn't just statistically significant; it's robustly so, implying the subpopulations are quite distinct. Saying they are "more distinguishable" than others could be reasonable, but it's better grounded if supported by additional context (e.g., effect size).

    Plagioclase (t-stat = 6.8, t-critical = 2.6)
    This is still significant, but the smaller excess indicates the difference in An values, while detectable, is less pronounced relative to the variability. You could say it's "less distinguishable" than olivine, but again, this is qualitative unless quantified further.

    Pyroxene (Wu: t-critical > t-stat; En/Fs: t-stat slightly exceeds t-critical)
    For Wu, the lack of significance (t-critical > t-stat) means you can't reject the null hypothesis—no distinguishable difference. For En and Fs, where t-stat just barely exceeds t-critical (e.g., 3.0 vs. 2.0 or 4.3 vs. 3.3), the difference is statistically significant, but the narrow margin suggests it's borderline. Characterizing this as "barely distinguishable" is intuitive and aligns with the result, but it's not a precise statistical term.

Is this an appropriate way to characterize t-test results?

It's not improper, but it's informal. The difference between t-stat and t-critical isn't a standard statistic for comparing across tests because it's influenced by sample size, variance, and degrees of freedom, which differ across your populations (olivine: >100, plagioclase: >300, etc.). A t-stat much larger than t-critical doesn't inherently mean a bigger "real-world" difference—it could reflect lower variability or larger sample sizes inflating the t-stat.

To make your comparisons more statistically rigorous, consider these alternatives:

    P-values: These quantify the strength of evidence against the null hypothesis and are comparable across tests (e.g., p < 0.0001 for olivine vs. p = 0.01 for pyroxene En values).

    Effect size: Metrics like Cohen's d measure the magnitude of the difference between means in standardized units, accounting for variability. This would let you say, e.g., "The olivine subpopulations show a larger effect size (d = 1.2) than plagioclase (d = 0.5)," which is more precise than comparing t-stat margins.

    Confidence intervals: Reporting the 95% CI of the mean difference for each pair gives a range of plausible differences, adding depth to your interpretation.

Practical advice for your description

You can point out the differing extents to which t-stat exceeds t-critical as a descriptive observation, especially for a general audience, but frame it cautiously:

    Acceptable: "The olivine subpopulations show a highly significant difference in Fo values (t = 14.2, t-critical = 2.3), suggesting a robust distinction, while the plagioclase subpopulations (t = 6.8, t-critical = 2.6) are also distinct but less markedly so. For pyroxene, the En and Fs values are only marginally distinguishable (e.g., t = 3.0, t-critical = 2.0), while Wu shows no significant difference."

    Avoid overreach: Don't treat the t-stat minus t-critical gap as a formal metric (e.g., "olivine is 6 times more distinguishable than plagioclase") without supporting it with p-values or effect sizes.

My thoughts

Your instinct to highlight these differences is valid and useful for storytelling in your dataset, especially in a geological context where distinguishing subpopulations has practical implications. Just avoid implying that the raw difference between t-stat and t-critical is a definitive measure of "distinguishability." If you have access to the raw data, I'd recommend calculating effect sizes or p-values to back up your narrative. For example, the olivine result (t = 14.2) likely corresponds to a tiny p-value and possibly a large effect, while pyroxene's borderline result (t = 3.0 vs. 2.0) might hover near p = 0.05 with a small effect—those would solidify your interpretation.

Does that help clarify things? If you've got the data handy and want a deeper dive (e.g., estimating effect sizes), feel free to share more details!

Grok's response seems pretty reasonable to me. Can anyone detect any "hallucinations"?

[slynton]

Interesting and helpful. Thanks. I also considered one other potential method for distinguishing between the larger and smaller differences between t-statistic and t-critical for the different populations and subpopulations. That method would be to change the alpha values successively from 0.05 to 0.01 (and then, perhaps, to still smaller values) to see whether there was an alpha value at which the t-statistic value for one of the populations no longer exceeded the t-critical threshold. That didn't initially work in this case, as t-statistic exceeded t-critical for both the olivine and plagioclase populations after changing the alpha value from 0.05 to 0.01. I don't know whether that method might work in other situations, and I'm not sure whether reducing the alpha value below 0.01 would be reasonable.