JEOL EDS data cube

[JonF]

For people with the newer MEC EDS interface, it's also possible to extract the EDS hypercube to produce dead time corrected maps in the same (well, similar) way.

Here's a spectrum from a map I took on our iHP200F using the JEOL EDS (combined SXES, WDS and EDS mapping from within the PC EPMA software):



and you can extract arbitrary layers from the map e.g.:



and pull out the dead time for the map, too (units are % dead time):



You can then integrate the various channels from the map and correct each pixel for deadtime (units are cps):



The above was from a 100x100 pixel map at 7kV, 30nA with a 600ms dwell time. 

[John Donovan]

Hi Jon,
Do you have any idea if the JEOL hypercube is the same file format for the 8230/8530 as the iSP100/iHP200F?

I have "spectrum imaging" API calls for the 8203/8530 EDS API to obtain net intensity maps, but not for the MEC EDS interface.  I will write Japan.

[JonF]

Hi Jon,
Do you have any idea if the JEOL hypercube is the same file format for the 8230/8530 as the iSP100/iHP200F?

I have "spectrum imaging" API calls for the 8203/8530 EDS API to obtain net intensity maps, but not for the MEC EDS interface.  I will write Japan.

Different file formats and stored in different places, annoyingly.
The 8530F keeps the EDS hypercube in a 2406PTTD.PTS file in the EDS folder of the EPMA side of things (i.e. where the map.cnd lives and all the WDS data).
The iHP200F keeps the EDS hypercube in a PTTD.TagData file in a directory separate from the EPMA stuff(C:\JDS\Data\), and is linked to from within the EPMA data directory from a EdsNodeId.cnd file, which contains a link to another directory, and then another directory... Not only does this make working with the EDS file more difficult (you've got to find it first!), but it also makes backing up data more difficult. 

[John Donovan]

Hi Jon,
Do you have any idea if the JEOL hypercube is the same file format for the 8230/8530 as the iSP100/iHP200F?

I have "spectrum imaging" API calls for the 8203/8530 EDS API to obtain net intensity maps, but not for the MEC EDS interface.  I will write Japan.

Different file formats and stored in different places, annoyingly.
The 8530F keeps the EDS hypercube in a 2406PTTD.PTS file in the EDS folder of the EPMA side of things (i.e. where the map.cnd lives and all the WDS data).
The iHP200F keeps the EDS hypercube in a PTTD.TagData file in a directory separate from the EPMA stuff(C:\JDS\Data\), and is linked to from within the EPMA data directory from a EdsNodeId.cnd file, which contains a link to another directory, and then another directory... Not only does this make working with the EDS file more difficult (you've got to find it first!), but it also makes backing up data more difficult. 

You've done some sleuthing I see!

I've written JEOL Japan to see if the 8230/8530 spectrum image processing API calls are also available in the MEC interface API.

Another question: are the dead time corrected net intensity maps above in cps (livetime) units?  That's what we need for quantification with (or without!) WDS maps in CalcImage.

If yes, then you are well on your way to getting all this imported in CalcImage for full quantification. You only need to convert the WDS maps into GRD files using Probe Image, then convert these EDS net intensity maps into GRD files using your code, then simply write a CalcImage CIP project file as shown here:

https://smf.probesoftware.com/index.php?topic=1844.0

And can quant your heart away!

[JonF]

Another question: are the dead time corrected net intensity maps above in cps (livetime) units?  That's what we need for quantification with (or without!) WDS maps in CalcImage.

The maps above are in deadtime corrected counts per second real time, but they can be in whatever format we wish. I've got real time, live time and dead time maps and it's simply a job of dividing each pixel by whichever map you like.
For the maps above, I've just divided each pixel of the integrated count maps by the corresponding pixel of the live time map (~400ms) to get counts per (real) ms, and then multiplied up to get cps. At that point, the dead time has effectively been removed from the data set, which needs accounting for in CalcImage (as it will see a HUGE number and try to deadtime correct it again).   

I've been doing quant EDS mapping alongside quant WDS (and quant SXES) on our 8530F (old JEOL EDS and SXES-LR) for a while, I gave a presentation on it at EMAS in Mataro last year, but the work wasn't included in the conference proceedings so I've been looking for somewhere else to publish it. It actually works really quite well! This thread has given me the push to crack on and get the same sorted for our iHP200F, which has the new MEC EDS interface and the SXES-ER.
I've gone about dealing with the EDS in a very different way from what Ben is doing with NeXL, so I'm very interested to see where that goes. 

[John Donovan]

Another question: are the dead time corrected net intensity maps above in cps (livetime) units?  That's what we need for quantification with (or without!) WDS maps in CalcImage.

The maps above are in deadtime corrected counts per second real time, but they can be in whatever format we wish. I've got real time, live time and dead time maps and it's simply a job of dividing each pixel by whichever map you like.

Wait, so you're already importing 8230/8530 EDS maps into CalcImage for quantification?  Why didn't you tell me? 

For the maps above, I've just divided each pixel of the integrated count maps by the corresponding pixel of the live time map (~400ms) to get counts per (real) ms, and then multiplied up to get cps. At that point, the dead time has effectively been removed from the data set, which needs accounting for in CalcImage (as it will see a HUGE number and try to deadtime correct it again). 

Why would this need "accounting for in CalcImage"?  All GRD files in CalcImage are in cps and EDS net intensities are NOT corrected for dead time if the element assigned is an EDS element (crystal type = EDS)

Here is the code for the dead time, beam drift and off-peak corrections for each pixel:

Code Select Expand

Sub CalcImageCorrectData(ipixelx As Long, ipixely As Long, npixelx As Long, npixely As Long)
' Perform deadtime, beam drift and off-peak correction for each pixel

ierror = False
On Error GoTo CalcImageCorrectDataError

Dim i As Integer, ip As Integer
Dim ihi As Integer, ilo As Integer
Dim tbeamon As Single, tbeamhi As Single, tbeamlo As Single

' Load unknown (on-peak) data for this pixel (GRD files from Probe Image are already in cps) (including EDS data!)
For i% = 1 To CalcImageOldSample(1).LastElm%
CalcImageOldSample(1).OnPeakCounts!(1, i%) = 0#                                             ' only one data row for image pixels
ip% = CalcImageIndex(Int(0), CalcImageNumberofImageFiles%, i%, CalcImageElementRows%(), CalcImageImageTypes%())
If ip% > 0 Then
CalcImageOldSample(1).OnPeakCounts!(1, i%) = ImageData(ip%).gData(ipixelx&, ipixely&)       ' only one data row for image pixels
CalcImageOldSample(1).OnPeakCounts_Raw_Cps!(1, i%) = ImageData(ip%).gData(ipixelx&, ipixely&) ' save for statistics calculations!!!!!!!
CalcImageOldSample(1).OnCountTimes!(1, i%) = CalcImageCountTimes(ip%)                         ' load x-ray map on peak count times (for possible detection limit calculations)
CalcImageOldSample(1).OnTimeData!(1, i%) = CalcImageCountTimes(ip%)                         ' load x-ray map on peak count times (for aggregate calculations)
End If
Next i%

' Load off peak data (if specified) (GRD files from Probe Image are already in cps)
For i% = 1 To CalcImageOldSample(1).LastElm%
If CalcImageOldSample(1).CrystalNames$(i%) <> EDS_CRYSTAL$ Then                             ' skip EDS spectra channels (no off-peak data for EDS!)

' Check for high and low off peaks images
CalcImageOldSample(1).HiPeakCounts!(1, i%) = 0#                                             ' only one data row for image pixels
CalcImageOldSample(1).LoPeakCounts!(1, i%) = 0#                                             ' only one data row for image pixels
ihi% = CalcImageIndex(Int(1), CalcImageNumberofImageFiles%, i%, CalcImageElementRows%(), CalcImageImageTypes%())
ilo% = CalcImageIndex(Int(2), CalcImageNumberofImageFiles%, i%, CalcImageElementRows%(), CalcImageImageTypes%())
If ihi% > 0 Then
CalcImageOldSample(1).HiPeakCounts!(1, i%) = ImageData(ihi%).gData(ipixelx&, ipixely&)
CalcImageOldSample(1).HiPeakCounts_Raw_Cps!(1, i%) = ImageData(ihi%).gData(ipixelx&, ipixely&) ' save for statistics calculations!!!!!!!
CalcImageOldSample(1).HiCountTimes!(1, i%) = CalcImageCountTimes(ihi%)                        ' load hi peak count times (for possible detection limit calculations)
CalcImageOldSample(1).HiTimeData!(1, i%) = CalcImageCountTimes(ihi%)                        ' load hi peak count times (for aggregate calculations)
End If

If ilo% > 0 Then
CalcImageOldSample(1).LoPeakCounts!(1, i%) = ImageData(ilo%).gData(ipixelx&, ipixely&)
CalcImageOldSample(1).LoPeakCounts_Raw_Cps!(1, i%) = ImageData(ilo%).gData(ipixelx&, ipixely&) ' save for statistics calculations!!!!!!!
CalcImageOldSample(1).LoCountTimes!(1, i%) = CalcImageCountTimes(ilo%)                        ' load lo peak count times (for possible detection limit calculations)
CalcImageOldSample(1).LoTimeData!(1, i%) = CalcImageCountTimes(ilo%)                        ' load lo peak count times (for aggregate calculations)
End If

End If      ' not EDS
Next i%

' Correct on and off peak intensities for deadtime (assume standard is the same)
For i% = 1 To CalcImageOldSample(1).LastElm%
If CalcImageOldSample(1).CrystalNames$(i%) <> EDS_CRYSTAL$ Then           ' skip EDS spectra channels (already deadtime corrected by hardware)
Call DataCorrectDataDeadTime(CalcImageOldSample(1).OnPeakCounts!(1, i%), CalcImageOldSample(1).DeadTimes!(i%))
If ierror Then Exit Sub

' Correct off-peak intensities for dead time
Call DataCorrectDataDeadTime(CalcImageOldSample(1).HiPeakCounts!(1, i%), CalcImageOldSample(1).DeadTimes!(i%))
If ierror Then Exit Sub
Call DataCorrectDataDeadTime(CalcImageOldSample(1).LoPeakCounts!(1, i%), CalcImageOldSample(1).DeadTimes!(i%))
If ierror Then Exit Sub

End If
Next i%

' Correct on-peak data for beam current (including EDS data!)
For i% = 1 To CalcImageOldSample(1).LastElm%
ip% = CalcImageIndex(Int(0), CalcImageNumberofImageFiles%, i%, CalcImageElementRows%(), CalcImageImageTypes%())
If ip% > 0 Then
Call CalcImageCalculateBeam(ipixelx&, ipixely&, npixelx&, npixely&, CalcImageBeamCurrents!(ip%), CalcImageBeamCurrents2!(ip%), tbeamon!)
If ierror Then Exit Sub
Call DataCorrectDataBeamDrift(CalcImageOldSample(1).OnPeakCounts!(1, i%), tbeamon!)
If ierror Then Exit Sub
End If
Next i%

' Correct off-peak data for beam current
For i% = 1 To CalcImageOldSample(1).LastElm%
If CalcImageOldSample(1).CrystalNames$(i%) <> EDS_CRYSTAL$ Then                ' skip EDS spectra channels (no off-peak data for EDS!)

' Correct off-peak intensities for beam current
ihi% = CalcImageIndex(Int(1), CalcImageNumberofImageFiles%, i%, CalcImageElementRows%(), CalcImageImageTypes%())
ilo% = CalcImageIndex(Int(2), CalcImageNumberofImageFiles%, i%, CalcImageElementRows%(), CalcImageImageTypes%())
If ihi% > 0 Then
Call CalcImageCalculateBeam(ipixelx&, ipixely&, npixelx&, npixely&, CalcImageBeamCurrents!(ihi%), CalcImageBeamCurrents2!(ihi%), tbeamhi!)
If ierror Then Exit Sub
Call DataCorrectDataBeamDrift(CalcImageOldSample(1).HiPeakCounts!(1, i%), tbeamhi!)
If ierror Then Exit Sub
End If
If ilo% > 0 Then
Call CalcImageCalculateBeam(ipixelx&, ipixely&, npixelx&, npixely&, CalcImageBeamCurrents!(ilo%), CalcImageBeamCurrents2!(ilo%), tbeamlo!)
If ierror Then Exit Sub
Call DataCorrectDataBeamDrift(CalcImageOldSample(1).LoPeakCounts!(1, i%), tbeamlo!)
If ierror Then Exit Sub
End If

' Override background correction type based on number of off-peak images (0=linear, 1=average, 2=high only, 3=low only, 4=exponential, 5=slope hi, 6=slope lo, 7=polynomial, 8=multi-point)
If CalcImageOldSample(1).BackgroundTypes%(i%) = 0 And CalcImageOldSample(1).OffPeakCorrectionTypes(i%) < MAXOFFBGDTYPES% Then
If ihi% > 0 And ilo% = 0 Then
If CalcImageOldSample(1).OffPeakCorrectionTypes%(i%) <> 2 And CalcImageOldSample(1).OffPeakCorrectionTypes%(i%) <> 5 Then
CalcImageOldSample(1).OffPeakCorrectionTypes%(i%) = 2     ' high side only
If Not OffPeakMissingWarned(i%) Then
msg$ = "Only high side off-peak background images were found for " & CalcImageOldSample(1).Elsyms$(i%) & " " & CalcImageOldSample(1).Xrsyms$(i%) & " on spectormeter " & Format$(CalcImageOldSample(1).MotorNumbers%(i%)) & ". Consider changing to a slope high background fit." & vbCrLf & vbCrLf & "Changing off-peak fit to high side only."
MiscMsgBoxTim FormMSGBOXTIME, "CalcImageCorrectData", msg$, 20#
'MsgBox msg$, vbOKOnly + vbExclamation, "CalcImageCorrectData"
OffPeakMissingWarned(i%) = True
End If
End If
End If

If ihi% = 0 And ilo% > 0 Then
If CalcImageOldSample(1).OffPeakCorrectionTypes%(i%) <> 3 And CalcImageOldSample(1).OffPeakCorrectionTypes%(i%) <> 6 Then
CalcImageOldSample(1).OffPeakCorrectionTypes%(i%) = 3     ' low side only
If Not OffPeakMissingWarned(i%) Then
msg$ = "Only low side off-peak background images were found for " & CalcImageOldSample(1).Elsyms$(i%) & " " & CalcImageOldSample(1).Xrsyms$(i%) & " on spectormeter " & Format$(CalcImageOldSample(1).MotorNumbers%(i%)) & ". Consider changing to a slope low background fit." & vbCrLf & vbCrLf & "Changing off-peak fit to low side only."
MiscMsgBoxTim FormMSGBOXTIME, "CalcImageCorrectData", msg$, 20#
'MsgBox msg$, vbOKOnly + vbExclamation, "CalcImageCorrectData"
OffPeakMissingWarned(i%) = True
End If
End If
End If

' If both off-peak images are missing and UseMANForOffPeakElementsFlag is set, the background type will be set to MAN in DataCorrectData when the sample setup is loaded (not here!)
'If ihi% = 0 And ilo% = 0 And UseMANForOffPeakElementsFlag Then CalcImageOldSample(1).OffPeakCorrectionTypes%(i%) = 1    ' set to MAN (0=off-peak, 1=MAN, 2=multipoint)
End If

End If
Next i%

' Load MPB (off-peak shared only!) intensities  (0=linear, 1=average, 2=high only, 3=low only, 4=exponential, 5=slope hi, 6=slope lo, 7=polynomial, 8=multi-point)
Call CalcImageCorrectDataMPB(ipixelx&, ipixely&, npixelx&, npixely&)
If ierror Then Exit Sub

' Correct on peak intensities for background (only one data row for image pixels)
For i% = 1 To CalcImageOldSample(1).LastElm%
If CalcImageOldSample(1).CrystalNames$(i%) <> EDS_CRYSTAL$ Then                             ' skip EDS spectra channels (already in net intensity)
CalcImageOldSample(1).BgdData!(1, i%) = 0#
CalcImageOldSample(1).CorData!(1, i%) = CalcImageOldSample(1).OnPeakCounts!(1, i%)

' Load count x-ray map beam currents (for possible detection limit calculations)
ip% = CalcImageIndex(Int(0), CalcImageNumberofImageFiles%, i%, CalcImageElementRows%(), CalcImageImageTypes%())
If ip% > 0 Then
CalcImageOldSample(1).OnBeamData!(1, i%) = CalcImageBeamCurrents!(ip%)                      ' note expanded dimension in case used for aggregate intensity correction
CalcImageOldSample(1).OnBeamDataArray!(1, i%) = CalcImageBeamCurrents!(ip%)
End If

' Perform off-peak correction if not MAN (0=off-peak, 1=MAN, 2=multipoint)
If CalcImageOldSample(1).BackgroundTypes%(i%) <> 1 Then                                     ' background type is set to MAN in DataCorrectData if UseMANForOffPeakElementsFlag is true
Call DataCorrectDataBackground(Int(1), i%, CalcImageOldSample())                            ' row is always 1 for image pixels
If ierror Then Exit Sub
End If

' If EDS element, just load with background correction (already is net intensity)
Else
CalcImageOldSample(1).BgdData!(1, i%) = 0#
CalcImageOldSample(1).CorData!(1, i%) = CalcImageOldSample(1).OnPeakCounts!(1, i%)
End If
Next i%

Exit Sub

' Errors
CalcImageCorrectDataError:
MsgBox Error$, vbOKOnly + vbCritical, "CalcImageCorrectData"
ierror = True
Exit Sub

End Sub

Note that intensity data for each pixel (WDS and EDS) are already in raw cps.

I've been doing quant EDS mapping alongside quant WDS (and quant SXES) on our 8530F (old JEOL EDS and SXES-LR) for a while, I gave a presentation on it at EMAS in Mataro last year, but the work wasn't included in the conference proceedings so I've been looking for somewhere else to publish it. It actually works really quite well! This thread has given me the push to crack on and get the same sorted for our iHP200F, which has the new MEC EDS interface and the SXES-ER.

That would be interesting, but sounds like the iHP200F JEOL data cube is different among other things...

I've gone about dealing with the EDS in a very different way from what Ben is doing with NeXL, so I'm very interested to see where that goes. 

The problem I see with Ben's NeXL method for quantification in CalcImage is that standards for use in CalcImage are acquired in Probe for EPMA and these net intenities are extracted using the JEOL net intensity function.

So I'm not sure these two different net intensity methods will give commensurate results.

[Nicholas Ritchie]

John, you see quant from a WDS-centric perspective (as well you ought.)  Net counts, the coin of the realm in WDS, are not the coin of the realm in EDS.  We work with spectra.  We fit reference spectra to measurand spectra to extract k-ratios directly.  k-ratios literally are the linear least squares fit coefficients that emerge when we fit references to the unknown.  We never need "net counts" in the process.  In theory, we can calculate net counts from the k-ratios but rarely is this useful.  As you frequently point out, we care about k-ratios.

Your approach to quantifying EDS spectra (using Net Counts) forces EDS into a WDS-centric framework.  This may be optimal for your product but (IMHO) isn't the best for an EDS-centric product.  (For one, it is over-optimistic about the count statistics uncertainty estimates.)

NeXL never uses the net intensities you report from the EDS vendor software.  It uses the spectra collected by the vendor's software.  It fits spectra to spectra.  The only challenge is that frequently EDS vendors use separate binning logic to record spectrum images and point spectra.  The energy axes frequently are not immediately commensurate. There are couple of different ways to handle this problem but it does require a little thought.

[John Donovan]

John, you see quant from a WDS-centric perspective (as well you ought.)  Net counts, the coin of the realm in WDS, are not the coin of the realm in EDS.  We work with spectra.  We fit reference spectra to measurand spectra to extract k-ratios directly.  k-ratios literally are the linear least squares fit coefficients that emerge when we fit references to the unknown.  We never need "net counts" in the process.  In theory, we can calculate net counts from the k-ratios but rarely is this useful.  As you frequently point out, we care about k-ratios.

Your approach to quantifying EDS spectra (using Net Counts) forces EDS into a WDS-centric framework.  This may be optimal for your product but (IMHO) isn't the best for an EDS-centric product.  (For one, it is over-optimistic about the count statistics uncertainty estimates.)

NeXL never uses the net intensities you report from the EDS vendor software.  It uses the spectra collected by the vendor's software.  It fits spectra to spectra.  The only challenge is that frequently EDS vendors use separate binning logic to record spectrum images and point spectra.  The energy axes frequently are not immediately commensurate. There are couple of different ways to handle this problem but it does require a little thought.

I was mostly making a funny. It seems like 6 or half a dozen of the other to me. Multiply by live time to get raw counts (for calculating statistics), or divide by live time to get cps.

But thanks for adding support for cps EMSA files.

[Ben Buse]

Ok in hyperspy I realised you can add plugins (python scripts to menus) (although it's python editor is picky), so here's showing the installed plugin for reading the PTS (based on Yuji) and saving a rpl file for NeXL and a live image map as csv (also used in NeXL for further processing)

Plugin loaded into menu


Displays data within hyperspy

Figure 3 is the live time map, where you see in the crack the live time = real time. The other two are the map and the spectrum for a pixel, which you can select on map

Save the files


Yuji or others, does hyperspy have a depository for sharing plugins?

[Ben Buse]

Great thing is julia can be called from python, so hoping to run entire process as one python script which can be loaded as plugin into hyperspy, giving .grd files and .CIP file for opening in calcimage. And also add spatial and energy calibration to plots shown above

[Ben Buse]

Ok here's the hyperspy plugin to convert PTS file for reading in CalcImage along with WDS images converted with ProbeImage, generating CIP file for easy opening in CalcImage ready to quantify map.

Thank you Nicholas for your help with NeXL and Yuji for your help with PTS file

Instructions to use are

Prior to use
Copy 'jeoltocalcimage.py' to your hyperspy plugin folder - In hyperspy: see Settings menu - Plugin Manager - to find folder location
In hyperspy console install required packages

Hyperspy plugin instructions
1.    Copy .PTS and .MDB to same folder
2.    Copy one of the WDS .cnd files to same folder as .PTS and .MDB
3.    Using Probe Image convert JEOL (WDS) maps to PrbImg format
4.    Using CalcImage convert above PrbImg maps to grd maps
5.    Copy above grd maps to same folder as .PTS and .MDB files
6.    In PFE export spectra to emsa (this will provide standard spectra to NeXL; standards with interfering elements are rejected)
7.    Run hyperspy plugin
8.    In Calcimage open the CIP file – open old project
9.    Proceed as normal

And latest example images

[Probeman]

This is very impressive Ben.

Which elements were WDS and which were EDS?

Also can we see a totals map?

john

[Ben Buse]

Hi John,

I had done it with H by diff, as I expected some hydrous phases, and the quartz is slightly high because I was using an Olivine standard for Si

[Ben Buse]

Ok I've done some comparisons with extraction from map and CAMECA analyses from same material but not same sample, and also have the 2 quant points from PFE.

[Probeman]

Ok I've done some comparisons with extraction from map and CAMECA analyses from same material but not same sample, and also have the 2 quant points from PFE.

I see, so P and Fe by EDS.  Well your P is pretty close to zero and also H by diff is close to zero...

As you know, the best way to test the accuracy of your EDS maps is to perform such WDS and EDS on a synthetic mineral such as Mg2SiO4 or MgAl2O4 (using pure oxides as primary standards) or whatever pure and stoichiometric synthetics you have. It will be interesting to see the accuracy of both the majors (and total) and a few traces (which should be zero) for a number of EDS mapped elements.