NeXL

[Nicholas Ritchie]

There is a recalibrate function that might work to change the energy offset.

Code Select Expand

"""
    recalibrate(s::Spectrum{T}, es::LinearEnergyScale)

Allows changing the energy scale on a spectrum from one LinearEnergyScale to another as though the spectrum were
measured on a different detector.  The algorithm uses a FFT-base scheme to rescale and shift the spectral data.
This scheme allows for fractional shifts of offset and fractional changes in the width.  It is limited in that
the change in width must produce an integral number of channels in the resulting spectrum.  The algorithm
maintains the total spectrum integral so the new spectrum can be used for quantitative purposes.
Plotting one spectrum over the other should maintain peak position but is likely to change the channel counts.
"""
function recalibrate(s::Spectrum{T}, es::LinearEnergyScale)

Example:

Code Select Expand

les = LinearEnergyScale(3.0, 10.1)
spec = loadspectrum("\path\to\spectrum\spec.msa")
new_spec = recalibrate(spec, les)


[Nicholas Ritchie]

It could be the 128 should be 128.0 (float instead of int) in

Code Select Expand

references([reference(n"Al",MapSum,mat"Ca3Mn57Al40Si60O240")],128)


[Ben Buse]

Hi Nicholas, thanks I'll give that a go. I've also removed the energy shift from the map dataset at source

Code Select Expand

https://smf.probesoftware.com/index.php?msg=14193.
And now have a quant map

Code Select Expand

refs3 = references([reference(n"Mn",joinpath(path12,"2406SPCtakeoff.emsa"),mat"Mn"),reference(n"Al",joinpath(path13,"2406SPCtakeoff.emsa"),mat"Al2O3"),reference(n"Mg",joinpath(path14,"2406SPCtakeoff.emsa"),mat"MgO"),reference(n"Fe",joinpath(path15,"2406SPCtakeoff.emsa"),mat"Fe2O3"),reference(n"Si",joinpath(path16,"2406SPCtakeoff.emsa"),mat"Si59Mg107Fe11O240"),reference(n"O",joinpath(path13,"2406SPCtakeoff.emsa"),mat"Al2O3"),],130)
hs7 = NeXLSpectrum.compress(HyperSpectrum(
                                 LinearEnergyScale(0.0,10.0),
                                Dict{Symbol,Any}(
                                  :TakeOffAngle => deg2rad(40.0),
                                                                          :ProbeCurrent => 9.993,
                                  :LiveTime => 0.372575,
                                  :BeamEnergy => 15.0e3,
                                  :Name => "spess"
                                ),
                                readrplraw(raw"C:\\Users\\glxbb\\julia\\spess"),
                                           fov = [ 60u"mm", 60u"mm"], offset= [ 0.0u"mm", 0.0u"mm" ]
                            ))
resmap7 = fit_spectrum(hs7,refs3)
bestresmap7 = optimizeks(SimpleKRatioOptimizer(2.0),resmap7)
resmap7q = quantify(resmap7)

where quant for first pixel gives

Code Select Expand

resmap7q[1,1]
Map[1,1][O=0.4264,Al=0.1058,Si=0.1672,Mn=0.3032,Fe=0.0024]


[Ben Buse]

I just now can't figure out how to export the quant map as csv for each element

Code Select Expand

resmap7q
Materials[Map, 60 × 60 × (O, Mg, Al, Si, Mn, Fe)]


Code Select Expand

asa(DataFrame,resmap7q)
ERROR: MethodError: no method matching asa(::Type{DataFrame}, ::Materials{Float32, Float64, 2})
The function `asa` exists, but no method is defined for this combination of argument types.

So the array of data for 1st element is

Code Select Expand

resmap7q.massfractions[:,:,1]But I can't see to use this with asa dataframe or CSV write

[Ben Buse]

You can specify per pixel live-times by adding the `livetime` argument to the HyperSpectrum constructor assigning it a matrix the same dimensions as the RPL/RAW data. (Remove the `:LiveTime => lt,`)

Code Select Expand

    readrplraw(....),
    livetime=fill(get(props, :LiveTime, 1.0), (512, 512))
)


Ok I've saved Live time data in hyperspy as csv text image

And loaded the data in Julia

Code Select Expand

testprop3 = CSV.read("LiveSpess.csv",Tables.matrix,header=0)
livems=testprop2*10
livesec=livems/1000

Array is

Code Select Expand

60×60 Matrix{Float64}
But trying to load array into live time gives an error

Code Select Expand

fill(get(livesecs, :LiveTime, 1), (60, 60))
ERROR: MethodError: no method matching get(::Matrix{Float64}, ::Symbol, ::Int64)
The function `get` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  get(::Base.TTY, ::Symbol, ::Any)
   @ Base terminfo.jl:394
  get(::Base.TermInfo, ::Symbol, ::Int64)
   @ Base terminfo.jl:218
  get(::Pkg.Types.Manifest, ::Any, ::Any)


[Nicholas Ritchie]

This should work to load the live-times.

Code Select Expand

hs7 = NeXLSpectrum.compress(HyperSpectrum(
                                 LinearEnergyScale(0.0,10.0),
                                Dict{Symbol,Any}(
                                  :TakeOffAngle => deg2rad(40.0),
                                                                          :ProbeCurrent => 9.993,
                                  :LiveTime => 0.372575,
                                  :BeamEnergy => 15.0e3,
                                  :Name => "spess"
                                ),
                                readrplraw(raw"C:\\Users\\glxbb\\julia\\spess"),
                                fov = [ 60u"mm", 60u"mm"], offset= [ 0.0u"mm", 0.0u"mm" ],
                                livetime = CSV.read("LiveSpess.csv",Tables.matrix,header=0)

   ))

I'll have to do a little work to figure out how to export the quant maps as a CSV.  It shouldn't be hard.  Usually, I export the data in a flat form since this is frequently easier to work with. 

Headers:  "Row, Col, O, Mg, Al, Si, Mn, Fe"

It can be shaped using DataFrames Split-Apply-Combine

[Ben Buse]

Thanks will give it go, I've just figured out export to csv

Code Select Expand

using DelimitedFiles
writedlm("TestResult.csv", resmap7q.massfractions[:,:,1],',')
How do you produce quant images?

[Ben Buse]

Hi Nicholas, great thanks live time corrected now good

Code Select Expand

hs8 = NeXLSpectrum.compress(HyperSpectrum(
                                               LinearEnergyScale(0.0,10.0),
                                              Dict{Symbol,Any}(
                                                                                :TakeOffAngle => deg2rad(40.0),
                                                                                                                                                                 :ProbeCurrent => 9.993,
                                                                                  :BeamEnergy => 15.0e3,
                                                :Name => "spess"
                                              ),
                                              readrplraw(raw"C:\\Users\\glxbb\\julia\\spess"),
                                                                                         livetime = livesec,fov = [ 60u"mm", 60u"mm"], offset= [ 0.0u"mm", 0.0u"mm" ]
                                          ))
 resmap8 = fit_spectrum(hs8,refs3)
bestresmap8 = optimizeks(SimpleKRatioOptimizer(2.0),resmap8)
resmap8q = quantify(resmap8)

And here's a couple of pixel comparisons using the average live time for the whole map and the pixel per pixel live time, it's a homogeneous area.

Code Select Expand

resmap7q[1,1]
Map[1,1][O=0.4264,Al=0.1058,Si=0.1672,Mn=0.3032,Fe=0.0024]

julia> resmap8q[1,1]
Map[1,1][O=0.4294,Al=0.1065,Si=0.1684,Mn=0.3053,Fe=0.0024]

julia> resmap7q[1,2]
Map[1,2][O=0.3456,Mg=0.0026,Al=0.1120,Si=0.1671,Mn=0.3031]

julia> resmap8q[1,2]
Map[1,2][O=0.3389,Mg=0.0026,Al=0.1098,Si=0.1639,Mn=0.2972]


[Nicholas Ritchie]

This should do what you want...

Code Select Expand

function NeXLUncertainties.asa(::Type{DataFrame}, mat::Materials;  uncertainties = false)
    function wu(v)
        isa(v, UncertainValue) ? (uncertainties ? uncertainty(v) : value(v)) : (uncertainties ? NaN64 : v)
    end
    ci = CartesianIndices(mat)
    elms = sort(collect(keys(mat)))
    res = DataFrame(
        "Row" => reshape(collect(map(ci->ci[1], ci)),(:)),
        "Col" => reshape(collect(map(ci->ci[2],ci)),(:)),
        map(elm->symbol(elm)=>collect(reshape(map(ci->wu(mat[ci][elm]),ci),(:))),elms)...
    )
end


[Nicholas Ritchie]

Code Select Expand

using Images
function grayscale(mats::Materials, elm::Element)
    vof(v::UncertainValue) = value(v)
    vof(v) = v
    Gray.(N0f8.(map(CartesianIndices(mats)) do ci
        clamp(vof(mats[ci][elm]),0.0,1.0)
    end))
end

function colorize(mats::Materials, elms::AbstractArray{Element})
    colori(i) = length(elms)>=i && haskey(mats.planes, elms[i]) ? grayscale(mats, elms[i]) :  zeros(N0f8, size(mats))
    colorview(RGB, colori(1), colori(2), colori(3))
end

This will display in a Jupyter notebook as an image.

[Ben Buse]

Thanks Nicholas will give a try.

With refs I have error

Code Select Expand

[ Info: A material containing Fe and O is not a suitable reference for "Fe L3-M5 + 13 others" due to a peak interference.

Is it possible to have a separate reference for Fe L from Fe K, like in DTSA2 for shape only?

To summarise where at: I've quantified maps using either emsa from tiny maps of each standard as refs, or using the spectra within PFE and it's emsa export function. For PFE spectra once loaded into julia I multiplied by live time to give counts rather than counts per sec.

Code Select Expand

sPFE_Al2O3 = sPFE*sPFE[:LiveTime]PFE emsa already contain take off angle. I also have the option of saving the kratio maps and loading into calcimage to combine with wds signal, provided I recast the kratio maps as unknown intensities. Conversely could PFE kratios maps for WDS, by combined with kratio EDS maps in NeXL for quantification.

And to check/compare map quantification and kratios median seems a good function where map a single phase

Code Select Expand

using Statistics
median(bestresmap8[1].kratios)
median(resmap8q.massfractions[:,:,3])
And to output kratio and quant maps as csv

Code Select Expand

for i in 1:length(bestresmapPFE2)
           writedlm(string(bestresmapPFE2[i].xrays)*"_kratios.csv",bestresmapPFE2[i].kratios,',')
       end
for i in 1:length(bestresmapPFE2)
           writedlm(string(bestresmapPFE2[i].xrays)*"_quant.csv",resmapPFE2q.massfractions[:,:,i],',')
       end


[Nicholas Ritchie]

Do I understand you correctly, that PFE exports EMSA files with the values in "Counts/s" rather than "Counts"?  Reading through the standard, I didn't see anything that forbids this but it would be a novel interpretation of the standard.  (In which case, I should add support for it to NeXL and DTSA-II.)  Would you post an example PFE EMSA file to the forum for me to reverse engineer?

If this is the case, it would be acceptable to say:

Code Select Expand

sPFE[:LiveTime] = 1.0
which would be equivalent to

Code Select Expand

sPFE[:LiveTime]=measured_live_time
sPFE_Al2o3=sPFE*sPFE[:LiveTime]

The Delimited files package solution to writing CSV files seems like a good one.

[Ben Buse]

Do I understand you correctly, that PFE exports EMSA files with the values in "Counts/s" rather than "Counts"?  Reading through the standard, I didn't see anything that forbids this but it would be a novel interpretation of the standard.  (In which case, I should add support for it to NeXL and DTSA-II.)  Would you post an example PFE EMSA file to the forum for me to reverse engineer?

Yes I've attached a file, and you see it has

Code Select Expand

#YUNITS      : cps

[Ben Buse]

So here's the NeXL maps quantified in CalcImage (with exception of Ca which is from hyperspy because I forgot to include it). Also attached square analyses



Interestingly NeXL appears to do a better job of Fe-Mn interference than the JEOL net intensity, JEOL net intensity giving ca. 1 wt % Fe, whereas NeXL has ca. 0.247 wt %. The PFE spot analyses using the JEOL net intensity are, with comparison WDS analysis in Standard.



Here because of the collimator on the JEOL EDS (with varying size apertures) we can analysis WDS trace elements at 1uA, whilst the concurrent EDS is running at 7% dead time for major elements

To achieve this the kratios where exported from NeXL to csv as shown above. Then in python recast to unknown intensity and grid files created.

Code Select Expand

import csv
import tkinter
from tkinter import filedialog
import numpy as np
import struct
import os
import glob
import re
import pandas as pd
mapbeamcurrent = 997.9
#maplivetime = 1319.49
#livetimepixel = 0.366525
#livetimepixel = maplivetime/(60*60)
#load info from wds cnd file
cndfile = tkinter.filedialog.askopenfile(title="Select WDS element .cnd file")
reader = csv.reader(open(cndfile.name), delimiter=" ")
d = list(reader)
Pos2_X1 = d[113][1]
Pos2_Y1 = d[113][2]
Pos2_Z1 = d[113][3]
Pos3_X2 = d[115][1]
Pos3_Y2 = d[115][2]
Pos3_Z2 = d[115][3]
ProjectName = ' '.join(d[29][1:len(d[29])])
#load each of the net intensity csv maps produced by hyperspy
stdintfile = tkinter.filedialog.askopenfile(title="Select exported PFE unknown .dat file containing net inten and kraw")
df = pd.read_csv(stdintfile,sep='\t')

path = filedialog.askdirectory(title="Select location of csv kratio map files")
directories = path.split('/')
for filename in glob.glob(os.path.join(path,'*kratios.csv')):
    with open(filename) as f:
        reader = csv.reader(f)
        #csvfile = tkinter.filedialog.askopenfile(title="Select map .csv file")
        #reader = csv.reader(csvfile)
        ImageList = list(reader)
        NumList = np.array(ImageList)
        NumFloat=NumList.astype(float)
        #NumFloat=NumFloat/livetimepixel/beamcurrent
        #NumFloat=NumFloat/livetimepixel
        #NumFloat from kratio to cps - using standard cps
        elementfile=re.sub("[]'[]",'',filename.split('\\')[1]).split('.csv')[0]
        print(filename)
        Element = elementfile.split(' ')[0]
        ElLine = elementfile.split(' ')[1]
        try:
            #df.filter(regex='EDS').filter(regex='NECNT').filter(regex='Al ka').mean()*100
            NECNT = df.filter(regex='EDS').filter(regex='NECNT').filter(regex=Element+' '+ElLine[:1].lower()+'a').mean()
            Kraw = df.filter(regex='EDS').filter(regex='K-RAW').filter(regex=Element+' '+ElLine[:1].lower()+'a').mean()
            StdInt = (1/Kraw.iloc[0])*NECNT.iloc[0]*mapbeamcurrent
            NumFloat = NumFloat*StdInt
            thead_tag = 1112691524
            thead_size = 4
            tGrid_size =72
            tGrid_ncol=NumFloat.shape[1]
            tGrid_nrow=NumFloat.shape[0]
            tGrid_xLL=Pos3_X2
            tGrid_yLL=Pos2_Y1
            tGrid_zmin=NumFloat.min()
            tGrid_zmax=NumFloat.max()
            tGrid_rotation=0
            tGrid_BlankValue=1.71041e38
            tData_tag=1096040772
            tData_size=tGrid_nrow * tGrid_ncol * 8
            tData_hData=NumFloat
            thead_version=1
            tGrid_tag=1145655879
            part1=(int(thead_tag),int(thead_size),int(thead_version),int(tGrid_tag),int(tGrid_size),int(tGrid_ncol),int(tGrid_nrow))
            tGrid_xsize=1/1000
            tGrid_ysize=1/1000
            tGrid_rotation=0
            tGrid_blank=1.71041e38
            part2=(float(tGrid_xLL),float(tGrid_yLL),float(tGrid_xsize),float(tGrid_ysize),float(tGrid_zmin),float(tGrid_zmax),float(tGrid_rotation),float(tGrid_blank))
            part3=(int(tData_tag),int(tData_size))
            part4=NumFloat
            fp = open(path + '/' +ProjectName+'_00001_WDS0_'+Element+'_'+'EDS'+'.grd','wb')
            fp.write(struct.pack('<iiiiiii',part1[0],part1[1],part1[2],part1[3],part1[4],part1[5],part1[6]))
            fp.write(struct.pack('<dddddddd',part2[0],part2[1],part2[2],part2[3],part2[4],part2[5],part2[6],part2[7]))
            fp.write(struct.pack('<ii',part3[0],part3[1]))
            fp.write(NumFloat.astype('<d').tobytes())
            fp.close()
        except:
            print("An error occured, element may not be present in PFE file")


[John Donovan]

Interestingly NeXL appears to do a better job of Fe-Mn interference than the JEOL net intensity, JEOL net intensity giving ca. 1 wt % Fe, whereas NeXL has ca. 0.247 wt %. The PFE spot analyses using the JEOL net intensity are, with comparison WDS analysis in Standard.

Note that PFE allows one to correct for spectral interferences between WDS and EDS elements.