EDD subpackage (CHAP.edd)
The EDD subpackage contains the modules that are unique to Energy Dispersive Diffraction (EDD) data processing workflows. This document describes how to run an detector energy calibration and strain analysis workflow in a Linux terminal.
Processing the data
A standard strain analysis in CHAP consists of three steps:
Performing the detector channel energy calibration. This is typically performed by fitting a set of fluorescence peak centers in an EDD experiment on a CeO2 sample and comparing the results to their known energy values.
Fine tuning the detector channel energy calibration (and optionally the takeoff angle \(2\theta\)) by fitting a set of Bragg peak centers in an EDD experiment on typically the same CeO2 sample and comparing the results to their known energy values for a given channel energy calibration and \(2\theta\) value.
Performing the strain analysis with an EDD experiment on a sample using the calibrated detector channel energies.
Running an EDD workflow on the CHESS Linux system
Navigate to your work directory.
Create the required CHAP pipeline file for the workflow (see below) and any additional workflow specific input files.
Run the workflow using the latest production release version:
$ /nfs/chess/sw/CHESS-software-releases/prod/CHAP_edd <pipelinefilename>
or the latest development release version:
$ /nfs/chess/sw/CHESS-software-releases/dev/CHAP_edd <pipelinefilename>
You may find it convenient to add an alias to your
~/.bascrcor~/.bash_aliases, for example for the CHAP EDD production release:alias CHAP_edd_prod='/nfs/chess/sw/CHESS-software-releases/prod/CHAP_edd'
(see: instructions on running
CHAPon the CHESS Linux system)Respond to any prompts that pop up if running interactively.
Running an EDD workflow on any Linux system (requires a local Conda environment and CHAP clone)
Create a base Conda environent and clone the
CHAPrepository according to steps 1 and 2 of the Conda installation instructions.Activate the base Conda environment:
$ source <path_to_CHAP_clone_dir>/bin/activate
Create the EDD conda environment:
(base) $ mamba env create -f <path_to_CHAP_clone_dir>/CHAP/edd/environment.yml
Activate the
CHAP_eddenvironment:(base) $ conda activate CHAP_edd
Run the workflow using your own
CHAP_eddconda environment:(CHAP_edd) $ CHAP <pipelinefilename>
Inspecting output
The output consists of a single NeXus (.nxs) file containing the strain analysis data as well as all metadata pertaining to the analysis. Additionally, optional output figures (.png) may be saved to an output directory specified in the pipeline file.
The optional output figures can be viewed directly by any PNG image viewer. The data in the NeXus output file can be viewed in NeXpy, a high-level python interface to HDF5 files, particularly those stored as NeXus data:
Open the NeXpy GUI by entering in your terminal:
$ /nfs/chess/sw/nexpy/anaconda/envs/nexpy/bin/nexpy &
You may find it convenient to add an alias to your
~/.bascrcor~/.bash_aliases:alias nexus='/nfs/chess/sw/nexpy/anaconda/envs/nexpy/bin/nexpy &'
After the GUI pops up, click File-> Open to navigate to the folder where your output
.nxsfile was saved, and select it.Navigate the filetree in the “NeXus Data” panel to inspect any output or metadata field.
Creating the pipeline file
Create a workflow pipeline.yaml file according to the CHAP pipeline instructions. A generic pipeline input file for an energy calibration and strain analysis workflow is as follows (note that spaces and indentation are important in .yaml files):
config:
root: . # Change as desired
inputdir: . # Change as desired
# Path can be relative to root (line 2) or absolute
outputdir: output # Change as desired
# Path can be relative to root (line 2) or absolute
interactive: true # Change as desired
log_level: info # Set to debug, info, warning, or error
energy:
# Energy calibration
- common.SpecReader:
config:
station: id1a3 # Change as needed
experiment_type: EDD
spec_scans: # Edit both SPEC log file path and EDD scan numbers
# Path can be relative to inputdir (line 3) or absolute
- spec_file: <your_raw_ceria_data_directory>/spec.log
scan_numbers: 1
- edd.MCAEnergyCalibrationProcessor:
config:
max_peak_index: 1 # Index in `peak_energies` with the highest peak amplitude
peak_energies: [34.276, 34.717, 39.255, 40.231]
materials: # Optional, using default CeO2 properties when omitted
- material_name: CeO2
sgnum: 225
lattice_parameters: 5.41153
detector_config:
baseline: true
mask_ranges: [[650, 850]]
detectors: # Choose the detectors
# Use all available detector elements when omitted
- id: 0
- id: 11
- id: 22
save_figures: true
schema: edd.models.MCAEnergyCalibrationConfig
- common.YAMLWriter:
filename: energy_calibration_result.yaml
# Energy calibration output filename, change as desired
force_overwrite: true
- common.ImageWriter:
outputdir: figures # Change as desired, unless an absolute path
# this will appear under 'outdutdir' (line 5)
force_overwrite: true # Do not set to false!
# Rename an existing file if you want to prevent
# it from being overwritten
twotheta:
# Twotheta calibration
- common.YAMLReader:
filename: energy_calibration_result.yaml
# Energy calibration filename, same as written to above
schema: edd.models.MCAEnergyCalibrationConfig
- common.SpecReader:
config:
station: id1a3 # Change as needed
experiment_type: EDD
spec_scans: # Edit both SPEC log file path and EDD scan numbers
# Path can be relative to inputdir (line 3) or absolute
- spec_file: <your_raw_ceria_data_directory>/spec.log
scan_numbers: 1
- edd.MCATthCalibrationProcessor:
config:
tth_initial_guess: 5.2 # Set to the initial tth angle from detector setup
detector_config:
energy_mask_ranges: [[65, 155]] # Change as needed
detectors: # The same as in the energy calibration when omitted
- id: 0
- id: 11
- id: 22
save_figures: true
- common.YAMLWriter:
filename: tth_calibration_result.yaml
# Twotheta calibration output filename, change as desired
force_overwrite: true
- common.ImageWriter:
outputdir: figures # Change as desired
force_overwrite: true # Do not set to false!
map:
# Create a CHESS style map
- edd.EddMapReader:
filename: <your_raw_data_directory>/<parfile_name> # The par-file name
dataset_id: 1 # The dataset ID, change as needed
schema: common.models.map.MapConfig
- common.MapProcessor:
detector_config:
detectors: # Use available detector elements when omitted
- id: 0
- id: 11
- id: 22
- common.NexusWriter:
filename: map.nxs # NeXus map output filename
force_overwrite: true
strain:
# Perform the strain analysis
- common.NexusReader:
filename: map.nxs # NeXus map, same file as written to above
- common.YAMLReader:
filename: tth_calibration_result.yaml
# Twotheta calibration filename, same as written to above
schema: edd.models.MCATthCalibrationConfig
- edd.StrainAnalysisProcessor:
config:
find_peak_cutoff: 0.02 # Change as desired
materials:
- material_name: Al
sgnum: 225
lattice_parameters: 4.046
rel_height_cutoff: 0.02 # Change as desired
skip_animation: false
detector_config:
energy_mask_ranges: [[75, 140]] # Change as desired
detectors: # Use available detector elements when omitted
- id: 0
- id: 11
- id: 22
save_figures: true
- common.NexusWriter:
# Write the strain analysis result to file
filename: strain.nxs # The NeXus output filename, change as desired
force_overwrite: true # Do not set to false!
- common.ImageWriter:
# Write the figures to file
outputdir: figures # Change as desired
force_overwrite: true # Do not set to false!
- common.ImageWriter:
# Write the animation to file, omit when skip_animation is true
outputdir: figures # Change as desired
force_overwrite: true # Do not set to false!
The “config” block defines the CHAP generic configuration parameters:
root: The work directory, defaults to the current directory (whereCHAP <pipelinefilename>is executed). Must be an absolute path or relative to the current directory.inputdir: The default directory for files read by any CHAP reader (must have read access), defaults toroot. Must be an absolute path or relative toroot.outputdir: The default directory for files written by any CHAP writter (must have write access, will be created if not existing), defaults toroot. Must be an absolute path or relative toroot.interactive: Allows for user interactions, defaults toFalse.log_level: The Python logging level.
The remainder of the file contains the actual workflow pipeline, in this example it consists of four blocks, energy, twotheta, map, and strain, which can be executed individually or all at once as described here. In addition to the readers and writers of the intermediate results, nine toplevel processes get executed successively in the combined four pipeline blocks:
The EDD/XRF energy calibration consists of two processes:
common.SpecReader: A reader that reads the raw detector calibration data.edd.MCAEnergyCalibrationProcessor: A processor that performs the detector channel energy calibration.
The EDD \(2\theta\) calibration consists of two processes:
common.SpecReader: A reader that reads the raw detector calibration data.edd.MCATthCalibrationProcessor: A processor that performs the \(2\theta\) calibration.
The following two processors read the strain analysis sample’s raw detector data and create a CHESS style map:
edd.EddMapReader: A reader that reads the sample’s raw detector data using a CHESS style experiment par-file.common.MapProcessor: A processor that creates a CHESS style map for the raw detector data.
The last three processors perform the actual strain analysis and write the output to a Nexus file:
edd.StrainAnalysisProcessor: A processor that perfroms the actual strain analysis and creates a single Nexus object with the strain analysis results as well as all metadata pertaining to the workflow.common.NexusWriter: A wrtiter that writes the strain analysis results to a NeXus file.common.ImageWriter: A wrtiter that writes a (set of) figure(s) or an animation to file (omit whenskip_animationis set totrue`).
Note that the energy calibration can also be obtained ahead of time and used for multiple strain analyses. In this case remove the first two blocks in the pipeline and read the detector channel energy/\(2\theta\) calibration info in what is now the third block in the pipeline, labelled map.
Creating the CHESS stype strain analysis map without a par-file
The above strain analysis workflow example can also be executed without availablity of a CHESS style experiment par-file, by reading the raw data information directly from the SPEC log file. In this case the map construction part description of the workflow in the pipeline.yaml file must be replaced by the following:
map:
# Create a CHESS style map
- common.MapProcessor:
config:
title: <experiment title> # Enter an appropriate name (like the BTR)
station: id1a3
experiment_type: EDD
sample:
name: <sample name> # Enter the sample name
description: ''
spec_scans: # Edit both SPEC log file path and EDD scan numbers
# Path can be relative to inputdir or absolute
- spec_file: <your_raw_data_directory>/spec.log
scan_numbers: 1-4
scalar_data: # Add or modify as appropriate
- label: SCAN_N
units: n/a
data_type: smb_par
name: SCAN_N # Change as needed
independent_dimensions:
# Add or modify as appropriate
- label: labx
units: mm
data_type: smb_par
name: labx # Change as needed
- label: laby
units: mm
data_type: smb_par
name: laby # Change as needed
- label: labz
units: mm
data_type: smb_par
name: labz # Change as needed
presample_intensity:
label: presample_intensity
units: counts
data_type: scan_column
name: a3ic1 # Change as needed
dwell_time_actual:
label: dwell_time_actual
units: s
data_type: scan_column
name: sec # Change as needed
postsample_intensity:
label: postsample_intensity
units: counts
data_type: scan_column
name: diode # Change as needed
detector_config:
detectors: # Use available detector elements when omitted
- id: 0
- id: 11
- id: 22
- common.NexusWriter:
filename: map.nxs # NeXus map output filename
force_overwrite: true
Specifying the detector configuration in the EDD workflow
Selecting detectors for detector calibration
In the example above, the raw data for both the energy and the \(2\theta\) calibration is read by the common.SpecReader pipeline item, while which detectors to include in the calibration is specified in the detectors field under detector_config in the edd.MCAEnergyCalibrationProcessor and edd.MCATthCalibrationProcessor pipeline items, respectively.
However, this is not the only way to specify the detectors to use. As can be seen from the documentation for SpecReader, the detectors for which raw data is read can also be specified by adding a detector_config field to the SpecReader. For example, by using the following in the pipeline file for the EDD workflow above:
- common.SpecReader:
config:
station: id1a3 # Change as needed
experiment_type: EDD
spec_scans: # Edit both SPEC log file path and EDD scan numbers
# Path can be relative to inputdir (line 3) or absolute
- spec_file: <your_raw_ceria_data_directory>/spec.log
scan_numbers: 1
detector_config:
detectors: # Choose the detectors
# Use all available detector elements when omitted
- id: 0
- id: 11
- id: 22
When the detector_config or detectors field is omitted in the MCAEnergyCalibrationProcessor configuration, the energy calibration will be performed for all detectors for which raw data is read. With the detectors field specified, the energy calibration is performed for only those detectors that are selected there that are also read by SpecReader. Any detector selected in the MCAEnergyCalibrationProcessor configuration that is not read by SpecReader will be omitted (resulting in a “no raw data” warning).
The same detector selection mechanism is valid for \(2\theta\) calibration, with the added feature that any detector selected during the energy calibration will be included automatically during \(2\theta\) calibration, irrespective of it being selected in the MCATthCalibrationProcessor configuration (contingent upon the reading of the appropriate raw data by the preceeding SpecReader).
Selecting detectors for Strain analysis
A similar detector selection approach is available for the strain analysis among the MapProcessor and the StrainAnalysisProcessor configurations and the availablity of calibrated detector data. When the detector_config or detectors field is omitted in the StrainAnalysisProcessor configuration, the strain analysis will be performed for all detectors for which both raw data is read by MapProcessor and for which calibrated detector data is read by the common.YAMLReader pipeline item under the strain block in the pipeline file above. With the detectors field specified, the detectors that are used are further restricted to only those that are listed in its field under the edd.StrainAnalysisProcessor pipeline item. Any detector selected in the StrainAnalysisProcessor configuration for which raw data from MapProcessor is missing is skipped accompanied by a “no raw data” warning. Similarly, any selected detector for which detector calibration input is missing is skipped accompanied by a “no calibration data” warning.
Selecting other parameters in the detector configuration
In addition to which detectors to include, the user also has the choice to pick many other parameters that affect the detector configuration and the EDD workflow. Some are specific to certain processors in the EDD workflow, but a certain set, those in the FitConfig configuration class, can be specified both on a per-detector-basis as well as on a for-all-detectors basis. This is evident from the fact that both MCADetectorConfig as well the list entries in its detectors field are derived from FitConfig. The generic behavior for each of these parameters is that those specified on the for-all-detector level, i.e. as items of the detector_config fields, superseed those that are specified on a per-detector-basis.
Take for example two scenarios for the detector configuration in the energy calibration step:
as specified in the pipeline file above:
detector_config: baseline: true mask_ranges: [[650, 850]] detectors: - id: 0 - id: 11 - id: 22
or specified as follows:
detector_config: mask_ranges: [[650, 850]] detectors: - id: 0 baseline: true - id: 11 mask_ranges: [[600, 850]] - id: 22 baseline: lam: 1.e-5 mask_ranges: [[650, 900]]
The first case will set baseline and mask_range for each detector to the same value specified by the respective values in the detector_config configuration. The same happens for mask_range in the second case, where its entries for detector 11 and 22 are ignored. Whereas, the baseline parameter for detector 0, 11 and 22 is set to True, False (its default value), and True with a smoothness parameter, lam, of \(10^{-5}\), respectively.
Additional notes on energy calibration
As mentioned above a standard EDD experiment needs calibration of the detector channel energies. Experiments have shown that the channel energies \(E_j\) vary linearly with the channel index \(j\) within the energy range of typical EDD experiments: \(E_j = mj+b\), where the slope \(m\) and intercept \(b\) can be determined in one or a combination of two experiments:
With an XRF experiment by fitting a set of flueorescence peak centers at known energies:
This uniquely determines \(m\) and \(b\) within the statistical errors of the experiment without having to know the actual takeoff angle \(2\theta\).
With a diffraction experiment by fitting a set of Bragg peak centers corresponding to known lattice spacings \(d_{hkl}\):
Given Bragg’s law, \(\lambda = 2d\sin(\theta)\), with \(E = hc/\lambda\), the Bragg peaks appear at channels with energies \(E_{hkl} = hc / (2d_{hkl}\sin(\theta)\). Rearranging this with the detector channel energy calibration relation gives:
which says that given a set of known Bragg peaks corresponding to lattice spacings \(d_{hkl}\) occuring at channel indices \(j_{hkl}\), a linear fit will uniquely determine \(m'\) and \(b'\). For a known takeoff angle \(2\theta\), this uniquely determines \(m\) and \(b\) as well. Note that this also implies that without an accurately known value of \(2`theta\), one cannot uniquely determine \(m\) and \(b\) from Bragg diffraction alone!
This leads to the above mentioned two-step detector channel energy calibration procedure:
Get nominal values for \(m\) and \(b\), by performing an EDD/XRF experiment on a Ce02 sample and fitting a set of fluorescence peak centers with known energies vs detector channel index.
Fine tuning the calibration by fitting a set of Bragg peak centers in an EDD experiment on (typically) the same CeO2 sample, where the channel indices for the initial Bragg peak positions are obtained from the known Bragg peak energies and the nominal values for \(m\) and \(b\):
If \(2\theta\) is known with sufficient accuracy, one can fit the peak centers vs detector channel index to get \(m'\) and \(b'\) and thus with the known \(2\theta\) convert those directly to fine tuned values for \(m\) and \(b\) over the entire energy range of interest.
If \(2\theta\) is not known with sufficient accuracy, one can fit the Bragg peak centers vs detector channel index to get \(m\), \(b\) and \(2\theta\) in a way that minimizes the RMS error between the fitted peak centers for a given fit parameter set \((m, b, 2\theta)\) and those obtained from Bragg’s law given their \(d_{hkl}\)’s. The latter will still need a sufficently decent initial guess for \(2\theta\), which can be given as an input to the CHAP \(2\theta\) calibration processor or picked interactively.
The choice between the latter two apporaches is set by the calibration_method field in the \(2\theta\) processor configuration in the pipeline yaml input file. Set calibration_method to direct_fit_bragg (default) to use a fixed given value of \(2\theta\), or set it to direct_fit_tth_ecc to fit for the unknown \(2\theta\) and the energy calibration coefficients \(m\) and \(b\).