compare_outputs
- ci_watson.artifactory_helpers.compare_outputs(outputs, raise_error=True, ignore_keywords=[], ignore_hdus=[], ignore_fields=[], rtol=0.0, atol=0.0, input_path=[], docopy=True, results_root=None, verbose=True)[source]
Compare output with “truth” using appropriate diff routine; namely:
fitsdiff
for FITS file comparisons.unified_diff
for ASCII products.
Only after all elements of
outputs
have been processed will the method report any success or failure, with failure of any one comparison not preventing the rest of the comparisons to be performed.- Parameters:
- outputslist of tuple or dict
This list defines what outputs from running the test will be compared. Three distinct types of values as list elements are supported:
2-tuple :
(test output filename, truth filename)
3-tuple :
(test output filename, truth filename, HDU names)
dict :
{'files': (output, truth), 'pars': {key: val}}
If filename contains extension such as
[hdrtab]
, it will be interpreted as specifying comparison of just that HDU.- raise_errorbool
Raise
AssertionError
if difference is found.- ignore_keywordslist of str
List of FITS header keywords to be ignored by
FITSDiff
andHDUDiff
.- ignore_hduslist of str
List of FITS HDU names to ignore by
FITSDiff
. This is only available forastropy>=3.1
.- ignore_fieldslist of str
List FITS table column names to be ignored by
FITSDiff
andHDUDiff
.- rtol, atolfloat
Relative and absolute tolerance to be used by
FITSDiff
andHDUDiff
.- input_pathlist or tuple
A series of sub-directory names under
get_bigdata_root()
that leads to the path of the ‘truth’ files to be compared against. If not provided, it assumes that ‘truth’ is in the working directory. For example, withget_bigdata_root()
pointing to/grp/test_data
, a file at:/grp/test_data/pipeline/dev/ins/test_1/test_a.py
would require
input_path
of:["pipeline", "dev", "ins", "test_1"]
- docopybool
If True, ‘truth’ will be copied to output directory before comparison is done.
- results_rootstr or None
If not None, for every failed comparison, the test output is automatically renamed to the given ‘truth’ in the output directory and
generate_upload_schema()
will be called to generate a JSON scheme for Artifactory upload. If you do not need this functionality, useresults_root=None
.- verbosebool
Print extra info to screen.
- Returns:
- creature_reportstr
Report from FITS or ASCII comparator. This is part of error message if
raise_error=True
.
Examples
There are multiple use cases for this method, specifically related to how
outputs
are defined upon calling this method. The specification of theoutputs
can be any combination of the following patterns:2-tuple inputs:
outputs = [('file1.fits', 'file1_truth.fits')]
This definition indicates that
file1.fits
should be compared as a whole withfile1_truth.fits
.2-tuple inputs with extensions:
outputs = [('file1.fits[hdrtab]', 'file1_truth.fits[hdrtab]')]
This definition indicates that only the HDRTAB extension from
file1.fits
will be compared to the HDRTAB extension fromfile1_truth.fits
.3-tuple inputs:
outputs = [('file1.fits', 'file1_truth.fits', ['primary', 'sci'])]
This definition indicates that only the PRIMARY and SCI extensions should be compared between the two files. This creates a temporary
HDUList
object comprising only the given extensions for comparison.Dictionary of inputs and parameters:
outputs = [{'files': ('file1.fits', 'file1_truth.fits'), 'pars': {'ignore_keywords': ['ROOTNAME']}}] This definition indicates that ROOTNAME will be ignored during the comparison between the files specified in ``'files'``. Any input parameter for ``FITSDiff`` or ``HDUDiff`` can be specified as part of the ``'pars'`` dictionary. In addition, the input files listed in ``'files'`` can also include an extension specification, such as ``[hdrtab]``, to limit the comparison to just that extension.
This example from an actual test definition demonstrates how multiple input defintions can be used at the same time:
outputs = [ ('jw99999_nircam_f140m-maskbar_psfstack.fits', 'jw99999_nircam_f140m-maskbar_psfstack_ref.fits' ), ('jw9999947001_02102_00002_nrcb3_a3001_crfints.fits', 'jw9999947001_02102_00002_nrcb3_a3001_crfints_ref.fits' ), {'files': ('jw99999_nircam_f140m-maskbar_i2d.fits', 'jw99999_nircam_f140m-maskbar_i2d_ref.fits'), 'pars': {'ignore_hdus': ['HDRTAB']}, {'files': ('jw99999_nircam_f140m-maskbar_i2d.fits', 'jw99999_nircam_f140m-maskbar_i2d_ref.fits', ['primary','sci','dq']), 'pars': {'rtol': 0.000001} }, {'files': ('jw99999_nircam_f140m-maskbar_i2d.fits[hdrtab]', 'jw99999_nircam_f140m-maskbar_i2d_ref.fits[hdrtab]'), 'pars': {'ignore_keywords': ['NAXIS1', 'TFORM*'], 'ignore_fields': ['COL1', 'COL2']} }]
Note
Each
outputs
entry in the list gets interpreted and processed separately.