compare_outputs

ci_watson.artifactory_helpers.compare_outputs(outputs, raise_error=True, ignore_keywords=[], ignore_hdus=[], ignore_fields=[], rtol=0.0, atol=0.0, input_path=[], docopy=True, results_root=None, verbose=True)[source]

Compare output with “truth” using appropriate diff routine; namely:

  • fitsdiff for FITS file comparisons.

  • unified_diff for ASCII products.

Only after all elements of outputs have been processed will the method report any success or failure, with failure of any one comparison not preventing the rest of the comparisons to be performed.

Parameters:
outputslist of tuple or dict

This list defines what outputs from running the test will be compared. Three distinct types of values as list elements are supported:

  • 2-tuple : (test output filename, truth filename)

  • 3-tuple : (test output filename, truth filename, HDU names)

  • dict : {'files': (output, truth), 'pars': {key: val}}

If filename contains extension such as [hdrtab], it will be interpreted as specifying comparison of just that HDU.

raise_errorbool

Raise AssertionError if difference is found.

ignore_keywordslist of str

List of FITS header keywords to be ignored by FITSDiff and HDUDiff.

ignore_hduslist of str

List of FITS HDU names to ignore by FITSDiff. This is only available for astropy>=3.1.

ignore_fieldslist of str

List FITS table column names to be ignored by FITSDiff and HDUDiff.

rtol, atolfloat

Relative and absolute tolerance to be used by FITSDiff and HDUDiff.

input_pathlist or tuple

A series of sub-directory names under get_bigdata_root() that leads to the path of the ‘truth’ files to be compared against. If not provided, it assumes that ‘truth’ is in the working directory. For example, with get_bigdata_root() pointing to /grp/test_data, a file at:

/grp/test_data/pipeline/dev/ins/test_1/test_a.py

would require input_path of:

["pipeline", "dev", "ins", "test_1"]
docopybool

If True, ‘truth’ will be copied to output directory before comparison is done.

results_rootstr or None

If not None, for every failed comparison, the test output is automatically renamed to the given ‘truth’ in the output directory and generate_upload_schema() will be called to generate a JSON scheme for Artifactory upload. If you do not need this functionality, use results_root=None.

verbosebool

Print extra info to screen.

Returns:
creature_reportstr

Report from FITS or ASCII comparator. This is part of error message if raise_error=True.

Examples

There are multiple use cases for this method, specifically related to how outputs are defined upon calling this method. The specification of the outputs can be any combination of the following patterns:

  1. 2-tuple inputs:

    outputs = [('file1.fits', 'file1_truth.fits')]
    

    This definition indicates that file1.fits should be compared as a whole with file1_truth.fits.

  2. 2-tuple inputs with extensions:

    outputs = [('file1.fits[hdrtab]', 'file1_truth.fits[hdrtab]')]
    

    This definition indicates that only the HDRTAB extension from file1.fits will be compared to the HDRTAB extension from file1_truth.fits.

  3. 3-tuple inputs:

    outputs = [('file1.fits', 'file1_truth.fits', ['primary', 'sci'])]
    

    This definition indicates that only the PRIMARY and SCI extensions should be compared between the two files. This creates a temporary HDUList object comprising only the given extensions for comparison.

  4. Dictionary of inputs and parameters:

       outputs = [{'files': ('file1.fits', 'file1_truth.fits'),
                   'pars': {'ignore_keywords': ['ROOTNAME']}}]
    
    This definition indicates that ROOTNAME will be ignored during
    the comparison between the files specified in ``'files'``.
    Any input parameter for ``FITSDiff`` or ``HDUDiff`` can be specified
    as part of the ``'pars'`` dictionary.
    In addition, the input files listed in ``'files'`` can also include
    an extension specification, such as ``[hdrtab]``, to limit the
    comparison to just that extension.
    

This example from an actual test definition demonstrates how multiple input defintions can be used at the same time:

outputs = [
    ('jw99999_nircam_f140m-maskbar_psfstack.fits',
     'jw99999_nircam_f140m-maskbar_psfstack_ref.fits'
    ),
    ('jw9999947001_02102_00002_nrcb3_a3001_crfints.fits',
     'jw9999947001_02102_00002_nrcb3_a3001_crfints_ref.fits'
    ),
    {'files': ('jw99999_nircam_f140m-maskbar_i2d.fits',
               'jw99999_nircam_f140m-maskbar_i2d_ref.fits'),
     'pars': {'ignore_hdus': ['HDRTAB']},
    {'files': ('jw99999_nircam_f140m-maskbar_i2d.fits',
               'jw99999_nircam_f140m-maskbar_i2d_ref.fits',
               ['primary','sci','dq']),
     'pars': {'rtol': 0.000001}
    },
    {'files': ('jw99999_nircam_f140m-maskbar_i2d.fits[hdrtab]',
               'jw99999_nircam_f140m-maskbar_i2d_ref.fits[hdrtab]'),
     'pars': {'ignore_keywords': ['NAXIS1', 'TFORM*'],
              'ignore_fields': ['COL1', 'COL2']}
    }]

Note

Each outputs entry in the list gets interpreted and processed separately.