Space Telescope Science Institute
2010 WFPC2 Data Handbook
help@stsci.edu
Table of Contents Previous Next Index Print


WFPC2 Data Handbook > Chapter 3: WFPC2 Calibration > 3.2 Data Processing Overview

3.2 Data Processing Overview
During pipeline calibration, raw WFPC2 data files were passed through the “OPUS1 pipeline;” all calibration steps were recorded in the trailer file2 (trl file) for each dataset. Figure 3.1 shows an example of a trailer file that identifies the major pipeline steps as described below:
(A) The data was partitioned; the POD file was separated into individual files representing the engineering and science data sections.
(B) Any missing pixel values were replaced with dummy values (an arbitrary assigned value given in the header keyword STDCFFP).
(C) The data was evaluated to determine discrepancies between planned and executed observation parameters.
(D) The data was converted to a format necessary for further processing, a step known as “generic conversion.” Header keywords were also populated at this stage.
(E) The data was calibrated using a standard WFPC2-specific calibration algorithm, calwp2, and the best-available reference files.
(F) A series of new Python routines were run to populate various header keywords and generate a drizzled quick-look image mosaic of all the chips.
(G) After processing, many of the header keyword values were also written to Archive databases (these steps are not shown in Figure 3.1)
Figure 3.1: Sample Trailer File for u5k10101r
(A) 2009010181506-I-INFO -------------- Data Partitioning Starts ---------- (1)3
2009010181506-I-INFO POD file name: /info/sthubbins/otfr/clark/dat/dct//lz_a22c_303_0000213465_u5k10101r.pod_proc (1)
2009010181506-I-INFO Observation name: u5k10101r (1)
2009010181506-I-INFO Finished partitioning obs u5k10101r (1)
2009010181506-I-INFO Expected 3201 packets, Received 3201 packets (1)
2009010181506-I-INFO (1 SHP, 0 UDL, 0 BAD, 3200 SCI). (1)
2009010181506-I-INFO -------------- Data Partitioning ends ---------- (1)
(B) 2009010181510-I-INFO ------------- Data Quality Editing Started: u5k10101r ---------- (1)
2009010181510-I-INFO ----------- Data Quality Editing Completed: u5k10101r ------- (1)
(C) 2009010181526-I-INFO -------------- Data Validation Started: u5k10101r ------------ (1)
2009010181526-W-WARNING PMDB-PDB comparison will NOT be done (1)
2009010181528-I-INFO ------------ Data Validation Completed: u5k10101r ----------- (1)
2009010181532-I-INFO --------- World Coordinate System Started: u5k10101r ------- (1)
2009010181532-I-INFO -------- World Coordinate System Ended: u5k10101r ------- (1)
(D) 2009010181548-I-INFO ------------- Generic Conversion Started: u5k10101r ---------- (1)
2009010181551-I-INFO ----------- Generic Conversion Completed: u5k10101r -------- (1)
2009010181558-I-INFO------ REF started for: u5k10101r -----
2009010181558-I-INFO-GRAPHTAB value changed: => mtab$scf1549pm_tmg.fits
2009010181558-I-INFO-COMPTAB value changed: => mtab$scg1705am_tmc.fits
2009010181558-I-INFO-BIASFILE value changed: => uref$kcd1557lu.r2h
2009010181558-I-INFO-DARKFILE value changed: => uref$ja614574u.r3h
2009010181558-I-INFO-FLATFILE value changed: => uref$m3c1004au.r4h
2009010181558-I-INFO-ATODFILE value changed: => uref$dbu1405iu.r1h
2009010181559-I-INFO-SHADFILE value changed: => uref$e371355iu.r5h
2009010181559-I-INFO-IDCTAB value changed: => uref$sad1946fu_idc.fits
2009010181559-I-INFO-OFFTAB value changed: => uref$s9518396u_off.fits
2009010181559-I-INFO-WF4TFILE value changed: => N/A
2009010181559-I-INFO-DGEOFILE value changed: N/A => uref$s8f1222cu_dxy.fits
2009010181559-I-INFO------ REF ended for: u5k10101r -----
(E) CALBEG ------------ Sat Jan 10 18:35:41 GMT 2009------------------
--- WFPC2 Calibration Starting: CALWP2 Version 2.5.3 (Sep 4, 2008)
--- Starting CALWP2: Input = u5k10101r Output = u5k10101r
*** WARNING From: CALWP-2 While Processing: u5k10101r
*** No WF4TFILE specified. Skipping WF4TCORR.
MASKFILE=uref$f8213081u.r0h MASKCORR=COMPLETED
PEDIGREE=INFLIGHT 01/01/1994 - 15/05/1995
DESCRIP=STATIC MASK - INCLUDES CHARGE TRANSFER TRAPS
uref$dbu1405iu.r1h has no PEDIGREE keyword
BIASFILE=uref$kcd1557lu.r2h BIASCORR=COMPLETED
PEDIGREE=INFLIGHT 26/08/99 - 29/08/00
DESCRIP=not significantly different from j9a1612mu.
DARKFILE=uref$ja614574u.r3h DARKCORR=COMPLETED
PEDIGREE=INFLIGHT 27/09/1999 - 27/09/1999
DESCRIP=Pipeline dark: 120 frame superdark with hotpixels from 27/09/99
FLATFILE=uref$m3c1004au.r4h FLATCORR=COMPLETED
PEDIGREE=INFLIGHT 30/10/1996 - 30/04/2001
DESCRIP=Improved Cyc6-9 flat, rms errors < 0.2-0.3% - see ISR 2002-02
--- Starting processing of element 1
photmode = WFPC2,1,A2D7,F439W,,CAL
photmode(cont) = WFPC2,1,A2D7,F439W,,cont#51454.191,CAL
--- Starting processing of element 2
photmode = WFPC2,2,A2D7,F439W,,CAL
photmode(cont) = WFPC2,2,A2D7,F439W,,cont#51454.191,CAL
--- Starting processing of element 3
photmode = WFPC2,3,A2D7,F439W,,CAL
photmode(cont) = WFPC2,3,A2D7,F439W,,cont#51454.191,CAL
--- Starting processing of element 4
photmode = WFPC2,4,A2D7,F439W,,CAL
photmode(cont) = WFPC2,4,A2D7,F439W,,cont#51454.191,CAL
--- Computing image statistics of element 1
--- Computing image statistics of element 2
--- Computing image statistics of element 3
--- Computing image statistics of element 4
--- WFPC2 Calibration Completed
CALEND ------------ Sat Jan 10 18:36:41 GMT 2009------------------
(F) Filename: u5k10101r_c0f.fits
No. Name Type Cards Dimensions Format
0 PRIMARY PrimaryHDU 251 () int16
1 SCI ImageHDU 143 (800, 800) float32
2 SCI ImageHDU 107 (800, 800) float32
3 SCI ImageHDU 107 (800, 800) float32
4 SCI ImageHDU 107 (800, 800) float32
Background computed to be: 7.0
CTE_1E2 = 0.278727
CTE_1E3 = 0.114256
CTE_1E4 = 0.056283
Background computed to be: 7.56305964487
CTE_1E2 = 0.273426
CTE_1E3 = 0.111878
CTE_1E4 = 0.0549846
Background computed to be: 7.0
CTE_1E2 = 0.278727
CTE_1E3 = 0.114256
CTE_1E4 = 0.056283
Background computed to be: 7.0
CTE_1E2 = 0.278727
CTE_1E3 = 0.114256
CTE_1E4 = 0.056283
Updating keywords in: u5k10101r_c0f.fits
+ MAKEWCS Version 1.1.0 (2 Sept 2008)
-Updating image u5k10101r_c0f.fits[sci,1]
- IDCTAB: Distortion model from row 18 for chip 1 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 17 and 21
- IDCTAB: Distortion model from row 158 for chip 3 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 19 and 23
-Updating image u5k10101r_c0f.fits[sci,2]
- IDCTAB: Distortion model from row 88 for chip 2 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 18 and 22
- IDCTAB: Distortion model from row 158 for chip 3 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 19 and 23
-Updating image u5k10101r_c0f.fits[sci,3]
- IDCTAB: Distortion model from row 158 for chip 3 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 19 and 23- IDCTAB: Distortion model from row 158 for chip 3 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 19 and 23
-Updating image u5k10101r_c0f.fits[sci,4]
- IDCTAB: Distortion model from row 228 for chip 4 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 20 and 24
- IDCTAB: Distortion model from row 158 for chip 3 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 19 and 23
Setting up output name: u5k10101r_drz.fits
Starting PyDrizzle Version 6.3.0 (15-Oct-2008) at 18:37:03 (10/01/2009)
- IDCTAB: Distortion model from row 18 for chip 1 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 17 and 21
- IDCTAB: Distortion model from row 88 for chip 2 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 18 and 22
- IDCTAB: Distortion model from row 158 for chip 3 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 19 and 23
- IDCTAB: Distortion model from row 228 for chip 4 : F439W and CLEAR
- OFFTAB: Offset interpolated from rows 20 and 24
Drizzle parameters have been calculated. Ready to .run()...
Finished calculating parameters at 18:37:10 (10/01/2009)
PyDrizzle: drizzle task started at 18:37:11 (10/01/2009)
Callable DRIZZLE Version 0.7 (4th Apr 2005)
-Opening coefficients file: u5k10101r_c0f_coeffs1.dat
-Using distortion reference point: [ 425.0000, 425.0000]
Callable DRIZZLE Version 0.7 (4th Apr 2005)
-Opening coefficients file: u5k10101r_c0f_coeffs2.dat
-Using distortion reference point: [ 425.0000, 425.0000]
Callable DRIZZLE Version 0.7 (4th Apr 2005)
-Opening coefficients file: u5k10101r_c0f_coeffs3.dat
-Using distortion reference point: [ 425.0000, 425.0000]
Callable DRIZZLE Version 0.7 (4th Apr 2005)
-Opening coefficients file: u5k10101r_c0f_coeffs4.dat
-Using distortion reference point: [ 425.0000, 425.0000]
! Warning, 253 points were outside the output image.
card is too long, comment is truncated.
card is too long, comment is truncated.
card is too long, comment is truncated.
card is too long, comment is truncated.
-Generating multi-extension output file: u5k10101r_drz.fits
PyDrizzle drizzling completed at 18:37:33 (10/01/2009)
3.2.1
calwp2 is the software used to calibrate WFPC2 images. It contains the steps used in standard astronomical CCD data reduction such as removing the effects of bias, dark current, and pixel response (flat-fielding). In the OPUS pipeline, calwp2 is run after raw science data are extracted from telemetry (POD) files, processed, and formatted. A version of calwp2 identical to that in the pipeline is available in STSDAS for users who wish to recalibrate their images.
For final reprocessing to create the WFPC2 Static Archive, the best reference files from the “Calibration Data Base (CDBS) at STScI were used in the final OPUS pipeline calibration. These reference files are available at the WFPC2 CDBS reference files Web page at:
http://www.stsci.edu/hst/observatory/cdbs/SIfileInfo/WFPC2/reftablequeryindex_wfpc2.
The processing steps for the data through calwp2 is presented in schematic form in Figure 3.2, and discussed in further detail in Section 3.3. The software takes as input the raw WFPC2 data files and any necessary calibration reference images and tables.
The calibration steps are determined by the values of the calibration switches (e.g., MASKCORR, BIASCORR, etc.) in the image headers. In raw image files (d0m.fits), calibration switches will indicate whether a specific calibration step will be performed or not:
PERFORM - calibration step to be applied by calwp2.
OMIT - correction is not performed by calwp2.
In calibrated images, the switches will indicate if a particular calibration step was done or not:
COMPLETE - calibration step has been completed.
SKIPPED - step was skipped because the reference file was a dummy
or placeholder (i.e., had no effect on the data). Or it was
specified by the user during recalibration.
An excerpt from the header of ub080101m_d0m.fits[0] showing the calibration switches:
MASKCORR= ’PERFORM ’ / Do mask correction: PERFORM, OMIT, COMPLETE
ATODCORR= ’PERFORM ’ / Do A-to-D correction: PERFORM, OMIT, COMPLETE
WF4TCORR= ’PERFORM ’ / correct WF4 data for temp-dependent amp error
BLEVCORR= ’PERFORM ’ / Do bias level correction
BIASCORR= ’PERFORM ’ / Do bias correction: PERFORM, OMIT, COMPLETE
DARKCORR= ’OMIT ’ / Do dark correction: PERFORM, OMIT, COMPLETE
FLATCORR= ’PERFORM ’ / Do flat field correction
SHADCORR= ’PERFORM ’ / Do shaded shutter correction
DOSATMAP= ’OMIT ’ / Output saturated pixel map
DOPHOTOM= ’PERFORM ’ / Fill photometry keywords
DOHISTOS= ’OMIT ’ / Make histograms: PERFORM, OMIT, COMPLETE
DRIZCORR= ’PERFORM ’ / drizzle processing
OUTDTYPE= ’REAL ’ / Output image datatype: REAL, LONG, SHORT
Figure 3.2: Pipeline Processing by calwp2
Likewise, the reference files to be used in the calibration of the data are set by the values in the reference file keywords. The appropriate settings of the calibration switches and reference file keywords depend on the instrument configuration used (such as gain, exposure time, readout mode, filters), the date of observation, and any special pre-specified observing constraints. They are initially set in the header of the raw data file during the generic conversion step (Figure 3.1, step D); if manual calibration with different calibration settings is necessary, the reference filenames and calibration switches can be changed by editing the raw header file (d0m.fits) using the STSDAS tasks hedit or chcalpar and then running calwp2 on the modified raw files.
An excerpt of the raw image header showing calibration reference files for dataset ub080101m_d0m.fits[0]:
MASKFILE= ’uref$f8213081u.r0h’ / name of the input DQF of known bad pixels
ATODFILE= ’uref$dbu1405fu.r1h’ / name of the A-to-D conversion file
WF4TFILE= ’uref$t721550cu.r7h’ / name of the WF4 correction reference file
BLEVFILE= ’ucal$ub080101m.x0h’ / Engineering file with extended register da
BLEVDFIL= ’ucal$ub080101m.q1h’ / Engineering file DQF
BIASFILE= ’uref$t7d1506ou.r2h’ / name of the bias frame reference file
BIASDFIL= ’uref$t7d1506ou.b2h’ / name of the bias frame reference DQF
DARKFILE= ’uref$t6h10308u.r3h’ / name of the dark reference file
DARKDFIL= ’uref$t6h10308u.b3h’ / name of the dark reference DQF
FLATFILE= ’uref$m3c10041u.r4h’ / name of the flat field reference file
FLATDFIL= ’uref$m3c10041u.b4h’ / name of the flat field reference DQF
SHADFILE= ’uref$e371355eu.r5h’ / name of the reference file for shutter sha
PHOTTAB = ’ ’ / name of the photometry calibration table
GRAPHTAB= ’mtab$t2605492m_tmg.fits’ / the HST graph table
COMPTAB = ’mtab$t6i1714pm_tmc.fits’ / the HST components table
IDCTAB = ’uref$sad1946fu_idc.fits’ / Distortion correction table
OFFTAB = ’uref$s9518396u_off.fits’ / Drift correction table
DGEOFILE= ’uref$s8f1222cu_dxy.fits’ / Distortion correction image
For convenience, the calibration keywords that hold the switches and reference file names are also reported in the trailer files (Figure 3.1), and in the HISTORY keywords at the end of the calibrated science data header (.c0h or c0m.fits[0]).
 
A note about recalibration: calwp2 can operate on GEIS or MEF raw datasets. However, image reference files used by calwp2 must always be converted from wFITS to GEIS regardless of whether the raw datasets are in GEIS or MEF formats. The format of calwp2 output files will always be the same as that of the input files. We recommend that you use MEF format raw datasets for recalibration; MEF files have additional keywords that may be useful for future data analysis.
As with other header keywords, the calibration keywords can be viewed using the STSDAS tasks imhead, hselect, or hedit. Alternatively, the chcalpar task in the STSDAS tools package can be used to view and change group calibration keyword values directly.
Figure 3.2 shows a flow chart that summarizes the sequence of calibration steps performed by calwp2, including the type of input calibration reference files and tables, and type of the output data files from each step. The purpose of each calibration step is briefly described in Table 3.1, and a more detailed explanation is provided in Section 3.3.
Table 3.1: Calibration Steps and Reference Files used for WFPC2 Pipeline Processing
Update the science data quality file using the static bad pixel mask reference file (MASKFILE), which flags defects in the CCD that degrade pixel performance. These are defects that are stable over time. (No changes are made to the science image.)
maskfile (r0h,r0d)
Correct the value of each science data pixel for the analog-to-digital conversion error using information in the A/D lookup reference file (ATODFILE).
atodfile (r1h,r1d)
wf4tfile (r7h,r7d)
Subtract the mean bias level from each pixel in the science data. Mean values are determined separately for even column pixels (group keyword value BIASEVEN) and odd column pixels (BIASODD) because the bias level exhibits a column-wise pattern that changes over time.
blevfile (x0h,x0d)
blevdfil (q1h,q1d)
Subtract the bias image reference file (BIASFILE) from the science image and update the science data quality file with bias image data quality (BIASDFIL).
biasfile (r2h,r2d)
biasdfil (b2h,b2d)
Correct for dark current: scale the normalized dark image reference file to the value in the header keyword DARKTIME, the total dark accumulation time. Then, subtract it from the science image.
darkfile (r3h,r3d) darkdfil (b3h,b3d)
flatfile (r4h,r4d)
flatdfil (b4h,b4d)
shadfile (r5h,.r5d)
Determine absolute sensitivity using SYNPHOT throughput tables identified in the GRAPHTAB and COMPTAB reference tables. Science data are not changed.
graphtab (tmg.fits)
comptab (tmc.fits)
An optional recalibration processing step: create a 3-row image (c2h, c2d) for each group. Row 1 is a histogram of raw science values, row 2 has the A/D correction histogram, and row 3 contains the calibrated image histogram. The default value for DOHISTOS is OMIT.

1
OPUS is the name of the pipeline software that controls the processing and archiving of data at STScI: converting telemetry into FITS data products, populating the Archive catalog, and performing housekeeping on the pipelines.

2
A few trailer files may show error messages near the beginning of the file. This may be a residual artifact from a previous processing run. The correct processing comments should be appended to it. If you suspect a problem with your data, please contact us at help@stsci.edu.

3
The text "---------- (1)" is part of the trailer file content.


Table of Contents Previous Next Index Print