Rawtran

Rawtran is a wrapper around dcraw designed to produce images in standard astronomical FITS format from raw (CR2, CRW, MRW, etc.) photos.

Introduction

Selected digital cameras offers possibility to save raw photos, eg. only slightly pre-processed photos intended for further image processing. Rawtran is developed as a wrapper around dcraw by D.Coffin to produce FITS files from the raw photos. FITS files are widely used by astronomers and can be processed by many standard astronomical software by analogical way as ones acquired by a regular scientific CCD camera. FITS files produced by Rawtran are suitable for all photometry, spectroscopy or astrometry measurements.

Overview

Proper conversion of data acquired by common digital cameras has two key features:

Metadata
or a description of physical parameters of frame like date and time, used optics etc. as integral part of the frame. It is absolutely necessary for regular scientific processing. Therefore rawtran carefully implements conversion of EXIF data enclosed to every raw photo and interprets its in scope of usual astronomical conventions.
Colors
and its proper handling is important for photometry or spectroscopy interpretation. Therefore rawtran implements conversion from CIE 1931 XYZ color space to common astronomical photometry bands (conversion from instrumental to XYZ is provided by dcraw). Saving frames in Color FITS format is just side effect of the conversions.

Rawtran also implements a few convenience functions common in astronomical photometry for using of dark frames.

Conversion to Color FITS

The output image is Color FITS image. Because, there is no widely accepted convention or definition of the color representation in FITS world, RAWTRAN uses Munipack's specification of Color FITS:

The purpose of the conversion type is a simple manipulation (and representation) of color images in FITS format. Color FITS can be displayed by xmunipack (GUI for Munipack) or converted to PNG by fitspng (used to generate all images on this page).

Instrumental (Bayer's) Data

The output image contains one 3D primary array without any photometric transformations. Four original bands are stored in the array with the structure: width ⨉ height ⨉ 4. The bands are storing Bayer's colors in the order: B (blue), G1 (green1), G2 (green2), R (red). According to astronomical terminology, the single bands are in an instrumental photometric system.

The conversion type provides absolutely raw access to data in FITS format. One offers a direct conversion between file formats (it rearranges bytes only) and does NOT modify data itself. To be useful, transform the data to any known color space by using of a transformation matrix (included in the source code of dcraw.c) or derive the transformation matrix yourself.

Conversion and selection of a single band

The conversion is intended for representation of the intensity in the selected color band. The output image is a gray-scale and data are stored as two dimensional FITS primary array.

Note, that all data are slightly modified as result of a color interpolation on Bayer's mask. Use -X "-h" to switch-off the interpolation.

All data must be supposed as RELATIVE quantities only.

A required photometry band can be selected with -c switch.

Instrumental bands (Ri,Gi1,Gi2,Gi,Bi)

Selected colors band is simply copied out from the the Bayer's mask. Gi is derived as the arithmetical mean Gi = (Gi1 + Gi2)/2. These instrumental filters are specific for an unique camera model.

CIE XYZ 1978 Color Bands

Instrumental bands are difficult to reconstruct a standard colors or photometric information. Therefore there is possibility to get a standard band. Colors are converted to primary color space XYZ.

Y component represents luminance. Its spectral profile is similar to humans eye spectral sensitivity under well-light conditions (photopic vision).

Johnson Photometric System (B,V,R)

With exact definition of XYZ spectral profiles, we can derive of color transformation matrix to another colorspace. Rawtran has built-in a transformation matrix from XYZ to Johnson UBVRI system (wide-band astronomical photometry system).

Derived Bands (unfiltered, scotopic)

It is also possible to specify of a wide-spectral band (like the human eye or an unfiltered CCD image) or to simulate scotopic vision (like human eye under low-light conditions). The visual difference between color images (or Y component) and the scotopic one illustrates Purkyňův effect.

Command line options

-c [X|Y|Z|R|V|B|u|s] (standard bands)
-c [Ri|Gi|Gi1|Gi2|Bi] (instrumental bands)
Selects a color band:
X,Y or Z
CIE XYZ 1931 colorspace. Y is the human eye sensitivity under well-light conditions.
B,V or R
Johnson astronomical photometry system
u
Unfiltered, integrated over full camera spectral sensitivity, u = X + Y + Z for all pixels. Recommended to use of -b 32.
s
Scotopic. The human eye sensitivity under low-light conditions, s = 0.36169 Z + 1.18214 Y - 0.80498 X for all pixels.
Ri,Gi,Gi1,Gi2 or Bi
Instrumental, camera specific color in Bayer's mask. The mean green is Gi=(Gi1+Gi2)/2 for all pixels.
-b [8|16|32|-32]
Select bits per pixel of an output image. Possible values are: 8 (0-255), 16 (0-65535), 32 (0-4294967296) and -32 (-10^38 .. 10^38), (values in braces indicates numerical range). An optimal bitpix for raw (instrumental) data is 16 (default) which covers a full range of digital cameras. Some out of range (rare) values will be suppressed (cut-off). The representation occupies width*height*2 bytes. The representing by float numbers (eg. -32) is recommended value for images intended for further processing because saves numerical precision and their numerical range (but occupies of twice more space with respect to 16). 8 bits reduces range (eg. suppress dynamical range) and 32 wastes a lot of storage place only.
-o file
Output file name. If the option is not used, the output is stored as rawtran.fits. Use dash '-' for redirection to the standard output. Precede the filename with exclamation point (!) to overwrite an existing file. Usually, to prevent shell expansion, it will be require to escape it: \!rawtran.fits.
-D RAW file
Use the raw photo as a dark frame. Rawtran will try to convert the file to PGM format by the recommended command dcraw -D -4 -j -t 0 file.RAW and pass it to dcraw -K file.pgm. Note that sometimes can be necessary to switch-off automatic frame rotation using -A "-t 0" option. See Dark Frame section for more info.
-E FITS file
Use the FITS file as a dark frame. The FITS file must be previously converted as dcraw -D -4 -j -t 0 dark.RAW See Dark Frame section for more info.
-C options
Conversions options for dcraw. Default is '-4 -o 5' for standard photometry filters and '-4 -D' for instrumental bands. Defaults are usually satisfactory.
-X options
Specify options to conversion by dcraw itself. One represents default parameters: '-q 3 -w'. See also -A.
-A options
Specify additional options to dcraw itself. If you need pass more options, enclose ones to quotes or apostrophes. In doubts, use -A "-v" to show the detailed trace of the conversion provided by dcraw.
-h, --help, --version
Show summary of options or current versions.

Splitting of conversion parameters for dcraw on three groups of options -C,-X,-A is conventional. Main purpose is separating important options -C,-X and optional -A to prevent potential mistakes. Changing of the important parameters is recommended only for experienced users.

Return value

The zero indicates successful conversion. A non-zero value is returned when an error is occurred during conversion. If the utility is launched without options or with -h switch, zero means that rawtran's internal checker can run dcraw binary and non-zero value otherwise.

Examples of usage

Compose a color FITS and show one in xmunipack:

bash$ rawtran -o IMG_0666.fits IMG_0666.CR2
bash$ xmunipack IMG_0666.fits

Convert an image to gray-scale as an equivalent to humans eye brightness sensitivity at day:

bash$ rawtran -c Y -o IMG_0666.fits IMG_0666.CR2

Convert an image to be equivalent to Johnson's B filter:

bash$ rawtran -c B -o IMG_0666.fits IMG_0666.CR2

Get (mean) green component of the raw photo:

bash$ rawtran -c Gi -o IMG_0666.fits IMG_0666.CR2

Overwrite an output file:

bash$ rawtran -o '!IMG_0666.fits' IMG_0666.CR2

A large set of raw photos can be converted with help of shell functions:

bash$ for A in *.CR2; do rawtran -o ${A%CR2}fits ${A}; done

Included examples has been generated as:

rawtran IMG_5952.CR2 -o IMG_5952.fits
rawtran IMG_5952.CR2 -o IMG_5952_X.fits -c X
rawtran IMG_5952.CR2 -o IMG_5952_Y.fits -c Y
rawtran IMG_5952.CR2 -o IMG_5952_Z.fits -c Z
rawtran IMG_5952.CR2 -o IMG_5952_B.fits -c B
rawtran IMG_5952.CR2 -o IMG_5952_V.fits -c V
rawtran IMG_5952.CR2 -o IMG_5952_R.fits -c R
rawtran IMG_5952.CR2 -o IMG_5952_Bi.fits -c Bi
rawtran IMG_5952.CR2 -o IMG_5952_Gi.fits -c Vi
rawtran IMG_5952.CR2 -o IMG_5952_Ri.fits -c Ri
rawtran IMG_5952.CR2 -o IMG_5952_s.fits -c s
rawtran IMG_5952.CR2 -o IMG_5952_u.fits -c u

fitspng IMG_5952.fits -fl 0,20000 -s 10 -o IMG_5952.png
fitspng IMG_5952_X.fits -fl 0,20000 -s 10 -o IMG_5952_X.png
fitspng IMG_5952_Y.fits -fl 0,20000 -s 10 -o IMG_5952_Y.png
fitspng IMG_5952_Z.fits -fl 0,20000 -s 10 -o IMG_5952_Z.png
fitspng IMG_5952_R.fits -fl 0,20000 -s 10 -o IMG_5952_R.png
fitspng IMG_5952_V.fits -fl 0,20000 -s 10 -o IMG_5952_V.png
fitspng IMG_5952_B.fits -fl 0,20000 -s 10 -o IMG_5952_B.png
fitspng IMG_5952_s.fits -fl 0,20000 -s 10 -o IMG_5952_s.png
fitspng IMG_5952_u.fits  -fl 0,40000 -s 10 -o IMG_5952_u.png
fitspng IMG_5952_Ri.fits -fl 128,1000 -s 10 -o IMG_5952_Ri.png
fitspng IMG_5952_Gi.fits -fl 128,1000 -s 10 -o IMG_5952_Gi.png
fitspng IMG_5952_Bi.fits -fl 128,1000 -s 10 -o IMG_5952_Bi.png
IMG_5952

Color FITS

IMG_5952_X

X component

IMG_5952_Y

Y component

IMG_5952_Z

Z component

IMG_5952_R

R filter

IMG_5952_V

V filter

IMG_5952_B

B filter

IMG_5952_Ri

R Bayer instrumental

IMG_5952_Gi

G Bayer instrumental

IMG_5952_Bi

B Bayer instrumental

IMG_5952_u

Unfiltered

IMG_5952_s

Night vision

Reference image

Dark Frame

Rawtran provides convenience functions for handling dark frames.

The basic way is use of -D option which launch dcraw to create PGM file representing of the dark. While rawtran sometimes save results to PPM file, transparent conversion to PGM can be required. Time duration of the procedure will depend on the conversion and generally one will twice slower against to passing a PGM file directly by -A "-K file.pgm". An user is responsible to provide correct dark frame.

Little bit advanced way is to use a FITS file as dark frame which is on the fly converted to PGN. Purpose of the option is to offer possibility for averaging of darks. A hypothetical session can be:

$ rawtran -c P -A "-j -t 0" -o dark1.fits dark1.CR2
$ rawtran -c P -A "-j -t 0" -o dark2.fits dark2.CR2
$ munipack dark -o dark.fits dark?.fits   # or equivalent
$ rawtran -E dark.fits -o light.fits light.CR2

First two lines (or possible more) converts dark frames to FITS format using of -c P option. Than the created FITSes are averaged (munipack utility is only an example, any another software can be used). The averaged file is used as the dark frame.

Download and Installation

Source code is freely available under GPL2 license. cfitsio library with developing support files (headers, static libraries) is necessary for building. Building does not depends on dcraw itself.

A recommended way of local installation is:

$ tar zxf rawtran-X.Y.Z.tar.gz
$ cd rawtran-X.Y.Z/
$ ./configure CFLAGS="-O4 -DNDEBUG"
$ make
# make install

Last step must be executed under root account. Both binary and man page are installed under /usr/local tree. It would be nice to keep the source directory for a case of later uninstalling.

Rawtran can be build also under any DEB-based GNU/Linux distribution (like Ubuntu, Mint or Debian). Install build-essential package and build it as:

$ tar zxf rawtran-X.Y.Z.tar.gz
$ cd rawtran-X.Y.Z/
$ export DEB_CFLAGS_SET="-O4 -DNDEBUG"
$ dpkg-buildpackage -us -uc
# dpkg -i ../rawtran_X.Y.Z-D_arch.deb

Building under RPM-bases distributions (RHELL, Fedora) requires place of rawtran's tarball to a directory where RPM places sources (rpmbuild), unpack the spec file and we can use the following commands to build:

$ rpmbuild -bb rawtran.spec

where the package must by placed in your RPM building directory.

Rawtran runs under any Unix-like operating system (all flavors of GNU/Linux and BSD, Solaris, Mac OS X, etc). Usage under Windows or DOS has not been reported yet.

Details

Actually, Rawtran is a wrapper around dcraw. One internally launches and process if its output. The info about image (EXIF information) is obtained as:

dcraw -i -v -c

The image data oneself are imported by:

dcraw -4 -c -o 5    (standard bands)
dcraw -4 -c -D      (instrumental bands)

Passed parameters can be changed with -C option (except -c).

Rawtran internally represents image data as 4-byte float in all numerical operations.

See Also

Fitspng is a converter from FITS to PNG. Munipack is a general utility to work with FITS images.

Development notes can be found in Hroch's diary.

License

Rawtran is free software licensed under the GNU General Public License. This gives you the freedom to use and modify Rawtran to suit your needs.

Copyright © 2007 — 2016, F. Hroch, Institute of Theoretical Physics and Astrophysics, Masaryk University, Brno, Czech Republic.