next up previous contents
Next: 4. Examining Extensions and Up: CFHT FITS Handling Library Previous: 2. Using `idx' Sorting   Contents

Subsections

3. Opening, Locking, Closing and Creating FITS files

3.1 fh_create()

Before calling fh_file() to read from a filename (or fh_read() to read from a file descriptor which you opened yourself) this function must be used to create a new HeaderUnit. The HeaderUnit will be empty, until fh_file(), fh_read(), fh_set*(), or fh_merge() functions are used to add cards to it.

3.2 fh_copy_hu()

This creates a new HeaderUnit from an existing one, copying all the FITS cards from the existing one into the otherwise empty new one.

3.3 fh_destroy()

For each HeaderUnit returned by fh_create(), don't forget to pass it to fh_destroy() when it is no longer needed. The following happens in fh_destroy()

  1. Any remaining advisory file locks are released.
  2. The file is closed, if and only if it was opened with fh_file().
  3. Any linked extension header units are fh_destroy()'d.
  4. All resources allocated to the list are freed.

The first two steps may have errors, so always check the return code from fh_destroy(). For example:


  if (fh_destroy(hu) != FH_SUCCESS) rtn = FAIL;

WARNING: fh_destroy() should not be used on HeaderUnit's returned by the fh_ehu*() functions. These are destroyed automatically when their parent header unit is fh_destroy'ed.

3.4 fh_read()

This is not needed if you use fh_file(). But if you open() your own file descriptor, pass it to this function, along with an empty HeaderUnit from fh_create() to read the header from a FITS file.

Whether you used fh_read() or fh_file(), at this point if the file happens to be an MEF file, you can find out using fh_extensions() and access each extension unit using fh_ehu*().

3.5 fh_file()

While it is possible to open your own file descriptor, and use the library's routines only to parse the header, using fh_file() to open the file for you will give all tools using this library a similar behavior. Several things happen when you call this function:

  1. If no filespec, or the special name '-' is given, the library will try to read from Standard Input. If stdin is a terminal device, an error will result.
  2. Next, if the filespec exists exactly as given, it is opened, the primary header unit is parsed and returned in HeaderUnit.
  3. Next, if the filespec string contains '[]', that portion of the string is removed and treated as an EXTNAME. The header unit of a matching extension is returned in HeaderUnit instead of the primary header. If the file exists, but no matching EXTNAME could be found, the error is the same as if the file was not found.
  4. Next, filespec + ".fits" is attempted (with EXTNAME, if any).
  5. Finally, if all of the above fail, filespec + "/" + filebase + extnum + ".fits" is tried. In this step, filespec does not include ".fits" if it had it, and extnum is extname with a leading "chip" or "im" removed. This hack allows our current split-file convention to be accessed as if it was MEF.

In addition to a new HeaderUnit, which you must obtain from fh_create() and the `filespec' parameter, a file mode is required. If you intend to use fh_rewrite() to update any FITS cards, you must choose FH_FILE_RDWR. Otherwise, use FH_FILE_RDONLY for this parameter.

3.5.1 Advisory File Locks

Advisory file locking will automatically be done before reading the header unit. If the file was opened RDWR, the lock is not released until the first call of fh_rewrite(). Otherwise, with RDONLY, the lock is already released by the time fh_file() returns. ( EXCEPTION: If the file contains extensions, such a RDONLY file lock is left in place until fh_destroy() is called.)

Tests between HP-UX 10.20, Solaris 2.6, and Linux-2.2.16-cfht (includes our NFS3 patches) have been completed. All combinations of NFS server and NFS client (and local file access) were tested including competing access from all three architectures and from six hosts to one file, at the same time. Per run, each test client makes 6 calls to fh_file() and inserts 6 new FITS cards, checks that they appear correctly (PID's are saved in the value field) and checks that other test client's cards are valid as well. The conclusions from the tests are:

Read performance was tested with /cfht/bin/fhtool -Vv file.fits, before, during, and after file locking:

  1. Before ever applying a lock to a file, the first time a file is accessed over NFS for reading, the speed is limited by either the remote disk, or the network (plus some NFS overheads). This is typically in the range of 3-8 MB/sec on our systems.

  2. A second access (even for full CFH12K mosaic images) is always much faster, as it gets read from the cache. Druid (Linux) gets about 300 MB/sec, Ohia (Linux) gets about 250 MB/sec, Mahina (Solaris) gets about 100 MB/sec, and Neptune gets 25 MB/sec (except for large files, which it doesn't seem to be able to cache, so they go back to the 3-8 MB/sec speed.)

  3. Access during an exclusive (RDWR) lock will not happen (unless a client ignores the lock.)

  4. Access after the lock is back to the speed of (1), except for when the file was not modified (no call to fh_rewrite()). In that case, the next access immediately returns to the cached speed. This means, with our current NFS implementations, use of the cache is apparently put ``on hold'' during a lock, and only invalidated if the file actually changed. At that point, there is no mechanism to determine which blocks of the file have been modified, so the whole thing must be read over the network again. This would actually be the same without locking, by the way.

    Local access to the file does figure out which individual blocks have changed, and so in that case the whole file does not have to be re-read into local buffers each time a FITS card is changed.

  5. After a file has been locked, changed, unlocked, read once over the network . . . subsequent reads immediately return to the fast cached speed of (2).

These tests have not revealed any problem with file locking on our system, nor have they shown any disadvantage to using file locks. Therefore, the default of the library will be to lock, since no harm is done. It should be safe to run programs which update FITS cards in parallel.

3.6 fh_reindex()

During an fh_read() or fh_file(), libfh assigns arbitrary index numbers to the keywords read (with specific ones only for the first few required cards). fh_reindex() reassigns the numbers from fh_registry.h to the cards in the provided HeaderUnit.

3.7 fh_file_desc()

fh_file() opens a file on a HeaderUnit. Pass this header unit to fh_file_desc() if you need to do any operations directly on the file descriptor. File position after a call to fh_file() is at the start of the image (or first extension header for MEF). To be certain of file position, see fh_ehu() for a way to seek to a specific section of data (including data associated with the primary/only header.)

3.8 fh_reserve()

Use this before calling fh_write() to select the number of reserved COMMENT cards in the header. These cards will be used by other programs to add more keywords to the FITS file.

(These other programs do not call fh_reserve() themselves. The just call fh_rewrite() and fail if there wasn't enough room.)

3.9 fh_write()

Write the FITS header to a file descriptor. After this, make a call to fh_write_padded_image() to add the data onto the new FITS file. Use fh_rewrite() instead of fh_write() for programs which only update FITS cards without generating a new FITS file.

3.10 fh_write_padding()

This function writes 0-byte padding intended to go after the image data. If the header unit contains XTENSION='TABLE' then ASCII space (0x20-byte) padding is used instead, to satisfy the requirements of an ASCII table.

You don't need this function if you use fh_write_padded_image() or fh_copy_padded_image(). If you choose to write the data to the file descriptor yourself, however, you might find it useful to let the library calculate the padding for you.

3.11 fh_write_padded_image()

This routine can be used to both image data and preformatted ASCII tables. For image data, the data can be either in host byte-order or FITS byte-order. `data' is a buffer of `size' bytes which you must allocate first. The library calculates its own value for `typesize' and `size' based on BITPIX and NAXIS (see fh_image_bytes()), which must match your value, or an error will be returned. This, `size' is completely redundant, but required as a sanity check, while `typesize' is critical in controlling byte-swapping.

File pointer is left at the end of the padding after the image data. When building MEF files, this is the right place to begin the next header unit. If not, just close the file here... and check for errors when calling close()!

The `typesize' parameter affects the byte-order in which `data' is written to the FITS file. For the most common case, where `data' is a buffer of host-byte-order 8, 16, 32, or 64 bit values, `typesize' should be set to sizeof(char), sizeof(short), or sizeof(long), or sizeof(double). The library will automatically determine if the data needs to be swapped when written to FITS-byte-order. This is the recommended way to let byte-swapping happen. If your program has already taken care of putting `data' in FITS-byte-order, then pass FH_TYPESIZE_RAW (a value of 0) for `typesize' to force no byte-swapping. Finally, to force byte-swapping to happen, even on a Sparc or other type of computer where host- and FITS-byte-order are the same, pass the negative of the sizeof($<$datatype$>$). Note that `typesize' is a number of bytes, i.e. is equal to |BITPIX|/8.


Table 6: Choices for `typesize' Parameter
BITPIX typesize= swap by: data written to file is:
8 sizeof(unsigned char)   unchanged
16 sizeof(signed short) 2 swapped if needed
32 sizeof(signed long) 4 swapped if needed
-32 sizeof(float) 4 swapped if needed
-64 sizeof(double) 8 swapped if needed
(any) FH_TYPESIZE_RAW   never swapped
16 -sizeof(signed short) 2 always swapped
32 -sizeof(signed long) 4 always swapped
-32 -sizeof(float) 4 always swapped
-64 -sizeof(double) 8 always swapped

Note that the -sizeof() options will swap even on architectures such as Sparc and HP-PA where the data is normally already stored in the correct byte order for FITS files.

Also note that typesize must be either 0, or equal to BITPIX/8 or library will fail and return FH_BAD_VALUE.

3.12 fh_read_padded_image()

See description for fh_write_padded_image(). Instead of writing data in your buffer, this reads the image from the file and places it in the buffer (which you must still allocate.) After reading image data, padding in the input file is verified, but not copied to your buffer. (I.e. `size' must still be exactly equal to fh_image_bytes() or the function will fail.)

Byte-swapping happens according to the table in fh_write_padded_image. Pass the correct size of the data (in bytes) or FH_TYPESIZE_RAW (a value of 0) if you want to read raw bytes from the FITS file in order.

3.13 fh_copy_padded_image()

This performs a straight copy from one FITS file to another. Both are expected to be at the start of their image data. The number of bytes to read and the padding are determined from HeaderUnit's BITPIX and NAXIS* values (see fh_image_bytes()).

This routine does not work on the primary header unit (it lacks image data.) To copy multiple image extensions, each extension must be copied by doing the following:

See the source code to fhtool.c, which splits and joins MEF, for some examples using this call. (But be warned, there's a lot of other cruft in fhtool.c.)

3.14 fh_rewrite()

Using the same `fd' obtained by fh_file(), or passed to fh_read(), this call seeks back to the start of the header and attempts to rewrite it to reflect any changes made by fh_remove() or fh_set() calls on the HeaderUnit.

fh_rewrite() does not take a file descriptor argument. It will fail if there was no previously successful fh_file() or fh_read() call from which it can obtain a file descriptor. It will also return delayed failure from any fh_set() calls which may have failure due to incorrect usage or out of memory errors.


next up previous contents
Next: 4. Examining Extensions and Up: CFHT FITS Handling Library Previous: 2. Using `idx' Sorting   Contents
Jim Thomas
2013-12-19