This is gnuastro.info, produced by makeinfo version 7.0.3 from
gnuastro.texi.

This book documents version 0.21 of the GNU Astronomy Utilities
(Gnuastro).  Gnuastro provides various programs and libraries for
astronomical data manipulation and analysis.

   Copyright © 2015-2023 Free Software Foundation, Inc.

     Permission is granted to copy, distribute and/or modify this
     document under the terms of the GNU Free Documentation License,
     Version 1.3 or any later version published by the Free Software
     Foundation; with no Invariant Sections, no Front-Cover Texts, and
     no Back-Cover Texts.  A copy of the license is included in the
     section entitled “GNU Free Documentation License”.
INFO-DIR-SECTION Astronomy
START-INFO-DIR-ENTRY
* Gnuastro: (gnuastro).       GNU Astronomy Utilities.
* libgnuastro: (gnuastro)Gnuastro library. Full Gnuastro library doc.

* help-gnuastro: (gnuastro)help-gnuastro mailing list. Getting help.

* bug-gnuastro: (gnuastro)Report a bug. How to report bugs

* Arithmetic: (gnuastro)Arithmetic. Arithmetic operations on pixels.
* astarithmetic: (gnuastro)Invoking astarithmetic. Options to Arithmetic.

* BuildProgram: (gnuastro)BuildProgram. Compile and run programs using Gnuastro’s library.
* astbuildprog: (gnuastro)Invoking astbuildprog. Options to BuildProgram.

* ConvertType: (gnuastro)ConvertType. Convert different file types.
* astconvertt: (gnuastro)Invoking astconvertt. Options to ConvertType.

* Convolve: (gnuastro)Convolve. Convolve an input file with kernel.
* astconvolve: (gnuastro)Invoking astconvolve. Options to Convolve.

* CosmicCalculator: (gnuastro)CosmicCalculator. For cosmological params.
* astcosmiccal: (gnuastro)Invoking astcosmiccal. Options to CosmicCalculator.

* Crop: (gnuastro)Crop. Crop region(s) from image(s).
* astcrop: (gnuastro)Invoking astcrop. Options to Crop.

* Fits: (gnuastro)Fits. View and manipulate FITS extensions and keywords.
* astfits: (gnuastro)Invoking astfits. Options to Fits.

* MakeCatalog: (gnuastro)MakeCatalog. Make a catalog from labeled image.
* astmkcatalog: (gnuastro)Invoking astmkcatalog. Options to MakeCatalog.

* MakeProfiles: (gnuastro)MakeProfiles. Make mock profiles.
* astmkprof: (gnuastro)Invoking astmkprof. Options to MakeProfiles.

* Match: (gnuastro)Match. Match two separate catalogs.
* astmatch: (gnuastro)Invoking astmatch. Options to Match.

* NoiseChisel: (gnuastro)NoiseChisel. Detect signal in noise.
* astnoisechisel: (gnuastro)Invoking astnoisechisel. Options to NoiseChisel.

* Segment: (gnuastro)Segment. Segment detections based on signal structure.
* astsegment: (gnuastro)Invoking astsegment. Options to Segment.

* Query: (gnuastro)Query. Access remote databases for downloading data.
* astquery: (gnuastro)Invoking astquery. Options to Query.

* Statistics: (gnuastro)Statistics. Get image Statistics.
* aststatistics: (gnuastro)Invoking aststatistics. Options to Statistics.

* Table: (gnuastro)Table. Read and write FITS binary or ASCII tables.
* asttable: (gnuastro)Invoking asttable. Options to Table.

* Warp: (gnuastro)Warp. Warp a dataset to a new grid.
* astwarp: (gnuastro)Invoking astwarp. Options to Warp.

* astscript: (gnuastro)Installed scripts. Gnuastro’s installed scripts.
* astscript-ds9-region: (gnuastro)Invoking astscript-ds9-region. Options to this script
* astscript-fits-view: (gnuastro)Invoking astscript-fits-view. Options to this script
* astscript-pointing-simulate: (gnuastro)Invoking astscript-pointing-simulate. Options to this script
* astscript-psf-scale-factor: (gnuastro)Invoking astscript-psf-scale-factor. Options to this script
* astscript-psf-select-stars: (gnuastro)Invoking astscript-psf-select-stars. Options to this script
* astscript-psf-stamp: (gnuastro)Invoking astscript-psf-stamp. Options to this script
* astscript-psf-subtract: (gnuastro)Invoking astscript-psf-subtract. Options to this script
* astscript-psf-unite: (gnuastro)Invoking astscript-psf-unite. Options to this script
* astscript-radial-profile: (gnuastro)Invoking astscript-radial-profile. Options to this script
* astscript-sort-by-night: (gnuastro)Invoking astscript-sort-by-night. Options to this script
* astscript-zeropoint: (gnuastro)Invoking astscript-zeropoint. Options to this script
END-INFO-DIR-ENTRY


File: gnuastro.info,  Node: Zero point estimation,  Next: Pointing pattern simulation,  Prev: Viewing FITS file contents with DS9 or TOPCAT,  Up: Installed scripts

10.5 Zero point estimation
==========================

Through the “zero point”, we are able to give physical units to the
pixel values of an image (often in units of “counts” or ADUs) and thus
compare them with other images (as well as measurements that are done on
them).  The zero point is therefore an important calibration of pixel
values (as astromerty is a calibration of the pixel positions).  The
fundamental concepts behind the zero point are described in *note
Brightness flux magnitude::.  We will therefore not go deeper into the
basics here and stick to the practical aspects of it.

   The purpose of Gnuastro’s ‘astscript-zeropoint’ script is to obtain
the zero point of an image by considering another image (where the zero
point is already known), or a catalog.  In the The operation involves
multiple lower-level programs in a standard series of steps.  For
example, when using another image, the script will take the following
steps:

  1. Download the Gaia catalog that overlaps with the input image using
     Gnuastro’s Query program (see *note Query::).  This is done to
     determine the stars within the image(1).
  2. Perform aperture photometry(2) with *note MakeProfiles:: *note
     MakeCatalog::.  We will assume a zero point of 0 for the input
     image.  If the reference is an image, then we should perform
     aperture photometry also in that image.
  3. Match the two catalogs(3) with *note Match::.
  4. The difference between the input and reference magnitudes should be
     independent of the magnitude of the stars.  This does not hold when
     the stars are saturated in one/both the images (giving us a
     bright-limit for the magnitude range to use) or for stars fainter
     than a certain magnitude, where the signal-to-noise ratio drops
     significantly in one/both images (giving us a faint limit for the
     magnitude range to use).
  5. Since a zero point of 0 was used for the input image, the magnitude
     difference above (in the reliable magnitude range) is the zero
     point of the input image.

   In the “Tutorials” chapter of this Gnuastro book, there are two
tutorials dedicated to the usage of this script.  The first uses an
image as a reference (*note Zero point tutorial with reference image::)
and the second uses a catalog (*note Zero point tutorial with reference
catalog::).  For the full set of options an a detailed description of
each, see *note Invoking astscript-zeropoint::.

* Menu:

* Invoking astscript-zeropoint::  How to call the script

   ---------- Footnotes ----------

   (1) Stars have an almost identical shape in the image (as opposed to
galaxies for example), using confirmed stars will produce a more
reliable result.

   (2) For a complete tutorial on aperture photometry, see *note
Aperture photometry::.

   (3) For a tutorial on matching catalogs, see *note Matching
catalogs::).


File: gnuastro.info,  Node: Invoking astscript-zeropoint,  Prev: Zero point estimation,  Up: Zero point estimation

10.5.1 Invoking astscript-zeropoint
-----------------------------------

This installed script will calculate the zero point of an input image to
calibrate it.  The reference can be an image or catalog (which have been
previously calibrated) The executable name is ‘astscript-zeropoint’,
with the following general template:

     ## Using a reference image in four apertures.
     $ astscript-zeropoint image.fits --hdu=1 \
                           --refimgs=ref-img1.fits,ref-img2.fits \
                           --refimgshdu=1,1 \
                           --refimgszp=22.5,22.5 \
                           --aperarcsec=1.5,2,2.5,3 \
                           --magnituderange=16,18 \
                           --output=output.fits

     ## Using a reference catalog
     $ astscript-zeropoint image.fits --hdu=1 \
                           --refcat=cat.fits \
                           --refcathdu=1 \
                           --aperarcsec=1.5,2,2.5,3 \
                           --magnituderange=16,18 \
                           --output=output.fits

   To learn more about the core concepts behind the zero point, please
see *note Brightness flux magnitude::.  For a practical review of how to
optimally use this script and ways to interpret its results, we have two
tutorials: *note Zero point tutorial with reference image:: and *note
Zero point tutorial with reference catalog::.

   To find the zero point of your input image, this script can use a
reference image (that already has a zero point) or a reference catalog
(that just has magnitudes).  In any case, it is mandatory to identify at
least one aperture for aperture photometry over the image (using
‘--aperarcsec’).  If reference image(s) is(are) given, it is mandatory
to specify its(their) zero point(s) using the ‘--refimgszp’ option (it
can take a separate value for each reference image).  When a catalog is
given, it should already contain the magnitudes of the object (you can
specify which column to use).

   This script will not estimate the zero point based on all the objects
in the reference image or catalog.  It will first query Gaia database
and only select objects have a significant parallax (because Gaia’s
algorithms sometimes confuse galaxies and stars based on pure
morphology).  You can bypass this step (which needs internet connection
and can only be used on real data, not simulations) using the
‘--starcat’ option described in *note zero point options::.  This script
will then match the catalog of stars (either through Gaia or
‘--starcat’) with the reference catalog and only use them.  If the
reference is an image, it will simply use the stars catalog to do
aperture photometry.

   By default, this script will estimate the number of available threads
and run all independent steps in parallel on those threads.  To control
this behavior (and allow it to only run on a certain number of threads),
you can use the ‘--numthreads’ option.

   During its operation, this script will build a temporary file in the
running directory that will be deleted once it is finished.  The
‘--tmpdir’ option can be used to manually set the temporary directory’s
location at any location in your file system.  The ‘--keeptmp’ option
can be used to stop the deletion of that directory (useful for when you
want to debug the script or better understand what it does).

* Menu:

* zero point output::           Format of the output.
* zero point options::          List and details of options.


File: gnuastro.info,  Node: zero point output,  Next: zero point options,  Prev: Invoking astscript-zeropoint,  Up: Invoking astscript-zeropoint

10.5.1.1 astscript-zeropoint output
...................................

The output will be a multi-extension FITS table.  The first table in the
output gives the zero point and its standard deviation for all the
requested apertures.  This gives you the ability to inspect them and
select the best.  The other table(s) give the exact measurements for
each star that was used (if you use ‘--keepzpap’, it will be for all
your apertures, if not, only for the aperture with the smallest standard
deviation).  For a full tutorial on how to interpret the output of this
script, see *note Zero point tutorial with reference image::

   If you just want the estimated zero point with the least standard
deviation, this script will write it as a FITS keyword in the first
table of the output.

‘ZPAPER’
     Read as “Zero Point APERture”.  This shows the aperture radius (in
     arcseconds) that had the smallest standard deviation in the
     estimated zero points.
‘ZPVALUE’
     The zero point estimation for the aperture of ‘ZPAPER’.
‘ZPSTD’
     The standard deviation of the zero point (for all the stars used,
     within the aperture of ‘ZPAPER’).
‘ZPMAGMIN’
     The minimum (brightest) magnitude used to estimate the zero point.
‘ZPMAGMAX’
     The maximum (faintest) magnitude used to estimate the zero point.

A simple way to see these keywords, or read the value of one is shown
below.  For more on viewing and manipulating FITS keywords, see *note
Keyword inspection and manipulation::.

     ## See all the keywords written by this script (that start with 'ZP')
     $ astfits out.fits -h1 | grep ^ZP

     ## If you just want the zero point
     $ astfits jplus-zeropoint.fits -h1 --keyvalue=ZPVALUE


File: gnuastro.info,  Node: zero point options,  Prev: zero point output,  Up: Invoking astscript-zeropoint

10.5.1.2 astscript-zeropoint options
....................................

All the operating phases of the this script can be customized through
the options below.

‘-h STR/INT’
‘--hdu=STR/INT’
     The HDU/extension of the input image to use.

‘-o STR’
‘--output=STR’
     The name of the output file produced by this script.  See *note
     zero point output:: for the format of its contents.

‘-N INT’
‘--numthreads=INT’
     The number of threads to use.  By default this script will attempt
     to find the number of available threads at run-time and will use
     them.

‘-a FLT,[FLT]’
‘--aperarcsec=FLT,[FLT]’
     The radius/radii (in arc seconds) of aperture(s) used in aperture
     photometry of the input image.  This option can take many values
     (to check different apertures and find the best for a more accurate
     zero point estimation).  If a reference image is used, the same
     aperture radii will be used for aperture photometry there.

‘-M FLT,FLT’
‘--magnituderange=FLT,FLT’
     Range of the magnitude for finding the best aperture and zero
     point.  Very bright stars get saturated and fainter stars are
     affected too much by noise.  Therefore, it is important to limit
     the range of magnitudes used in estimating the zero point.  A full
     tutorial is given in *note Zero point tutorial with reference
     image::.

‘-S STR’
‘--starcat=STR’
     Name of catalog containing the RA and Dec of positions for aperture
     photometry in the input image and reference (catalog or image).  If
     not given, the Gaia database will be queried for all stars that
     overlap with the input image (see *note Available databases::).

     This option is therefore useful in the following scenarios (among
     others):
        • No internet connection.
        • Many images having a major overlap in the sky, making it
          inefficient to query Gaia for every image separately: you can
          query the larger area (containing all the images) once, and
          directly give the downloaded table to all the separate runs of
          this script.  Especially if the field is wide, the download
          time can be the slowest part of this script.
        • In simulations (where you have a pre-defined list of stars).

     Through the ‘--starcathdu’, ‘--starcatra’ and ‘--starcatdec’
     options described below, you can specify the HDU, RA column and Dec
     Column within this file.

     The reference image or catalog probably have many objects that are
     not stars.  But it is only stars that have the same shape (the PSF)
     across the image(1).  Therefore

‘--starcathdu=STR/INT’
     The HDU name or number in file given to ‘--starcat’ (described
     above) that contains the table of RA and Dec positions for aperture
     photometry.  If not given, it is assumed that the table is in HDU
     number 1 (counting from 0).

‘--starcatra=STR/INT’
     The column name or number (in the table given to ‘--starcat’) that
     contains the Right Ascension.

‘--starcatdec=STR/INT’
     The column name or number (in the table given to ‘--starcat’) that
     contains the Declination.

‘-c STR’
‘--refcat=STR’
     Reference catalog used to estimate the zero point of the input
     image.  This option is mutually exclusive with (cannot be given at
     the same time as) ‘--refimgs’.  This catalog should have RA, Dec
     and Magnitude of the stars (that match with Gaia or ‘--starcat’).

‘-C STR/INT’
‘--refcathdu=STR/INT’
     The HDU/extension of the reference catalog will be calculated.

‘-r STR’
‘--refcatra=STR’
     Right Ascension column name of the reference catalog.

‘-d STR’
‘--refcatdec=STR’
     Declination column name of the reference catalog.

‘-m STR’
‘--refcatmag=STR’
     Magnitude column name of the reference catalog.

‘-s FLT’
‘--matchradius=FLT’
     Matching radius of stars (in arc seconds) and reference catalog in
     arc-seconds.  By default it is 0.2 arc seconds.

‘-R STR,[STR]’
‘--refimgs=STR,[STR]’
     Reference image(s) for estimating the zero point.  This option can
     take any number of separate file names, separated by a comma.  The
     HDUs of each reference image should be given to the ‘refimgshdu’
     option.  In case the images are in separate HDUs of the same file,
     you need to repeat the file name here.  This option is mutually
     exclusive with (cannot be given at the same time as) ‘--refimgs’.

‘-H STR/INT’
‘--refimgshdu=STR/INT’
     HDU/Extension name of number of the reference files.  The number of
     values given to this option should be the same as the number of
     reference image(s).

‘-z FLT,[FLT]’
‘--refimgszp=FLT,[FLT]’
     Zero point of the reference image(s).  The number of values given
     to this should be the same as the number of names given to
     ‘--refimgs’.

‘-K’
‘--keepzpap’
     Keep the table of separate zero points found for each star for all
     apertures.  By default, this table is only present for the aperture
     that had the least standard deviation in the estimated zero point.

‘-t’
‘--tmpdir’
     Directory to keep temporary files during the execution of the
     script.  If the directory does not exist at run$-$time, this script
     will create it.  By default, upon completion of the script, this
     directory will be deleted.  However, if you would like to keep the
     intermediate files, you can use the ‘--keeptmp’ option.

‘-k’
‘--keeptmp’
     Its recommended to not remove the temporary directory (see
     description of ‘--keeptmp’).  This option is useful for debugging
     and checking the outputs of internal steps.

‘--mksrc=STR’
     Use a non-standard Makefile for the Makefile to call.  This option
     is primarily useful during the development of this script and its
     Makefile, not for normal/regular user.  So if you are not
     developing this script, you can safely ignore this option.  When
     this option is given, the default installed Makefile will not be
     used: the file given to this option will be read by ‘make’ (within
     the script) instead.

   ---------- Footnotes ----------

   (1) The PSF itself can vary across the field of view; but that is
second-order for this analysis.


File: gnuastro.info,  Node: Pointing pattern simulation,  Next: PSF construction and subtraction,  Prev: Zero point estimation,  Up: Installed scripts

10.6 Pointing pattern simulation
================================

Astronomical images are often composed of many single exposures.  When
the science topic does not depend on the time of observation (for
example galaxy evolution), after completing the observations, we stack
those single exposures into one “deep” image.  Designing the strategy to
take those single exposures is therefore a very important aspect of
planning your astronomical observation.  There are many reasons for
taking many short exposures instead of one long exposure:

   • Modern astronomical telescopes have very high precision (with
     pixels that are often much smaller than an arc-second or 1/3600
     degrees.  However, the Earth is orbiting the Sun at a very high
     speed of roughly 15 degrees every hour!  Keeping the (often very
     large!)  telescopes in track with this fast moving sky is not easy;
     such that most cannot continue accurate tracking more than 10
     minutes.
   • For ground-based observations, the turbulence of the atmosphere
     changes very fast (on the scale of minutes!).  So if you plan to
     observe at 10 minutes and at the start of your observations the
     seeing is good, it may happen that on the 8th minute, it becomes
     bad.  This will affect the quality of your final exposure!
   • When an exposure is taken, the instrument/environment imprint a lot
     of artifacts on it.  One common example that we also see in normal
     cameras is vignetting (https://en.wikipedia.org/wiki/Vignetting);
     where the center receives a larger fraction of the incoming light
     than the periphery).  In order to characterize and remove such
     artifacts (which depend on many factors at the precision that we
     need in astronomy!), we need to take many exposures of our science
     target.
   • By taking many exposures we can build a stack that has a higher
     resolution; this is often done in under-sampled data, like those in
     the Hubble Space Telescope (HST) or James Webb Space Telescope
     (JWST).
   • The scientific target can be larger than the field of view of your
     telescope and camera.

   In the jargon of observational astronomers, each exposure is also
known as a “dither” (literally/generally meaning “trembling” or
“vibration”).  This name was chosen because two exposures are not
usually taken on exactly the same position of the sky (known as
“pointing”).  In order to improve all the item above, we often move the
center of the field of view from one exposure to the next.  In most
cases this movement is small compared to the field of view, so most of
the central part of the final stack has a fixed depth, but the edges are
shallower (conveying a sense of vibration).  When the spacing between
pointings is large, they are known as an “offset”.  A “pointing” is used
to refer to either a dither or an offset.

   For example see Figures 3 and 4 of Illingworth et al.  2013
(https://arxiv.org/pdf/1305.1931.pdf) which show the exposures that went
into the XDF survey.  The pointing pattern can also be large compared to
the field of view, for example see Figure 1 of Trujillo et al.  2021
(https://arxiv.org/pdf/2109.07478.pdf), which show the pointing strategy
for the LIGHTS survey.  These types of images (where each pixel contains
the number of exposures, or time, that were used in it) are known as
exposure maps.

   The pointing pattern therefore is strongly defined by the science
case (high-level purpose of the observation) and your telescope’s field
of view.  For example in the XDF survey is focused on very high redshift
(most distant!)  galaxies.  These are very small objects and within that
small footprint (of just 1 arcmin) we have thousands of them.  However,
the LIGHTS survey is focused on the halos of large nearby galaxies (that
can be more than 10 arcminutes wide!).

   In *note Invoking astscript-pointing-simulate:: of Gnuastro’s *note
Installed scripts:: is described in detail.  This script is designed to
simplify the process of selecting the best pointing pattern for your
observation strategy.  For a practical tutorial on using this script,
see *note Pointing pattern design::.

* Menu:

* Invoking astscript-pointing-simulate::  Options and running mode.


File: gnuastro.info,  Node: Invoking astscript-pointing-simulate,  Prev: Pointing pattern simulation,  Up: Pointing pattern simulation

10.6.1 Invoking astscript-pointing-simulate
-------------------------------------------

This installed script will simulate a final stacked image from a certain
pointing pattern (given as a table).  A general overview of this script
has been published in Akhlaghi (2023)
(https://ui.adsabs.harvard.edu/abs/2023RNAAS...7..211A); please cite it
if this script proves useful in your research.  The executable name is
‘astscript-pointing-simulate’, with the following general template:

     $ astscript-pointing-simulate [OPTION...] pointings.fits

Examples (for a tutorial, see *note Pointing pattern design::):

     $ astscript-pointing-simulate pointing.fits --output=stack.fits \
                --img=image.fits --center=10,10 --width=1,1

   The default output of this script is a stacked image that results
from placing the given image (given to ‘--img’) in the pointings of a
pointing pattern.  The Right Ascension (RA) and Declination (Dec) of
each pointing is given in the main input catalog (‘pointing.fits’ in the
example above).  The center and width of the final stack (both in
degrees by default) should be specified using the ‘--width’ option.
Therefore, in order to successfully run, this script at least needs the
following four inputs:
Pointing positions
     A table containing the RA and Dec of each pointing (the only input
     argument).  The actual column names that contain them can be set
     with the ‘--racol’ and ‘--deccol’ options (see below).
An image
     This is used for its distortion and rotation, its pixel values and
     position on the sky will be ignored.  The file containing the image
     should be given to the ‘--img’ option.
Stack’s central coordinate
     The central RA and Dec of the finally produced stack (given to the
     ‘--center’ option).
Stack’s width
     The width (in degrees) of the final stack (given to the ‘--width’
     option).

   This script will ignore the pixel values of the reference image
(given to ‘--img’) and the Reference coordinates (values to ‘CRVAL1’ and
‘CRVAL2’ in its WCS keywords).  For each pointing pointing, this script
will put the given RA and Dec into the ‘CRVAL1’ and ‘CRVAL2’ keywords of
a copy of the input (not changing the input in anyway), and reset that
input’s pixel values to 1.  The script will then warp the modified copy
into the final pixel grid (correcting any rotation and distortions that
are used from the original input).  This process is done for all the
pointing points in parallel.  Finally, all the exposures in the pointing
list are stacked together to produce an exposure map (showing how many
exposures go into each pixel of the final stack.

   Except for the table of pointing positions, the rest of the inputs
and settings are configured through *note Options::, just note the
restrictions in *note Installed scripts::.

‘-o STR’
‘--output=STR’
     Name of the output.  The output is an image of the requested size
     (‘--width’) and position (‘--center’) in the sky, but each pixel
     will contain the number of exposures that go into it after the
     pointing has been done.  See description above for more.

‘-h STR/INT’
‘--hdu=STR/INT’
     The name or counter (counting from zero; from the FITS standard) of
     HDU containing the table of pointing pointing positions (the file
     name of this table is the main input argument to this script).  For
     more, see the description of this option in *note Input output
     options::.

‘-i STR’
‘--img=STR’
     The references image.  The pixel values and central location in
     this image will be ignored by the script.  The only relevant
     information within this script are the WCS properties (except for
     ‘CRVAL1’ and ‘CRVAL2’, which connect it to a certain position on
     the sky) and image size.  See the description above for more.

‘-H STR/INT’
‘--imghdu=STR/INT’
     The name or counter (counting from zero; from the FITS standard) of
     the HDU containing the reference image (file name should be given
     to the ‘--img’ option).  If not given, a default value of ‘1’ is
     assumed; so this is not a mandatory option.

‘-r STR/INT’
‘--racol=STR/INT’
     The name or counter (counting from 1; from the FITS standard) of
     the column containing the Right Ascension (RA) of each pointing to
     be used in the pointing pattern.  The file containing the table is
     given to this script as its only argument.

‘-d STR/INT’
‘--deccol=STR/INT’
     The name or counter (counting from 1; from the FITS standard) of
     the column containing the Declination (Dec) of each pointing to be
     used in the pointing pattern.  The file containing the table is
     given to this script as its only argument.

‘-C FLT,FLT’
‘--center=FLT,FLT’
     The central RA and Declination of the final stack in degrees.

‘-w FLT,FLT’
‘--width=FLT,FLT’
     The width of the final stack in degrees.  If ‘--widthinpix’ is
     given, the two values given to this option will be interpreted as
     degrees.

‘--widthinpix’
     Interpret the values given to ‘--width’ as number of pixels along
     each dimension), and not as degrees.

‘--ctype=STR,STR’
     The projection of the output stack (‘CTYPEi’ keyword in the FITS
     WCS standard).  For more, see the description of the same option in
     *note Align pixels with WCS considering distortions::.

‘--hook-warp-before='STR'’
     Command to run before warping each exposure into the output pixel
     grid.  By default, the exposure is immediately warped to the final
     pixel grid, but in some scenarios it is necessary to do some
     operations on the exposure before warping (for example account for
     vignetting; see *note Accounting for non-exposed pixels::).  The
     warping of each exposure is done in parallel by default; therefore
     there are pre-defined variables that you should use for the input
     and output file names of your command:
     ‘$EXPOSURE’
          Input: name of file with the same size as the reference image
          with all pixels having a fixed value of 1.  The WCS has also
          been corrected based on the pointing pattern.
     ‘$TOWARP’
          Output: name of the expected output of your hook.  If it is
          not created by your script, the script will complain and
          abort.  This file will be given to Warp to be warped into the
          output pixel grid.

     For an example of using hooks with an extended discussion, see
     *note Pointing pattern design:: and *note Accounting for
     non-exposed pixels::.

     To develop your command, you can use ‘--hook-warp-before='...; echo
     GOOD; exit 1'’ (where ‘...’ can be replaced by any command) and run
     the script on a single thread (with ‘--numthreads=1’) to produce a
     single file and simplify the checking that your desired operation
     works as expected.  All the files will be within the temporary
     directory (see ‘--tmpdir’).

‘--hook-warp-after='STR'’
     Command to run after the warp of each exposure into the output
     pixel grid, but before the stacking of all exposures.  For more on
     hooks, see the description of ‘--hook-warp-before’, *note Pointing
     pattern design:: and *note Accounting for non-exposed pixels::.

     ‘$WARPED’
          Input: name of file containing the warped exposure in the
          output pixel grid.
     ‘$TOWARP’
          Output: name of the expected output of your hook.  If it is
          not created by your script, the script will complain and
          abort.  This file will be stacked from the same file for all
          exposures into the final output.

‘--stack-operator="STR"’
     The operator to use for stacking the warped individual exposures
     into the final output of this script.  For the full list, see *note
     Stacking operators::.  By default it is the ‘sum’ operator (to
     produce an output exposure map).  For an example usage, see the
     tutorial in *note Pointing pattern design::.

‘--mksrc=STR’
     Use a non-standard Makefile for the Makefile to call.  This option
     is primarily useful during the development of this script and its
     Makefile, not for normal/regular usage.  So if you are not
     developing this script, you can safely ignore this option.  When
     this option is given, the default installed Makefile will not be
     used: the file given to this option will be read by ‘make’ (within
     the script) instead.

‘-t STR’
‘--tmpdir=STR’
     Name of directory containing temporary files.  If not given, a
     temporary directory will be created in the running directory with a
     long name using some of the input options.  By default, this
     temporary directory will be deleted after the output is created.
     You can disable the deletion of the temporary directory (useful for
     debugging!)  with the ‘--keeptmp’ option.

     Using this option has multiple benefits in larger pipelines:
        • You can avoid conflicts in case the used inputs in the default
          name are the same.
        • You can put this directory somewhere else in the running file
          system to avoid mixing output files with your source, or to
          use other storage hardware that are mounted on the running
          file system.

‘-k’
‘--keeptmp’
     Keep the temporary directory (and do not delete it).

‘-?’
‘--help’
     Print a list of all the options, along with a short description and
     context for the program.  For more, see ‘Operating mode options’.

‘-N INT’
‘--numthreads=INT’
     The number of threads to use for parallel operations (warping the
     input into the different pointing points).  If not given (by
     default), the script will try to find the number of available
     threads on the running system and use that.  For more, see
     ‘Operating mode options’.

‘--cite’
     Give BibTeX and acknowledgment information for citing this script
     within your paper.  For more, see ‘Operating mode options’.

‘-q’
‘--quiet’
     Do not print the series of commands or their outputs in the
     terminal.  For more, see ‘Operating mode options’.

‘-V’
‘--version’
     Print the version of the running Gnuastro along with a copyright
     notice and list of authors that contributed to this script.  For
     more, see ‘Operating mode options’.


File: gnuastro.info,  Node: PSF construction and subtraction,  Prev: Pointing pattern simulation,  Up: Installed scripts

10.7 PSF construction and subtraction
=====================================

The point spread function (PSF) describes how the light of a point-like
source is affected by several optical scattering effects (atmosphere,
telescope, instrument, etc.).  Since the light of all astrophysical
sources undergoes all these effects, characterizing the PSF is key in
astronomical analysis (for small and large objects).  Consequently,
having a good characterization of the PSF is fundamental to any
analysis.

   In some situations(1) a parametric (analytical) model is sufficient
for the PSF (such as Gaussian or Moffat, see *note PSF::).  However,
once you are interested in objects that are larger than a handful of
pixels, it is almost impossible to find an analytic function to
adequately characterize the PSF. Therefore, it is necessary to obtain an
empirical (non-parametric) and extended PSF. In this section we describe
a set of installed scrips in Gnuastro that will let you construct the
non-parametric PSF using point-like sources.  They allow you to derive
the PSF from the same astronomical images that the science is derived
from (without assuming any analytical function).

   The scripts are based on the concepts described in Infante-Sainz et
al.  (2020, <https://arxiv.org/abs/1911.01430>).  But to be complete, we
first give a summary of the logic and overview of their combined usage
in *note Overview of the PSF scripts::.  Furthermore, before going into
the technical details of each script, we encourage you to go through the
tutorial that is devoted to this at *note Building the extended PSF::.
The tutorial uses a real dataset and includes all the logic and
reasoning behind every step of the usage in every installed script.

* Menu:

* Overview of the PSF scripts::  Summary of concepts and methods
* Invoking astscript-psf-select-stars::  Select good starts within an image.
* Invoking astscript-psf-stamp::  Make a stamp of each star to stack.
* Invoking astscript-psf-unite::  Merge stacks of different regions of PSF.
* Invoking astscript-psf-scale-factor::  Calculate factor to scale PSF to star.
* Invoking astscript-psf-subtract::  Put the PSF in the image to subtract.

   ---------- Footnotes ----------

   (1) An example scenario where a parametric PSF may be enough: you are
only interested in very small, high redshift objects that only extended
a handful of pixels.


File: gnuastro.info,  Node: Overview of the PSF scripts,  Next: Invoking astscript-psf-select-stars,  Prev: PSF construction and subtraction,  Up: PSF construction and subtraction

10.7.1 Overview of the PSF scripts
----------------------------------

To obtain an extended and non-parametric PSF, several steps are
necessary and we will go through them here.  The fundamental ideas of
the following methodology are thoroughly described in Infante-Sainz et
al.  (2020, <https://arxiv.org/abs/1911.01430>).  A full tutorial is
also available in *note Building the extended PSF::.  The tutorial will
go through the full process on a pre-selected dataset, but will describe
the logic behind every step in away that can easily be
modified/generalized to other datasets.

   This section is basically just a summary of that tutorial.  We could
have put all these steps into one large program (installed script),
however this would introduce several problems.  The most prominent of
these problems are:

   • The command would require _many_ options, making it very complex to
     run every time.
   • You usually have many stars in an image, and many of the steps can
     be optimized or parallelized depending on the particular analysis
     scenario.  Predicting all the possible optimizations for all the
     possible usage scenarios would make the code extremely complex
     (filled with many unforeseen bugs!).

   Therefore, following the modularity principle of software
engineering, after several years of working on this, we have broken the
full job into the smallest number of independent steps as separate
scripts.  All scripts are independent of each other, meaning this that
you are free to use all of them as you wish (for example, only some of
them, using another program for a certain step, using them for other
purposes, or running independent parts in parallel).

   For constructing the PSF from your dataset, the first step is to
obtain a catalog of stars within it (you cannot use galaxies to build
the PSF!). But you cannot blindly use all the stars either!  For
example, we do not want contamination from other bright, and nearby
objects.  The first script below is therefore designed for selecting
only good star candidates in your image.  It will use different
criteria, for example, good parallax (where available, to avoid
confusion with galaxies), not being near to bright stars, axis ratio,
etc.  For more on this script, see *note Invoking
astscript-psf-select-stars::.

   Once the catalog of stars is constructed, another script is in charge
of making appropriate stamps of the stars.  Each stamp is a cropped
image of the star with the desired size, normalization of the flux, and
mask of the contaminant objects.  For more on this script, see *note
Invoking astscript-psf-stamp:: After obtaining a set of star stamps,
they can be stacked for obtaining the combined PSF from many stars (for
example, with *note Stacking operators::).

   In the combined PSF, the masked background objects of each star’s
image will be covered and the signal-to-noise ratio will increase,
giving a very nice view of the “clean” PSF. However, it is usually
necessary to obtain different regions of the same PSF from different
stars.  For example, to construct the far outer wings of the PSF, it is
necessary to consider very bright stars.  However, these stars will be
saturated in the most inner part, and immediately outside of the
saturation level, they will be deformed due to non-linearity effects.
Consequently, fainter stars are necessary for the inner regions.

   Therefore, you need to repeat the steps above for certain stars (in a
certain magnitude range) to obtain the PSF in certain radial ranges.
For example, in Infante-Sainz et al.  (2020,
<https://arxiv.org/abs/1911.01430>), the final PSF was constructed from
three regions (and thus, using stars from three ranges in magnitude).
In other cases, we even needed four groups of stars!  But in the example
dataset from the tutorial, only two groups are necessary (see *note
Building the extended PSF::).

   Once clean stacks of different parts of the PSF have been constructed
through the steps above, it is therefore necessary to blend them all
into one.  This is done by finding a common radial region in both, and
scaling the inner region by a factor to add with the outer region.  This
is not trivial, therefore, a third script is in charge of it, see *note
Invoking astscript-psf-unite::.

   Having constructed the PSF as described above (or by any other
procedure), it can be scaled to the magnitude of the various stars in
the image to get subtracted (and thus remove the extended/bright wings;
better showing the background objects of interest).  Note that the
absolute flux of a PSF is meaningless (and in fact, it is usually
normalized to have a total sum of unity!), so it should be scaled.  We
therefore have another script that will calculate the scale
(multiplication) factor of the PSF for each star.  For more on the
scaling script, see *note Invoking astscript-psf-scale-factor::.

   Once the flux factor has been computed, a final script is in charge
of placing the scaled PSF over the proper location in the image, and
subtracting it.  It is also possible to only obtain the modeled star by
the PSF. For more on the scaling and positioning script, see *note
Invoking astscript-psf-subtract::.

   As mentioned above, in the following sections, each script has its
own documentation and list of options for very detailed customization
(if necessary).  But if you are new to these scripts, before continuing,
we recommend that you do the tutorial *note Building the extended PSF::.
Just do not forget to run every command, and try to tweak its steps
based on the logic to nicely understand it.


File: gnuastro.info,  Node: Invoking astscript-psf-select-stars,  Next: Invoking astscript-psf-stamp,  Prev: Overview of the PSF scripts,  Up: PSF construction and subtraction

10.7.2 Invoking astscript-psf-select-stars
------------------------------------------

This installed script will select good star candidates for constructing
a PSF. It will consider stars within a given range of magnitudes without
nearby contaminant objects.  To do that, it allows to the user to
specify different options described here.  A complete tutorial is
available to show the operation of this script as a modular component to
extract the PSF of a dataset: *note Building the extended PSF::.  The
executable name is ‘astscript-psf-select-stars’, with the following
general template:

     $ astscript-psf-select-stars [OPTION...] FITS-file

Examples:

     ## Select all stars within 'image.fits' with magnitude in range
     ## of 6 to 10; only keeping those that are less than 0.02 degrees
     ## from other nearby stars.
     $ astscript-psf-select-stars image.fits \
                --magnituderange=6,10 --mindistdeg=0.02

   The input of this script is an image, and the output is a catalog of
stars with magnitude in the requested range of magnitudes (provided with
‘--magnituderange’).  The output catalog will also only contain stars
that are sufficiently distant (‘--mindistdeg’) from all other brighter,
and some fainter stars.  It is possible to consider different datasets
with the option ‘--dataset’ (by default, Gaia DR3 dataset is considered)
All stars that are ‘--faintmagdiff’ fainter than the faintest limit will
also be accounted for, when selecting good stars.  The
‘--magnituderange’, and ‘--mindistdeg’ are mandatory: if not specified
the code will abort.

   The output of this script is a file whose name can be specified with
the (optional) ‘--output’ option.  If not given, an automatically
generated name will be used for the output.  A full description of each
option is given below.

‘-h STR/INT’
‘--hdu=STR/INT’
     The HDU/extension of the input image to use.

‘-S STR’
‘--segmented=STR’
     Optional segmentation file obtained by *note Segment::.  It should
     have two extensions (‘CLUMPS’ and ‘OBJECTS’).  If given, a catalog
     of ‘CLUMPS’ will be computed and matched with the Gaia catalog to
     reject those objects that are too elliptical (see
     ‘--minaxisratio’).  The matching will occur on an aperture (in
     degrees) specified by ‘--matchaperturedeg’.

‘-a FLT’
‘--matchaperturedeg=FLT’
     This option determines the aperture (in degrees) for matching the
     catalog from gaia with the clumps catalog that is produced by the
     segmentation image given to ‘--segmented’.  The default value is 10
     arc-seconds.

‘-c STR’
‘--catalog=STR’
     Optional reference catalog to use for selecting stars (instead of
     querying an external catalog like Gaia).  When this option is
     given, ‘--dataset’ (described below) will be ignored and no
     internet connection will be necessary.

‘-d STR’
‘--dataset=STR’
     Optional dataset to query (see *note Query::).  It should contain
     the database and dataset entries to Query.  Its value will be
     immediately given to ‘astquery’.  By default, its value is ‘gaia
     --dataset=dr3’ (so it connects to the Gaia database and requests
     the data release 3).  For example, if you want to use VizieR’s Gaia
     DR3 instead (for example due to a maintenance on ESA’s Gaia
     servers), you should use ‘--dataset="vizier --dataset=gaiadr3"’.

     It is possible to specify a different dataset from which the
     catalog is downloaded.  In that case, the necessary column names
     may also differ, so you also have to set ‘--refcatra’,
     ‘--refcatdec’ and ‘--field’.  See their description for more.

‘-r STR’
‘--refcatra=STR’
     The name of the column containing the Right Ascension (RA) in the
     requested dataset (‘--dataset’).  If the user does not determine
     this option, the default value is assumed to be ‘ra’.

‘-d STR’
‘--refcatdec=STR’
     The name of the column containing the Declination (Dec) in the
     requested dataset (‘--dataset’).  If the user does not determine
     this option, the default value is assumed to be ‘dec’.

‘-f STR’
‘--field=STR’
     The name of the column containing the magnitude in the requested
     dataset (‘--dataset’).  The output will only contain stars that
     have a value in this column, between the values given to
     ‘--magnituderange’ (see below).  By default, the value of this
     option is ‘phot_g_mean_mag’ (that corresponds to the name of the
     magnitude of the G-band in the Gaia catalog).

‘-m FLT,FLT’
‘--magnituderange=FLT,FLT’
     The acceptable range of values for the column in ‘--field’.  This
     option is mandatory and no default value is assumed.

‘-p STR,STR’
‘--parallaxanderrorcolumn=STR,STR’
     With this option the user can provide the parallax and parallax
     error column names in the requested dataset.  When given, the
     output will only contain stars for which the parallax value is
     smaller than three times the parallax error.  If the user does not
     provide this option, the script will not use parallax information
     for selecting the stars.  In the case of Gaia, if you want to use
     parallax to further limit the good stars, you can pass
     ‘parallax,parallax_error’.

‘-D FLT’
‘--mindistdeg=FLT’
     Stars with nearby bright stars closer than this distance are
     rejected.  The default value is 1 arc minute.  For fainter stars
     (when constructing the center of the PSF), you should decrease the
     value.

‘-b INT’
‘--brightmag=INT’
     The brightest star magnitude to avoid (should be brighter than the
     brightest of ‘--magnituderange’).  The basic idea is this: if a
     user asks for stars with magnitude 6 to 10 and one of those stars
     is near a magnitude 3 star, that star (with a magnitude of 6 to 10)
     should be rejected because it is contaminated.  But since the
     catalog is constrained to stars of magnitudes 6-10, the star with
     magnitude 3 is not present and cannot be compared with!  Therefore,
     when considering proximity to nearby stars, it is important to use
     a larger magnitude range than the user’s requested magnitude range
     for good stars.  The acceptable proximity is defined by
     ‘--mindistdeg’.

     With this option, you specify the brightest limit for the proximity
     check.  The default value is a magnitude of $-10$, so you’ll rarely
     need to change or customize this option!

     The faint limit of the proximity check is specified by
     ‘--faintmagdiff’.  As the name suggests, this is a “diff” or
     relative value.  The default value is 4.  Therefore if the user
     wants to build the PSF with stars in the magnitude range of 6 to
     10, the faintest stars used for the proximity check will have a
     magnitude of 14: $10+4$.  In summary, by default, the proximity
     check will be done with stars in the magnitude range $-10$ to $14$.

‘-F INT’
‘--faintmagdiff’
     The magnitude difference of the faintest star used for proximity
     checks to the faintest limit of ‘--magnituderange’.  For more, see
     the description of ‘--brightmag’.

‘-Q FLT’
‘--minaxisratio=FLT’
     Minimum acceptable axis ratio for the selected stars.  In other
     words, only stars with axis ratio between ‘--minaxisratio’ to 1.0
     will be selected.  Default value is ‘--minaxisratio=0.9’.  Recall
     that the axis ratio is only used when you also give a segmented
     image with ‘--segmented’.

‘-t’
‘--tmpdir’
     Directory to keep temporary files during the execution of the
     script.  If the directory does not exist at run-time, this script
     will create it.  By default, upon completion of the script, this
     directory will be deleted.  However, if you would like to keep the
     intermediate files, you can use the ‘--keeptmp’ option.

‘-k’
‘--keeptmp’
     Do not remove the temporary directory (see description of
     ‘--keeptmp’).  This option is useful for debugging and checking the
     outputs of internal steps.

‘-o STR’
‘--output=STR’
     The output name of the final catalog containing good stars.


File: gnuastro.info,  Node: Invoking astscript-psf-stamp,  Next: Invoking astscript-psf-unite,  Prev: Invoking astscript-psf-select-stars,  Up: PSF construction and subtraction

10.7.3 Invoking astscript-psf-stamp
-----------------------------------

This installed script will generate a stamp of fixed size, centered at
the provided coordinates (performing sub-pixel re-gridding if necessary)
and normalized at a certain normalization radius.  Optionally, it will
also mask all the other background sources.  A complete tutorial is
available to show the operation of this script as a modular component to
extract the PSF of a dataset: *note Building the extended PSF::.  The
executable name is ‘astscript-psf-stamp’, with the following general
template:

     $ astscript-psf-stamp [OPTION...] FITS-file

Examples:

     ## Make a stamp around (x,y)=(53,69) of width=151 pixels.
     ## Normalize the stamp within the radii 20 and 30 pixels.
     $ astscript-psf-stamp image.fits --mode=img \
           --center=53,69 --widthinpix=151,151 --normradii=20,30 \
           --output=stamp.fits

     ## Iterate over a catalog with positions of stars that are
     ## in the input image. Use WCS coordinates.
     $ asttable catalog.fits | while read -r ra dec mag; do \
         astscript-psf-stamp image.fits \
             --mode=wcs \
             --center=$ra,$dec \
             --normradii=20,30 \
             --widthinpix=150,150 \
             --output=stamp-"$ra"-"$dec".fits; done


   The input is an image from which the stamp of the stars are
constructed.  The output image will have the following properties:
   • A certain width (specified by ‘--widthinpix’ in pixels).
   • Centered at the coordinate specified by the option ‘--center’ (it
     can be in image/pixel or WCS coordinates, see ‘--mode’).  If no
     center is specified, then it is assumed that the object of interest
     is already in the center of the image.
   • If the given coordinate has sub-pixel elements (for example, pixel
     coordinates 1.234,4.567), the pixel grid of the output will be
     warped so your given coordinate falls in the center of the central
     pixel of the final output.  This is very important for building the
     central parts of the PSF, but not too effective for the middle or
     outer parts (to speed up the program in such cases, you can disable
     it with the ‘--nocentering’ option).
   • Normalized “normalized” by the value computed within the ring
     around the center (at a radial distance between the two radii
     specified by the option ‘--normradii’).  If no normalization ring
     is considered, the output will not be normalized.

   In the following cases, this script will produce a fully NaN-valued
stamp (of the size given to ‘--widthinpix’).  A fully NaN image can
safely be used with the stacking operators of Arithmetic (see *note
Stacking operators::) because they will be ignored.  In case you do not
want to waste storage with fully NaN images, you can compress them with
‘gzip --best output.fits’, and give the resulting ‘.fits.gz’ file to
Arithmetic.
   • The requested box (center coordinate with desired width) is not
     within the input image at all.
   • If a normalization radius is requested, and all the pixels within
     the normalization radii are NaN. Here are some scenarios that this
     can happen: 1) You have a saturated star (where the saturated
     pixels are NaN), and your normalization radius falls within the
     saturated region.  2) The star is outside the image by more than
     your larger normalization radius (so there are no pixels for doing
     normalization), but the full stamp width still overlaps part of the
     image.

The full set of options are listed below for optimal customization in
different scenarios:

‘-h STR’
‘--hdu=STR’
     The HDU/extension of the input image to use.

‘-O STR’
‘--mode=STR’
     Interpret the center position of the object (values given to
     ‘--center’) in image or WCS coordinates.  This option thus accepts
     only two values: ‘img’ or ‘wcs’.

‘-c FLT,FLT’
‘--center=FLT,FLT’
     The central position of the object.  This option is used for
     placing the center of the stamp.  This parameter is used in *note
     Crop:: to center and crop the image.  The positions along each
     dimension must be separated by a comma (<,>).  The units of the
     coordinates are read based on the value to the ‘--mode’ option, see
     the examples above.

     The given coordinate for the central value can have sub-pixel
     elements (for example, it falls on coordinate 123.4,567.8 of the
     input image pixel grid).  In such cases, after cropping, this
     script will use Gnuastro’s *note Warp:: to shift (or translate) the
     pixel grid by $-0.4$ pixels along the horizontal and $1-0.8=0.2$
     pixels along the vertical.  Finally the newly added pixels (due to
     the warping) will be trimmed to have your desired coordinate
     exactly in the center of the central pixel of the output.  This is
     very important (critical!)  when you are constructing the central
     part of the PSF. But for the outer parts it is not too effective,
     so to avoid wasting time for the warping, you can simply use
     ‘--nocentering’ to disable it.

‘-d’
‘--nocentering’
     Do Not do the sub-pixel centering to a new pixel grid.  See the
     description of the ‘--center’ option for more.

‘-W INT,INT’
‘--widthinpix=INT,INT’
     Size (width) of the output image stamp in pixels.  The size of the
     output image will be always an odd number of pixels.  As a
     consequence, if the user specify an even number, the final size
     will be the specified size plus 1 pixel.  This is necessary to
     place the specified coordinate (given to ‘--center’) in the center
     of the central pixel.  This is very important (and necessary) in
     the case of the centers of stars, therefore a sub-pixel translation
     will be performed internally to ensure this.

‘-n FLT,FLT’
‘--normradii=FLT,FLT’
     Minimum and maximum radius of ring to normalize the image.  This
     option takes two values, separated by a comma (<,>).  The first
     value is the inner radius, the second is the outer radius.

‘-S STR’
‘--segment=STR’
     Optional filename of a segmentation image from Segment’s output
     (must contain the ‘CLUMPS’ and ‘OBJECT’ HDUs).  For more on the
     definition of “objects” and “clumps”, see *note Segment::.  If
     given, Segment’s output is used to mask all background sources from
     the large foreground object (a bright star):
        • Objects that are not the central object.
        • Clumps (within the central object) that are not the central
          clump.
     The result is that all objects and clumps that contaminate the
     central source are masked, while the diffuse flux of the central
     object remains.  The non masked object and clump labels are kept
     into the header of the output image.  The keywords are ‘CLABEL’ and
     ‘OLABEL’.  If no segmentation image is used, then their values are
     set to ‘none’.

‘-T FLT’
‘--snthresh=FLT’
     Mask all the pixels below the given signal-to-noise ratio (S/N)
     threshold.  This option is only valid with the ‘--segment’ option
     (it will use the ‘SKY_STD’ extension of the *note Segment output::.
     This threshold is applied prior to the possible normalization or
     centering of the stamp.  After all pixels below the given threshold
     are masked, the mask is also dilated by one level to avoid single
     pixels above the threshold (which are mainly due to noise when the
     threshold is lower).

     After applying the signal-to-noise threshold (if it is requested),
     any extra pixels that are not connected to the central target are
     also masked.  Such pixels can remain in rivers between bright
     clumps and will cause problem in the final stack, if they are not
     masked.

     This is useful for increasing the S/N of inner parts of each region
     of the finally stacked PSF. As the stars (that are to be stacked)
     become fainter, the S/N of their outer parts (at a fixed radius)
     decreases.  The stack of a higher-S/N image with a lower-S/N image
     will have an S/N that is lower than the higher one.  But we can
     still use the inner parts of those fainter stars (that have
     sufficiently high S/N).

‘-N STR’
‘--normop=STR’
     The operator for measuring the values within the ring defined by
     the option ‘--normradii’.  The operator given to this option will
     be directly passed to the radial profile script
     ‘astscript-radial-profile’, see *note Generate radial profile::.
     As a consequence, all MakeCatalog measurements (median, mean,
     sigclip-mean, sigclip-number, etc.)  can be used here.  For a full
     list of MakeCatalog’s measurements, please run ‘astmkcatalog
     --help’.  The final normalization value is saved into the header of
     the output image with the keyword ‘NORMVAL’.  If no normalization
     is done, then the value is set to ‘1.0’.

‘-Q FLT’
‘--axis-ratio=FLT’
     The axis ratio of the radial profiles for computing the
     normalization value.  By default (when this option is not given),
     the radial profile will be circular (axis ratio of 1).  This
     parameter is used directly in the ‘astscript-radial-profile’
     script.

‘-p FLT’
‘--position-angle=FLT’
     The position angle (in degrees) of the profiles relative to the
     first FITS axis (horizontal when viewed in SAO DS9).  By default,
     it is ‘--position-angle=0’, which means that the semi-major axis of
     the profiles will be parallel to the first FITS axis.  This
     parameter is used directly in the ‘astscript-radial-profile’
     script.

‘-s FLT,FLT’
‘--sigmaclip=FLT,FLT’
     Sigma clipping parameters: only relevant if sigma-clipping
     operators are requested by ‘--normop’.  For more on sigma-clipping,
     see *note Sigma clipping::.

‘-t’
‘--tmpdir’
     Directory to keep temporary files during the execution of the
     script.  If the directory does not exist at run-time, this script
     will create it.  By default, upon completion of the script, this
     directory will be deleted.  However, if you would like to keep the
     intermediate files, you can use the ‘--keeptmp’ option.

‘-k’
‘--keeptmp’
     Do not remove the temporary directory (see description of
     ‘--keeptmp’).  This option is useful for debugging and checking the
     outputs of internal steps.

‘-o STR’
‘--output=STR’
     Filename of stamp image.  By default the name of the stamp will be
     a combination of the input image name, the name of the script, and
     the coordinates of the center.  For example, if the input image is
     named image.fits and the center is ‘--center=33,78’, then the
     output name wil be: image_stamp_33_78.fits The main reason of
     setting this name is to have an unique name for each stamp by
     default.


File: gnuastro.info,  Node: Invoking astscript-psf-unite,  Next: Invoking astscript-psf-scale-factor,  Prev: Invoking astscript-psf-stamp,  Up: PSF construction and subtraction

10.7.4 Invoking astscript-psf-unite
-----------------------------------

This installed script will join two PSF images at a given radius.  This
operation is commonly used when merging (uniting) the inner and outer
parts of the PSF. A complete tutorial is available to show the operation
of this script as a modular component to extract the PSF of a dataset:
*note Building the extended PSF::.  The executable name is
‘astscript-psf-unite’, with the following general template:

     $ astscript-psf-unite [OPTION...] FITS-file

Examples:

     ## Multiply inner.fits by 3 and put it in the center (within a radius of
     ## 25 pixels) of outer.fits. The core goes up to a radius of 25 pixels.
     $ astscript-psf-unite outer.fits \
                --core=inner.fits --scale=3 \
                --radius=25 --output=joined.fits

     ## Same example than the above, but considering an
     ## ellipse (instead of a circle).
     $ astscript-psf-unite outer.fits \
                --core=inner.fits --scale=3 \
                --radius=25 --axis-ratio=0.5 \
                --position-angle=40 --output=joined.fits


   The junction is done by considering the input image as the outer
part.  The central part is specified by FITS image given to ‘--inner’
and it is multiplied by the factor ‘--scale’.  All pixels within
‘--radius’ (in pixels) of the center of the outer part are then replaced
with the inner image.

   The scale factor to multiply with the inner part has to be explicitly
provided (see the description of ‘--scale’ below).  Note that this
script assumes that PSF is centered in both images.  More options are
available with the goal of obtaining a good junction.  A full
description of each option is given below.

‘-h STR’
‘--hdu=STR’
     The HDU/extension of the input image to use.

‘-i STR’
‘--inner=STR’
     Filename of the inner PSF. This image is considered to be the
     central part of the PSF. It will be cropped at the radius specified
     by the option ‘--radius’, and multiplied by the factor specified by
     ‘--scale’.  After that, it will be appended to the outer part
     (input image).

‘-I STR’
‘--innerhdu=STR’
     The HDU/extension of the inner PSF (option ‘--inner’).

‘-f FLT’
‘--scale=FLT’
     Factor by which the inner part (‘--inner’) is multiplied.  This
     factor is necessary to put the two different parts of the PSF at
     the same flux level.  A convenient way of obtaining this value is
     by using the script ‘astscript-model-scale-factor’, see *note
     Invoking astscript-psf-scale-factor::.  There is also a full
     tutorial on using all the ‘astscript-psf-*’ installed scripts
     together, see *note Building the extended PSF::.  We recommend
     doing that tutorial before starting to work on your own datasets.

‘-r FLT’
‘--radius=FLT’
     Radius (in pixels) at which the junction of the images is done.
     All pixels in the outer image within this radius (from its center)
     will be replaced with the pixels of the inner image (that has been
     scaled).  By default, a circle is assumed for the shape of the
     inner region, but this can be tweaked with ‘--axis-ratio’ and
     ‘--position-angle’ (see below).

‘-Q FLT’
‘--axisratio=FLT’
     Axis ratio of ellipse to define the inner region.  By default this
     option has a value of 1.0, so all central pixels (of the outer
     image) within a circle of radius ‘--radius’ are replaced with the
     scaled inner image pixels.  With this option, you can customize the
     shape of pixels to take from the inner and outer profiles.

     For a PSF, it will usually not be necessary to change this option:
     even if the PSF is non-circular, the inner and outer parts will
     both have the same ellipticity.  So if the scale factor is chosen
     accurately, using a circle to select which pixels from the inner
     image to use in the outer image will be irrelevant.

‘-p FLT’
‘--position-angle=FLT’
     Position angle of the ellipse (in degrees) to define which central
     pixels of the outer image to replace with the scaled inner image.
     Similar to ‘--axis-ratio’ (see above).

‘-t’
‘--tmpdir’
     Directory to keep temporary files during the execution of the
     script.  If the directory does not exist at run-time, this script
     will create it.  By default, upon completion of the script, this
     directory will be deleted.  However, if you would like to keep the
     intermediate files, you can use the ‘--keeptmp’ option.

‘-k’
‘--keeptmp’
     Do not remove the temporary directory (see description of
     ‘--keeptmp’).  This option is useful for debugging and checking the
     outputs of internal steps.


File: gnuastro.info,  Node: Invoking astscript-psf-scale-factor,  Next: Invoking astscript-psf-subtract,  Prev: Invoking astscript-psf-unite,  Up: PSF construction and subtraction

10.7.5 Invoking astscript-psf-scale-factor
------------------------------------------

This installed script will compute the multiplicative factor (scale)
that is necessary to match the PSF to a given star.  The match in flux
is done within a ring of pixels.  It can also be used to compute the
scale factor to multiply the inner part of the PSF with the outer part
during the creation of a PSF. A complete tutorial is available to show
the operation of this script as a modular component to extract the PSF
of a dataset: *note Building the extended PSF::.  The executable name is
‘astscript-psf-scale-factor’, with the following general template:

     $ astscript-psf-scale-factor [OPTION...] FITS-file

Examples:

     ## Compute the scale factor for the object at (x,y)=(53,69) for
     ## the PSF (psf.fits). Compute it in the ring 20-30 pixels.
     $ astscript-psf-scale-factor image.fits --mode=img \
           --center=53,69 --normradii=20,30 --psf=psf.fits

     ## Iterate over a catalog with RA,Dec positions of stars that are in
     ## the input image to compute their scale factors.
     $ asttable catalog.fits | while read -r ra dec mag; do \
         astscript-psf-scale-factor image.fits \
             --mode=wcs \
             --psf=psf.fits \
             --center=$ra,$dec --quiet \
             --normradii=20,30 > scale-"$ra"-"$dec".txt; done


   The input should be an image containing the star that you want to
match in flux with the PSF. The output will be a single number that is
printed on the command-line.  That number is the multiplicative factor
to scale the PSF image (given to ‘--psf’) to match in flux with the
given star (which is located in ‘--center’ coordinate of the input
image).  The scale factor will be calculated within the ring of pixels
specified by the option ‘--normradii’.

   All the pixels within this ring will be separated from both the PSF
and input images.  For the input image, around the selected coordinate;
while masking all other sources (see ‘--segment’).  The finally selected
pixels of the input image will then be divided by those of the PSF
image.  This gives us an image containing one scale factor per pixel.
The finally reported value is the sigma-clipped median of all the scale
factors in the finally-used pixels.  To fully understand the process on
first usage, we recommend that you run this script with ‘--keeptmp’ and
inspect the files inside of the temporary directory.

   The most common use-cases of this scale factor are:
  1. To find the factor for joining two different parts of the same PSF,
     see *note Invoking astscript-psf-unite::.
  2. When modeling a star in order to subtract it using the PSF, see
     *note Invoking astscript-psf-subtract::.

   For a full tutorial on how to use this script along with the other
‘astscript-psf-*’ scripts in Gnuastro, please see *note Building the
extended PSF::.  To allow full customizability, the following options
are available with this script.

‘-h STR’
‘--hdu=STR’
     The HDU/extension of the input image to use.

‘-p STR’
‘--psf=STR’
     Filename of the PSF image.  The PSF is assumed to be centered in
     this image.

‘-P STR’
‘--psfhdu=STR’
     The HDU/extension of the PSF image.

‘-c FLT,FLT’
‘--center=FLT,FLT’
     The central position of the object to scale with the PSF. This
     parameter is passed to Gnuastro’s Crop program make a crop for
     further processing (see *note Crop::).  The positions along each
     dimension must be separated by a comma (<,>).  The units of the
     coordinates are interpreted based on the value to the ‘--mode’
     option (see below).

‘-O STR’
‘--mode=STR’
     Interpret the center position of the object (values given to
     ‘--center’) in image or WCS coordinates.  This option thus accepts
     only two values: ‘img’ or ‘wcs’.

‘-n INT,INT’
‘--normradii=INT,INT’
     Inner (inclusive) and outer (exclusive) radii (in units of pixels)
     around the central position in which the scale factor is computed.
     The option takes two values separated by a comma (<,>).  The first
     value is the inner radius, the second is the outer radius.  These
     two radii define a ring of pixels around the center that is used
     for obtaining the scale factor value.

‘-W INT,INT’
‘--widthinpix=INT,INT’
     Size (width) of the image stamp in pixels.  This is an intermediate
     product computed internally by the script.  By default, the size of
     the stamp is automatically set to be as small as possible (i.e.,
     two times the external radius of the ring specified by
     ‘--normradii’) to make the computation fast.  As a consequence,
     this option is only relevant for checking and testing that
     everything is fine (debugging; it will usually not be necessary).

‘-S STR’
‘--segment=STR’
     Optional filename of a segmentation image from Segment’s output
     (must contain the ‘CLUMPS’ and ‘OBJECT’ HDUs).  For more on the
     definition of “objects” and “clumps”, see *note Segment::.  If
     given, Segment’s output is used to mask all background sources from
     the large foreground object (a bright star):
        • Objects that are not the central object.
        • Clumps (within the central object) that are not the central
          clump.
     The result is that all objects and clumps that contaminate the
     central source are masked, while the diffuse flux of the central
     object remains.

‘-s FLT,FLT’
‘--sigmaclip=FLT,FLT’
     Sigma clipping parameters used in the end to find the final scale
     factor from the distribution of all pixels used.  For more on
     sigma-clipping, see *note Sigma clipping::.

‘-t’
‘--tmpdir’
     Directory to keep temporary files during the execution of the
     script.  If the directory does not exist at run-time, this script
     will create it.  By default, upon completion of the script, this
     directory will be deleted.  However, if you would like to keep the
     intermediate files, you can use the ‘--keeptmp’ option.

‘-k’
‘--keeptmp’
     Do not remove the temporary directory (see description of
     ‘--keeptmp’).  This option is useful for debugging and checking the
     outputs of internal steps.


File: gnuastro.info,  Node: Invoking astscript-psf-subtract,  Prev: Invoking astscript-psf-scale-factor,  Up: PSF construction and subtraction

10.7.6 Invoking astscript-psf-subtract
--------------------------------------

This installed script will put the provided PSF into a given position
within the input image (implementing sub-pixel adjustments where
necessary), and then it will subtract it.  It is aimed at modeling and
subtracting the scattered light field of an input image.  It is also
possible to obtain the modeled star with the PSF (and not make the
subtraction of it from the original image).

   A complete tutorial is available to show the operation of this script
as a modular component to extract the PSF of a dataset: *note Building
the extended PSF::.  The executable name is ‘astscript-psf-subtract’,
with the following general template:

     $ astscript-psf-subtract [OPTION...] FITS-file

Examples:

     ## Multiply the PSF (psf.fits) by 3 and subtract it from the
     ## input image (image.fits) at the pixel position (x,y)=(53,69).
     $ astscript-psf-subtract image.fits \
         --psf=psf.fits \
         --mode=img \
         --scale=3 \
         --center=53,69 \
         --output=star-53_69.fits

     ## Iterate over a catalog with positions of stars that are
     ## in the input image. Use WCS coordinates.
     $ asttable catalog.fits | while read -r ra dec mag; do
              scale=$(cat scale-"$ra"_"$dec".txt)
              astscript-psf-subtract image.fits \
                  --mode=wcs \
                  --psf=psf.fits \
                  --scale=$scale \
                  --center=$ra,$dec; done


   The input is an image from which the star is considered.  The result
is the same image but with the star subtracted (modeled by the PSF). The
modeling of the star is done with the PSF image specified with the
option ‘--psf’, and flux-scaled with the option ‘--scale’ at the
position defined by ‘--center’.  Instead of obtaining the PSF-subtracted
image, it is also possible to obtain the modeled star by the PSF. To do
that, use the option ‘--modelonly’.  With this option, the output will
be an image with the same size as the original one with the PSF situated
in the star coordinates and flux-scaled.  In this case, the region not
covered by the PSF are set to zero values.

   Note that this script works over individual objects.  As a
consequence, to generate a scattered light field of many stars, it is
necessary to make multiple calls.  A full description of each option is
given below.

‘-h STR’
‘--hdu=STR’
     The HDU/extension of the input image to use.

‘-p STR’
‘--psf=STR’
     Filename of the PSF image.  The PSF is assumed to be centered in
     this image.

‘-P STR’
‘--psfhdu=STR’
     The HDU/extension of the PSF image.

‘-O STR’
‘--mode=STR’
     Interpret the center position of the object (values given to
     ‘--center’) in image or WCS coordinates.  This option thus accepts
     only two values: ‘img’ or ‘wcs’.

‘-c FLT,FLT’
‘--center=FLT,FLT’
     The central position of the object.  This parameter is used in
     *note Crop:: to center and crop the image.  The positions along
     each dimension must be separated by a comma (<,>).  The number of
     values given to this option must be the same as the dimensions of
     the input dataset.  The units of the coordinates are read based on
     the value to the ‘--mode’ option, see above.

     If the central position does not fall in the center of a pixel in
     the input image, the PSF is resampled with sub-pixel change in the
     pixel grid before subtraction.

‘-s FLT’
‘--scale=FLT’
     Factor by which the PSF (‘--psf’) is multiplied.  This factor is
     necessary to put the PSF with the desired flux level.  A convenient
     way of obtaining this value is by using the script
     ‘astscript-scale-factor’, see *note Invoking
     astscript-psf-scale-factor::.  For a full tutorial on using the
     ‘astscript-psf-*’ scripts together, see *note Building the extended
     PSF::.

‘-t’
‘--tmpdir’
     Directory to keep temporary files during the execution of the
     script.  If the directory does not exist at run-time, this script
     will create it.  By default, upon completion of the script, this
     directory will be deleted.  However, if you would like to keep the
     intermediate files, you can use the ‘--keeptmp’ option.

‘-k’
‘--keeptmp’
     Do not remove the temporary directory (see description of
     ‘--keeptmp’).  This option is useful for debugging and checking the
     outputs of internal steps.

‘-m’
‘--modelonly’
     Do Not make the subtraction of the modeled star by the PSF. This
     option is useful when the user wants to obtain the scattered light
     field given by the PSF modeled star.


File: gnuastro.info,  Node: Makefile extensions,  Next: Library,  Prev: Installed scripts,  Up: Top

11 Makefile extensions (for GNU Make)
*************************************

Make (https://en.wikipedia.org/wiki/Make_(software)) is a build
automation tool.  It can greatly help manage your analysis workflow,
even very complex projects with thousands of files and hundreds of
processing steps.  In this book, we have discussed Make previously in
the context of parallelization (see *note How to run simultaneous
operations::).  It has also been used in

   GNU Make is the most common and powerful implementation of Make, with
many unique additions to the core POSIX standard of Make.  One of those
features is the ability to add extensions using a dynamic library (that
Gnuastro provides).  For the details of this feature from GNU Make’s own
manual, see its Loading dynamic objects
(https://www.gnu.org/software/make/manual/html_node/Loading-Objects.html)
section.  Through this feature, Gnuastro provides additional Make
functions that are useful in the context of data analysis.

   To use this feature, Gnuastro has to be built in shared library more.
Gnuastro’s Make extensions will not work if you build Gnuastro without
shared libraries (for example, when you configure Gnuastro with
‘--disable-shared’ or ‘--debug’).

* Menu:

* Loading the Gnuastro Make functions::  How to find and load Gnuastro’s Make library.
* Makefile functions of Gnuastro::  The available functions.


File: gnuastro.info,  Node: Loading the Gnuastro Make functions,  Next: Makefile functions of Gnuastro,  Prev: Makefile extensions,  Up: Makefile extensions

11.1 Loading the Gnuastro Make functions
========================================

To load Gnuastro’s Make functions in your Makefile, you should use the
‘load’ command of GNU Make in your Makefile.  The load command should be
given Gnuastro’s ‘libgnuastro_make.so’ dynamic library, which has been
specifically written for being called by GNU Make.  The generic command
looks like this (the ‘/PATH/TO’ part should be changed):

     load /PATH/TO/lib/libgnuastro_make.so

Here are the possible replacements of the ‘/PATH/TO’ component:
‘/usr/local’
     If you installed Gnuastro from source and did not use the
     ‘--prefix’ option at configuration time, you should use this base
     directory.
‘/usr/’
     If you installed Gnuastro through your operating system’s package
     manager, it is highly likely that Gnuastro’s library is here.
‘~/.local’
     If you installed Gnuastro from source, but used ‘--prefix’ to
     install Gnuastro in your home directory (as described in *note
     Installation directory::).

   If you cannot find ‘libgnuastro_make.so’ in the locations above, the
command below should give you its location.  It assumes that the
libraries are in the same base directory as the programs (which is
usually the case).

     $ which astfits | sed -e's|bin/astfits|lib/libgnuastro_make.so|'


File: gnuastro.info,  Node: Makefile functions of Gnuastro,  Prev: Loading the Gnuastro Make functions,  Up: Makefile extensions

11.2 Makefile functions of Gnuastro
===================================

All Gnuastro Make functions start with the ‘ast-’ prefix (similar to the
programs on the command-line, but with a dash).  After you have loaded
Gnuastro’s shared library for Makefiles within your Makefile, you can
call these functions just like any Make function.  For instructions on
how to load Gnuastro’s Make functions, see *note Loading the Gnuastro
Make functions::.

   The Make functions in Gnuastro have been recently added (in August
2022), and will be gradually increasing, as we find the need for more
specialized functions.

*Difference between ‘‘=’’ or ‘‘:=’’ for variable definition* When you
define a variable with ‘‘=’’, its value is expanded only when used, not
when defined.  However, when you use ‘‘:=’’, it is immediately expanded
when defined.  Therefore the location of a ‘‘:=’’ variable in the
Makefile matters: if used before its definition, it will be empty!
Those defined by ‘‘=’’ can be used even before they are defined!  On the
other hand, if your variable invokes functions (like ‘foreach’ or
‘wildcard’), it is better to use ‘‘:=’’.  Otherwise, each time the value
is used, the function will be expanded (possibly may times) and this
will reduce the speed of your pipeline.  For more, see the The two
flavors of variables
(https://www.gnu.org/software/make/manual/html_node/Flavors.html) in the
GNU Make manual.

‘$(ast-version-is STRING)’
     Returns ‘1’ if the version of the used Gnuastro is equal to
     ‘STRING’, and ‘0’ otherwise.  This is useful/critical for obtaining
     reproducible results on different systems.  It can be used in
     combination with Conditionals in Make
     (https://www.gnu.org/software/make/manual/html_node/Conditionals.html)
     to ensure the required version of Gnuastro is going to be used in
     your workflow.

     For example, in the minimal working Makefile below, we are using it
     to specify if the default (first) target (‘all’) should have any
     prerequisites (and let the workflow start), or if it should simply
     print a message (that the required version of Gnuastro isn’t
     installed) and abort (without any prerequisites).

          load /usr/local/lib/libgnuastro_make.so

          gnuastro-version = 0.19
          ifeq ($(ast-version-is $(gnuastro-version)),1)
          all: paper.pdf
          else
          all:; @echo "Please use Gnuastro $(gnuastro-version)"
          endif

          result.fits: input.fits
                  astnoisechisel $< --output=$@

          paper.pdf: result.fits
                  pdflatex --halt-on-error paper.tex

‘$(ast-text-contains STRING, TEXT)’
     Returns all white-space-separated words in ‘TEXT’ that contain the
     ‘STRING’, removing any words that _do not_ match.  For example, the
     following minimal Makefile will only print the ‘bAaz Aah’ word of
     the list.

          load /usr/local/lib/libgnuastro_make.so

          list = fooo baar bAaz uggh Aah
          all:
               echo $(ast-text-contains Aa, $(list))

     This can be thought of as Make’s own ‘filter’ function, but if it
     would accept two patterns in a format like this ‘$(filter
     %Aa%,$(list))’ (for the example above).  In fact, the first
     sentence describing this function is taken from the Make manual’s
     first sentence that describes the ‘filter’ function!  However,
     unfortunately Make’s ‘filter’ function only accepts a single ‘%’,
     not two!

‘$(ast-text-not-contains STRING, TEXT)’
     Returns all white-space-separated words in ‘TEXT’ that _do not_
     contain the ‘STRING’, removing any words that _do not_ match.  This
     is the inverse of the ‘ast-text-contains’ function.  For example,
     the following minimal Makefile will print ‘fooo baar uggh’ word of
     the list.

          load /usr/local/lib/libgnuastro_make.so

          list = fooo baar bAaz uggh Aah
          all:
               echo $(ast-text-not-contains Aa, $(list))

‘$(ast-fits-with-keyvalue KEYNAME, KEYVALUES, HDU, FITS_FILES)’
     Will select only the FITS files (from a list of many in
     ‘FITS_FILES’, non-FITS files are ignored), where the ‘KEYNAME’
     keyword has the value(s) given in ‘KEYVALUES’.  Only the HDU given
     in the ‘HDU’ argument will be checked.  According to the FITS
     standard, the keyword name is not case sensitive, but the keyword
     value is.

     For example, if you have many FITS files in the ‘/datasets/images’
     directory, the minimal Makefile below will put those with a value
     of ‘BAR’ or ‘BAZ’ for the ‘FOO’ keyword in HDU number ‘1’ in the
     ‘selected’ Make variable.  Notice how there is no comma between
     ‘BAR’ and ‘BAZ’: you can specify any series of values.

     load /usr/local/lib/libgnuastro_make.so
     
     files := $(wildcard /datasets/images/*.fits)
     selected := $(ast-fits-with-keyvalue FOO, BAR BAZ, 1, $(files))
     
     all:
     	echo "Full:     $(words $(files)) files";
     	echo "Selected: $(words $(selected)) files"

‘$(ast-fits-unique-keyvalues KEYNAME, HDU, FITS_FILES)’
     Will return the unique values given to the given FITS keyword
     (‘KEYNAME’) in the given HDU of all the input FITS files (non-FITS
     files are ignored).  For example, after the commands below, the
     ‘keyvalues’ variable will contain the unique values given to the
     ‘FOO’ keyword in HDU number 1 of all the FITS files in
     ‘/datasets/images/*.fits’.

          files := $(wildcard /datasets/images/*.fits)
          keyvalues := $(ast-fits-unique-keyvalues FOO, 1, $(files))

     This is useful when you do not know the full range of values
     a-priori.  For example, let’s assume that you are looking at a
     night’s observations with a telescope and the purpose of the FITS
     image is written in the ‘OBJECT’ keyword of the image (which we can
     assume is in HDU number 1).  This keyword can have the name of the
     various science targets (for example, ‘NGC123’ and ‘M31’) and
     calibration targets (for example, ‘BIAS’ and ‘FLAT’).  The list of
     science targets is different from project to project, such that in
     one night, you can observe multiple projects.  But the calibration
     frames have unique names.  Knowing the calibration keyword values,
     you can extract the science keyword values of the night with the
     command below (feeding the output of this function to Make’s
     ‘filter-out’ function).

          calib = BIAS FLAT
          files := $(wildcard /datasets/images/*.fits)
          science := $(filter-out $(calib), \
                       $(ast-fits-unique-keyvalues OBJECT, 1, $(files)))

     The ‘science’ variable will now contain the unique science targets
     that were observed in your selected FITS images.  You can use it to
     group the various exposures together in the next stages to make
     separate stacks of deep images for each science target (you can
     select FITS files based on their keyword values using the
     ‘ast-fits-with-keyvalue’ function, which is described separately in
     this section).


File: gnuastro.info,  Node: Library,  Next: Developing,  Prev: Makefile extensions,  Up: Top

12 Library
**********

Each program in Gnuastro that was discussed in the prior chapters (or
any program in general) is a collection of functions that is compiled
into one executable file which can communicate directly with the outside
world.  The outside world in this context is the operating system.  By
communication, we mean that control is directly passed to a program from
the operating system with a (possible) set of inputs and after it is
finished, the program will pass control back to the operating system.
For programs written in C and C++, the unique ‘main’ function is in
charge of this communication.

   Similar to a program, a library is also a collection of functions
that is compiled into one executable file.  However, unlike programs,
libraries do not have a ‘main’ function.  Therefore they cannot
communicate directly with the outside world.  This gives you the chance
to write your own ‘main’ function and call library functions from within
it.  After compiling your program into a binary executable, you just
have to _link_ it to the library and you are ready to run (execute) your
program.  In this way, you can use Gnuastro at a much lower-level, and
in combination with other libraries on your system, you can
significantly boost your creativity.

   This chapter starts with a basic introduction to libraries and how
you can use them in *note Review of library fundamentals::.  The
separate functions in the Gnuastro library are then introduced
(classified by context) in *note Gnuastro library::.  If you end up
routinely using a fixed set of library functions, with a well-defined
input and output, it will be much more beneficial if you define a
program for the job.  Therefore, in its *note Version controlled
source::, Gnuastro comes with the *note The TEMPLATE program:: to easily
define your own programs(s).

* Menu:

* Review of library fundamentals::  Guide on libraries and linking.
* BuildProgram::                Link and run source files with this library.
* Gnuastro library::            Description of all library functions.
* Library demo programs::       Demonstration for using libraries.


File: gnuastro.info,  Node: Review of library fundamentals,  Next: BuildProgram,  Prev: Library,  Up: Library

12.1 Review of library fundamentals
===================================

Gnuastro’s libraries are written in the C programming language.  In
*note Why C::, we have thoroughly discussed the reasons behind this
choice.  C was actually created to write Unix, thus understanding the
way C works can greatly help in effectively using programs and libraries
in all Unix-like operating systems.  Therefore, in the following
subsections some important aspects of C, as it relates to libraries (and
thus programs that depend on them) on Unix are reviewed.  First we will
discuss header files in *note Headers:: and then go onto *note
Linking::.  This section finishes with *note Summary and example on
libraries::.  If you are already familiar with these concepts, please
skip this section and go directly to *note Gnuastro library::.

   In theory, a full operating system (or any software) can be written
as one function.  Such a software would not need any headers or linking
(that are discussed in the subsections below).  However, writing that
single function and maintaining it (adding new features, fixing bugs,
documentation, etc.)  would be a programmer or scientist’s worst
nightmare!  Furthermore, all the hard work that went into creating it
cannot be reused in other software: every other programmer or scientist
would have to re-invent the wheel.  The ultimate purpose behind
libraries (which come with headers and have to be linked) is to address
this problem and increase modularity: “the degree to which a system’s
components may be separated and recombined” (from Wikipedia).  The more
modular the source code of a program or library, the easier maintaining
it will be, and all the hard work that went into creating it can be
reused for a wider range of problems.

* Menu:

* Headers::                     Header files included in source.
* Linking::                     Linking the compiled source files into one.
* Summary and example on libraries::  A summary and example on using libraries.


File: gnuastro.info,  Node: Headers,  Next: Linking,  Prev: Review of library fundamentals,  Up: Review of library fundamentals

12.1.1 Headers
--------------

C source code is read from top to bottom in the source file, therefore
program components (for example, variables, data structures and
functions) should all be _defined_ or _declared_ closer to the top of
the source file: before they are used.  _Defining_ something in C or C++
is jargon for providing its full details.  _Declaring_ it, on the
other-hand, is jargon for only providing the minimum information needed
for the compiler to pass it temporarily and fill in the detailed
definition later.

   For a function, the _declaration_ only contains the inputs and their
data-types along with the output’s type(1).  The _definition_ adds to
the declaration by including the exact details of what operations are
done to the inputs to generate the output.  As an example, take this
simple summation function:

     double
     sum(double a, double b)
     {
       return a + b;
     }
What you see above is the _definition_ of this function: it shows you
(and the compiler) exactly what it does to the two ‘double’ type inputs
and that the output also has a ‘double’ type.  Note that a function’s
internal operations are rarely so simple and short, it can be
arbitrarily long and complicated.  This unreasonably short and simple
function was chosen here for ease of reading.  The declaration for this
function is:

     double
     sum(double a, double b);

You can think of a function’s declaration as a building’s address in the
city, and the definition as the building’s complete blueprints.  When
the compiler confronts a call to a function during its processing, it
does not need to know anything about how the inputs are processed to
generate the output.  Just as the postman does not need to know the
inner structure of a building when delivering the mail.  The declaration
(address) is enough.  Therefore by _declaring_ the functions once at the
start of the source files, we do not have to worry about _defining_ them
after they are used.

   Even for a simple real-world operation (not a simple summation like
above!), you will soon need many functions (for example, some for
reading/preparing the inputs, some for the processing, and some for
preparing the output).  Although it is technically possible, managing
all the necessary functions in one file is not easy and is contrary to
the modularity principle (see *note Review of library fundamentals::),
for example, the functions for preparing the input can be usable in your
other projects with a different processing.  Therefore, as we will see
later (in *note Linking::), the functions do not necessarily need to be
defined in the source file where they are used.  As long as their
definitions are ultimately linked to the final executable, everything
will be fine.  For now, it is just important to remember that the
functions that are called within one source file must be declared within
the source file (declarations are mandatory), but not necessarily
defined there.

   In the spirit of modularity, it is common to define contextually
similar functions in one source file.  For example, in Gnuastro,
functions that calculate the median, mean and other statistical
functions are defined in ‘lib/statistics.c’, while functions that deal
directly with FITS files are defined in ‘lib/fits.c’.

   Keeping the definition of similar functions in a separate file
greatly helps their management and modularity, but this fact alone does
not make things much easier for the caller’s source code: recall that
while definitions are optional, declarations are mandatory.  So if this
was all, the caller would have to manually copy and paste (_include_)
all the declarations from the various source files into the file they
are working on now.  To address this problem, programmers have adopted
the header file convention: the header file of a source code contains
all the declarations that a caller would need to be able to use any of
its functions.  For example, in Gnuastro, ‘lib/statistics.c’ (file
containing function definitions) comes with ‘lib/gnuastro/statistics.h’
(only containing function declarations).

   The discussion above was mainly focused on functions, however, there
are many more programming constructs such as preprocessor macros and
data structures.  Like functions, they also need to be known to the
compiler when it confronts a call to them.  So the header file also
contains their definitions or declarations when they are necessary for
the functions.

   Preprocessor macros (or macros for short) are replaced with their
defined value by the preprocessor before compilation.  Conventionally
they are written only in capital letters to be easily recognized.  It is
just important to understand that the compiler does not see the macros,
it sees their fixed values.  So when a header specifies macros you can
do your programming without worrying about the actual values.  The
standard C types (for example, ‘int’, or ‘float’) are very low-level and
basic.  We can collect multiple C types into a _structure_ for a
higher-level way to keep and pass-along data.  See *note Generic data
container:: for some examples of macros and data structures.

   The contents in the header need to be _include_d into the caller’s
source code with a special preprocessor command: ‘#include
<path/to/header.h>’.  As the name suggests, the _preprocessor_ goes
through the source code prior to the processor (or compiler).  One of
its jobs is to include, or merge, the contents of files that are
mentioned with this directive in the source code.  Therefore the
compiler sees a single entity containing the contents of the main file
and all the included files.  This allows you to include many (sometimes
thousands of) declarations into your code with only one line.  Since the
headers are also installed with the library into your system, you do not
even need to keep a copy of them for each separate program, making
things even more convenient.

   Try opening some of the ‘.c’ files in Gnuastro’s ‘lib/’ directory
with a text editor to check out the include directives at the start of
the file (after the copyright notice).  Let’s take ‘lib/fits.c’ as an
example.  You will notice that Gnuastro’s header files (like
‘gnuastro/fits.h’) are indeed within this directory (the ‘fits.h’ file
is in the ‘gnuastro/’ directory).  You will notice that files like
‘stdio.h’, or ‘string.h’ are not in this directory (or anywhere within
Gnuastro).

   On most systems the basic C header files (like ‘stdio.h’ and
‘string.h’ mentioned above) are located in ‘/usr/include/’(2).  Your
compiler is configured to automatically search that directory (and
possibly others), so you do not have to explicitly mention these
directories.  Go ahead, look into the ‘/usr/include’ directory and find
‘stdio.h’ for example.  When the necessary header files are not in those
specific libraries, the preprocessor can also search in places other
than the current directory.  You can specify those directories with this
preprocessor option(3):

‘-I DIR’
     “Add the directory ‘DIR’ to the list of directories to be searched
     for header files.  Directories named by ’-I’ are searched before
     the standard system include directories.  If the directory ‘DIR’ is
     a standard system include directory, the option is ignored to
     ensure that the default search order for system directories and the
     special treatment of system headers are not defeated...” (quoted
     from the GNU Compiler Collection manual).  Note that the space
     between <I> and the directory is optional and commonly not used.

   If the preprocessor cannot find the included files, it will abort
with an error.  In fact a common error when building programs that
depend on a library is that the compiler does not know where a library’s
header is (see *note Known issues::).  So you have to manually tell the
compiler where to look for the library’s headers with the ‘-I’ option.
For a small software with one or two source files, this can be done
manually (see *note Summary and example on libraries::).  However, to
enhance modularity, Gnuastro (and most other bin/libraries) contain many
source files, so the compiler is invoked many times(4).  This makes
manual addition or modification of this option practically impossible.

   To solve this problem, in the GNU build system, there are
conventional environment variables for the various kinds of compiler
options (or flags).  These environment variables are used in every call
to the compiler (they can be empty).  The environment variable used for
the C preprocessor (or CPP) is ‘CPPFLAGS’.  By giving ‘CPPFLAGS’ a value
once, you can be sure that each call to the compiler will be affected.
See *note Known issues:: for an example of how to set this variable at
configure time.

   As described in *note Installation directory::, you can select the
top installation directory of a software using the GNU build system,
when you ‘./configure’ it.  All the separate components will be put in
their separate sub-directory under that, for example, the programs,
compiled libraries and library headers will go into ‘$prefix/bin’
(replace ‘$prefix’ with a directory), ‘$prefix/lib’, and
‘$prefix/include’ respectively.  For enhanced modularity, libraries that
contain diverse collections of functions (like GSL, WCSLIB, and
Gnuastro), put their header files in a sub-directory unique to
themselves.  For example, all Gnuastro’s header files are installed in
‘$prefix/include/gnuastro’.  In your source code, you need to keep the
library’s sub-directory when including the headers from such libraries,
for example, ‘#include <gnuastro/fits.h>’(5).  Not all libraries need to
follow this convention, for example, CFITSIO only has one header
(‘fitsio.h’) which is directly installed in ‘$prefix/include’.

   ---------- Footnotes ----------

   (1) Recall that in C, functions only have one output.

   (2) The ‘include/’ directory name is taken from the pre-processor’s
‘#include’ directive, which is also the motivation behind the ‘I’ in the
‘-I’ option to the pre-processor.

   (3) Try running Gnuastro’s ‘make’ and find the directories given to
the compiler with the ‘-I’ option.

   (4) Nearly every command you see being executed after running ‘make’
is one call to the compiler.

   (5) the top ‘$prefix/include’ directory is usually known to the
compiler


File: gnuastro.info,  Node: Linking,  Next: Summary and example on libraries,  Prev: Headers,  Up: Review of library fundamentals

12.1.2 Linking
--------------

To enhance modularity, similar functions are defined in one source file
(with a ‘.c’ suffix, see *note Headers:: for more).  After running
‘make’, each human-readable, ‘.c’ file is translated (or compiled) into
a computer-readable “object” file (ending with ‘.o’).  Note that object
files are also created when building programs, they are not particular
to libraries.  Try opening Gnuastro’s ‘lib/’ and ‘bin/progname/’
directories after running ‘make’ to see these object files(1).
Afterwards, the object files are _linked_ together to create an
executable program or a library.

   The object files contain the full definition of the functions in the
respective ‘.c’ file along with a list of any other function (or
generally “symbol”) that is referenced there.  To get a list of those
functions you can use the ‘nm’ program which is part of GNU Binutils.
For example, from the top Gnuastro directory, run:

     $ nm bin/arithmetic/arithmetic.o

This will print a list of all the functions (more generally, ‘symbols’)
that were called within ‘bin/arithmetic/arithmetic.c’ along with some
further information (for example, a ‘T’ in the second column shows that
this function is actually defined here, ‘U’ says that it is undefined
here).  Try opening the ‘.c’ file to check some of these functions for
yourself.  Run ‘info nm’ for more information.

   To recap, the _compiler_ created the separate object files mentioned
above for each ‘.c’ file.  The _linker_ will then combine all the
symbols of the various object files (and libraries) into one program or
library.  In the case of Arithmetic (a program) the contents of the
object files in ‘bin/arithmetic/’ are copied (and re-ordered) into one
final executable file which we can run from the operating system.

   There are two ways to _link_ all the necessary symbols: static and
dynamic/shared.  When the symbols (computer-readable function
definitions in most cases) are copied into the output, it is called
_static_ linking.  When the symbols are kept in their original file and
only a reference to them is kept in the executable, it is called
_dynamic_, or _shared_ linking.

   Let’s have a closer look at the executable to understand this better:
we will assume you have built Gnuastro without any customization and
installed Gnuastro into the default ‘/usr/local/’ directory (see *note
Installation directory::).  If you tried the ‘nm’ command on one of
Arithmetic’s object files above, then with the command below you can
confirm that all the functions that were defined in the object file
above (had a ‘T’ in the second column) are also defined in the
‘astarithmetic’ executable:

     $ nm /usr/local/bin/astarithmetic

These symbols/function have been statically linked (copied) in the final
executable.  But you will notice that there are still many undefined
symbols in the executable (those with a ‘U’ in the second column).  One
class of such functions are Gnuastro’s own library functions that start
with ‘‘gal_’’:

     $ nm /usr/local/bin/astarithmetic | grep gal_

   These undefined symbols (functions) are present in another file and
will be linked to the Arithmetic program every time you run it.
Therefore they are known as dynamically _linked_ libraries (2).  As we
saw above, static linking is done when the executable is being built.
However, when a program is dynamically linked to a library, at
build-time, the library’s symbols are only checked with the available
libraries: they are not actually copied into the program’s executable.
Every time you run the program, the (dynamic) linker will be activated
and will try to link the program to the installed library before the
program starts.

   If you want all the libraries to be statically linked to the
executables, you have to tell Libtool (which Gnuastro uses for the
linking) to disable shared libraries at configure time(3):

     $ configure --disable-shared

Try configuring Gnuastro with the command above, then build and install
it (as described in *note Quick start::).  Afterwards, check the ‘gal_’
symbols in the installed Arithmetic executable like before.  You will
see that they are actually copied this time (have a ‘T’ in the second
column).  If the second column does not convince you, look at the
executable file size with the following command:

     $ ls -lh /usr/local/bin/astarithmetic

It should be around 4.2 Megabytes with this static linking.  If you
configure and build Gnuastro again with shared libraries enabled (which
is the default), you will notice that it is roughly 100 Kilobytes!

   This huge difference would have been very significant in the old
days, but with the roughly Terabyte storage drives commonly in use
today, it is negligible.  Fortunately, output file size is not the only
benefit of dynamic linking: since it links to the libraries at run-time
(rather than build-time), you do not have to rebuild a higher-level
program or library when an update comes for one of the lower-level
libraries it depends on.  You just install the new low-level library and
it will automatically be used/linked next time in the programs that use
it.  To be fair, this also creates a few complications(4):

   • Reproducibility: Even though your high-level tool has the same
     version as before, with the updated library, you might not get the
     same results.
   • Broken links: if some functions have been changed or removed in the
     updated library, then the linker will abort with an error at
     run-time.  Therefore you need to re-build your higher-level program
     or library.

   To see a list of all the shared libraries that are needed for a
program or a shared library to run, you can use GNU C library’s ‘ldd’(5)
program, for example:

     $ ldd /usr/local/bin/astarithmetic

   Library file names (in their installation directory) start with a
‘lib’ and their ending (suffix) shows if they are static (‘.a’) or
dynamic (‘.so’), as described below.  The name of the library is in the
middle of these two, for example, ‘libgsl.a’ or ‘libgnuastro.a’ (GSL and
Gnuastro’s static libraries), and ‘libgsl.so.23.0.0’ or
‘libgnuastro.so.4.0.0’ (GSL and Gnuastro’s shared library, the numbers
may be different).

   • A static library is known as an archive file and has the ‘.a’
     suffix.  A static library is not an executable file.

   • A shared library ends with the ‘.so.X.Y.Z’ suffix and is
     executable.  The three numbers in the suffix, describe the version
     of the shared library.  Shared library versions are defined to
     allow multiple versions of a shared library simultaneously on a
     system and to help detect possible updates in the library and
     programs that depend on it by the linker.

     It is very important to mention that this version number is
     different from the software version number (see *note Version
     numbering::), so do not confuse the two.  See the “Library
     interface versions” chapter of GNU Libtool for more.

     For each shared library, we also have two symbolic links ending
     with ‘.so.X’ and ‘.so’.  They are automatically set by the
     installer, but you can change them (point them to another version
     of the library) when you have multiple versions of a library on
     your system.

   Libraries that are built with GNU Libtool (including Gnuastro and its
dependencies), build both static and dynamic libraries by default and
install them in ‘prefix/lib/’ directory (for more on ‘prefix’, see *note
Installation directory::).  In this way, programs depending on the
libraries can link with them however they prefer.  See the contents of
‘/usr/local/lib’ with the command below to see both the static and
shared libraries available there, along with their executable nature and
the symbolic links:

     $ ls -l /usr/local/lib/

   To link with a library, the linker needs to know where to find the
library.  _At compilation time_, these locations can be passed to the
linker with two separate options (see *note Summary and example on
libraries:: for an example) as described below.  You can see these
options and their usage in practice while building Gnuastro (after
running ‘make’):

‘-L DIR’
     Will tell the linker to look into ‘DIR’ for the libraries.  For
     example, ‘-L/usr/local/lib’, or ‘-L/home/yourname/.local/lib’.  You
     can make multiple calls to this option, so the linker looks into
     several directories at compilation time.  Note that the space
     between <L> and the directory is optional and commonly ignored
     (written as ‘-LDIR’).

‘-lLIBRARY’
     Specify the unique library identifier/name (not containing
     directory or shared/dynamic nature) to be linked with the
     executable.  As discussed above, library file names have fixed
     parts which must not be given to this option.  So ‘-lgsl’ will
     guide the linker to either look for ‘libgsl.a’ or ‘libgsl.so’
     (depending on the type of linking it is suppose to do).  You can
     link many libraries by repeated calls to this option.

     *Very important: * The place of this option on the compiler’s
     command matters.  This is often a source of confusion for
     beginners, so let’s assume you have asked the linker to link with
     library A using this option.  As soon as the linker confronts this
     option, it looks into the list of the undefined symbols it has
     found until that point and does a search in library A for any of
     those symbols.  If any pending undefined symbol is found in library
     A, it is used.  After the search in undefined symbols is complete,
     the contents of library A are completely discarded from the
     linker’s memory.  Therefore, if a later object file or library uses
     an unlinked symbol in library A, the linker will abort after it has
     finished its search in all the input libraries or object files.

     As an example, Gnuastro’s ‘gal_fits_img_read’ function depends on
     the ‘fits_read_pix’ function of CFITSIO (specified with
     ‘-lcfitsio’, which in turn depends on the cURL library, called with
     ‘-lcurl’).  So the proper way to link something that uses this
     function is ‘-lgnuastro -lcfitsio -lcurl’.  If instead, you give:
     ‘-lcfitsio -lgnuastro’ the linker will complain and abort.  To
     avoid such linking complexities when using Gnuastro’s library, we
     recommend using *note BuildProgram::.

   If you have compiled and linked your program with a dynamic library,
then the dynamic linker also needs to know the location of the libraries
after building the program: _every time_ the program is run afterwards.
Therefore, it may happen that you do not get any errors when
compiling/linking a program, but are unable to run your program because
of a failure to find a library.  This happens because the dynamic linker
has not found the dynamic library _at run time_.

   To find the dynamic libraries at run-time, the linker looks into the
paths, or directories, in the ‘LD_LIBRARY_PATH’ environment variable.
For a discussion on environment variables, especially search paths like
‘LD_LIBRARY_PATH’, and how you can add new directories to them, see
*note Installation directory::.

   ---------- Footnotes ----------

   (1) Gnuastro uses GNU Libtool for portable library creation.  Libtool
will also make a ‘.lo’ file for each ‘.c’ file when building libraries
(‘.lo’ files are human-readable).

   (2) Do not confuse dynamically _linked_ libraries with dynamically
_loaded_ libraries.  The former (that is discussed here) are only loaded
once at the program startup.  However, the latter can be loaded anytime
during the program’s execution, they are also known as plugins.

   (3) Libtool is very common and is commonly used.  Therefore, you can
use this option to configure on most programs using the GNU build system
if you want static linking.

   (4) Both of these can be avoided by joining the mailing lists of the
lower-level libraries and checking the changes in newer versions before
installing them.  Updates that result in such behaviors are generally
heavily emphasized in the release notes.

   (5) If your operating system is not using the GNU C library, you
might need another tool.


File: gnuastro.info,  Node: Summary and example on libraries,  Prev: Linking,  Up: Review of library fundamentals

12.1.3 Summary and example on libraries
---------------------------------------

After the mostly abstract discussions of *note Headers:: and *note
Linking::, we will give a small tutorial here.  But before that, let’s
recall the general steps of how your source code is prepared, compiled
and linked to the libraries it depends on so you can run it:

  1. The *preprocessor* includes the header (‘.h’) files into the
     function definition (‘.c’) files, expands preprocessor macros.
     Generally the preprocessor prepares the human-readable source for
     compilation (reviewed in *note Headers::).

  2. The *compiler* will translate (compile) the human-readable contents
     of each source (merged ‘.c’ and the ‘.h’ files, or generally the
     output of the preprocessor) into the computer-readable code of ‘.o’
     files.

  3. The *linker* will link the called function definitions from various
     compiled files to create one unified object.  When the unified
     product has a ‘main’ function, this function is the product’s only
     entry point, enabling the operating system or user to directly
     interact with it, so the product is a program.  When the product
     does not have a ‘main’ function, the linker’s product is a library
     and it is exported functions can be linked to other executables (it
     has many entry points).

   The GNU Compiler Collection (or GCC for short) will do all three
steps.  So as a first example, from Gnuastro’s source, go to
‘tests/lib/’.  This directory contains the library tests, you can use
these as some simple tutorials.  For this demonstration, we will compile
and run the ‘arraymanip.c’.  This small program will call Gnuastro
library for some simple operations on an array (open it and have a
look).  To compile this program, run this command inside the directory
containing it.

     $ gcc arraymanip.c -lgnuastro -lm -o arraymanip

The two ‘-lgnuastro’ and ‘-lm’ options (in this order) tell GCC to first
link with the Gnuastro library and then with C’s math library.  The ‘-o’
option is used to specify the name of the output executable, without it
the output file name will be ‘a.out’ (on most OSs), independent of your
input file name(s).

   If your top Gnuastro installation directory (let’s call it ‘$prefix’,
see *note Installation directory::) is not recognized by GCC, you will
get preprocessor errors for unknown header files.  Once you fix it, you
will get linker errors for undefined functions.  To fix both, you should
run GCC as follows: additionally telling it which directories it can
find Gnuastro’s headers and compiled library (see *note Headers:: and
*note Linking::):

     $ gcc -I$prefix/include -L$prefix/lib arraymanip.c -lgnuastro -lm     \
           -o arraymanip

This single command has done all the preprocessor, compilation and
linker operations.  Therefore no intermediate files (object files in
particular) were created, only a single output executable was created.
You are now ready to run the program with:

     $ ./arraymanip

   The Gnuastro functions called by this program only needed to be
linked with the C math library.  But if your program needs WCS
coordinate transformations, needs to read a FITS file, needs special
math operations (which include its linear algebra operations), or you
want it to run on multiple CPU threads, you also need to add these
libraries in the call to GCC: ‘-lgnuastro -lwcs -lcfitsio -lgsl
-lgslcblas -pthread -lm’.  In *note Gnuastro library::, where each
function is documented, it is mentioned which libraries (if any) must
also be linked when you call a function.  If you feel all these linkings
can be confusing, please consider Gnuastro’s *note BuildProgram::
program.


File: gnuastro.info,  Node: BuildProgram,  Next: Gnuastro library,  Prev: Review of library fundamentals,  Up: Library

12.2 BuildProgram
=================

The number and order of libraries that are necessary for linking a
program with Gnuastro library might be too confusing when you need to
compile a small program for one particular job (with one source file).
BuildProgram will use the information gathered during configuring
Gnuastro and link with all the appropriate libraries on your system.
This will allow you to easily compile, link and run programs that use
Gnuastro’s library with one simple command and not worry about which
libraries to link to, or the linking order.

   BuildProgram uses GNU Libtool to find the necessary libraries to link
against (GNU Libtool is the same program that builds all of Gnuastro’s
libraries and programs when you run ‘make’).  So in the future, if
Gnuastro’s prerequisite libraries change or other libraries are added,
you do not have to worry, you can just run BuildProgram and internal
linking will be done correctly.

*BuildProgram requires GNU Libtool:* BuildProgram depends on GNU
Libtool, other implementations do not have some necessary features.  If
GNU Libtool is not available at Gnuastro’s configure time, you will get
a notice at the end of the configuration step and BuildProgram will not
be built or installed.  Please see *note Optional dependencies:: for
more information.

* Menu:

* Invoking astbuildprog::       Options and examples for using this program.


File: gnuastro.info,  Node: Invoking astbuildprog,  Prev: BuildProgram,  Up: BuildProgram

12.2.1 Invoking BuildProgram
----------------------------

BuildProgram will compile and link a C source program with Gnuastro’s
library and all its dependencies, greatly facilitating the compilation
and running of small programs that use Gnuastro’s library.  The
executable name is ‘astbuildprog’ with the following general template:

     $ astbuildprog [OPTION...] C_SOURCE_FILE

One line examples:

     ## Compile, link and run `myprogram.c':
     $ astbuildprog myprogram.c

     ## Similar to previous, but with optimization and compiler warnings:
     $ astbuildprog -Wall -O2 myprogram.c

     ## Compile and link `myprogram.c', then run it with `image.fits'
     ## as its argument:
     $ astbuildprog myprogram.c image.fits

     ## Also look in other directories for headers and linking:
     $ astbuildprog -Lother -Iother/dir myprogram.c

     ## Just build (compile and link) `myprogram.c', do not run it:
     $ astbuildprog --onlybuild myprogram.c

   If BuildProgram is to run, it needs a C programming language source
file as input.  By default it will compile and link the given source
into a final executable program and run it.  The built executable name
can be set with the optional ‘--output’ option.  When no output name is
set, BuildProgram will use Gnuastro’s *note Automatic output:: system to
remove the suffix of the input source file (usually ‘.c’) and use the
resulting name as the built program name.

   For the full list of options that BuildProgram shares with other
Gnuastro programs, see *note Common options::.  You may also use
Gnuastro’s *note Configuration files:: to specify other
libraries/headers to use for special directories and not have to type
them in every time.

   The C compiler can be chosen with the ‘--cc’ option, or environment
variables, please see the description of ‘--cc’ for more.  The two
common ‘LDFLAGS’ and ‘CPPFLAGS’ environment variables are also checked
and used in the build by default.  Note that they are placed after the
values to the corresponding options ‘--includedir’ and ‘--linkdir’.
Therefore BuildProgram’s own options take precedence.  Using environment
variables can be disabled with the ‘--noenv’ option.  Just note that
BuildProgram also keeps the important flags in these environment
variables in its configuration file.  Therefore, in many cases, even
though you may needed them to build Gnuastro, you will not need them in
BuildProgram.

   The first argument is considered to be the C source file that must be
compiled and linked.  Any other arguments (non-option tokens on the
command-line) will be passed onto the program when BuildProgram wants to
run it.  Recall that by default BuildProgram will run the program after
building it.  This behavior can be disabled with the ‘--onlybuild’
option.

   When the ‘--quiet’ option (see *note Operating mode options::) is not
called, BuildPrograms will print the compilation and running commands.
Once your program grows and you break it up into multiple files (which
are much more easily managed with Make), you can use the linking flags
of the non-quiet output in your ‘Makefile’.

‘-c STR’
‘--cc=STR’
     C compiler to use for the compilation, if not given environment
     variables will be used as described in the next paragraph.  If the
     compiler is in your system’s search path, you can simply give its
     name, for example, ‘--cc=gcc’.  If it is not in your system’s
     search path, you can give its full path, for example,
     ‘--cc=/path/to/your/custom/cc’.

     If this option has no value after parsing the command-line and all
     configuration files (see *note Configuration file precedence::),
     then BuildProgram will look into the following environment
     variables in the given order ‘CC’ and ‘GCC’.  If they are also not
     defined, BuildProgram will ultimately default to the ‘gcc’ command
     which is present in many systems (sometimes as a link to other
     compilers).

‘-I STR’
‘--includedir=STR’
     Directory to search for files that you ‘#include’ in your C
     program.  Note that headers relating to Gnuastro and its
     dependencies do not need this option.  This is only necessary if
     you want to use other headers.  It may be called multiple times and
     order matters.  This directory will be searched before those of
     Gnuastro’s build and also the system search directories.  See *note
     Headers:: for a thorough introduction.

     From the GNU C preprocessor manual: “Add the directory ‘STR’ to the
     list of directories to be searched for header files.  Directories
     named by ‘-I’ are searched before the standard system include
     directories.  If the directory ‘STR’ is a standard system include
     directory, the option is ignored to ensure that the default search
     order for system directories and the special treatment of system
     headers are not defeated”.

‘-L STR’
‘--linkdir=STR’
     Directory to search for compiled libraries to link the program
     with.  Note that all the directories that Gnuastro was built with
     will already be used by BuildProgram (GNU Libtool).  This option is
     only necessary if your libraries are in other directories.
     Multiple calls to this option are possible and order matters.  This
     directory will be searched before those of Gnuastro’s build and
     also the system search directories.  See *note Linking:: for a
     thorough introduction.

‘-l STR’
‘--linklib=STR’
     Library to link with your program.  Note that all the libraries
     that Gnuastro was built with will already be linked by BuildProgram
     (GNU Libtool).  This option is only necessary if you want to link
     with other directories.  Multiple calls to this option are possible
     and order matters.  This library will be linked before Gnuastro’s
     library or its dependencies.  See *note Linking:: for a thorough
     introduction.

‘-O INT/STR’
‘--optimize=INT/STR’
     Compiler optimization level: 0 (for no optimization, good
     debugging), 1, 2, 3 (for the highest level of optimizations).  From
     the GNU Compiler Collection (GCC) manual: “Without any optimization
     option, the compiler’s goal is to reduce the cost of compilation
     and to make debugging produce the expected results.  Statements are
     independent: if you stop the program with a break point between
     statements, you can then assign a new value to any variable or
     change the program counter to any other statement in the function
     and get exactly the results you expect from the source code.
     Turning on optimization flags makes the compiler attempt to improve
     the performance and/or code size at the expense of compilation time
     and possibly the ability to debug the program.” Please see your
     compiler’s manual for the full list of acceptable values to this
     option.

‘-g’
‘--debug’
     Emit extra information in the compiled binary for use by a
     debugger.  When calling this option, it is best to explicitly
     disable optimization with ‘-O0’.  To combine both options you can
     run ‘-gO0’ (see *note Options:: for how short options can be merged
     into one).

‘-W STR’
‘--warning=STR’
     Print compiler warnings on command-line during compilation.
     “Warnings are diagnostic messages that report constructions that
     are not inherently erroneous but that are risky or suggest there
     may have been an error.” (from the GCC manual).  It is always
     recommended to compile your programs with warnings enabled.

     All compiler warning options that start with ‘W’ are usable by this
     option in BuildProgram also, see your compiler’s manual for the
     full list.  Some of the most common values to this option are:
     ‘pedantic’ (Warnings related to standard C) and ‘all’ (all issues
     the compiler confronts).

‘-t’
‘--tag=STR’
     The language configuration information.  Libtool can build objects
     and libraries in many languages.  In many cases, it can identify
     the language automatically, but when it does not you can use this
     option to explicitly notify Libtool of the language.  The
     acceptable values are: ‘CC’ for C, ‘CXX’ for C++, ‘GCJ’ for Java,
     ‘F77’ for Fortran 77, ‘FC’ for Fortran, ‘GO’ for Go and ‘RC’ for
     Windows Resource.  Note that the Gnuastro library is not yet fully
     compatible with all these languages.

‘-b’
‘--onlybuild’
     Only build the program, do not run it.  By default, the built
     program is immediately run afterwards.

‘-d’
‘--deletecompiled’
     Delete the compiled binary file after running it.  This option is
     only relevant when the compiled program is run after being built.
     In other words, it is only relevant when ‘--onlybuild’ is not
     called.  It can be useful when you are busy testing a program or
     just want a fast result and the actual binary/compiled file is not
     of later use.

‘-a STR’
‘--la=STR’
     Use the given ‘.la’ file (Libtool control file) instead of the one
     that was produced from Gnuastro’s configuration results.  The
     Libtool control file keeps all the necessary information for
     building and linking a program with a library built by Libtool.
     The default ‘prefix/lib/libgnuastro.la’ keeps all the information
     necessary to build a program using the Gnuastro library gathered
     during configure time (see *note Installation directory:: for
     prefix).  This option is useful when you prefer to use another
     Libtool control file.

‘-e’
‘--noenv’
     Do Not use environment variables in the build, just use the values
     given to the options.  As described above, environment variables
     like ‘CC’, ‘GCC’, ‘LDFLAGS’, ‘CPPFLAGS’ will be read by default and
     used in the build if they have been defined.


File: gnuastro.info,  Node: Gnuastro library,  Next: Library demo programs,  Prev: BuildProgram,  Up: Library

12.3 Gnuastro library
=====================

Gnuastro library’s programming constructs (function declarations,
macros, data structures, or global variables) are classified by context
into multiple header files (see *note Headers::)(1).  In this section,
the functions in each header will be discussed under a separate
sub-section, which includes the name of the header.  Assuming a function
declaration is in ‘headername.h’, you can include its declaration in
your source code with:

     # include <gnuastro/headername.h>

The names of all constructs in ‘headername.h’ are prefixed with
‘gal_headername_’ (or ‘GAL_HEADERNAME_’ for macros).  The ‘gal_’ prefix
stands for _G_NU _A_stronomy _L_ibrary.

   Gnuastro library functions are compiled into a single file which can
be linked on the command-line with the ‘-lgnuastro’ option.  See *note
Linking:: and *note Summary and example on libraries:: for an
introduction on linking and some fully working examples of the
libraries.

   Gnuastro’s library is a high-level library which depends on lower
level libraries for some operations (see *note Dependencies::).
Therefore if at least one of Gnuastro’s functions in your program use
functions from the dependencies, you will also need to link those
dependencies after linking with Gnuastro.  See *note BuildProgram:: for
a convenient way to deal with the dependencies.  BuildProgram will take
care of the libraries to link with your program (which uses the Gnuastro
library), and can even run the built program afterwards.  Therefore it
allows you to conveniently focus on your exciting science/research when
using Gnuastro’s libraries.

*Libraries are still under heavy development: * Gnuastro was initially
created to be a collection of command-line programs.  However, as the
programs and their the shared functions grew, internal (not installed)
libraries were added.  Since the 0.2 release, the libraries are
install-able.  Hence the libraries are currently under heavy development
and will significantly evolve between releases and will become more
mature and stable in due time.  It will stabilize with the removal of
this notice.  Check the ‘NEWS’ file for interface changes.  If you use
the Info version of this manual (see *note Info::), you do not have to
worry: the documentation will correspond to your installed version.

* Menu:

* Configuration information::   General information about library config.
* Multithreaded programming::   Tools for easy multi-threaded operations.
* Library data types::          Definitions and functions for types.
* Pointers::                    Wrappers for easy working with pointers.**
* Library blank values::        Blank values and functions to deal with them.
* Library data container::      General data container in Gnuastro.
* Dimensions::                  Dealing with coordinates and dimensions.
* Linked lists::                Various types of linked lists.
* Array input output::          Reading and writing images or cubes.
* Table input output::          Reading and writing table columns.
* FITS files::                  Working with FITS data.
* File input output::           Reading and writing to various file formats.
* World Coordinate System::     Dealing with the world coordinate system.
* Arithmetic on datasets::      Arithmetic operations on a dataset.
* Tessellation library::        Functions for working on tiles.
* Bounding box::                Finding the bounding box.
* Polygons::                    Working with the vertices of a polygon.
* Qsort functions::             Helper functions for Qsort.
* K-d tree::                    Space partitioning in K dimensions.
* Permutations::                Re-order (or permute) the values in a dataset.
* Matching::                    Matching catalogs based on position.
* Statistical operations::      Functions for basic statistics.
* Fitting functions::           Fit independent and measured variables.
* Binary datasets::             Datasets that can only have values of 0 or 1.
* Labeled datasets::            Working with Segmented/labeled datasets.
* Convolution functions::       Library functions to do convolution.
* Pooling functions::           Reduce size of input by statistical methods.
* Interpolation::               Interpolate (over blank values possibly).
* Warp library::                Warp pixel grid to a new one.
* Color functions::             Definitions and operations related to colors.
* Git wrappers::                Wrappers for functions in libgit2.
* Python interface::            Functions to help in writing Python wrappers.
* Unit conversion library::     Converting between recognized units.
* Spectral lines library::      Functions for operating on Spectral lines.
* Cosmology library::           Cosmological calculations.
* SAO DS9 library::             Take inputs from files generated by SAO DS9.

   ---------- Footnotes ----------

   (1) Within Gnuastro’s source, all installed ‘.h’ files in
‘lib/gnuastro/’ are accompanied by a ‘.c’ file in ‘/lib/’.


File: gnuastro.info,  Node: Configuration information,  Next: Multithreaded programming,  Prev: Gnuastro library,  Up: Gnuastro library

12.3.1 Configuration information (‘config.h’)
---------------------------------------------

The ‘gnuastro/config.h’ header contains information about the full
Gnuastro installation on your system.  Gnuastro developers should note
that this is the only header that is not available within Gnuastro, it
is only available to a Gnuastro library user _after_ installation.
Within Gnuastro, ‘config.h’ (which is included in every Gnuastro ‘.c’
file, see *note Coding conventions::) has more than enough information
about the overall Gnuastro installation.

 -- Macro: GAL_CONFIG_VERSION
     This macro can be used as a string literal(1) containing the
     version of Gnuastro that is being used.  See *note Version
     numbering:: for the version formats.  For example:

          printf("Gnuastro version: %s\n", GAL_CONFIG_VERSION);

     or

          char *gnuastro_version=GAL_CONFIG_VERSION;

 -- Macro: GAL_CONFIG_HAVE_GSL_INTERP_STEFFEN
     GNU Scientific Library (GSL) is a mandatory dependency of Gnuastro
     (see *note GNU Scientific Library::).  The Steffen interpolation
     function that can be used in Gnuastro was introduced in GSL version
     2.0 (released in October 2015).  This macro will have a value of
     ‘1’ if the host GSL contains this feature at configure time, and
     ‘0’ otherwise.

 -- Macro: GAL_CONFIG_HAVE_FITS_IS_REENTRANT
     This macro will have a value of 1 when the CFITSIO of the host
     system has the ‘fits_is_reentrant’ function (available from CFITSIO
     version 3.30).  This function is used to see if CFITSIO was
     configured to read a FITS file simultaneously on different threads.

 -- Macro: GAL_CONFIG_HAVE_WCSLIB_VERSION
     WCSLIB is the reference library for world coordinate system
     transformation (see *note WCSLIB:: and *note World Coordinate
     System::).  However, only more recent versions of WCSLIB also
     provide its version number.  If the WCSLIB that is installed on the
     system provides its version (through the possibly existing
     ‘wcslib_version’ function), this macro will have a value of one,
     otherwise it will have a value of zero.

 -- Macro: GAL_CONFIG_HAVE_WCSLIB_DIS_H
     This macro has a value of 1 if the host’s WCSLIB has the
     ‘wcslib/dis.h’ header for distortion-related operations.

 -- Macro: GAL_CONFIG_HAVE_WCSLIB_MJDREF
     This macro has a value of 1 if the host’s WCSLIB reads and stores
     the ‘MJDREF’ FITS header keyword as part of its core ‘wcsprm’
     structure.

 -- Macro: GAL_CONFIG_HAVE_WCSLIB_OBSFIX
     This macro has a value of 1 if the host’s WCSLIB supports the
     ‘OBSFIX’ feature (used by ‘wcsfix’ function to parse the input WCS
     for known errors).

 -- Macro: GAL_CONFIG_HAVE_PTHREAD_BARRIER
     The POSIX threads standard define barriers as an optional
     requirement.  Therefore, some operating systems choose to not
     include it.  As one of the ‘./configure’ step checks, Gnuastro we
     check if your system has this POSIX thread barriers.  If so, this
     macro will have a value of ‘1’, otherwise it will have a value of
     ‘0’.  see *note Implementation of pthread_barrier:: for more.

 -- Macro: GAL_CONFIG_SIZEOF_LONG
 -- Macro: GAL_CONFIG_SIZEOF_SIZE_T
     The size of (number of bytes in) the system’s ‘long’ and ‘size_t’
     types.  Their values are commonly either 4 or 8 for 32-bit and
     64-bit systems.  You can also get this value with the expression
     ‘‘sizeof size_t’’ for example, without having to include this
     header.

 -- Macro: GAL_CONFIG_HAVE_LIBGIT2
     Libgit2 is an optional dependency of Gnuastro (see *note Optional
     dependencies::).  When it is installed and detected at configure
     time, this macro will have a value of ‘1’ (one).  Otherwise, it
     will have a value of ‘0’ (zero).  Gnuastro also comes with some
     wrappers to make it easier to use libgit2 (see *note Git
     wrappers::).

 -- Macro: GAL_CONFIG_HAVE_PYTHON
     Gnuastro can optionally provide a set of basic functions to
     facilitate wrapper libraries in Python (see *note Python
     interface::).  If a version of Python 3.X was found on the host
     system that has the necessary Numpy headers, this macro will be
     given a value of ‘1’.  Otherwise, it will be given a value of ‘0’
     and the the Python interface functions won’t be available in the
     host’s Gnuastro library.

 -- Macro: GAL_CONFIG_HAVE_GNUMAKE_H
     Gnuastro provides a set of GNU Make extension functions (see *note
     Makefile extensions::).  In order to use those, the host should
     have ‘gnumake.h’ in its include paths.  This check is done at
     Gnuastro’s configuration time.  If it was found, this macro is
     given a value of ‘1’, otherwise, it will have a value of ‘0’.

   ---------- Footnotes ----------

   (1) <https://en.wikipedia.org/wiki/String_literal>


File: gnuastro.info,  Node: Multithreaded programming,  Next: Library data types,  Prev: Configuration information,  Up: Gnuastro library

12.3.2 Multithreaded programming (‘threads.h’)
----------------------------------------------

In recent years, newer CPUs do not have significantly higher frequencies
any more.  However, CPUs are being manufactured with more cores,
enabling more than one operation (thread) at each instant.  This can be
very useful to speed up many aspects of processing and in particular
image processing.

   Most of the programs in Gnuastro utilize multi-threaded programming
for the CPU intensive processing steps.  This can potentially lead to a
significant decrease in the running time of a program, see *note A note
on threads::.  In terms of reading the code, you do not need to know
anything about multi-threaded programming.  You can simply follow the
case where only one thread is to be used.  In these cases, threads are
not used and can be completely ignored.

   When the C language was defined (the K&R’s book was written), using
threads was not common, so C’s threading capabilities are not introduced
there.  Gnuastro uses POSIX threads for multi-threaded programming,
defined in the ‘pthread.h’ system wide header.  There are various
resources for learning to use POSIX threads.  An excellent tutorial
(https://computing.llnl.gov/tutorials/pthreads/) is provided by the
Lawrence Livermore National Laboratory, with abundant figures to better
understand the concepts, it is a very good start.  The book ‘Advanced
programming in the Unix environment’(1), by Richard Stevens and Stephen
Rago, Addison-Wesley, 2013 (Third edition) also has two chapters
explaining the POSIX thread constructs which can be very helpful.

   An alternative to POSIX threads was OpenMP, but POSIX threads are low
level, allowing much more control, while being easier to understand, see
*note Why C::.  All the situations where threads are used in Gnuastro
currently are completely independent with no need of coordination
between the threads.  Such problems are known as “embarrassingly
parallel” problems.  They are some of the simplest problems to solve
with threads and are also the ones that benefit most from them, see the
LLNL introduction(2).

   One very useful POSIX thread concept is ‘pthread_barrier’.
Unfortunately, it is only an optional feature in the POSIX standard, so
some operating systems do not include it.  Therefore in *note
Implementation of pthread_barrier::, we introduce our own
implementation.  This is a rather technical section only necessary for
more technical readers and you can safely ignore it.  Following that, we
describe the helper functions in this header that can greatly simplify
writing a multi-threaded program, see *note Gnuastro's thread related
functions:: for more.

* Menu:

* Implementation of pthread_barrier::  Some systems do not have pthread_barrier
* Gnuastro's thread related functions::  Functions for managing threads.

   ---------- Footnotes ----------

   (1) Do Not let the title scare you!  The two chapters on
Multi-threaded programming are very self-sufficient and do not need any
more knowledge than K&R.

   (2) <https://computing.llnl.gov/tutorials/parallel_comp/>


File: gnuastro.info,  Node: Implementation of pthread_barrier,  Next: Gnuastro's thread related functions,  Prev: Multithreaded programming,  Up: Multithreaded programming

12.3.2.1 Implementation of ‘pthread_barrier’
............................................

One optional feature of the POSIX Threads standard is the
‘pthread_barrier’ concept.  It is a very useful high-level construct
that allows for independent threads to “wait” behind a “barrier” for the
rest after they finish.  Barriers can thus greatly simplify the code in
a multi-threaded program, so they are heavily used in Gnuastro.
However, since it is an optional feature in the POSIX standard, some
operating systems do not include it.  So to make Gnuastro portable, we
have written our own implementation of those ‘pthread_barrier’
functions.

   At ‘./configure’ time, Gnuastro will check if ‘pthread_barrier’
constructs are available on your system or not.  If ‘pthread_barrier’ is
not available, our internal implementation will be compiled into the
Gnuastro library and the definitions and declarations below will be
usable in your code with ‘#include <gnuastro/threads.h>’.

 -- Type: pthread_barrierattr_t
     Type to specify the attributes of a POSIX threads barrier.

 -- Type: pthread_barrier_t
     Structure defining the POSIX threads barrier.

 -- Function:
          int
          pthread_barrier_init (pthread_barrier_t *b,
          pthread_barrierattr_t *attr, unsigned int limit)
     Initialize the barrier ‘b’, with the attributes ‘attr’ and total
     ‘limit’ (a number of) threads that must wait behind it.  This
     function must be called before spinning off threads.

 -- Function:
          int
          pthread_barrier_wait (pthread_barrier_t *b)
     This function is called within each thread, just before it is ready
     to return.  Once a thread’s function hits this, it will “wait”
     until all the other functions are also finished.

 -- Function:
          int
          pthread_barrier_destroy (pthread_barrier_t *b)
     Destroy all the information in the barrier structure.  This should
     be called by the function that spun-off the threads after all the
     threads have finished.

     *Destroy a barrier before re-using it:* It is very important to
     destroy the barrier before (possibly) reusing it.  This destroy
     function not only destroys the internal structures, it also waits
     (in 1 microsecond intervals, so you will not notice!)  until all
     the threads do not need the barrier structure any more.  If you
     immediately start spinning off new threads with a not-destroyed
     barrier, then the internal structure of the remaining threads will
     get mixed with the new ones and you will get very strange and
     apparently random errors that are extremely hard to debug.


File: gnuastro.info,  Node: Gnuastro's thread related functions,  Prev: Implementation of pthread_barrier,  Up: Multithreaded programming

12.3.2.2 Gnuastro’s thread related functions
............................................

The POSIX Threads functions offered in the C library are very low-level
and offer a great range of control over the properties of the threads.
So if you are interested in customizing your tools for complicated
thread applications, it is strongly encouraged to get a nice familiarity
with them.  Some resources were introduced in *note Multithreaded
programming::.

   However, in many cases used in astronomical data analysis, you do not
need communication between threads and each target operation can be done
independently.  Since such operations are very common, Gnuastro provides
the tools below to facilitate the creation and management of jobs
without any particular knowledge of POSIX Threads for such operations.
The most interesting high-level functions of this section are the
‘gal_threads_number’ and ‘gal_threads_spin_off’ that identify the number
of threads on the system and spin-off threads.  You can see a
demonstration of using these functions in *note Library demo -
multi-threaded operation::.

 -- C struct: gal_threads_params
     Structure keeping the parameters of each thread.  When each thread
     is created, a pointer to this structure is passed to it.  The
     ‘params’ element can be the pointer to a structure defined by the
     user which contains all the necessary parameters to pass onto the
     worker function.  The rest of the elements within this structure
     are set internally by ‘gal_threads_spin_off’ and are relevant to
     the worker function.
          struct gal_threads_params
          {
            size_t            id; /* Id of this thread.                  */
            void         *params; /* User-identified pointer.            */
            size_t       *indexs; /* Target indices given to this thread. */
            pthread_barrier_t *b; /* Barrier for all threads.            */
          };

 -- Function:
          size_t
          gal_threads_number ()
     Return the number of threads that the operating system has
     available for your program.  This number is usually fixed for a
     single machine and does not change.  So this function is useful
     when you want to run your program on different machines (with
     different CPUs).

 -- Function:
          void
          gal_threads_spin_off (void *(*worker)(void *), void
          *caller_params, size_t numactions, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Distribute ‘numactions’ jobs between ‘numthreads’ threads and
     spin-off each thread by calling the ‘worker’ function.  The
     ‘caller_params’ pointer will also be passed to ‘worker’ as part of
     the ‘gal_threads_params’ structure.  For a fully working example of
     this function, please see *note Library demo - multi-threaded
     operation::.

     If there are many jobs (millions or billions) to organize, memory
     issues may become important.  With ‘minmapsize’ you can specify the
     minimum byte-size to allocate the necessary space in a
     memory-mapped file or alternatively in RAM. If ‘quietmmap’ is
     non-zero, then a warning will be printed upon creating a
     memory-mapped file.  For more on Gnuastro’s memory management, see
     *note Memory management::.

 -- Function:
          void
          gal_threads_attr_barrier_init (pthread_attr_t *attr,
          pthread_barrier_t *b, size_t limit)
     This is a low-level function in case you do not want to use
     ‘gal_threads_spin_off’.  It will initialize the general thread
     attribute ‘attr’ and the barrier ‘b’ with ‘limit’ threads to wait
     behind the barrier.  For maximum efficiency, the threads
     initialized with this function will be detached.  Therefore no
     communication is possible between these threads and in particular
     ‘pthread_join’ will not work on these threads.  You have to use the
     barrier constructs to wait for all threads to finish.

 -- Function:
          char *
          gal_threads_dist_in_threads (size_t numactions, size_t
          numthreads, size_t minmapsize, int quietmmap, size_t **indexs,
          size_t *icols)
     This is a low-level function in case you do not want to use
     ‘gal_threads_spin_off’.  The job of this function is to distribute
     ‘numactions’ jobs/actions in ‘numthreads’ threads.  To do this, it
     will assign each job an ID, ranging from 0 to ‘numactions’-1.  The
     output is the allocated ‘*indexs’ array and the ‘*icols’ number.
     In memory, it is just a simple 1D array that has ‘numthreads’
     $\times$ ‘*icols’ elements.  But you can visualize it as a 2D array
     with ‘numthreads’ rows and ‘*icols’ columns.  For more on the logic
     of the distribution, see below.

     When you have millions/billions of jobs to distribute, ‘indexs’
     will become very large.  For memory management (when to use a
     memory-mapped file, and when to use RAM), you need to specify the
     ‘minmapsize’ and ‘quietmmap’ arguments.  For more on memory
     management, see *note Memory management::.  In general, if your
     distributed jobs will not be on the scale of billions (and you want
     everything to always be written in RAM), just set ‘minmapsize=-1’
     and ‘quietmmap=1’.

     When ‘indexs’ is actually in a memory-mapped file, this function
     will return a string containing the name of the file (that you can
     later give to ‘gal_pointer_mmap_free’ to free/delete).  When
     ‘indexs’ is in RAM, this function will return a ‘NULL’ pointer.  So
     after you are finished with ‘indexs’, you can free it like this:

          char *mmapname;
          int quietmmap=1;
          size_t *indexs, thrdcols;
          size_t numactions=5000, minmapsize=-1;
          size_t numthreads=gal_threads_number();

          /* Distribute the jobs. */
          mmapname=gal_threads_dist_in_threads(numactions, numthreads,
                                               minmapsize, quietmmap,
                                               &indexs, &thrdcols);

          /* Do any processing you want... */

          /* Free the 'indexs' array. */
          if(mmapname) gal_pointer_mmap_free(&mmapname, quietmmap);
          else         free(indexs);

     Here is a brief description of the reasoning behind the ‘indexs’
     array and how the jobs are distributed.  Let’s assume you have $A$
     actions (where there is only one function and the input values
     differ for each action) and $T$ threads available to the system
     with $A>T$ (common values for these two would be $A>1000$ and
     $T<10$).  Spinning off a thread is not a cheap job and requires a
     significant number of CPU cycles.  Therefore, creating $A$ threads
     is not the best way to address such a problem.  The most efficient
     way to manage the actions is such that only $T$ threads are
     created, and each thread works on a list of actions identified for
     it in series (one after the other).  This way your CPU will get all
     the actions done with minimal overhead.

     The purpose of this function is to do what we explained above: each
     row in the ‘indexs’ array contains the indices of actions which
     must be done by one thread (so it has ‘numthreads’ rows with
     ‘*icols’ columns).  However, when using ‘indexs’, you do not have
     to know the number of columns.  It is guaranteed that all the rows
     finish with ‘GAL_BLANK_SIZE_T’ (see *note Library blank values::).
     The ‘GAL_BLANK_SIZE_T’ macro plays a role very similar to a
     string’s ‘\0’: every row finishes with this macro, so can easily
     stop parsing the indexes in the row as soon as you confront
     ‘GAL_BLANK_SIZE_T’.  For some real examples, please see the example
     program in ‘tests/lib/multithread.c’ for a demonstration.


File: gnuastro.info,  Node: Library data types,  Next: Pointers,  Prev: Multithreaded programming,  Up: Gnuastro library

12.3.3 Library data types (‘type.h’)
------------------------------------

Data in astronomy can have many types, numeric (numbers) and strings
(names, identifiers).  The former can also be divided into integers and
floats, see *note Numeric data types:: for a thorough discussion of the
different numeric data types and which one is useful for different
contexts.

   To deal with the very large diversity of types that are available
(and used in different contexts), in Gnuastro each type is identified
with global integer variable with a fixed name, this variable is then
passed onto functions that can work on any type or is stored in
Gnuastro’s *note Generic data container:: as one piece of meta-data.

   The actual values within these integer constants is irrelevant and
you should never rely on them.  When you need to check, explicitly use
the named variable in the table below.  If you want to check with more
than one type, you can use C’s ‘switch’ statement.

   Since Gnuastro heavily deals with file input-output, the types it
defines are fixed width types, these types are portable to all systems
and are defined in the standard C header ‘stdint.h’.  You do not need to
include this header, it is included by any Gnuastro header that deals
with the different types.  However, the most commonly used types in a C
(or C++) program (for example, ‘int’ or ‘long’) are not defined by their
exact width (storage size), but by their minimum storage.  So for
example, on some systems, ‘int’ may be 2 bytes (16-bits, the minimum
required by the standard) and on others it may be 4 bytes (32-bits,
common in modern systems).

   With every type, a unique “blank” value (or place-holder showing the
absence of data) can be defined.  Please see *note Library blank
values:: for constants that Gnuastro recognizes as a blank value for
each type.  See *note Numeric data types:: for more explanation on the
limits and particular aspects of each type.

 -- Global integer: GAL_TYPE_INVALID
     This is just a place-holder to specifically mark that no type has
     been set.

 -- Global integer: GAL_TYPE_BIT
     Identifier for a bit-stream.  Currently no program in Gnuastro
     works directly on bits, but features will be added in the future.

 -- Global integer: GAL_TYPE_UINT8
     Identifier for an unsigned, 8-bit integer type: ‘uint8_t’ (from
     ‘stdint.h’), or an ‘unsigned char’ in most modern systems.

 -- Global integer: GAL_TYPE_INT8
     Identifier for a signed, 8-bit integer type: ‘int8_t’ (from
     ‘stdint.h’), or an ‘signed char’ in most modern systems.

 -- Global integer: GAL_TYPE_UINT16
     Identifier for an unsigned, 16-bit integer type: ‘uint16_t’ (from
     ‘stdint.h’), or an ‘unsigned short’ in most modern systems.

 -- Global integer: GAL_TYPE_INT16
     Identifier for a signed, 16-bit integer type: ‘int16_t’ (from
     ‘stdint.h’), or a ‘short’ in most modern systems.

 -- Global integer: GAL_TYPE_UINT32
     Identifier for an unsigned, 32-bit integer type: ‘uint32_t’ (from
     ‘stdint.h’), or an ‘unsigned int’ in most modern systems.

 -- Global integer: GAL_TYPE_INT32
     Identifier for a signed, 32-bit integer type: ‘int32_t’ (from
     ‘stdint.h’), or an ‘int’ in most modern systems.

 -- Global integer: GAL_TYPE_UINT64
     Identifier for an unsigned, 64-bit integer type: ‘uint64_t’ (from
     ‘stdint.h’), or an ‘unsigned long’ in most modern 64-bit systems.

 -- Global integer: GAL_TYPE_INT64
     Identifier for a signed, 64-bit integer type: ‘int64_t’ (from
     ‘stdint.h’), or an ‘long’ in most modern 64-bit systems.

 -- Global integer: GAL_TYPE_INT
     Identifier for a ‘int’ type.  This is just an alias to ‘int16’, or
     ‘int32’ types, depending on the system.

 -- Global integer: GAL_TYPE_UINT
     Identifier for a ‘unsigned int’ type.  This is just an alias to
     ‘uint16’, or ‘uint32’ types, depending on the system.

 -- Global integer: GAL_TYPE_ULONG
     Identifier for a ‘unsigned long’ type.  This is just an alias to
     ‘uint32’, or ‘uint64’ types for 32-bit, or 64-bit systems
     respectively.

 -- Global integer: GAL_TYPE_LONG
     Identifier for a ‘long’ type.  This is just an alias to ‘int32’, or
     ‘int64’ types for 32-bit, or 64-bit systems respectively.

 -- Global integer: GAL_TYPE_SIZE_T
     Identifier for a ‘size_t’ type.  This is just an alias to ‘uint32’,
     or ‘uint64’ types for 32-bit, or 64-bit systems respectively.

 -- Global integer: GAL_TYPE_FLOAT32
     Identifier for a 32-bit single precision floating point type or
     ‘float’ in C.

 -- Global integer: GAL_TYPE_FLOAT64
     Identifier for a 64-bit double precision floating point type or
     ‘double’ in C.

 -- Global integer: GAL_TYPE_COMPLEX32
     Identifier for a complex number composed of two ‘float’ types.
     Note that the complex type is not yet fully implemented in all
     Gnuastro’s programs.

 -- Global integer: GAL_TYPE_COMPLEX64
     Identifier for a complex number composed of two ‘double’ types.
     Note that the complex type is not yet fully implemented in all
     Gnuastro’s programs.

 -- Global integer: GAL_TYPE_STRING
     Identifier for a string of characters (‘char *’).

 -- Global integer: GAL_TYPE_STRLL
     Identifier for a linked list of string of characters
     (‘gal_list_str_t’, see *note List of strings::).

The functions below are defined to make working with the integer
constants above easier.  In the functions below, the constants above can
be used for the ‘type’ input argument.

 -- Function:
          size_t
          gal_type_sizeof (uint8_t type)
     Return the number of bytes occupied by ‘type’.  Internally, this
     function uses C’s ‘sizeof’ operator to measure the size of each
     type.  For strings, this function will return the size of ‘char *’.

 -- Function:
          char *
          gal_type_name (uint8_t type, int long_name)
     Return a string literal that contains the name of ‘type’.  It can
     return both short and long formats of the type names (for example,
     ‘f32’ and ‘float32’).  If ‘long_name’ is non-zero, the long format
     will be returned, otherwise the short name will be returned.  The
     output string is statically allocated, so it should not be freed.
     This function is the inverse of the ‘gal_type_from_name’ function.
     For the full list of names/strings that this function will return,
     see *note Numeric data types::.

 -- Function:
          uint8_t
          gal_type_from_name (char *str)
     Return the Gnuastro integer constant that corresponds to the string
     ‘str’.  This function is the inverse of the ‘gal_type_name’
     function and accepts both the short and long formats of each type.
     For the full list of names/strings that this function will return,
     see *note Numeric data types::.

 -- Function:
          void
          gal_type_min (uint8_t type, void *in)
     Put the minimum possible value of ‘type’ in the space pointed to by
     ‘in’.  Since the value can have any type, this function does not
     return anything, it assumes the space for the given type is
     available to ‘in’ and writes the value there.  Here is one example
          int32_t min;
          gal_type_min(GAL_TYPE_INT32, &min);

     Note: Do not use the minimum value for a blank value of a general
     (initially unknown) type, please use the constants/functions
     provided in *note Library blank values:: for the definition and
     usage of blank values.

 -- Function:
          void
          gal_type_max (uint8_t type, void *in)
     Put the maximum possible value of ‘type’ in the space pointed to by
     ‘in’.  Since the value can have any type, this function does not
     return anything, it assumes the space for the given type is
     available to ‘in’ and writes the value there.  Here is one example
          uint16_t max;
          gal_type_max(GAL_TYPE_INT16, &max);

     Note: Do not use the maximum value for a blank value of a general
     (initially unknown) type, please use the constants/functions
     provided in *note Library blank values:: for the definition and
     usage of blank values.

 -- Function:
          int
          gal_type_is_int (uint8_t type)
     Return 1 if the type is an integer (any width and any sign).

 -- Function:
          int
          gal_type_is_list (uint8_t type)
     Return 1 if the type is a linked list and zero otherwise.

 -- Function:
          int
          gal_type_out (int first_type, int second_type)
     Return the larger of the two given types which can be used for the
     type of the output of an operation involving the two input types.

 -- Function:
          char *
          gal_type_bit_string (void *in, size_t size)
     Return the bit-string in the ‘size’ bytes that ‘in’ points to.  The
     string is dynamically allocated and must be freed afterwards.  You
     can use it to inspect the bits within one region of memory.  Here
     is one short example:

          int32_t a=2017;
          char *bitstr=gal_type_bit_string(&a, 4);
          printf("%d: %s (%X)\n", a, bitstr, a);
          free(bitstr);

     which will produce:
          2017: 11100001000001110000000000000000  (7E1)

     As the example above shows, the bit-string is not the most
     efficient way to inspect bits.  If you are familiar with
     hexadecimal notation, it is much more compact, see
     <https://en.wikipedia.org/wiki/Hexadecimal>.  You can use
     ‘printf’’s ‘%x’ or ‘%X’ to print integers in hexadecimal format.

 -- Function:
          char *
          gal_type_to_string (void *ptr, uint8_t type, int
          quote_if_str_has_space);
     Read the contents of the memory that ‘ptr’ points to (assuming it
     has type ‘type’ and print it into an allocated string which is
     returned.

     If the memory is a string of characters and
     ‘quote_if_str_has_space’ is non-zero, the output string will have
     double-quotes around it if it contains space characters.  Also,
     note that in this case, ‘ptr’ must be a pointer to an array of
     characters (or ‘char **’), as in the example below (which will put
     ‘"sample string"’ into ‘out’):

          char *out, *string="sample string"
          out = gal_type_to_string(&string, GAL_TYPE_STRING, 1);

 -- Function:
          int
          gal_type_from_string (void **out, char *string, uint8_t type)
     Read a string as a given data type and put a pointer to it in
     ‘*out’.  When ‘*out!=NULL’, then it is assumed to be already
     allocated and the value will be simply put there.  If ‘*out==NULL’,
     then space will be allocated for the given type and the string will
     be read into that type.

     Note that when we are dealing with a string type, ‘*out’ should be
     interpreted as ‘char **’ (one element in an array of pointers to
     different strings).  In other words, ‘out’ should be ‘char ***’.

     This function can be used to fill in arrays of numbers from strings
     (in an already allocated data structure), or add nodes to a linked
     list (if the type is a list type).  For an array, you have to pass
     the pointer to the ‘i’th element where you want the value to be
     stored, for example, ‘&(array[i])’.

     If the string was successfully parsed to the requested type, this
     function will return a ‘0’ (zero), otherwise it will return ‘1’
     (one).  This output format will help you check the status of the
     conversion in a code like the example below where we will try
     reading a string as a single precision floating point number.

          float out;
          void *outptr=&out;
          if( gal_type_from_string(&outptr, string, GAL_TYPE_FLOAT32) )
            {
              fprintf(stderr, "%s could not be read as float32\n", string);
              exit(EXIT_FAILURE);
            }

     When you need to read many numbers into an array, ‘out’ would be an
     array, and you can simply increment ‘outptr=out+i’ (where you
     increment ‘i’).

 -- Function:
          void *
          gal_type_string_to_number (char *string, uint8_t *type)
     Read ‘string’ into smallest type that can host the number, the
     allocated space for the number will be returned and the type of the
     number will be put into the memory that ‘type’ points to.  If
     ‘string’ could not be read as a number, this function will return
     ‘NULL’.

     This function first calls the C library’s ‘strtod’ function to read
     ‘string’ as a double-precision floating point number.  When
     successful, it will check the value to put it in the smallest
     numerical data type that can handle it; for example, ‘120’ and
     ‘50000’ will be read as a signed 8-bit integer and unsigned 16-bit
     integer types.  When reading as an integer, the C library’s
     ‘strtol’ function is used (in base-10) to parse the string again.
     This re-parsing as an integer is necessary because integers with
     many digits (for example, the Unix epoch seconds) will not be
     accurately stored as a floating point and we cannot use the result
     of ‘strtod’.

     When ‘string’ is successfully parsed as a number _and_ there is ‘.’
     in ‘string’, it will force the number into floating point types.
     For example, ‘"5"’ is read as an integer, while ‘"5."’ or ‘"5.0"’,
     or ‘"5.00"’ will be read as a floating point (single-precision).

     For floating point types, this function will count the number of
     significant digits and determine if the given string is single or
     double precision as described in *note Numeric data types::.

     For integers, negative numbers will always be placed in signed
     types (as expected).  If a positive integer falls below the maximum
     of a signed type of a certain width, it will be signed (for
     example, ‘10’ and ‘150’ will be defined as a signed and unsigned
     8-bit integer respectively).  In other words, even though ‘10’ can
     be unsigned, it will be read as a signed 8-bit integer.  This is
     done to respect the C implicit type conversion in binary operators,
     where signed integers will be interpreted as unsigned, when the
     other operand is an unsigned integer of the same width.

     For example, see the short program below.  It will print ‘-50 is
     larger than 100000’ (which is wrong!).  This happens because when a
     negative number is parsed as an unsigned, the value is effectively
     subtracted from the maximum and $4294967295-50$ is indeed larger
     than 100000 (recall that $4294967295$ is the largest unsigned
     32-bit integer, see *note Numeric data types::).

          #include <stdio.h>
          #include <stdlib.h>
          #include <stdint.h>

          int
          main(void)
          {
            int32_t  a=-50;
            uint32_t b=100000;
            printf("%d is %s than %d\n", a,
                   a>b ? "larger" : "less or equal", b);
            return 0;
          }

     However, if we read 100000 as a signed 32-bit integer, there will
     not be any problem and the printed sentence will be logically
     correct (for someone who does not know anything about numeric data
     types: users of your programs).  For the advantages of integers,
     see *note Integer benefits and pitfalls::.


File: gnuastro.info,  Node: Pointers,  Next: Library blank values,  Prev: Library data types,  Up: Gnuastro library

12.3.4 Pointers (‘pointer.h’)
-----------------------------

Pointers play an important role in the C programming language.  As the
name suggests, they _point_ to a byte in memory (like an address in a
city).  The C programming language gives you complete freedom in how to
use the byte (and the bytes that follow it).  Pointers are thus a very
powerful feature of C. However, as the saying goes: “With great power
comes great responsibility”, so they must be approached with care.  The
functions in this header are not very complex, they are just wrappers
over some basic pointer functionality regarding pointer arithmetic and
allocation (in memory or HDD/SSD).

 -- Function:
          void *
          gal_pointer_increment (void *pointer, size_t increment,
          uint8_t type)
     Return a pointer to an element that is ‘increment’ elements ahead
     of ‘pointer’, assuming each element has type of ‘type’.  For the
     type codes, see *note Library data types::.

     When working with the ‘array’ elements of ‘gal_data_t’, we are
     actually dealing with ‘void *’ pointers.  However, pointer
     arithmetic does not apply to ‘void *’, because the system does not
     know how many bytes there are in each element to increment the
     pointer respectively.  This function will use the given ‘type’ to
     calculate where the incremented element is located in memory.

 -- Function:
          size_t
          gal_pointer_num_between (void *earlier, void *later, uint8_t
          type)
     Return the number of elements (in the given ‘type’) between
     ‘earlier’ and ‘later’.  For the type codes, see *note Library data
     types::).

 -- Function:
          void *
          gal_pointer_allocate (uint8_t type, size_t size, int clear,
          const char *funcname, const char *varname)
     Allocate an array of type ‘type’ with ‘size’ elements in RAM (for
     the type codes, see *note Library data types::).  If ‘clear!=0’,
     then the allocated space is set to zero (cleared).

     This is effectively just a wrapper around C’s ‘malloc’ or ‘calloc’
     functions but takes Gnuastro’s integer type codes and will also
     abort with a clear error if there the allocation was not
     successful.  The number of allocated bytes is the value given to
     ‘size’ that is multiplied by the returned value of
     ‘gal_type_sizeof’ for the given type.  So if you want to allocate
     space for an array of strings you should pass the type
     ‘GAL_TYPE_STRING’.  Otherwise, if you just want space for one
     string (for example, 6 bytes for ‘hello’, including the
     string-termination character), you should set the type
     ‘GAL_TYPE_UINT8’.

     When space cannot be allocated, this function will abort the
     program with a message containing the reason for the failure.
     ‘funcname’ (name of the function calling this function) and
     ‘varname’ (name of variable that needs this space) will be used in
     this error message if they are not ‘NULL’.  In most modern
     compilers, you can use the generic ‘__func__’ variable for
     ‘funcname’.  In this way, you do not have to manually copy and
     paste the function name or worry about it changing later
     (‘__func__’ was standardized in C99).  To use this function
     effectively and avoid memory leaks, make sure to free the allocated
     array after you are done with it.  Also, be mindful of any
     functions that make use of this function as they should also free
     any allocated arrays to maintain memory management and prevent
     issues with the system.

 -- Function:
          void *
          gal_pointer_allocate_ram_or_mmap (uint8_t type, size_t size,
          int clear, size_t minmapsize, char **mmapname, int quietmmap,
          const char *funcname, const char *varname)
     Allocate the given space either in RAM or in a memory-mapped file.
     This function is just a high-level wrapper to
     ‘gal_pointer_allocate’ (to allocate in RAM) or
     ‘gal_pointer_mmap_allocate’ (to use a memory-mapped file).  For
     more on memory management in Gnuastro, please see *note Memory
     management::.  The various arguments are more fully explained in
     the two functions above.

 -- Function:
          void *
          gal_pointer_mmap_allocate (size_t size, uint8_t type, int
          clear, char **mmapname)
     Allocate the necessary space to keep ‘size’ elements of type ‘type’
     in HDD/SSD (a file, not in RAM). For the type codes, see *note
     Library data types::.  If ‘clear!=0’, then the allocated space will
     also be cleared.  The allocation is done using C’s ‘mmap’ function.
     The name of the file containing the allocated space is an allocated
     string that will be put in ‘*mmapname’.

     Note that the kernel does not allow an infinite number of memory
     mappings to files.  So it is not recommended to use this function
     with every allocation.  The best-case scenario to use this function
     is for arrays that are very large and can fill up the RAM. Keep the
     smaller arrays in RAM, which is faster and can have a
     (theoretically) unlimited number of allocations.

     When you are done with the dataset and do not need it anymore, do
     not use ‘free’ (the dataset is not in RAM). Just delete the file
     (and the allocated space for the filename) with the commands below,
     or simply use ‘gal_pointer_mmap_free’.

          remove(mmapname);
          free(mmapname);

 -- Function:
          void
          gal_pointer_mmap_free (char **mmapname, int quietmmap)
     “Free” (actually delete) the memory-mapped file that is named
     ‘*mmapname’, then free the string.  If ‘quietmmap’ is non-zero,
     then a warning will be printed for the user to know that the given
     file has been deleted.


File: gnuastro.info,  Node: Library blank values,  Next: Library data container,  Prev: Pointers,  Up: Gnuastro library

12.3.5 Library blank values (‘blank.h’)
---------------------------------------

When the position of an element in a dataset is important (for example,
a pixel in an image), a place-holder is necessary for the element if we
do not have a value to fill it with (for example, the CCD cannot read
those pixels).  We cannot simply shift all the other pixels to fill in
the one we have no value for.  In other cases, it often occurs that the
field of sky that you are studying is not a clean rectangle to nicely
fit into the boundaries of an image.  You need a way to separate the
pixels outside your scientific field from those inside it.  Blank values
act as these place holders in a dataset.  They have no usable value but
they have a position.

   Every type needs a corresponding blank value (see *note Numeric data
types:: and *note Library data types::).  Floating point types have a
unique value identified by IEEE known as Not-a-Number (or NaN) which is
a unique value that is recognized by the compiler.  However, integer and
string types do not have any standard value.  For integers, in Gnuastro
we take an extremum of the given type: for signed types (that allow
negatives), the minimum possible value is used as blank and for unsigned
types (that only accept positives), the maximum possible value is used.
To be generic and easy to read/write we define a macro for these blank
values and strongly encourage you only use these, and never make any
assumption on the value of a type’s blank value.

   The IEEE NaN blank value type is defined to fail on any comparison,
so if you are dealing with floating point types, you cannot use equality
(a NaN will _not_ be equal to a NaN). If you know your dataset is
floating point, you can use the ‘isnan’ function in C’s ‘math.h’ header.
For a description of numeric data types see *note Numeric data types::.
For the constants identifying integers, please see *note Library data
types::.

 -- Global integer: GAL_BLANK_UINT8
     Blank value for an unsigned, 8-bit integer.

 -- Global integer: GAL_BLANK_INT8
     Blank value for a signed, 8-bit integer.

 -- Global integer: GAL_BLANK_UINT16
     Blank value for an unsigned, 16-bit integer.

 -- Global integer: GAL_BLANK_INT16
     Blank value for a signed, 16-bit integer.

 -- Global integer: GAL_BLANK_UINT32
     Blank value for an unsigned, 32-bit integer.

 -- Global integer: GAL_BLANK_INT32
     Blank value for a signed, 32-bit integer.

 -- Global integer: GAL_BLANK_UINT64
     Blank value for an unsigned, 64-bit integer.

 -- Global integer: GAL_BLANK_INT64
     Blank value for a signed, 64-bit integer.

 -- Global integer: GAL_BLANK_INT
     Blank value for ‘int’ type (‘int16_t’ or ‘int32_t’ depending on the
     system.

 -- Global integer: GAL_BLANK_UINT
     Blank value for ‘int’ type (‘int16_t’ or ‘int32_t’ depending on the
     system.

 -- Global integer: GAL_BLANK_LONG
     Blank value for ‘long’ type (‘int32_t’ or ‘int64_t’ in 32-bit or
     64-bit systems).

 -- Global integer: GAL_BLANK_ULONG
     Blank value for ‘unsigned long’ type (‘uint32_t’ or ‘uint64_t’ in
     32-bit or 64-bit systems).

 -- Global integer: GAL_BLANK_SIZE_T
     Blank value for ‘size_t’ type (‘uint32_t’ or ‘uint64_t’ in 32-bit
     or 64-bit systems).

 -- Global integer: GAL_BLANK_FLOAT32
     Blank value for a single precision, 32-bit floating point type
     (IEEE NaN value).

 -- Global integer: GAL_BLANK_FLOAT64
     Blank value for a double precision, 64-bit floating point type
     (IEEE NaN value).

 -- Global integer: GAL_BLANK_STRING
     Blank value for string types (this is itself a string, it is not
     the ‘NULL’ pointer).

The functions below can be used to work with blank pixels.

 -- Function:
          void
          gal_blank_write (void *pointer, uint8_t type)
     Write the blank value for the given ‘type’ into the space that
     ‘pointer’ points to.  This can be used when the space is already
     allocated (for example, one element in an array or a statically
     allocated variable).

 -- Function:
          void *
          gal_blank_alloc_write (uint8_t type)
     Allocate the space required to keep the blank for the given data
     type ‘type’, write the blank value into it and return the pointer
     to it.

 -- Function:
          void
          gal_blank_initialize (gal_data_t *input)
     Initialize all the elements in the ‘input’ dataset to the blank
     value that corresponds to its type.  If ‘input’ is not a string,
     and is a tile over a larger dataset, only the region that the tile
     covers will be set to blank.  For strings, the full dataset will be
     initialized.

 -- Function:
          void
          gal_blank_initialize_array (void *array, size_t size, uint8_t
          type)
     Initialize all the elements in the ‘array’ to the blank value that
     corresponds to its type (identified with ‘type’), assuming the
     array has ‘size’ elements.

 -- Function:
          char *
          gal_blank_as_string (uint8_t type, int width)
     Write the blank value for the given data type ‘type’ into a string
     and return it.  The space for the string is dynamically allocated
     so it must be freed after you are done with it.  If ‘width!=0’,
     then the final string will be padded with white space characters to
     have the requested width if it is smaller.

 -- Function:
          int
          gal_blank_is (void *pointer, uint8_t type)
     Return 1 if the contents of ‘pointer’ (assuming a type of ‘type’)
     is blank.  Otherwise, return 0.  Note that this function only works
     on one element of the given type.  So if ‘pointer’ is an array,
     only its first element will be checked.  Therefore for strings, the
     type of ‘pointer’ is assumed to be ‘char *’.  To check if an
     array/dataset has blank elements or to find which elements in an
     array are blank, you can use ‘gal_blank_present’ or
     ‘gal_blank_flag’ respectively (described below).

 -- Function:
          int
          gal_blank_present (gal_data_t *input, int updateflag)
     Return 1 if the dataset has a blank value and zero if it does not.
     Before checking the dataset, this function will look at ‘input’’s
     flags.  If the ‘GAL_DATA_FLAG_BLANK_CH’ bit of ‘input->flag’ is on,
     this function will not do any check and will just use the
     information in the flags.  This can greatly speed up processing
     when a dataset needs to be checked multiple times.

     When the dataset’s flags were not used and ‘updateflags’ is
     non-zero, this function will set the flags appropriately to avoid
     having to re-check the dataset in future calls.  When
     ‘updateflags==0’, this function has no side-effects on the dataset:
     it will not toggle the flags.

     If you want to re-check a dataset with the blank-value-check flag
     already set (for example, if you have made changes to it), then
     explicitly set the ‘GAL_DATA_FLAG_BLANK_CH’ bit to zero before
     calling this function.  When there are no other flags, you can just
     set the flags to zero (‘input->flag=0’), otherwise you can use this
     expression:

          input->flag &= ~GAL_DATA_FLAG_BLANK_CH;

 -- Function:
          size_t
          gal_blank_number (gal_data_t *input, int updateflag)
     Return the number of blank elements in ‘input’.  If
     ‘updateflag!=0’, then the dataset blank keyword flags will be
     updated.  See the description of ‘gal_blank_present’ (above) for
     more on these flags.  If ‘input==NULL’, then this function will
     return ‘GAL_BLANK_SIZE_T’.

 -- Function:
          gal_data_t *
          gal_blank_flag (gal_data_t *input)
     Return a “flag” dataset with the same size as the input, but with
     an ‘uint8_t’ type that has a value of 1 for data elements that are
     blank and 0 for those that are not.

 -- Function:
          gal_data_t *
          gal_blank_flag_not (gal_data_t *input)
     Return a “flag” dataset with the same size as the input, but with
     an ‘uint8_t’ type that has a value of 1 for data elements that are
     _not_ blank and 0 for those that are blank.

 -- Function:
          size_t *
          gal_blank_not_minmax_coords (gal_data_t *input)
     Find the minimum and maximum coordinates of the non-blank regions
     within the input dataset.  The coordinates are in C order: starting
     from 0, and with the slowest dimension being first.  The output is
     an allocated array (that should be freed later) with $2\times N$
     elements; where $N$ is the number of dimensions.  The first two
     elements contain the minimum and maximum of regions containing
     non-blank elements along the 0-th dimension (the slowest), the
     second two elements contain the next dimension’s extrema; and so
     on.

 -- Function:
          gal_data_t *
          gal_blank_trim (gal_data_t *input, int inplace)
     Trim all the outer layers of blank values from the input dataset.
     For example in the 2D image, “layers” would correspond to columns
     or rows that are fully blank and touching the edge of the image.
     For a more complete description, see the description of the ‘trim’
     operator in *note Dimensionality changing operators::.

 -- Function:
          void
          gal_blank_flag_apply (gal_data_t *input, gal_data_t *flag)
     Set all non-zero and non-blank elements of ‘flag’ to blank in
     ‘input’.  ‘flag’ has to have an unsigned 8-bit type and be the same
     size as ‘input’.

 -- Function:
          void
          gal_blank_flag_remove (gal_data_t *input, gal_data_t *flag)
     Remove all elements within ‘input’ that are flagged, convert it to
     a 1D dataset and adjust the size properly (the number of
     non-flagged elements).  In practice this function does not‘realloc’
     the input array (see ‘gal_blank_remove_realloc’ for
     shrinking/re-allocating also), it just shifts the blank elements to
     the end and adjusts the size elements of the ‘gal_data_t’, see
     *note Generic data container::.

     Note that elements that are blank, but not flagged will not be
     removed.  This function will only remove flagged elements.

     If all the elements were flagged, then ‘input->size’ will be zero.
     This is thus a good parameter to check after calling this function
     to see if there actually were any non-flagged elements in the input
     or not and take the appropriate measure.  This check is highly
     recommended because it will avoid strange bugs in later steps.

 -- Function:
          void
          gal_blank_remove (gal_data_t *input)
     Remove blank elements from a dataset, convert it to a 1D dataset,
     adjust the size properly (the number of non-blank elements), and
     toggle the blank-value-related bit-flags.  In practice this
     function does not‘realloc’ the input array (see
     ‘gal_blank_remove_realloc’ for shrinking/re-allocating also), it
     just shifts the blank elements to the end and adjusts the size
     elements of the ‘gal_data_t’, see *note Generic data container::.

     If all the elements were blank, then ‘input->size’ will be zero.
     This is thus a good parameter to check after calling this function
     to see if there actually were any non-blank elements in the input
     or not and take the appropriate measure.  This check is highly
     recommended because it will avoid strange bugs in later steps.

 -- Function:
          void
          gal_blank_remove_realloc (gal_data_t *input)
     Similar to ‘gal_blank_remove’, but also shrinks/re-allocates the
     dataset’s allocated memory.

 -- Function:
          gal_data_t *
          gal_blank_remove_rows (gal_data_t *columns, gal_list_sizet_t
          *column_indexs, int onlydim0)
     Remove (in place) any row that has at least one blank value in any
     of the input columns and return a “flag” dataset (that should be
     freed later).  The input ‘columns’ is a list of ‘gal_data_t’s (see
     *note List of gal_data_t::).  When ‘onlydim0!=0’ the vector columns
     (with 2 dimensions) will not be checked for the presence of blank
     values.

     After this function, all the elements in ‘columns’ will still have
     the same size as each other, but if any of the searched columns has
     blank elements, all their sizes will decrease together.

     The returned flag dataset has the same size as the original input
     dataset, with a type of ‘uint8_t’.  Every row that has been removed
     from the original dataset has a value of 1, and the rest have a
     value of 0.

     When ‘column_indexs!=NULL’, only the columns whose index (counting
     from zero) is in ‘column_indexs’ will be used to check for blank
     values (see *note List of size_t::.  Therefore, if you want to
     check all columns, just set this to ‘NULL’.  In any case (no matter
     which columns are checked for blanks), the selected rows from all
     columns will be removed.


File: gnuastro.info,  Node: Library data container,  Next: Dimensions,  Prev: Library blank values,  Up: Gnuastro library

12.3.6 Data container (‘data.h’)
--------------------------------

Astronomical datasets have various dimensions, for example, 1D spectra
or table columns, 2D images, or 3D Integral field data cubes.  Datasets
can also have various numeric data types, depending on the
operation/purpose, for example, processed images are commonly stored in
floating point format, but their mask images are integers (allowing
bit-wise flags to identify certain classes of pixels to keep or mask,
see *note Numeric data types::).  Certain other information about a
dataset are also commonly necessary, for example, the units of the
dataset, the name of the dataset and some comments.  To deal with any
generic dataset, Gnuastro defines the ‘gal_data_t’ as input or output.

* Menu:

* Generic data container::      Definition of Gnuastro’s generic container.
* Dataset allocation::          Allocate, initialize and free a dataset.
* Arrays of datasets::          Functions to help with array of datasets.
* Copying datasets::            Functions to copy a dataset to a new one.


File: gnuastro.info,  Node: Generic data container,  Next: Dataset allocation,  Prev: Library data container,  Up: Library data container

12.3.6.1 Generic data container (‘gal_data_t’)
..............................................

To be able to deal with any dataset (various dimensions, numeric data
types, units and higher-level structures), Gnuastro defines the
‘gal_data_t’ type which is the input/output container of choice for many
of Gnuastro library’s functions.  It is defined in ‘gnuastro/data.h’.
If you will be using (‘‘# include’’ing) those libraries, you do not need
to include this header explicitly, it is already included by any library
header that uses ‘gal_data_t’.

 -- Type (C struct): gal_data_t
     The main container for datasets in Gnuastro.  It can host data of
     any dimensions, with any numeric data type.  It is actually a
     structure, but ‘typedef’’d as a new type to avoid having to write
     the ‘struct’ before any declaration.  The actual structure is shown
     below which is followed by a description of each element.

          typedef struct gal_data_t
          {
            void     *restrict array;  /* Basic array information.   */
            uint8_t             type;
            size_t              ndim;
            size_t            *dsize;
            size_t              size;
            int            quietmmap;
            char           *mmapname;
            size_t        minmapsize;

            int                 nwcs;  /* WCS information.           */
            struct wcsprm       *wcs;

            uint8_t             flag;  /* Content description.       */
            int               status;
            char               *name;
            char               *unit;
            char            *comment;

            int             disp_fmt;  /* For text printing.         */
            int           disp_width;
            int       disp_precision;

            struct gal_data_t  *next;  /* For higher-level datasets. */
            struct gal_data_t *block;
          } gal_data_t;

The list below contains a description for each ‘gal_data_t’ element.

‘void *restrict array’
     This is the pointer to the main array of the dataset containing the
     raw data (values).  All the other elements in this data-structure
     are actually meta-data enabling us to use/understand the series of
     values in this array.  It must allow data of any type (see *note
     Numeric data types::), so it is defined as a ‘void *’ pointer.  A
     ‘void *’ array is not directly usable in C, so you have to cast it
     to proper type before using it, please see *note Library demo -
     reading a image:: for a demonstration.

     The ‘restrict’ keyword was formally introduced in C99 and is used
     to tell the compiler that at any moment only this pointer will
     modify what it points to (a pixel in an image for example)(1).
     This extra piece of information can greatly help in compiler
     optimizations and thus the running time of the program.  But older
     compilers might not have this capability, so at ‘./configure’ time,
     Gnuastro checks this feature and if the user’s compiler does not
     support ‘restrict’, it will be removed from this definition.

‘uint8_t type’
     A fixed code (integer) used to identify the type of data in ‘array’
     (see *note Numeric data types::).  For the list of acceptable
     values to this variable, please see *note Library data types::.

‘size_t ndim’
     The dataset’s number of dimensions.

‘size_t *dsize’
     The size of the dataset along each dimension.  This is an array
     (with ‘ndim’ elements), of positive integers in row-major order(2)
     (based on C). When a data file is read into memory with Gnuastro’s
     libraries, this array is dynamically allocated based on the number
     of dimensions that the dataset has.

     It is important to remember that C’s row-major ordering is the
     opposite of the FITS standard which is in column-major order: in
     the FITS standard the fastest dimension’s size is specified by
     ‘NAXIS1’, and slower dimensions follow.  The FITS standard was
     defined mainly based on the FORTRAN language which is the opposite
     of C’s approach to multi-dimensional arrays (and also starts
     counting from 1 not 0).  Hence if a FITS image has ‘NAXIS1==20’ and
     ‘NAXIS2==50’, the ‘dsize’ array must be filled with ‘dsize[0]==50’
     and ‘dsize[1]==20’.

     The fastest dimension is the one that is contiguous in memory: to
     increment by one along that dimension, just go to the next element
     in the array.  As we go to slower dimensions, the number of memory
     cells we have to skip for an increment along that dimension becomes
     larger.

‘size_t size’
     The total number of elements in the dataset.  This is actually a
     multiplication of all the values in the ‘dsize’ array, so it is not
     an independent parameter.  However, low-level operations with the
     dataset (irrespective of its dimensions) commonly need this number,
     so this element is designed to avoid calculating it every time.

‘int quietmmap’
     When this value is zero, and the dataset must not be allocated in
     RAM (see ‘mmapname’ and ‘minmapsize’ below), a warning will be
     printed to inform the user when the file is created and when it is
     deleted.  The warning includes the filename, the size in bytes, and
     the fact that they can toggle this behavior through ‘--minmapsize’
     option in Gnuastro’s programs.

‘char *mmapname’
     Name of file hosting the ‘mmap’’d contents of ‘array’.  If the
     value of this variable is ‘NULL’, then the contents of ‘array’ are
     actually stored in RAM, not in a file on the HDD/SSD. See the
     description of ‘minmapsize’ below for more.

     If a file is used, it will be kept in the ‘gnuastro_mmap’ directory
     of the running directory.  Its name is randomly selected to allow
     multiple arrays at the same time, see description of ‘--minmapsize’
     in *note Processing options::.  When ‘gal_data_free’ is called the
     randomly named file will be deleted.

‘size_t minmapsize’
     The minimum size of an array (in bytes) to store the contents of
     ‘array’ as a file (on the non-volatile HDD/SSD), not in RAM. This
     can be very useful for large datasets which can be very memory
     intensive and the user’s RAM might not be sufficient to
     keep/process it.  A random filename is assigned to the array which
     is available in the ‘mmapname’ element of ‘gal_data_t’ (above), see
     there for more.  ‘minmapsize’ is stored in each ‘gal_data_t’, so it
     can be passed on to subsequent/derived datasets.

     See the description of the ‘--minmapsize’ option in *note
     Processing options:: for more on using this value.

‘nwcs’
     The number of WCS coordinate representations (for WCSLIB).

‘struct wcsprm *wcs’
     The main WCSLIB structure keeping all the relevant information
     necessary for WCSLIB to do its processing and convert data-set
     positions into real-world positions.  When it is given a ‘NULL’
     value, all possible WCS calculations/measurements will be ignored.

‘uint8_t flag’
     Bit-wise flags to describe general properties of the dataset.  The
     number of bytes available in this flag is stored in the
     ‘GAL_DATA_FLAG_SIZE’ macro.  Note that you should use bit-wise
     operators(3) to check these flags.  The currently recognized bits
     are stored in these macros:

     ‘GAL_DATA_FLAG_BLANK_CH’
          Marking that the dataset has been checked for blank values or
          not.  When a dataset does not have any blank values, the
          ‘GAL_DATA_FLAG_HASBLANK’ bit will be zero.  But upon
          initialization, all bits also get a value of zero.  Therefore,
          a checker needs this flag to see if the value in
          ‘GAL_DATA_FLAG_HASBLANK’ is reliable (dataset has actually
          been parsed for a blank value) or not.

          Also, if it is necessary to re-check the presence of flags,
          you just have to set this flag to zero and call
          ‘gal_blank_present’ for example, to parse the dataset and
          check for blank values.  Note that for improved efficiency,
          when this flag is set, ‘gal_blank_present’ will not actually
          parse the dataset, it will just use ‘GAL_DATA_FLAG_HASBLANK’.

     ‘GAL_DATA_FLAG_HASBLANK’
          This bit has a value of ‘1’ when the given dataset has blank
          values.  If this bit is ‘0’ and ‘GAL_DATA_FLAG_BLANK_CH’ is
          ‘1’, then the dataset has been checked and it did not have any
          blank values, so there is no more need for further checks.

     ‘GAL_DATA_FLAG_SORT_CH’
          Marking that the dataset is already checked for being sorted
          or not and thus that the possible ‘0’ values in
          ‘GAL_DATA_FLAG_SORTED_I’ and ‘GAL_DATA_FLAG_SORTED_D’ are
          meaningful.  The logic behind this is similar to that in
          ‘GAL_DATA_FLAG_BLANK_CH’.

     ‘GAL_DATA_FLAG_SORTED_I’
          This bit has a value of ‘1’ when the given dataset is sorted
          in an increasing manner.  If this bit is ‘0’ and
          ‘GAL_DATA_FLAG_SORT_CH’ is ‘1’, then the dataset has been
          checked and was not sorted (increasing), so there is no more
          need for further checks.

     ‘GAL_DATA_FLAG_SORTED_D’
          This bit has a value of ‘1’ when the given dataset is sorted
          in a decreasing manner.  If this bit is ‘0’ and
          ‘GAL_DATA_FLAG_SORT_CH’ is ‘1’, then the dataset has been
          checked and was not sorted (decreasing), so there is no more
          need for further checks.

     The macro ‘GAL_DATA_FLAG_MAXFLAG’ contains the largest internally
     used bit-position.  Higher-level flags can be defined with the
     bit-wise shift operators using this macro to define internal flags
     for libraries/programs that depend on Gnuastro without causing any
     possible conflict with the internal flags discussed above or having
     to check the values manually on every release.

‘int status’
     A context-specific status values for this data-structure.  This
     integer will not be set by Gnuastro’s libraries.  You can use it
     keep some additional information about the dataset (with integer
     constants) depending on your applications.

‘char *name’
     The name of the dataset.  If the dataset is a multi-dimensional
     array and read/written as a FITS image, this will be the value in
     the ‘EXTNAME’ FITS keyword.  If the dataset is a one-dimensional
     table column, this will be the column name.  If it is set to ‘NULL’
     (by default), it will be ignored.

‘char *unit’
     The units of the dataset (for example, ‘BUNIT’ in the standard FITS
     keywords) that will be read from or written to files/tables along
     with the dataset.  If it is set to ‘NULL’ (by default), it will be
     ignored.

‘char *comment’
     Any further explanation about the dataset which will be written to
     any output file if present.

‘disp_fmt’
     Format to use for printing each element of the dataset to a plain
     text file, the acceptable values to this element are defined in
     *note Table input output::.  Based on C’s ‘printf’ standards.

‘disp_width’
     Width of printing each element of the dataset to a plain text file,
     the acceptable values to this element are defined in *note Table
     input output::.  Based on C’s ‘printf’ standards.

‘disp_precision’
     Precision of printing each element of the dataset to a plain text
     file, the acceptable values to this element are defined in *note
     Table input output::.  Based on C’s ‘printf’ standards.

‘gal_data_t *next’
     Through this pointer, you can link a ‘gal_data_t’ with other
     datasets related datasets, for example, the different columns in a
     dataset each have one ‘gal_data_t’ associate with them and they are
     linked to each other using this element.  There are several
     functions described below to facilitate using ‘gal_data_t’ as a
     linked list.  See *note Linked lists:: for more on these wonderful
     high-level constructs.

‘gal_data_t *block’
     Pointer to the start of the complete allocated block of memory.
     When this pointer is not ‘NULL’, the dataset is not treated as a
     contiguous patch of memory.  Rather, it is seen as covering only a
     portion of the larger patch of memory that ‘block’ points to.  See
     *note Tessellation library:: for a more thorough explanation and
     functions to help work with tiles that are created from this
     pointer.

   ---------- Footnotes ----------

   (1) Also see <https://en.wikipedia.org/wiki/Restrict>.

   (2) Also see
<https://en.wikipedia.org/wiki/Row-_and_column-major_order>.

   (3) See <https://en.wikipedia.org/wiki/Bitwise_operations_in_C>.


File: gnuastro.info,  Node: Dataset allocation,  Next: Arrays of datasets,  Prev: Generic data container,  Up: Library data container

12.3.6.2 Dataset allocation
...........................

Gnuastro’s main data container was defined in *note Generic data
container::.  The functions listed in this section describe the most
basic operations on ‘gal_data_t’: those related to allocation and
freeing.  These functions are declared in ‘gnuastro/data.h’ which is
also visible from the function names (see *note Gnuastro library::).

 -- Function:
          gal_data_t *
          gal_data_alloc (void *array, uint8_t type, size_t ndim, size_t
          *dsize, struct wcsprm *wcs, int clear, size_t minmapsize, int
          quietmmap, char *name, char *unit, char *comment)

     Dynamically allocate a ‘gal_data_t’ and initialize it will all the
     given values.  See the description of ‘gal_data_initialize’ and
     *note Generic data container:: for more information.  This function
     will often be the most frequently used because it allocates the
     ‘gal_data_t’ hosting all the values _and_ initializes it.  Once you
     are done with the dataset, be sure to clean up all the allocated
     spaces with ‘gal_data_free’.

 -- Function:
          void
          gal_data_initialize (gal_data_t *data, void *array, uint8_t
          type, size_t ndim, size_t *dsize, struct wcsprm *wcs, int
          clear, size_t minmapsize, int quietmmap, char *name, char
          *unit, char *comment)

     Initialize the given data structure (‘data’) with all the given
     values.  Note that the raw input ‘gal_data_t’ must already have
     been allocated before calling this function.  For a description of
     each variable see *note Generic data container::.  It will set the
     values and do the necessary allocations.  If they are not ‘NULL’,
     all input arrays (‘dsize’, ‘wcs’, ‘name’, ‘unit’, ‘comment’) are
     separately copied (allocated) by this function for usage in ‘data’,
     so you can safely use one value to initialize many datasets or use
     statically allocated variables in this function call.  Once you are
     done with the dataset, you can free all the allocated spaces with
     ‘gal_data_free_contents’.

     If ‘array’ is not ‘NULL’, it will be directly copied into
     ‘data->array’ (based on the total number of elements calculated
     from ‘dsize’) and no new space will be allocated for the array of
     this dataset, this has many low-level advantages and can be used to
     work on regions of a dataset instead of the whole allocated array
     (see the description under ‘block’ in *note Generic data
     container:: for one example).  If the given pointer is not the
     start of an allocated block of memory or it is used in multiple
     datasets, be sure to set it to ‘NULL’ (with ‘data->array=NULL’)
     before cleaning up with ‘gal_data_free_contents’.

     ‘ndim’ may be zero.  In this case no allocation will occur,
     ‘data->array’ and ‘data->dsize’ will be set to ‘NULL’ and
     ‘data->size’ will be zero.  However (when necessary) ‘dsize’ must
     not have any zero values (a dimension of length zero is not
     defined).

 -- Function:
          gal_data_t *
          gal_data_alloc_empty (size_t ndim, size_t minmapsize, int
          quietmmap)
     Allocate an empty dataset with a certain number of dimensions, but
     no ’array’ component.  The ‘size’ element will be set to zero and
     the ‘dsize’ array will be properly allocated (based on the number
     of dimensions), but all elements will be zero.  This is useful in
     scenarios where you just need a ‘gal_data_t’ for metadata.

 -- Function:
          void
          gal_data_free_contents (gal_data_t *data)
     Free all the non-‘NULL’ pointers in ‘gal_data_t’ except for ‘next’
     and ‘block’.  All freed arrays are set to ‘NULL’.  If ‘data’ is
     actually a tile (‘data->block!=NULL’, see *note Tessellation
     library::), then ‘data->array’ is not freed.  For a complete
     description of ‘gal_data_t’ and its contents, see *note Generic
     data container::.

 -- Function:
          void
          gal_data_free (gal_data_t *data)
     Free all the non-‘NULL’ pointers in ‘gal_data_t’, then free the
     actual data structure.


File: gnuastro.info,  Node: Arrays of datasets,  Next: Copying datasets,  Prev: Dataset allocation,  Up: Library data container

12.3.6.3 Arrays of datasets
...........................

Gnuastro’s generic data container (‘gal_data_t’) is a very versatile
structure that can be used in many higher-level contexts.  One such
higher-level construct is an array of ‘gal_data_t’ structures to
simplify the allocation (and later cleaning) of several ‘gal_data_t’s
that are related.

   For example, each column in a table is usually represented by one
‘gal_data_t’ (so it has its own name, data type, units, etc.).  A table
(with many columns) can be seen as an array of ‘gal_data_t’s (when the
number of columns is known a-priori).  The functions below are defined
to create a cleared array of data structures and to free them when none
are necessary any more.  These functions are declared in
‘gnuastro/data.h’ which is also visible from the function names (see
*note Gnuastro library::).

 -- Function:
          gal_data_t *
          gal_data_array_calloc (size_t size)
     Allocate an array of ‘gal_data_t’ with ‘size’ elements.  This
     function will also initialize all the values (‘NULL’ for pointers
     and 0 for other types).  You can use ‘gal_data_initialize’ to fill
     each element of the array afterwards.  The following code snippet
     is one example of doing this.

          size_t i;
          gal_data_t *dataarr;
          dataarr=gal_data_array_calloc(10);
          for(i=0;i<10;++i) gal_data_initialize(&dataarr[i], ...);
          ...
          gal_data_array_free(dataarr, 10, 1);

 -- Function:
          void
          gal_data_array_free (gal_data_t *dataarr, size_t num, int
          free_array)
     Free all the ‘num’ elements within ‘dataarr’ and the actual
     allocated array.  If ‘free_array’ is not zero, then the ‘array’
     element of all the datasets will also be freed, see *note Generic
     data container::.

 -- Function:
          gal_data_t **
          gal_data_array_ptr_calloc (size_t size)
     Allocate an array of pointers to Gnuastro’s generic data structure
     and initialize all pointers to ‘NULL’.  This is useful when you
     want to allocate individual datasets later (for example, with
     ‘gal_data_alloc’).

 -- Function:
          void
          gal_data_array_ptr_free (gal_data_t **dataptr, size_t size,
          int free_array);
     Free all the individual datasets within the elements of ‘dataptr’,
     then free ‘dataptr’ itself (the array of pointers that was probably
     allocated with ‘gal_data_array_ptr_calloc’.


File: gnuastro.info,  Node: Copying datasets,  Prev: Arrays of datasets,  Up: Library data container

12.3.6.4 Copying datasets
.........................

The functions in this section describes Gnuastro’s facilities to copy a
given dataset into another.  The new dataset can have a different type
(including a string), it can be already allocated (in which case only
the values will be written into it).  In all these cases, if the input
dataset is a tile or a list, only the data within the given tile, or the
given node in a list, are copied.  If the input is a list, the ‘next’
pointer will also be copied to the output, see *note List of
gal_data_t::.

   In many of the functions here, it is possible to copy the dataset to
a new numeric data type (see *note Numeric data types::.  In such cases,
Gnuastro’s library is going to use the native conversion by C. So if you
are converting to a smaller type, it is up to you to make sure that the
values fit into the output type.

 -- Function:
          gal_data_t *
          gal_data_copy (gal_data_t *in)
     Return a new dataset that is a copy of ‘in’, all of ‘in’’s
     meta-data will also copied into the output, except for ‘block’.  If
     the dataset is a tile/list, only the given tile/node will be
     copied, the ‘next’ pointer will also be copied however.

 -- Function:
          gal_data_t *
          gal_data_copy_to_new_type (gal_data_t *in, uint8_t newtype)
     Return a copy of the dataset ‘in’, converted to ‘newtype’, see
     *note Library data types:: for Gnuastro library’s type identifiers.
     The returned dataset will have all meta-data except their type and
     ‘block’ equal to the input’s metadata.  If the dataset is a
     tile/list, only the given tile/node will be copied, the ‘next’
     pointer will also be copied however.

 -- Function:
          gal_data_t *
          gal_data_copy_to_new_type_free (gal_data_t *in, uint8_t
          newtype)
     Return a copy of the dataset ‘in’ that is converted to ‘newtype’
     and free the input dataset.  See *note Library data types:: for
     Gnuastro library’s type identifiers.  The returned dataset will
     have all meta-data, except their type, equal to the input’s
     metadata (including ‘next’).  Note that if the input is a tile
     within a larger block, it will not be freed.  This function is
     similar to ‘gal_data_copy_to_new_type’, except that it will free
     the input dataset.

 -- Function:
          void
          gal_data_copy_to_allocated (gal_data_t *in, gal_data_t *out)
     Copy the contents of the array in ‘in’ into the already allocated
     array in ‘out’.  The types of the input and output may be
     different, type conversion will be done internally.  When ‘in->size
     != out->size’ this function will behave as follows:

     ‘out->size < in->size’
          This function will not re-allocate the necessary space, it
          will abort with an error, so please check before calling this
          function.

     ‘out->size > in->size’
          This function will write the values in ‘out->size’ and
          ‘out->dsize’ from the same values of ‘in’.  So if you want to
          use a pre-allocated space/dataset multiple times with varying
          input sizes, be sure to reset ‘out->size’ before every call to
          this function.

 -- Function:
          gal_data_t *
          gal_data_copy_string_to_number (char *string)
     Read ‘string’ into the smallest type that can store the value (see
     *note Numeric data types::).  This function is just a wrapper for
     the ‘gal_type_string_to_number’, but will put the value into a
     single-element dataset.


File: gnuastro.info,  Node: Dimensions,  Next: Linked lists,  Prev: Library data container,  Up: Gnuastro library

12.3.7 Dimensions (‘dimension.h’)
---------------------------------

An array is a contiguous region of memory.  Hence, at the lowest level,
every element of an array just has one single-valued position: the
number of elements that lie between it and the first element in the
array.  This is also known as the _index_ of the element within the
array.  A dataset’s number of dimensions is high-level abstraction
(meta-data) that we project onto that contiguous patch of memory.  When
the array is interpreted as a one-dimensional dataset, this index is
also the _coordinate_ of the element.  But once we associate the patch
of memory with a higher dimension, there must also be one coordinate for
each dimension.

   The functions and macros in this section provide you with the tools
to convert an index into a coordinate and vice-versa along with several
other issues for example, issues with the neighbors of an element in a
multi-dimensional context.

 -- Function:
          size_t
          gal_dimension_total_size (size_t ndim, size_t *dsize)
     Return the total number of elements for a dataset with ‘ndim’
     dimensions that has ‘dsize’ elements along each dimension.

 -- Function:
          int
          gal_dimension_is_different (gal_data_t *first, gal_data_t
          *second)
     Return ‘1’ (one) if the two datasets do not have the same size
     along all dimensions.  This function will also return ‘1’ when the
     number of dimensions of the two datasets are different.

 -- Function:
          size_t *
          gal_dimension_increment (size_t ndim, size_t *dsize)
     Return an allocated array that has the number of elements necessary
     to increment an index along every dimension.  For example, along
     the fastest dimension (last element in the ‘dsize’ and returned
     arrays), the value is ‘1’ (one).

 -- Function:
          size_t
          gal_dimension_num_neighbors (size_t ndim)
     The maximum number of neighbors (any connectivity) that a data
     element can have in ‘ndim’ dimensions.  Effectively, this function
     just returns $3^n-1$ (where $n$ is the number of dimensions).

 -- Function-like macro: GAL_DIMENSION_FLT_TO_INT (FLT)
     Calculate the integer pixel position that the floating point ‘FLT’
     number belongs to.  In the FITS format (and thus in Gnuastro), the
     center of each pixel is allocated on an integer (not it edge), so
     the pixel which hosts a floating point number cannot simply be
     found with internal type conversion.

 -- Function:
          void
          gal_dimension_add_coords (size_t *c1, size_t *c2, size_t *out,
          size_t ndim)
     For every dimension, add the coordinates in ‘c1’ with ‘c2’ and put
     the result into ‘out’.  In other words, for dimension ‘i’ run
     ‘out[i]=c1[i]+c2[i];’.  Hence ‘out’ may be equal to any one of ‘c1’
     or ‘c2’.

 -- Function:
          size_t
          gal_dimension_coord_to_index (size_t ndim, size_t *dsize,
          size_t *coord)
     Return the index (counting from zero) from the coordinates in
     ‘coord’ (counting from zero) assuming the dataset has ‘ndim’
     elements and the size of the dataset along each dimension is in the
     ‘dsize’ array.

 -- Function:
          void
          gal_dimension_index_to_coord (size_t index, size_t ndim,
          size_t *dsize, size_t *coord)
     Fill in the ‘coord’ array with the coordinates that correspond to
     ‘index’ assuming the dataset has ‘ndim’ elements and the size of
     the dataset along each dimension is in the ‘dsize’ array.  Note
     that both ‘index’ and each value in ‘coord’ are assumed to start
     from ‘0’ (zero).  Also that the space which ‘coord’ points to must
     already be allocated before calling this function.

 -- Function:
          size_t
          gal_dimension_dist_manhattan (size_t *a, size_t *b, size_t
          ndim)
     Return the manhattan distance (see Wikipedia
     (https://en.wikipedia.org/wiki/Taxicab_geometry)) between the two
     coordinates ‘a’ and ‘b’ (each an array of ‘ndim’ elements).

 -- Function:
          float
          gal_dimension_dist_radial (size_t *a, size_t *b, size_t ndim)
     Return the radial distance between the two coordinates ‘a’ and ‘b’
     (each an array of ‘ndim’ elements).

 -- Function:
          float
          gal_dimension_dist_elliptical (double *center, double *pa_deg,
          double *q, size_t ndim, double *point)
     Return the elliptical/ellipsoidal distance of the single point
     ‘point’ (containing ‘ndim’ values: coordinates of the point in each
     dimension) from an ellipse that is defined by ‘center’, ‘pa_deg’
     and ‘q’.  ‘center’ is the coordinates of the ellipse center (also
     with ‘ndim’ elements).  ‘pa’ is the position-angle in degrees (the
     angle of the semi-major axis from the first dimension in a 2D
     ellipse) and ‘q’ is the axis ratio.

     In a 2D ellipse, ‘pa’ and ‘q’ are a single-element array.  However,
     in a 3D ellipsoid, ‘pa’ must have three elements, and ‘q’ must have
     2 elements.  For more see *note Defining an ellipse and
     ellipsoid::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sum (gal_data_t *in, size_t c_dim,
          gal_data_t *weight)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by summing all elements in that direction.  If
     ‘weight!=NULL’, it must be a single-dimensional array, with the
     same size as the dimension to be collapsed.  The respective weight
     will be multiplied to each element during the collapse.

     For generality, the returned dataset will have a ‘GAL_TYPE_FLOAT64’
     type.  See *note Copying datasets:: for converting the returned
     dataset to a desired type.  Also, for more on the application of
     this function, see the Arithmetic program’s ‘collapse-sum’ operator
     (which uses this function) in *note Arithmetic operators::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mean (gal_data_t *in, size_t c_dim,
          gal_data_t *weight)
     Similar to ‘gal_dimension_collapse_sum’ (above), but the collapse
     will be done by calculating the mean along the requested dimension,
     not summing over it.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_number (gal_data_t *in, size_t c_dim)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by counting how many non-blank elements there are along
     that dimension.

     For generality, the returned dataset will have a ‘GAL_TYPE_INT32’
     type.  See *note Copying datasets:: for converting the returned
     dataset to a desired type.  Also, for more on the application of
     this function, see the Arithmetic program’s ‘collapse-number’
     operator (which uses this function) in *note Arithmetic
     operators::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_minmax (gal_data_t *in, size_t c_dim,
          int max1_min0)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by using the largest/smallest non-blank value along
     that dimension.  If ‘max1_min0’ is non-zero, then the collapsed
     dataset will have the maximum value along the given dimension and
     if it is zero, the minimum.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_median (gal_data_t *in, size_t c_dim,
          size_t numthreads, size_t minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the median non-blank value along that
     dimension.  Since the median involves sorting, this operator
     benefits from many threads (which needs to be set with
     ‘numthreads’).  For more on ‘minmapsize’ and ‘quietmmap’ see *note
     Memory management::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_std (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the standard deviation of pixels along that
     dimension after sigma-clipping.  Since sigma-clipping involves
     sorting, this operator benefits from many threads (which needs to
     be set with ‘numthreads’).  For more on ‘minmapsize’ and
     ‘quietmmap’ see *note Memory management::.  For more on sigma
     clipping, see *note Sigma clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_mean (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the mean of pixels along that dimension
     after sigma-clipping.  Since sigma-clipping involves sorting, this
     operator benefits from many threads (which needs to be set with
     ‘numthreads’).  For more on ‘minmapsize’ and ‘quietmmap’ see *note
     Memory management::.  For more on sigma clipping, see *note Sigma
     clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_median (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the median of pixels along that dimension
     after sigma-clipping.  Since sigma-clipping involves sorting, this
     operator benefits from many threads (which needs to be set with
     ‘numthreads’).  For more on ‘minmapsize’ and ‘quietmmap’ see *note
     Memory management::.  For more on sigma clipping, see *note Sigma
     clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_number (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the number of pixels along that dimension
     after sigma-clipping.  Since sigma-clipping involves sorting, this
     operator benefits from many threads (which needs to be set with
     ‘numthreads’).  For more on ‘minmapsize’ and ‘quietmmap’ see *note
     Memory management::.  For more on sigma clipping, see *note Sigma
     clipping::.

 -- Function:
          size_t
          gal_dimension_remove_extra (size_t ndim, size_t *dsize, struct
          wcsprm *wcs)
     Remove extra dimensions (those that only have a length of 1) from
     the basic size information of a dataset.  ‘ndim’ is the number of
     dimensions and ‘dsize’ is an array with ‘ndim’ elements containing
     the size along each dimension in the C dimension order.  When
     ‘wcs!=NULL’, the respective dimension will also be removed from the
     WCS.

     This function will return the new number of dimensions and the
     ‘dsize’ elements will contain the length along each new dimension.

 -- Function-like macro: GAL_DIMENSION_NEIGHBOR_OP (index, ndim, dsize,
          connectivity, dinc, operation)
     Parse the neighbors of the element located at ‘index’ and do the
     requested operation on them.  This is defined as a macro to allow
     easy definition of any operation on the neighbors of a given
     element without having to use loops within your source code (the
     loops are implemented by this macro).  For an example of using this
     function, please see *note Library demo - inspecting neighbors::.
     The input arguments to this function-like macro are described
     below:

     ‘index’
          Distance of this element from the first element in the array
          on a contiguous patch of memory (starting from 0), see the
          discussion above.
     ‘ndim’
          The number of dimensions associated with the contiguous patch
          of memory.
     ‘dsize’
          The full array size along each dimension.  This must be an
          array and is assumed to have the same number elements as
          ‘ndim’.  See the discussion under the same element in *note
          Generic data container::.
     ‘connectivity’
          Most distant neighbors to consider.  Depending on the number
          of dimensions, different neighbors may be defined for each
          element.  This function-like macro distinguish between these
          different neighbors with this argument.  It has a value
          between ‘1’ (one) and ‘ndim’.  For example, in a 2D dataset,
          4-connected neighbors have a connectivity of ‘1’ and
          8-connected neighbors have a connectivity of ‘2’.  Note that
          this is inclusive, so in this example, a connectivity of ‘2’
          will also include connectivity ‘1’ neighbors.
     ‘dinc’
          An array keeping the length necessary to increment along each
          dimension.  You can make this array with the following
          function.  Just do not forget to free the array after you are
          done with it:

               size_t *dinc=gal_dimension_increment(ndim, dsize);
               free(dinc);

          ‘dinc’ depends on ‘ndim’ and ‘dsize’, but it must be defined
          outside this function-like macro since it involves allocation
          to help in performance.

     ‘operation’
          Any C operation that you would like to do on the neighbor.
          This macro will provide you a ‘nind’ variable that can be used
          as the index of the neighbor that is currently being studied.
          It is defined as ‘‘size_t ndim;’’.  Note that ‘operation’ will
          be repeated the number of times there is a neighbor for this
          element.

     This macro works fully within its own ‘{}’ block and except for the
     ‘nind’ variable that shows the neighbor’s index, all the variables
     within this macro’s block start with ‘gdn_’.


File: gnuastro.info,  Node: Linked lists,  Next: Array input output,  Prev: Dimensions,  Up: Gnuastro library

12.3.8 Linked lists (‘list.h’)
------------------------------

An array is a contiguous region of memory that is very efficient and
easy to use for recording and later accessing any random element as fast
as any other.  This makes array the primary data container when you have
many elements (for example, an image which has millions of pixels).  One
major problem with an array is that the number of elements that go into
it must be known in advance and adding or removing an element will
require a re-set of all the other elements.  For example, if you want to
remove the 3rd element in a 1000 element array, all 997 subsequent
elements have to pulled back by one position, the reverse will happen if
you need to add an element.

   In many contexts such situations never come up, for example, you do
not want to shift all the pixels in an image by one or two pixels from
some random position in the image: their positions have scientific
value.  But in other contexts you will find yourself frequently
adding/removing an a-priori unknown number of elements.  Linked lists
(or _lists_ for short) are the data-container of choice in such
situations.  As in a chain, each _node_ in a list is an independent C
structure, keeping its own data along with pointer(s) to its immediate
neighbor(s).  Below, you can see one simple linked list node structure
along with an ASCII art schematic of how we can use the ‘next’ pointer
to add any number of elements to the list that we want.  By convention,
a list is terminated when ‘next’ is the ‘NULL’ pointer.

     struct list_float          /*     ---------    ---------           */
     {                          /*     | Value |    | Value |           */
       float             value; /*     |  ---  |    |  ---  |           */
       struct list_float *next; /*     |  next-|--> |  next-|--> NULL   */
     }                          /*     ---------    ---------           */

   The schematic shows another great advantage of linked lists: it is
very easy to add or remove/pop a node anywhere in the list.  If you want
to modify the first node, you just have to change one pointer.  If it is
in the middle, you just have to change two.  You initially define a
variable of this type with a ‘NULL’ pointer as shown below:

     struct list_float *mylist=NULL;

To add or remove/pop a node from the list you can use functions provided
for the respective type in the sections below.

When you add an element to the list, it is conventionally added to the
“top” of the list: the general list pointer will point to the newly
created node, which will point to the previously created node and so on.
So when you “pop” from the top of the list, you are actually retrieving
the last value you put in and changing the list pointer to the next
youngest node.  This is thus known as a “last-in-first-out” list.  This
is the most efficient type of linked list (easier to implement and
faster to process).  Alternatively, you can add each newly created node
at the end of the list.  If you do that, you will get a
“first-in-first-out” list.  But that will force you to go through the
whole list for each new element that is created (this will slow down the
processing)(1).

   The node example above creates the simplest kind of a list.  We can
define each node with two pointers to both the next and previous
neighbors, this is called a “Doubly linked list”.  In general, lists are
very powerful and simple constructs that can be very useful.  But going
into more detail would be out of the scope of this short introduction in
this book.  Wikipedia (https://en.wikipedia.org/wiki/Linked_list) has a
nice and more thorough discussion of the various types of lists.  To
appreciate/use the beauty and elegance of these powerful constructs even
further, see Chapter 2 (Information Structures, in volume 1) of Donald
Knuth’s “The art of computer programming”.

   In this section we will review the functions and structures that are
available in Gnuastro for working on lists.  They differ by the type of
data that each node can keep.  For each linked-list node structure, we
will first introduce the structure, then the functions for working on
the structure.  All these structures and functions are defined and
declared in ‘gnuastro/list.h’.

* Menu:

* List of strings::             Simply linked list of strings.
* List of int32_t::             Simply linked list of int32_ts.
* List of size_t::              Simply linked list of size_ts.
* List of float::               Simply linked list of floats.
* List of double::              Simply linked list of doubles
* List of void::                Simply linked list of void * pointers.
* Ordered list of size_t::      Simply linked, ordered list of size_t.
* Doubly linked ordered list of size_t::  Definition and functions.
* List of gal_data_t::          Simply linked list Gnuastro’s generic datatype.

   ---------- Footnotes ----------

   (1) A better way to get a first-in-first-out is to first keep the
data as last-in-first-out until they are all read.  Afterwards, reverse
the list by popping each node and immediately add it to the new list.
This practically reverses the last-in-first-out list to a
first-in-first-out one.  All the list types discussed in this chapter
have a function with a ‘_reverse’ suffix for this job.


File: gnuastro.info,  Node: List of strings,  Next: List of int32_t,  Prev: Linked lists,  Up: Linked lists

12.3.8.1 List of strings
........................

Probably one of the most common lists you will be using are lists of
strings.  They are the best tools when you are reading the user’s
inputs, or when adding comments to the output files.  Below you can see
Gnuastro’s string list type and several functions to help in adding,
removing/popping, reversing and freeing the list.

 -- Type (C struct): gal_list_str_t
     A single node in a list containing a string of characters.
          typedef struct gal_list_str_t
          {
            char *v;
            struct gal_list_str_t *next;
          } gal_list_str_t;

 -- Function:
          void
          gal_list_str_add (gal_list_str_t **list, char *value, int
          allocate)
     Add a new node to the list of strings (‘list’) and update it.  The
     new node will contain the string ‘value’.  If ‘allocate’ is not
     zero, space will be allocated specifically for the string of the
     new node and the contents of ‘value’ will be copied into it.  This
     can be useful when your string may be changed later in the program,
     but you want your list to remain.  Here is one short/simple example
     of initializing and adding elements to a string list:

          gal_list_str_t *strlist=NULL;
          gal_list_str_add(&strlist, "bottom of list.", 1);
          gal_list_str_add(&strlist, "second last element of list.", 1);

 -- Function:
          char *
          gal_list_str_pop (gal_list_str_t **list)
     Pop the top element of ‘list’, change ‘list’ to point to the next
     node in the list, and return the string that was in the popped
     node.  If ‘*list==NULL’, then this function will also return a
     ‘NULL’ pointer.

 -- Function:
          size_t
          gal_list_str_number (gal_list_str_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          gal_list_str_t *
          gal_list_str_last (gal_list_str_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_str_print (gal_list_str_t *list)
     Print the strings within each node of ‘*list’ on the standard
     output in the same order that they are stored.  Each string is
     printed on one line.  This function is mainly good for
     checking/debugging your program.  For program outputs, it is best
     to make your own implementation with a better, more user-friendly,
     format.  For example, the following code snippet.

          size_t i;
          gal_list_str_t *tmp;
          for(tmp=list; tmp!=NULL; tmp=tmp->next)
            printf("String %zu: %s\n", i, tmp->v);

 -- Function:
          void
          gal_list_str_reverse (gal_list_str_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          void
          gal_list_str_free (gal_list_str_t *list, int freevalue)
     Free every node in ‘list’.  If ‘freevalue’ is not zero, also free
     the string within the nodes.

 -- Function:
          gal_list_str_t *
          gal_list_str_extract (char *string)
     Extract space-separated components of the input string.  If any
     space element should be kept (and not considered as a delimiter
     between two tokens), precede it with a backslash (‘\’).

 -- Function:
          char *
          gal_list_str_cat (gal_list_str_t *list, char delimiter)
     Concatenate (append) the input list of strings into a single string
     where each node is separated from the next with the given
     ‘delimiter’.  The space for the output string is allocated by this
     function and should be freed when you have finished with it.

     If there is any delimiter characters are present in any of the
     elements, a backslash (‘\’) will be printed before the SPACE
     character.  This is necessary, otherwise, a function like
     ‘gal_list_str_extract’ will not be able to extract the elements
     back into separate elements in a list.


File: gnuastro.info,  Node: List of int32_t,  Next: List of size_t,  Prev: List of strings,  Up: Linked lists

12.3.8.2 List of ‘int32_t’
..........................

Signed integers are the best types when you are dealing with a positive
or negative integers.  The are generally useful in many contexts, for
example when you want to keep the order of a series of states (each
state stored as a given number in an ‘enum’ for example).  On many
modern systems, ‘int32_t’ is just an alias for ‘int’, so you can use
them interchangeably.  To make sure, check the size of ‘int’ on your
system:

 -- Type (C struct): gal_list_i32_t
     A single node in a list containing a 32-bit signed integer (see
     *note Numeric data types::).
          typedef struct gal_list_i32_t
          {
            int32_t v;
            struct gal_list_i32_t *next;
          } gal_list_i32_t;

 -- Function:
          void
          gal_list_i32_add (gal_list_i32_t **list, int32_t value)
     Add a new node (containing ‘value’) to the top of the ‘list’ of
     ‘int32_t’s (‘uint32_t’ is equal to ‘int’ on many modern systems),
     and update ‘list’.  Here is one short example of initializing and
     adding elements to a string list:

          gal_list_i32_t *i32list=NULL;
          gal_list_i32_add(&i32list, 52);
          gal_list_i32_add(&i32list, -4);

 -- Function:
          int32_t
          gal_list_i32_pop (gal_list_i32_t **list)
     Pop the top element of ‘list’ and return the value.  This function
     will also change ‘list’ to point to the next node in the list.  If
     ‘*list==NULL’, then this function will also return
     ‘GAL_BLANK_INT32’ (see *note Library blank values::).

 -- Function:
          size_t
          gal_list_i32_number (gal_list_i32_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          size_t
          gal_list_i32_last (gal_list_i32_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_i32_print (gal_list_i32_t *list)
     Print the integers within each node of ‘*list’ on the standard
     output in the same order that they are stored.  Each integer is
     printed on one line.  This function is mainly good for
     checking/debugging your program.  For program outputs, it is best
     to make your own implementation with a better, more user-friendly
     format.  For example, the following code snippet.  You can also
     modify it to print all values in one line, etc., depending on the
     context of your program.

          size_t i;
          gal_list_i32_t *tmp;
          for(tmp=list; tmp!=NULL; tmp=tmp->next)
            printf("String %zu: %s\n", i, tmp->v);

 -- Function:
          void
          gal_list_i32_reverse (gal_list_i32_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          int32_t *
          gal_list_i32_to_array (gal_list_i32_t *list, int reverse,
          size_t *num)
     Dynamically allocate an array and fill it with the values in
     ‘list’.  The function will return a pointer to the allocated array
     and put the number of elements in the array into the ‘num’ pointer.
     If ‘reverse’ has a non-zero value, the array will be filled in the
     opposite order of elements in ‘list’.  This function can be useful
     after you have finished reading an initially unknown number of
     values and want to put them in an array for easy random access.

 -- Function:
          void
          gal_list_i32_free (gal_list_i32_t *list)
     Free every node in ‘list’.


File: gnuastro.info,  Node: List of size_t,  Next: List of float,  Prev: List of int32_t,  Up: Linked lists

12.3.8.3 List of ‘size_t’
.........................

The ‘size_t’ type is a unique type in C: as the name suggests it is
defined to store sizes, or more accurately, the distances between memory
locations.  Hence it is always positive (an ‘unsigned’ type) and it is
directly related to the address-able spaces on the host system: on
32-bit and 64-bit systems it is an alias for ‘uint32_t’ and ‘uint64_t’,
respectively (see *note Numeric data types::).

   ‘size_t’ is the default compiler type to index an array (recall that
an array index in C is just a pointer increment of a given _size_).
Since it is unsigned, it is a great type for counting (where negative is
not defined), you are always sure it will never exceed the system’s
(virtual) memory and since its name has the word “size” inside it, it
provides a good level of documentation(1).  In Gnuastro, we do all
counting and array indexing with this type, so this list is very handy.
As discussed above, ‘size_t’ maps to different types on different
machines, so a portable way to print them with ‘printf’ is to use C99’s
‘%zu’ format.

 -- Type (C struct): gal_list_sizet_t
     A single node in a list containing a ‘size_t’ value (which maps to
     ‘uint32_t’ or ‘uint64_t’ on 32-bit and 64-bit systems), see *note
     Numeric data types::.
          typedef struct gal_list_sizet_t
          {
            size_t v;
            struct gal_list_sizet_t *next;
          } gal_list_sizet_t;

 -- Function:
          void
          gal_list_sizet_add (gal_list_sizet_t **list, size_t value)
     Add a new node (containing ‘value’) to the top of the ‘list’ of
     ‘size_t’s and update ‘list’.  Here is one short example of
     initializing and adding elements to a string list:

          gal_list_sizet_t *slist=NULL;
          gal_list_sizet_add(&slist, 45493);
          gal_list_sizet_add(&slist, 930484);

 -- Function:
          sizet_t
          gal_list_sizet_pop (gal_list_sizet_t **list)
     Pop the top element of ‘list’ and return the value.  This function
     will also change ‘list’ to point to the next node in the list.  If
     ‘*list==NULL’, then this function will also return
     ‘GAL_BLANK_SIZE_T’ (see *note Library blank values::).

 -- Function:
          size_t
          gal_list_sizet_number (gal_list_sizet_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          size_t
          gal_list_sizet_last (gal_list_sizet_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_sizet_print (gal_list_sizet_t *list)
     Print the values within each node of ‘*list’ on the standard output
     in the same order that they are stored.  Each integer is printed on
     one line.  This function is mainly good for checking/debugging your
     program.  For program outputs, it is best to make your own
     implementation with a better, more user-friendly format.  For
     example, the following code snippet.  You can also modify it to
     print all values in one line, etc., depending on the context of
     your program.

          size_t i;
          gal_list_sizet_t *tmp;
          for(tmp=list; tmp!=NULL; tmp=tmp->next)
            printf("String %zu: %zu\n", i, tmp->v);

 -- Function:
          void
          gal_list_sizet_reverse (gal_list_sizet_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          size_t *
          gal_list_sizet_to_array (gal_list_sizet_t *list, int reverse,
          size_t *num)
     Dynamically allocate an array and fill it with the values in
     ‘list’.  The function will return a pointer to the allocated array
     and put the number of elements in the array into the ‘num’ pointer.
     If ‘reverse’ has a non-zero value, the array will be filled in the
     inverse of the order of elements in ‘list’.  This function can be
     useful after you have finished reading an initially unknown number
     of values and want to put them in an array for easy random access.

 -- Function:
          void
          gal_list_sizet_free (gal_list_sizet_t *list)
     Free every node in ‘list’.

   ---------- Footnotes ----------

   (1) So you know that a variable of this type is not used to store
some generic state for example.


File: gnuastro.info,  Node: List of float,  Next: List of double,  Prev: List of size_t,  Up: Linked lists

12.3.8.4 List of ‘float’
........................

Single precision floating point numbers can accurately store real number
until 7.2 decimals and only consume 4 bytes (32-bits) of memory, see
*note Numeric data types::.  Since astronomical data rarely reach that
level of precision, single precision floating points are the type of
choice to keep and read data.  However, when processing the data, it is
best to use double precision floating points (since errors propagate).

 -- Type (C struct): gal_list_f32_t
     A single node in a list containing a 32-bit single precision
     ‘float’ value: see *note Numeric data types::.
          typedef struct gal_list_f32_t
          {
            float v;
            struct gal_list_f32_t *next;
          } gal_list_f32_t;

 -- Function:
          void
          gal_list_f32_add (gal_list_f32_t **list, float value)
     Add a new node (containing ‘value’) to the top of the ‘list’ of
     ‘float’s and update ‘list’.  Here is one short example of
     initializing and adding elements to a string list:

          gal_list_f32_t *flist=NULL;
          gal_list_f32_add(&flist, 3.89);
          gal_list_f32_add(&flist, 1.23e-20);

 -- Function:
          float
          gal_list_f32_pop (gal_list_f32_t **list)
     Pop the top element of ‘list’ and return the value.  This function
     will also change ‘list’ to point to the next node in the list.  If
     ‘*list==NULL’, then this function will return ‘GAL_BLANK_FLOAT32’
     (NaN, see *note Library blank values::).

 -- Function:
          size_t
          gal_list_f32_number (gal_list_f32_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          size_t
          gal_list_f32_last (gal_list_f32_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_f32_print (gal_list_f32_t *list)
     Print the values within each node of ‘*list’ on the standard output
     in the same order that they are stored.  Each floating point number
     is printed on one line.  This function is mainly good for
     checking/debugging your program.  For program outputs, it is best
     to make your own implementation with a better, more user-friendly
     format.  For example, in the following code snippet.  You can also
     modify it to print all values in one line, etc., depending on the
     context of your program.

          size_t i;
          gal_list_f32_t *tmp;
          for(tmp=list; tmp!=NULL; tmp=tmp->next)
            printf("Node %zu: %f\n", i, tmp->v);

 -- Function:
          void
          gal_list_f32_reverse (gal_list_f32_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          float *
          gal_list_f32_to_array (gal_list_f32_t *list, int reverse,
          size_t *num)
     Dynamically allocate an array and fill it with the values in
     ‘list’.  The function will return a pointer to the allocated array
     and put the number of elements in the array into the ‘num’ pointer.
     If ‘reverse’ has a non-zero value, the array will be filled in the
     inverse of the order of elements in ‘list’.  This function can be
     useful after you have finished reading an initially unknown number
     of values and want to put them in an array for easy random access.

 -- Function:
          void
          gal_list_f32_free (gal_list_f32_t *list)
     Free every node in ‘list’.


File: gnuastro.info,  Node: List of double,  Next: List of void,  Prev: List of float,  Up: Linked lists

12.3.8.5 List of ‘double’
.........................

Double precision floating point numbers can accurately store real number
until 15.9 decimals and consume 8 bytes (64-bits) of memory, see *note
Numeric data types::.  This level of precision makes them very good for
serious processing in the middle of a program’s execution: in many
cases, the propagation of errors will still be insignificant compared to
actual observational errors in a data set.  But since they consume 8
bytes and more CPU processing power, they are often not the best choice
for storing and transferring of data.

 -- Type (C struct): gal_list_f64_t
     A single node in a list containing a 64-bit double precision
     ‘double’ value: see *note Numeric data types::.
          typedef struct gal_list_f64_t
          {
            double v;
            struct gal_list_f64_t *next;
          } gal_list_f64_t;

 -- Function:
          void
          gal_list_f64_add (gal_list_f64_t **list, double value)
     Add a new node (containing ‘value’) to the top of the ‘list’ of
     ‘double’s and update ‘list’.  Here is one short example of
     initializing and adding elements to a string list:

          gal_list_f64_t *dlist=NULL;
          gal_list_f64_add(&dlist, 3.8129395763193);
          gal_list_f64_add(&dlist, 1.239378923931e-20);

 -- Function:
          double
          gal_list_f64_pop (gal_list_f64_t **list)
     Pop the top element of ‘list’ and return the value.  This function
     will also change ‘list’ to point to the next node in the list.  If
     ‘*list==NULL’, then this function will return ‘GAL_BLANK_FLOAT64’
     (NaN, see *note Library blank values::).

 -- Function:
          size_t
          gal_list_f64_number (gal_list_f64_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          size_t
          gal_list_f64_last (gal_list_f64_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_f64_print (gal_list_f64_t *list)
     Print the values within each node of ‘*list’ on the standard output
     in the same order that they are stored.  Each floating point number
     is printed on one line.  This function is mainly good for
     checking/debugging your program.  For program outputs, it is best
     to make your own implementation with a better, more user-friendly
     format.  For example, in the following code snippet.  You can also
     modify it to print all values in one line, etc., depending on the
     context of your program.

          size_t i;
          gal_list_f64_t *tmp;
          for(tmp=list; tmp!=NULL; tmp=tmp->next)
            printf("Node %zu: %f\n", i, tmp->v);

 -- Function:
          void
          gal_list_f64_reverse (gal_list_f64_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          double *
          gal_list_f64_to_array (gal_list_f64_t *list, int reverse,
          size_t *num)
     Dynamically allocate an array and fill it with the values in
     ‘list’.  The function will return a pointer to the allocated array
     and put the number of elements in the array into the ‘num’ pointer.
     If ‘reverse’ has a non-zero value, the array will be filled in the
     inverse of the order of elements in ‘list’.  This function can be
     useful after you have finished reading an initially unknown number
     of values and want to put them in an array for easy random access.

 -- Function:
          gal_data_t *
          gal_list_f64_to_data (gal_list_f64_t *list, uint8_t type,
          size_t minmapsize, int quietmmap)
     Write the values in the given ‘list’ into a ‘gal_data_t’ dataset of
     the requested ‘type’.  The order of the values in the dataset will
     be the same as the order from the top of the list.

 -- Function:
          void
          gal_list_f64_free (gal_list_f64_t *list)
     Free every node in ‘list’.


File: gnuastro.info,  Node: List of void,  Next: Ordered list of size_t,  Prev: List of double,  Up: Linked lists

12.3.8.6 List of ‘void *’
.........................

In C, ‘void *’ is the most generic pointer.  Usually pointers are
associated with the type of content they point to.  For example, ‘int *’
means a pointer to an integer.  This ancillary information about the
contents of the memory location is very useful for the compiler,
catching bad errors and also documentation (it helps the reader see what
the address in memory actually contains).  However, ‘void *’ is just a
raw address (pointer), it contains no information on the contents it
points to.

   These properties make the ‘void *’ very useful when you want to treat
the contents of an address in different ways.  You can use the ‘void *’
list defined in this section and its function on any kind of data: for
example you can use it to keep a list of custom data structures that you
have built for your own separate program.  Each node in the list can
keep anything and this gives you great versatility.  But in using ‘void
*’, please beware that “with great power comes great responsibility”.

 -- Type (C struct): gal_list_void_t
     A single node in a list containing a ‘void *’ pointer.
          typedef struct gal_list_void_t
          {
            void *v;
            struct gal_list_void_t *next;
          } gal_list_void_t;

 -- Function:
          void
          gal_list_void_add (gal_list_void_t **list, void *value)
     Add a new node (containing ‘value’) to the top of the ‘list’ of
     ‘void *’s and update ‘list’.  Here is one short example of
     initializing and adding elements to a string list:

          gal_list_void_t *vlist=NULL;
          gal_list_f64_add(&vlist, some_pointer);
          gal_list_f64_add(&vlist, another_pointer);

 -- Function:
          void *
          gal_list_void_pop (gal_list_void_t **list)
     Pop the top element of ‘list’ and return the value.  This function
     will also change ‘list’ to point to the next node in the list.  If
     ‘*list==NULL’, then this function will return ‘NULL’.

 -- Function:
          size_t
          gal_list_void_number (gal_list_void_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          size_t
          gal_list_void_last (gal_list_void_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_void_reverse (gal_list_void_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          void
          gal_list_void_free (gal_list_void_t *list)
     Free every node in ‘list’.


File: gnuastro.info,  Node: Ordered list of size_t,  Next: Doubly linked ordered list of size_t,  Prev: List of void,  Up: Linked lists

12.3.8.7 Ordered list of ‘size_t’
.................................

Positions/sizes in a dataset are conventionally in the ‘size_t’ type
(see *note List of size_t::) and it sometimes occurs that you want to
parse and read the values in a specific order.  For example, you want to
start from one pixel and add pixels to the list based on their distance
to that pixel.  So that ever time you pop an element from the list, you
know it is the nearest that has not yet been studied.  The
‘gal_list_osizet_t’ type and its functions in this section are designed
to facilitate such operations.

 -- Type (C struct): gal_list_osizet_t
     Each node in this singly-linked list contains a ‘size_t’ value and
     a floating point value.  The floating point value is used as a
     reference to add new nodes in a sorted manner.  At any moment, the
     first popped node in this list will have the smallest ‘tosort’
     value, and subsequent nodes will have larger to values.
     typedef struct gal_list_osizet_t
     {
       size_t v;                       /* The actual value. */
       float  s;                       /* The parameter to sort by. */
       struct gal_list_osizet_t *next;
     } gal_list_osizet_t;

 -- Function:
          void
          gal_list_osizet_add (gal_list_osizet_t **list, size_t value,
          float tosort)
     Allocate space for a new node in ‘list’, and store ‘value’ and
     ‘tosort’ into it.  The new node will not necessarily be at the
     “top” of the list.  If ‘*list!=NULL’, then the ‘tosort’ values of
     existing nodes is inspected and the given node is placed in the
     list such that the top element (which is popped with
     ‘gal_list_osizet_pop’) has the smallest ‘tosort’ value.

 -- Function:
          size_t
          gal_list_osizet_pop (gal_list_osizet_t **list, float
          *sortvalue)
     Pop a node from the top of ‘list’, return the node’s ‘value’ and
     put its sort value in the space that ‘sortvalue’ points to.  This
     function will also free the allocated space for the popped node and
     after this function, ‘list’ will point to the next node (which has
     a larger ‘tosort’ element).

 -- Function:
          void
          gal_list_osizet_to_sizet_free (gal_list_osizet_t *in,
          gal_list_sizet_t **out)
     Convert the ordered list of ‘size_t’s into an ordinary ‘size_t’
     linked list.  This can be useful when all the elements have been
     added and you just need to pop-out elements and do not care about
     the sorting values any more.  After the conversion is done, this
     function will free the input list.  Note that the ‘out’ list does
     not have to be empty.  If it already contains some nodes, the new
     nodes will be added on top of them.


File: gnuastro.info,  Node: Doubly linked ordered list of size_t,  Next: List of gal_data_t,  Prev: Ordered list of size_t,  Up: Linked lists

12.3.8.8 Doubly linked ordered list of ‘size_t’
...............................................

An ordered list of indices is required in many contexts, one example was
discussed at the beginning of *note Ordered list of size_t::.  But the
list that was introduced there only has one point of entry: you can
always only parse the list from smallest to largest.  In this section,
the doubly-linked ‘gal_list_dosizet_t’ node is defined which will allow
us to parse the values in ascending or descending order.

 -- Type (C struct): gal_list_dosizet_t

     Doubly-linked, ordered ‘size_t’ list node structure.  Each node in
     this Doubly-linked list contains a ‘size_t’ value and a floating
     point value.  The floating point value is used as a reference to
     add new nodes in a sorted manner.  In the functions here, this
     linked list can be pointed to by two pointers (largest and
     smallest) with the following format:
                      largest pointer
                      |
             NULL <-- (v0,s0) <--> (v1,s1) <--> ... (vn,sn) --> NULL
                                                    |
                                     smallest pointer
     At any moment, the two pointers will point to the nodes containing
     the “largest” and “smallest” values and the rest of the nodes will
     be sorted.  This is useful when an unknown number of nodes are
     being added continuously and during the operations it is important
     to have the nodes in a sorted format.

          typedef struct gal_list_dosizet_t
          {
            size_t v;                       /* The actual value. */
            float s;                        /* The parameter to sort by. */
            struct gal_list_dosizet_t *prev;
            struct gal_list_dosizet_t *next;
          } gal_list_dosizet_t;

 -- Function:
          void
          gal_list_dosizet_add (gal_list_dosizet_t **largest,
          gal_list_dosizet_t **smallest, size_t value, float tosort)
     Allocate space for a new node in ‘list’, and store ‘value’ and
     ‘tosort’ into it.  If the list is empty, both ‘largest’ and
     ‘smallest’ must be ‘NULL’.

 -- Function:
          size_t
          gal_list_dosizet_pop_smallest (gal_list_dosizet_t **largest,
          gal_list_dosizet_t **smallest, float tosort)
     Pop the value with the smallest reference from the doubly linked
     list and store the reference into the space pointed to by ‘tosort’.
     Note that even though only the smallest pointer will be popped,
     when there was only one node in the list, the ‘largest’ pointer
     also has to change, so we need both.

 -- Function:
          void
          gal_list_dosizet_print (gal_list_dosizet_t *largest,
          gal_list_dosizet_t *smallest)
     Print the largest and smallest values sequentially until the list
     is parsed.

 -- Function:
          void
          gal_list_dosizet_to_sizet (gal_list_dosizet_t *in,
          gal_list_sizet_t **out)
     Convert the doubly linked, ordered ‘size_t’ list into a
     singly-linked list of ‘size_t’.

 -- Function:
          void
          gal_list_dosizet_free (gal_list_dosizet_t *largest)
     Free the doubly linked, ordered ‘sizet_t’ list.


File: gnuastro.info,  Node: List of gal_data_t,  Prev: Doubly linked ordered list of size_t,  Up: Linked lists

12.3.8.9 List of ‘gal_data_t’
.............................

Gnuastro’s generic data container has a ‘next’ element which enables it
to be used as a singly-linked list (see *note Generic data container::).
The ability to connect the different data containers offers great
advantages.  For example, each column in a table in an independent
dataset: with its own name, units, numeric data type (see *note Numeric
data types::).  Another application is in Tessellating an input dataset
into separate tiles or only studying particular regions, or tiles, of a
larger dataset (see *note Tessellation:: and *note Tessellation
library::).  Each independent tile over the dataset can be connected to
the others as a linked list and thus any number of tiles can be
represented with one variable.

 -- Function:
          void
          gal_list_data_add (gal_data_t **list, gal_data_t *newnode)
     Add an already allocated dataset (‘newnode’) to top of ‘list’.
     Note that if ‘newnode->next!=NULL’ (‘newnode’ is itself a list),
     then ‘list’ will be added to its end.

     In this example multiple images are linked together as a list:
          int quietmmap=1;
          size_t minmapsize=-1;
          gal_data_t *tmp, *list=NULL;
          tmp = gal_fits_img_read("file1.fits", "1", minmapsize, quietmmap);
          gal_list_data_add( &list, tmp );
          tmp = gal_fits_img_read("file2.fits", "1", minmapsize, quietmmap);
          gal_list_data_add( &list, tmp );

 -- Function:
          void
          gal_list_data_add_alloc (gal_data_t **list, void *array,
          uint8_t type, size_t ndim, size_t *dsize, struct wcsprm *wcs,
          int clear, size_t minmapsize, int quietmmap, char *name, char
          *unit, char *comment)
     Allocate a new dataset (with ‘gal_data_alloc’ in *note Dataset
     allocation::) and put it as the first element of ‘list’.  Note that
     if this is the first node to be added to the list, ‘list’ must be
     ‘NULL’.

 -- Function:
          gal_data_t *
          gal_list_data_pop (gal_data_t **list)
     Pop the top node from ‘list’ and return it.

 -- Function:
          void
          gal_list_data_remove (gal_data_t **list, gal_data_t *node)
     Remove ‘node’ from the given ‘list’.  After finding the given node,
     this function will just set ‘node->next=NULL’ and correct the
     ‘next’ node of its previous element to its next element (thus
     “removing” it from the list).  If ‘node’ doesn’t exist in the list,
     this function won’t make any change to list.

 -- Function:
          gal_data_t *
          gal_list_data_select_by_name (gal_data_t *list, char *name)
     Select the dataset within the list, that has a ‘name’ element that
     is identical (case-sensitive) to the given ‘name’.  If not found, a
     ‘NULL’ pointer will be returned.

     Note that this dataset will not be popped from the list, only a
     pointer to it will be returned and if you free it or change its
     ‘next’ element, it may harm your original list.

 -- Function:
          gal_data_t *
          gal_list_data_select_by_id (gal_data_t *table, char *idstr,
          size_t *index)
     Select the dataset within the list that can be identified with the
     string given to ‘idstr’ (which can be a counter, starting from 1,
     or a name).  If not found, a ‘NULL’ pointer will be returned.

     Note that this dataset will not be popped from the list, only a
     pointer to it will be returned and if you free it or change its
     ‘next’ element, it may harm your original list.

 -- Function:
          void
          gal_list_data_reverse (gal_data_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          gal_data_t **
          gal_list_data_to_array_ptr (gal_data_t *list, size_t *num)
     Allocate and return an array of ‘gal_data_t *’ pointers with the
     same number of elements as the nodes in ‘list’.  The pointers will
     be put in the same order that the list is parsed.  Hence the N-th
     element in the array will point to the same dataset that the N-th
     node in the list points to.

 -- Function:
          size_t
          gal_list_data_number (gal_data_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          gal_data_t *
          gal_list_data_last (gal_data_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_data_free (gal_data_t *list)
     Free all the datasets in ‘list’ along with all the allocated spaces
     in each.


File: gnuastro.info,  Node: Array input output,  Next: Table input output,  Prev: Linked lists,  Up: Gnuastro library

12.3.9 Array input output
-------------------------

Getting arrays (commonly images or cubes) from a file into your program
or writing them after the processing into an output file are some of the
most common operations.  The functions in this section are designed for
such operations with the known file types.  The functions here are thus
just wrappers around functions of lower-level file type functions of
this library, for example, *note FITS files:: or *note TIFF files::.  If
the file type of the input/output file is already known, you can use the
functions in those sections respectively.

 -- Function:
          int
          gal_array_name_recognized (char *filename)
     Return 1 if the given file name corresponds to one of the
     recognized file types for reading arrays.

 -- Function:
          int
          gal_array_name_recognized_multiext (char *filename)
     Return 1 if the given file name corresponds to one of the
     recognized file types for reading arrays which may contain multiple
     extensions (for example FITS or TIFF) formats.

 -- Function:
          int
          gal_array_file_recognized (char *filename)
     Similar to ‘gal_array_name_recognized’, but for FITS files, it will
     also check the contents of the file if the recognized file name
     suffix is not found.  See the description of
     ‘gal_fits_file_recognized’ for more (*note FITS macros errors
     filenames::).

 -- Function:
          gal_data_t
          gal_array_read (char *filename, char *extension,
          gal_list_str_t *lines, size_t minmapsize, int quietmmap, char
          *ext_option_name)
     Read the array within the given extension (‘extension’) of
     ‘filename’, or the ‘lines’ list (see below).  If the array is
     larger than ‘minmapsize’ bytes, then it will not be read into RAM,
     but a file on the HDD/SSD (no difference for the programmer).
     Messages about the memory-mapped file can be disabled with
     ‘quietmmap’.

     ‘extension’ will be ignored for files that do not support them (for
     example JPEG or text).  For FITS files, ‘extension’ can be a number
     or a string (name of the extension), but for TIFF files, it has to
     be number.  In both cases, counting starts from zero.

     For multi-channel formats (like RGB images in JPEG or TIFF), this
     function will return a *note List of gal_data_t::: one data
     structure per channel.  Thus if you just want a single array (and
     want to check if the user has not given a multi-channel input), you
     can check the ‘next’ pointer of the returned ‘gal_data_t’.

     ‘lines’ is a list of strings with each node representing one line
     (including the new-line character), see *note List of strings::.
     It will mostly be the output of ‘gal_txt_stdin_read’, which is used
     to read the program’s input as separate lines from the standard
     input (see *note Text files::).  Note that ‘filename’ and ‘lines’
     are mutually exclusive and one of them must be ‘NULL’.

     ‘ext_option_name’ is used in error messages related to the
     extensions (e.g., HDUs in FITS) and is expected to the command-line
     option name that users to your program can specify the HDU for this
     particular input.  For example if the given ‘extension’ doesn’t
     exist in ‘filename’, a descriptive error message is printed which
     instructs the users how to find and fix the problem.  In that error
     message it also suggests the option to use to help the users of
     your program.

 -- Function:
          void
          gal_array_read_to_type (char *filename, char *extension,
          gal_list_str_t *lines, uint8_t type, size_t minmapsize, int
          quietmmap, char *ext_option_name)
     Similar to ‘gal_array_read’, but the output data structure(s) will
     have a numeric data type of ‘type’, see *note Numeric data types::.

 -- Function:
          void
          gal_array_read_one_ch (char *filename, char *extension,
          gal_list_str_t *lines, size_t minmapsize, int quietmmap, char
          *ext_option_name)
     Read the dataset within ‘filename’ (extension/hdu/dir ‘extension’)
     and make sure it is only a single channel.  This is just a simple
     wrapper around ‘gal_array_read’ that checks if there was more than
     one dataset and aborts with an informative error if there is more
     than one channel in the dataset.

     Formats like JPEG or TIFF support multiple channels per input, but
     it may happen that your program only works on a single dataset.
     This function can be a convenient way to make sure that the data
     that comes into your program is only one channel.

     Regarding ‘*ext_option_name’, see the description for the same
     argument in the description of ‘gal_array_read’.

 -- Function:
          void
          gal_array_read_one_ch_to_type (char *filename, char
          *extension, gal_list_str_t *lines, uint8_t type, size_t
          minmapsize, int quietmmap, char *ext_option_name)
     Similar to ‘gal_array_read_one_ch’, but the output data structure
     will has a numeric data type of ‘type’, see *note Numeric data
     types::.


File: gnuastro.info,  Node: Table input output,  Next: FITS files,  Prev: Array input output,  Up: Gnuastro library

12.3.10 Table input output (‘table.h’)
--------------------------------------

Tables are a collection of one dimensional datasets that are packed
together into one file.  They are the single most common format to store
high-level (processed) information, hence they play a very important
role in Gnuastro.  For a more thorough introduction, please see *note
Table::.  Gnuastro’s Table program, and all the other programs that can
read from and write into tables, use the functions of this section for
reading and writing their input/output tables.  For a simple
demonstration of using the constructs introduced here, see *note Library
demo - reading and writing table columns::.

   Currently only plain text (see *note Gnuastro text table format::)
and FITS (ASCII and binary) tables are supported by Gnuastro.  However,
the low-level table infra-structure is written such that accommodating
other formats is also possible and in future releases more formats will
hopefully be supported.  Please do not hesitate to suggest your favorite
format so it can be implemented when possible.

 -- Macro: GAL_TABLE_DEF_WIDTH_STR
 -- Macro: GAL_TABLE_DEF_WIDTH_INT
 -- Macro: GAL_TABLE_DEF_WIDTH_LINT
 -- Macro: GAL_TABLE_DEF_WIDTH_FLT
 -- Macro: GAL_TABLE_DEF_WIDTH_DBL
 -- Macro: GAL_TABLE_DEF_PRECISION_INT
 -- Macro: GAL_TABLE_DEF_PRECISION_FLT
 -- Macro: GAL_TABLE_DEF_PRECISION_DBL
     The default width and precision for generic types to use in writing
     numeric types into a text file (plain text and FITS ASCII tables).
     When the dataset does not have any pre-set width and precision (see
     ‘disp_width’ and ‘disp_precision’ in *note Generic data
     container::) these will be directly used in C’s ‘printf’ command to
     write the number as a string.

 -- Macro: GAL_TABLE_DISPLAY_FMT_STRING
 -- Macro: GAL_TABLE_DISPLAY_FMT_DECIMAL
 -- Macro: GAL_TABLE_DISPLAY_FMT_UDECIMAL
 -- Macro: GAL_TABLE_DISPLAY_FMT_OCTAL
 -- Macro: GAL_TABLE_DISPLAY_FMT_HEX
 -- Macro: GAL_TABLE_DISPLAY_FMT_FIXED
 -- Macro: GAL_TABLE_DISPLAY_FMT_EXP
 -- Macro: GAL_TABLE_DISPLAY_FMT_GENERAL
     The display format used in C’s ‘printf’ to display data of
     different types.  The ‘_STRING’ and ‘_DECIMAL’ are unique for
     printing strings and signed integers, they are mainly here for
     completeness.  However, unsigned integers and floating points can
     be displayed in multiple formats:

     Unsigned integer
          For unsigned integers, it is possible to choose from
          ‘_UDECIMAL’ (unsigned decimal), ‘_OCTAL’ (octal notation, for
          example, ‘125’ in decimal will be displayed as ‘175’), and
          ‘_HEX’ (hexadecimal notation, for example, ‘125’ in decimal
          will be displayed as ‘7D’).

     Floating point
          For floating point, it is possible to display the number in
          ‘_FLOAT’ (floating point, for example, ‘1500.345’), ‘_EXP’
          (exponential, for example, ‘1.500345e+03’), or ‘_GENERAL’
          which is the best of the two for the given number.

 -- Macro: GAL_TABLE_FORMAT_INVALID
 -- Macro: GAL_TABLE_FORMAT_TXT
 -- Macro: GAL_TABLE_FORMAT_AFITS
 -- Macro: GAL_TABLE_FORMAT_BFITS
     All the current acceptable table formats to Gnuastro.  The ‘AFITS’
     and ‘BFITS’ represent FITS ASCII tables and FITS Binary tables.
     You can use these anywhere you see the ‘tableformat’ variable.

 -- Macro: GAL_TABLE_SEARCH_INVALID
 -- Macro: GAL_TABLE_SEARCH_NAME
 -- Macro: GAL_TABLE_SEARCH_UNIT
 -- Macro: GAL_TABLE_SEARCH_COMMENT
     When the desired column is not a number, these values determine if
     the string to match, or regular expression to search, be in the
     _name_, _units_ or _comments_ of the column metadata.  These values
     should be used for the ‘searchin’ variables of the functions.

 -- Function:
          uint8_t
          gal_table_displayflt_from_str (char *string)
     Convert the input ‘string’ into one of the
     ‘GAL_TABLE_DISPLAY_FMT_FIXED’ (for fixed-point notation) or
     ‘GAL_TABLE_DISPLAY_FMT_EXP’ (for exponential notation).

 -- Function:
          char *
          gal_table_displayflt_to_str (uint8_t id)
     Convert the input identifier (one of the
     ‘GAL_TABLE_DISPLAY_FMT_FIXED’; for fixed-point notation, or
     ‘GAL_TABLE_DISPLAY_FMT_EXP’; for exponential notation) into a
     standard string that is used to identify them.

 -- Function:
          gal_data_t *
          gal_table_info (char *filename, char *hdu, gal_list_str_t
          *lines, size_t *numcols, size_t *numrows, int *tableformat)

     Store the information of each column of a table into an array of
     meta-data ‘gal_data_t’s.  In a metadata ‘gal_data_t’, the size
     elements are zero (‘ndim=size=0’ and ‘dsize=NULL’) but other
     relevant elements are filled).  See the end of this description for
     the exact components of each ‘gal_data_t’ that are filled.

     The returned array of ‘gal_data_t’s has ‘numcols’ datasets (one
     data structure for each column).  The number of rows in each
     dataset is stored in ‘numrows’ (in a table, all the columns have
     the same number of rows).  The format of the table (e.g., ASCII
     text file, or FITS binary or ASCII table) will be put in
     ‘tableformat’ (macros defined above).  If the ‘filename’ is not a
     FITS file, then ‘hdu’ will not be used (can be ‘NULL’).

     The input must be either a file (specified by ‘filename’) or a list
     of strings (‘lines’).  ‘lines’ is a list of strings with each node
     representing one line (including the new-line character), see *note
     List of strings::.  It will mostly be the output of
     ‘gal_txt_stdin_read’, which is used to read the program’s input as
     separate lines from the standard input (see *note Text files::).
     Note that ‘filename’ and ‘lines’ are mutually exclusive and one of
     them must be ‘NULL’.

     In the output datasets, only the meta-data strings (column name,
     units and comments), will be allocated and set as shown below.
     This function is just for column information (meta-data), not
     column contents.

          *restrict array  ->  Blank value (if present, in col's own type).
                     type  ->  Type of column data.
                     ndim  ->  0
                   *dsize  ->  NULL
                     size  ->  0
                quietmmap  ->  ------------
                *mmapname  ->  ------------
               minmapsize  ->  Repeat (length of vector; 1 if not vector).
                     nwcs  ->  ------------
                     *wcs  ->  ------------
                     flag  ->  'GAL_TABLEINTERN_FLAG_*' macros.
                   status  ->  ------------
                    *name  ->  Column name.
                    *unit  ->  Column unit.
                 *comment  ->  Column comments.
                 disp_fmt  ->  'GAL_TABLE_DISPLAY_FMT' macros.
               disp_width  ->  Width of string columns.
           disp_precision  ->  ------------
                    *next  ->  Pointer to next column's metadata
                   *block  ->  ------------

 -- Function:
          void
          gal_table_print_info (gal_data_t *allcols, size_t numcols,
          size_t numrows, char *hdu_option_name)
     Print the column information for all the columns (output of
     ‘gal_table_info’) to standard output.  The output is in the same
     format as this command with Gnuastro Table program (see *note
     Invoking asttable::):
          $ asttable --info table.fits

 -- Function:
          gal_data_t *
          gal_table_read (char *filename, char *hdu, gal_list_str_t
          *lines, gal_list_str_t *cols, int searchin, int ignorecase,
          size_t numthreads, size_t minmapsize, int quietmmap, size_t
          *colmatch, char *hdu_option_name)

     Read the specified columns in a file (named ‘filename’), or list of
     strings (‘lines’) into a linked list of data structures.  If the
     file is FITS, then ‘hdu’ will also be used, otherwise, ‘hdu’ is
     ignored.  For more on ‘hdu_option_name’ see the description of
     ‘gal_fits_hdu_open’ in *note FITS HDUs::.

     ‘lines’ is a list of strings with each node representing one line
     (including the new-line character), see *note List of strings::.
     It will mostly be the output of ‘gal_txt_stdin_read’, which is used
     to read the program’s input as separate lines from the standard
     input (see *note Text files::).  Note that ‘filename’ and ‘lines’
     are mutually exclusive and one of them must be ‘NULL’.

     The information to search for columns should be specified by the
     ‘cols’ list of strings (see *note List of strings::).  The string
     in each node of the list may be a number, an exact match to a
     column name, or a regular expression (in GNU AWK format) enclosed
     in ‘/ /’.  The ‘searchin’ value must be one of the macros defined
     above.  If ‘cols’ is NULL, then this function will read the full
     table.  Also, the ‘ignorecase’ value should be 1 if you want to
     ignore the case of alphabetic characters while matching/searching
     column meta-data (see *note Input output options::).

     For FITS tables, each column will be read independently.  Therefore
     they will be read in ‘numthreads’ CPU threads to greatly speed up
     the reading when there are many columns and rows.  However, this
     only happens if CFITSIO was configured with ‘--enable-reentrant’.
     This test has been done at Gnuastro’s configuration time; if so,
     ‘GAL_CONFIG_HAVE_FITS_IS_REENTRANT’ will have a value of 1,
     otherwise, it will have a value of 0.  For more on this macro, see
     *note Configuration information::).  Multi-threaded table reading
     is not currently applicable to other table formats (only for FITS
     tables).

     The output is an individually allocated list of datasets (see *note
     List of gal_data_t::) with the same order of the ‘cols’ list.  Note
     that one column node in the ‘cols’ list might give multiple columns
     (for example, from regular expressions), in this case, the order of
     output columns that correspond to that one input, are in order of
     the table (which column was read first).  So the first requested
     column is the first popped data structure and so on.

     if ‘colmatch!=NULL’, it is assumed to be an array that has at least
     the same number of elements as nodes in the ‘cols’ list.  The
     number of columns that matched each input column will be stored in
     each element.

 -- Function:
          gal_list_sizet_t *
          gal_table_list_of_indexs (gal_list_str_t *cols, gal_data_t
          *allcols, size_t numcols, int searchin, int ignorecase, char
          *filename, char *hdu, size_t *colmatch)
     Returns a list of indices (starting from 0) of the input columns
     that match the names/numbers given to ‘cols’.  This is a low-level
     operation which is called by ‘gal_table_read’ (described above),
     see there for more on each argument’s description.  ‘allcols’ is
     the returned array of ‘gal_table_info’.

 -- Function:
          void
          gal_table_comments_add_intro (gal_list_str_t **comments, char
          *program_string, time_t *rawtime)
     Add some basic information to the list of ‘comments’.  This basic
     information includes the following information
        • If the program is run in a Git version controlled directory,
          Git’s description is printed (see description under ‘COMMIT’
          in *note Output FITS files::).
        • The calendar time that is stored in ‘rawtime’ (‘time_t’ is C’s
          calendar time format defined in ‘time.h’).  You can calculate
          the time in this format with the following expressions:
               time_t rawtime;
               time(&rawtime);
        • The name of your program in ‘program_string’.  If it is
          ‘NULL’, this line is ignored.

 -- Function:
          void
          gal_table_write (gal_data_t *cols, struct gal_fits_list_key_t
          **keywords, gal_list_str_t *comments, int tableformat, char
          *filename, char *extname, uint8_t colinfoinstdout)

     Write ‘cols’ (a list of datasets, see *note List of gal_data_t::)
     into a table stored in ‘filename’.  The format of the table can be
     determined with ‘tableformat’ that accepts the macros defined
     above.  When ‘filename==NULL’, the column information will be
     printed on the standard output (command-line).

     If ‘comments!=NULL’, the list of comments (see *note List of
     strings::) will also be printed into the output table.  When the
     output table is a plain text file, every node of ‘comments’ will be
     printed after a ‘#’ (so it can be considered as a comment) and in
     FITS table they will follow a ‘COMMENT’ keyword.

     If a file named ‘filename’ already exists, the operation depends on
     the type of output.  When ‘filename’ is a FITS file, the table will
     be added as a new extension after all existing extensions.  If
     ‘filename’ is a plain text file, this function will abort with an
     error.

     If ‘filename’ is a FITS file, the table extension will have the
     name ‘extname’.

     When ‘colinfoinstdout!=0’ and ‘filename==NULL’ (columns are printed
     in the standard output), the dataset metadata will also printed in
     the standard output.  When printing to the standard output, the
     column information can be piped into another program for further
     processing and thus the meta-data (lines starting with a ‘#’) must
     be ignored.  In such cases, you only print the column values by
     passing ‘0’ to ‘colinfoinstdout’.

 -- Function:
          void
          gal_table_write_log (gal_data_t *logll, char *program_string,
          time_t *rawtime, gal_list_str_t *comments, char *filename, int
          quiet)
     Write the ‘logll’ list of datasets into a table in ‘filename’ (see
     *note List of gal_data_t::).  This function is just a wrapper
     around ‘gal_table_comments_add_intro’ and ‘gal_table_write’ (see
     above).  If ‘quiet’ is non-zero, this function will print a message
     saying that the ‘filename’ has been created.

 -- Function:
          gal_data_t *
          gal_table_col_vector_extract (gal_data_t *vector,
          gal_list_sizet_t *indexs)
     Given the “vector” column ‘vector’ (which is assumed to be a 2D
     dataset), extract the tokens that are identified in the ‘indexs’
     list into a list of one dimensional datasets.  For more on vector
     columns in tables, see *note Vector columns::.

 -- Function:
          gal_data_t *
          gal_table_cols_to_vector (gal_data_t *list)
     Merge the one-dimensional datasets in the given list into one
     2-dimensional dataset that can be treated as a vector column.  All
     the input datasets have to have the same size and type.  For more
     on vector columns in tables, see *note Vector columns::.


File: gnuastro.info,  Node: FITS files,  Next: File input output,  Prev: Table input output,  Up: Gnuastro library

12.3.11 FITS files (‘fits.h’)
-----------------------------

The FITS format is the most common format to store data (images and
tables) in astronomy.  The CFITSIO library already provides a very good
low-level collection of functions for manipulating FITS data.  The
low-level nature of CFITSIO is defined for versatility and portability.
As a result, even a simple and basic operation, like reading an image or
table column into memory, will require a special sequence of CFITSIO
function calls which can be inconvenient and buggy to manage in separate
locations.  To ease this process, Gnuastro’s library provides wrappers
for CFITSIO functions.  With these, it much easier to read, write, or
modify FITS file data, header keywords and extensions.  Hence, if you
feel these functions do not exactly do what you want, we strongly
recommend reading the CFITSIO manual to use its great features directly
(afterwards, send us your wrappers so we can include it here for others
to benefit also).

   All the functions and macros introduced in this section are declared
in ‘gnuastro/fits.h’.  When you include this header, you are also
including CFITSIO’s ‘fitsio.h’ header.  So you do not need to explicitly
include ‘fitsio.h’ anymore and can freely use any of its macros or
functions in your code along with those discussed here.

* Menu:

* FITS macros errors filenames::  General macros, errors and checking names.
* CFITSIO and Gnuastro types::  Conversion between FITS and Gnuastro types.
* FITS HDUs::                   Opening and getting information about HDUs.
* FITS header keywords::        Reading and writing FITS header keywords.
* FITS arrays::                 Reading and writing FITS images/arrays.
* FITS tables::                 Reading and writing FITS tables.

